What each setting did to accuracy

WixQA, 200 questions, judged correct against the arm without a cache. The dot is the change, the line its 95% interval.

interval excludes zero interval includes zero
−50+5+10+15+20+25worsebetter than no cacheOracle trails (upper bound)+16.5 (+11.5 to +21.2)Self-recorded, no gate+4.5 (−0.8 to +9.8)Ideal gate (judge sees reference)+5.8 (+1.0 to +10.8)Verifier v1+4.2 (−0.8 to +9.2)Verifier v2+5.5 (+0.8 to +10.2)points of judged-correct, against no cache
Only the oracle trails, which no deployment has, move accuracy clearly. Every realistic setting sits between about 4 and 6 points, the intervals overlap each other, and two of them include zero.
Table view
SettingChange (points)95% interval
Oracle trails (upper bound)+16.5+11.5 to +21.2
Self-recorded, no gate+4.5-0.8 to +9.8
Ideal gate (judge sees reference)+5.8+1.0 to +10.8
Verifier v1+4.2-0.8 to +9.2
Verifier v2+5.5+0.8 to +10.2