[Bug-gnubg] TEST RESULT: Variance Reduction

[Ian Shaw]

I've re-tested rollouts of the initial position now that the variance reduction 
has been corrected. I'd like to get some comment on before posting the test to 
GammOnLine.

My observations are:

1) Wins reduced
The winner of the opening roll wins less often. This is to be expected since 
doubles would always be a good opening roll. The old VR was factoring in some 
rolls being doubles and overcorrecting for what it saw as a poor set of rolls 
during the rollout.

2) Crawford 2-away Match Equity
Leader's MEQ at Crawford 2-away is even lower than the original rollout 
suggests, 68.2% rather than 68.3%. At 2-ply, my rollouts suggested that Trailer 
benefited even more, so we can expect Leaders' MEQ to drop below my current 
best value of 67.7%, which is already lower than any published MET. (Oh how I 
want to get multi-processing before I repeat THAT rollout!)

3) Standard Error
The reported overall std error is a LOT lower (see result below). I would have 
expected some reduction because, without doubles, there is simply less 
variation in the opening rolls. There is a huge swing on 66 for starters - it 
wins about 2/3 of games (according to 2-ply evaluation).
Even so, I was surprised to find such a large reduction in the std error. The 
individual std errors for w, wg etc have not changed much, so it's hard to see 
how the overall value would change so much. Could you have a look at this, 
please.

4) Rotating opening rolls
I rolled out 7776 times (6 * 36 * 36). Then I wondered how the stratification 
is affected when it's the opening roll. Will the rollout have done any 
stratification at all? Should I have done a multiple of 21 * 36 = 756? Will the 
stratification work if I do?

5) Match Equity Calculation
Finally, I'm a bit suspicious of the reported match equity values. I calculated 
them by hand from the win percentages, and got different results. 

Match equities reported by gnubg and calculated by hand:
Wrong variance reduction
Trailer: 33.784%        33.635%
Leader:  69.939%        70.265%
Corrected variance reduction                                    
Trailer: 69.081%        69.080%
Leader: 69.081% 69.080%

The ones from the corrected variance reduction rollout look OK, with the 
discrepancy attributable to rounding error. The earlier results look suspect 
though. I don't think rounding error would give such a large discrepancy. (E.g. 
the 70.265 value calculates to 70.25 and 70.5 if I round the win percentages to 
3 and 2 dps, so I can't see how it can get from 69.939 to 70.265.)
Leader's MEQ calculated as w + (l - lg) * 0.5
Trailer's MEQ calculated as wg + (w - wg) * 0.5


Rollout results

0-ply   trials/side     7776    Score: Crawford 2-away                          
                
        
Wrong variance reduction                                                        
        
1st Roll        w               wg              wbg             l               
lg              lbg             cubeless        cubeful
Trailer 0.5258  0.1469  0.0186  0.4742  0.1923  0.0300  33.784% 33.784%
std err 0.0010  0.0009  0.0010                  0.0023  0.0013  0.280%  0.280%  
Leader  0.5280  0.2233  0.0324  0.4720  0.1227  0.0124  69.939% 69.939%
std err 0.0010  0.0024  0.0013                  0.0009  0.0008  0.280%  0.280%
Leader's MEQ                                                                    
Combined        0.5011  0.2078  0.0312  0.4989  0.1348  0.0155          68.315% 
                                        Combined Std Err        0.198%  95%CI   
0.388%          
        
Corrected variance reduction                                    
1st Roll        w               wg              wbg             l               
lg              lbg             cubeless        cubeful
Trailer 0.5114  0.1424  0.0183  0.4886  0.2001  0.0299  32.687% 32.687%
std err 0.0010  0.0008  0.0010                  0.0023  0.0013  0.069%  0.069%  
Leader  0.5123  0.2176  0.0323  0.4877  0.1307  0.0138  69.081% 69.081%
std err 0.0010  0.0023  0.0013                  0.0008  0.0009  0.069%  0.069%
Leader's MEQ                                                                    
Combined        0.5005  0.2089  0.0311  0.4996  0.1366  0.0161          68.195% 
                                        Combined Std Err        0.049%  95%CI   
0.096%  


--Ian