Re: [Bug-gnubg] Luck analysis question

[Joern Thyssen]

On Tue, Jun 10, 2003 at 02:48:35PM -0300, Albert Silver wrote
> Something is bugging me regarding the analysis that GNU and Snowie are
> of exactly equal strength. I can't argue the math with you so I'll just
> explain it:
> 
> GNU won the 100 match series quite convincingly, granted. It also
> evaluated the relative luck of each player. Again, no argument. Its
> 2-ply analysis (the luck was done at 0-ply) will consistently say that
> it played perfectly and that Snowie invariably screwed up at some degree
> whether by a minuscule fraction or more. So ALL analysis it presents
> comparing its play to Snowie's should reveal it is stronger than Snowie,
> no? The analysis showed the players to be exactly equal, but what
> exactly was analyzed? Did it analyze that neither player gave up more
> equity than it benefitted luck so that the result was consistent. I.e. I
> was 15% luckier but only gave up 1% equity throughout the match thus the
> result is consistent. Also, wouldn't a 2-ply luck analysis, agreeing
> with all of GNU's 2-ply plays, reveal it is in effect stronger than
> Snowie? I'm not declaring it IS stronger just that such an analysis
> would inevitably say as much.

What I'm really after is some unbiased measure of the luck in each of
the 100 matches. The best tool available for measuring the luck is to
use gnubg or Snowie.

The calculation of the luck is independent of the actual moves made.
It's just the deviation of the equity of the actual roll from the
average of equities for all possible rolls. 

Note that I didn't analyse the moves or cubes in the matches. This makes
little sense for players very close in strength to the bot. You can read
more about that in Zare's excellent article:

   http://math.columbia.edu/~zare/vrskill.html

where he introduced "Variance Reduction of Skill", which is what I did
to the matches. Zare writes:

     Snowie estimates the luck in a roll by estimating the equity of the
     best play it sees after rolling, and subtracting the equity of the
     position before rolling. This is a good estimate, and is equivalent
     to measuring skill by summing up the errors compared with what
     Snowie thinks is the best play. I have found this to be
     tremendously helpful, but because Snowie does not estimate the
     equity perfectly, this method is biased, and unsuitable for
     analyzing matches between players close to Snowie's level or in
     positions where Snowie is less reliable. However, one can fix any
     estimate of match winning chances to produce an unbiased estimate.
     An evaluation as good as Snowie's may be corrected to eliminate
     most of the luck in backgammon without bias.


If I analyse the matches with gnubg, it will say that snowie is giving
up equity /relative/ to gnubg, and vice versa. Since we do not have a
perfect bot for analysing cube decisions and chequerplays, the best
thing we can do is to play a huge number of matches between the two. I
calculated the luck adjusted results to reduce the variance similar to
what we do in rollouts. 

Jørn