RE: [Bug-gnubg] Luck analysis question

[Nis]

--On 11 June 2003 11:28 -0300 Albert Silver <address@hidden> wrote:

> Perhaps one interesting way to examine this would be to run GNU 0.12,
> the version that played sgainst Snowie 3.2, and have it analyze the luck
> factor and see what the results are.
>
> The reason I was a bit concerned is this: the luck is NOT independent of
> the evaluation of the moves and positions as I see it.

Of course the measured luck is not independent of the measured skill. This 
follows from the equation

luck + skill = final - initial

(found in Zare's article).

> It is entirely
> dependent in fact. Suppose in a given position, I say the equity is
> 0.400, move A is the best giving that 0.400 equity, and move B is a
> blunder leading to only 0.250. Mind you, this is because I do not
> realize that mave B is in fact stronger and leads to 0.450.

I'm with you so far.

To summarize:

Move               A       B
Gnubg Equity    0.400   0.250
True Equity(TM) 0.400   0.450

> You
> correctly play move B, I play a move that does nothing still thinking it
> is only 0.250, and you play a move that improves the position by a mere
> 0.050. When I analyze this new resulting position, I now see the truth
> of it and see that the current equity is 0.500, however having misjudged
> it previously will declare you're a lucky guy whose move led to a 0.250
> gain.

I lost you here. I will try to draft the sequence of events (P is a 
position, R is a roll, M is a move. M' and P' are the moves and resulting 
positions used by gnubg for luck analysis:

P0 (starting position)
R1
M1 | M'1
P1 | P'1
R2
M2 | M'2
P2 | P'2
...

Rn
Mn | M'n
Pn = P'n (Clear win or loss)

Luck = sum from i= 1 to n of P'i - P(i-1)

Skill = sum from i = 1 to n of Pi - P'i

Thus Luck + Skill = Pn - P0

The nice thing about the luck calculation is that it is mathematically 
guaranteed to be unbiased. This is obtained by having Pi evaluated one ply 
deeper than P'i. This is not to say that it is always correct - only that 
the errors average to zero. Thus the same goes for the measurement of skill 
calculated like this - it is unbiased, but imprecise.

Note that the calculation of skill given above is different from the one 
normally used for analysing single moves - since P'i is on a lower ply than 
Pi.

Assume we have reached position Pn.
Assume that the value of P'(n+1) when rolling 4-3 is misevaluated by -0.200 
(possibly because gnubg chooses the wrong move).
Assume that the other rolls are evaluated correct. This affects the value 
of P(n-1) by -0.200 * 1/18 ALWAYS, no matter if we roll 4-3 (since this is 
a 1-ply evaluation. It affects the value of P'(n+1) by 0.200 IF we roll a 
4-3, for an average value of 0.200 * 1/18. The average influence on luck is 
thus 0.

> My point being only that the luck factor IS dependent on GNU's
> evaluation as far as I can see. As to the variance reduction in
> rollouts, it is different. Variance reduction doesn't guarantee the
> rollouts will be correct, it just reduces the number of trials needed to
> reach the same result of a larger number of trials without variance
> reduction. Thus I can get the right or wrong result with only 1000
> trials instead of 10,000.

Well, luck analysis is exactly the same as variance reduction. It increases 
the accuracy of the measurement of skill provided by every game. What we 
are doing is (in the simplest case) equivalent to a one-trial rollout of 
the value of the starting position (given the two players).

--
Nis Jorgensen
Greenpeace
Amsterdam