Re: [gnugo-devel] twin endgame match

[Paul Pogonyshev]

alain Baeckeroot wrote:
> Hi
> 
> Following Arend advice, gg378 and twin-378 had a 85 games endgame-match:
> - twin 26 win (1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 3 5 7 10 14 15 21 25 28)
> - GNU Go 14 win(-9 -3 -3 -3 -2 -2 -2 -1 -1 -1 -1 -1 -1 -1)
> - 45 unchanged
> The sum is +135, the average on 85 games +1.6
> 
> [...]
> 
> By construction, the twin "knows" exactly how gg378 evaluates the game, and 
> the twin may steal a big point before gg378 plays it, but it is still 
> gnugo-logic. So i wonder if this endgame match is significant or if it is 
> just a systematic error.
> 
> In other words, a reliable endgame comparison should imply an other engine, 
> good at endgame, and compare the results of both against the reference 
> engine.
> 
> Am i right, or just paranoid ?
> Is there such an engine available ?

This is a known problem that meta-engines show maximum strength increase when
playing against the underlying engine, because they know the way it plays.
However, (properly coded) meta-engines still should be stronger than the
underlying program since they can detect blunders of the underlying code.  Of
course, there is the downside when they choose a worse move just to trigger
a known blunder on the opponent side.

So, your tests, unfortunately, say little.  Your pointed out the flaw of
testing correctly, but that doesn't mean that the results of the tests are
necessarily wrong.  Yes, the right way would be to compare against a third,
independent, engine, but I don't know any readily available one...

Paul