Re: [Bug-gnubg] Bot comparison 2-ply v 2-ply pruned.

[Jim Segrave]

On Mon 26 Apr 2010 (21:32 +0000), Michael Depreli wrote:
> 
> Hi Guys
> For my bot comparison pruning was not used for any ply with gnubg although 
> it's kind of a default setting.
> I'm mildly curious to compare the error rates between them for 2-ply.
> 
> Is there anyone on the team that could write some code that could compare the 
> analysis between 2-ply and 2-ply pruned and output
> the instances where they disagree?
> 
> I'm not curious enough to manually go through the 500 games again but would 
> do it if someone could automate it for me.
> And if it turns out there aren't many differences I'd probably do 3 and 4 ply 
> too.

I wrote some scripts to do just that when pruning was first added. I
had a colection of a lot of sgf's analysed two different ways. The
files for any given match should have the same names in two different
directories, one directory with one analysis, one with the same
matches with the other analysis. 

IT does a move by move comparision of the two files, checking that
they both agree on the best move (if not, then it reports the equity
difference betwen the fist analysis's best move and the first
analysis' evaluation of what the second file considered the best move.

It accumulates mean and std err of the difference in evaluation of
wins, win-gammons, win-backgammons, lose-gammons, lose-backgammons and 
euity (cubeful) for every move evaluated in both files.

It keeps separate mean and stderr for the same items for moves where
the two files disagree in what the best move is.

Usage is simple - say you have ~/eval1/ containing a.sgf, b.sgf and c.sgg
  where a, b and c are three matches evaluated with one standard
~eval2 has the same 3 sgf files, but this time evaluated with a
  different standard.
You run the program, specfying the two directories and supply the
  names of the .sgf files on stdin for all the files to be analysed
  (which is most easily done by cd'ing to the first directory and
  using ls -1 *.sgf | comparegun , path-to-other-sgffiles. In this
  case that would be:

cd eval1
ls -1 *.sgf | comparegnu . ~/eval2



All output is to stdout.

IT's all in Perl, commented, but assumes you've read the perlref man
page (there are some commented out 


-- 
Jim Segrave           address@hidden

comparegnu
Description: Text document