RE: [Bug-gnubg] Bot comparison 2-ply v 2-ply pruned.

[Michael Depreli]

Hi Jim
What I need is the following:
I have 50 money game sessions (48*10 games and 1*20 games).
I need the script to compare the sessions and output the following where there are differences:

Filename:
Game#:
Move# :
Player:
Cube And/Or Checker Play:  (It's important that the script doesn't overlook when a cube error and checker error occur on the same move)

I don't need to know the equity differences etc because I will be using my own rollout data but it may obviously be useful for others.

Also I see from the mailing list that there are moves afoot to strengthen the net so if we wanted to compare the old net vs the new one on my benchmark, this tool would be very useful.

I'm really not familiar with Perl etc, so if you could write me a script and run the sessions through for me it would be a great help

Thanks

Michael














> Date: Tue, 27 Apr 2010 01:08:25 +0200
> From: address@hidden
> To: address@hidden
> CC: address@hidden
> Subject: Re: [Bug-gnubg] Bot comparison 2-ply v 2-ply pruned.
>
> On Mon 26 Apr 2010 (21:32 +0000), Michael Depreli wrote:
> >
> > Hi Guys
> > For my bot comparison pruning was not used for any ply with gnubg although it's kind of a default setting.
> > I'm mildly curious to compare the error rates between them for 2-ply.
> >
> > Is there anyone on the team that could write some code that could compare the analysis between 2-ply and 2-ply pruned and output
> > the instances where they disagree?
> >
> > I'm not curious enough to manually go through the 500 games again but would do it if someone could automate it for me.
> > And if it turns out there aren't many differences I'd probably do 3 and 4 ply too.
>
> I wrote some scripts to do just that when pruning was first added. I
> had a colection of a lot of sgf's analysed two different ways. The
> files for any given match should have the same names in two different
> directories, one directory with one analysis, one with the same
> matches with the other analysis.
>
> IT does a move by move comparision of the two files, checking that
> they both agree on the best move (if not, then it reports the equity
> difference betwen the fist analysis's best move and the first
> analysis' evaluation of what the second file considered the best move.
>
> It accumulates mean and std err of the difference in evaluation of
> wins, win-gammons, win-backgammons, lose-gammons, lose-backgammons and
> euity (cubeful) for every move evaluated in both files.
>
> It keeps separate mean and stderr for the same items for moves where
> the two files disagree in what the best move is.
>
> Usage is simple - say you have ~/eval1/ containing a.sgf, b.sgf and c.sgg
> where a, b and c are three matches evaluated with one standard
> ~eval2 has the same 3 sgf files, but this time evaluated with a
> different standard.
> You run the program, specfying the two directories and supply the
> names of the .sgf files on stdin for all the files to be analysed
> (which is most easily done by cd'ing to the first directory and
> using ls -1 *.sgf | comparegun , path-to-other-sgffiles. In this
> case that would be:
>
> cd eval1
> ls -1 *.sgf | comparegnu . ~/eval2
>
>
>
> All output is to stdout.
>
> IT's all in Perl, commented, but assumes you've read the perlref man
> page (there are some commented out
>
>
> --
> Jim Segrave address@hidden
>
>

---

Get a free e-mail account with Hotmail. Sign-up now.