Re: [Bug-gnubg] Huge evaluation difference

[Øystein Schønning-Johansen]

I wrote a little tool some years ago, that calculate the "exact" value for such racing position. It calculates the probabilities with a top-down dynamic programming way and uses *-minimax algorithm to fill in the values and prune useless moves.

It is really unusable in real life situation as the evaluation usually takes a few minutes (depending on the position, of course), and the calculation uses a lot of memory! However I was hoping this method could have been used in establishing a training dataset, but I've not build such a dataset yet. :-o

I managed to calculate the gammon saving probability of the posted position in a few minutes using about 15GB of memory. (I also have a slower version of the tool that uses sparse matrices to store the probabilities, but I did not try that today)

address@hidden:~/octopus/twoside$ ./lopsided --mode=gammonsave 2000036300001 2231

#off:  0  board: 2----3|63----|1-----|-----

        chk(bb): x----x|xx----|x-----|----- back: 13 #out: 10 #cross: 26

#off:  7  board: 2231--|------|------|-----

        chk(bb): xxxx--|------|------|----- back:  4 #out:  0 #cross:  8

Cubeless prob. of saving gammon: 0.129424

If anyone is interested, I'll share some code.

-Øystein

On Tue, Mar 19, 2019 at 10:36 PM Philippe Michel <address@hidden> wrote:

On Mon, Mar 18, 2019 at 10:25:52AM +0100, Terje Pedersen wrote:

> I just ran into a huge evaluation difference between XG and Gnu BG:
>
> XGID=-B----CFC----A-------acbb-:1:1:1:16:0:0:0:5:10
>
> I got dinged with a -0.684 error here while XG says it is a -0.0013
> error to play 13/6.
>
> Desktop version says it is:
>
>     7. Cubeful 0-ply    13/6                         Eq.:  -2,264 ( -0,328)
>        0,000 0,000 0,000 - 1,000 0,964 0,000
>         0-ply cubeful prune [expert]
>
> but still wildly off. Perhaps difficult to fix these kind of problems?

The position is of a kind that was probably not common in the racing
neural net training database. X's formation just doesn't happen in real
play, does it ?

 GNU Backgammon  Position ID: 2wUAAAb3OwgAAA
                 Match ID   : UQmnAAAAAAAE
 +13-14-15-16-17-18------19-20-21-22-23-24-+     O: GNUbg
 | X                |   |       O  O  O  O | OO  0 points
 |                  |   |          O  O  O | OO 
 |                  |   |          O       | O   
 |                  |   |                  | O   
 |                  |   |                  | O 
v|                  |BAR|                  |     5 point match
 |                6 |   |                  |   
 |                X |   |                  |     
 |             X  X |   | X                |     
 |             X  X |   | X              X |     Rolled 61
 |             X  X |   | X              X |     0 points
 +12-11-10--9--8--7-------6--5--4--3--2--1-+     X: You (Cube: 2)
                    Pip counts : O 19, X 99

"Fixing" this at the neural net level doesn't look easy. Adding a few
hundreds or thousands of similar positions in the training data would
probably help but it is difficult to guess in advance if the improvement
would be significant.

It wouldn't change the program's strength (it doesn't get into these
positions) but may avoid or at least mitigate this kind of embarrassing
result when analysing.

What you can do if it is important for you (for the analysis feature of
Backgamon Studio, I suppose) is to use the largest one-sided bearoff
database, available there :
ftp://ftp.demon.nl/pub/Museum/Demon/games/gnubg/databases/os/

If the download size is an issue, you can compute it yourself with the
makebearoff utility from gnubg ; os13.bd (with the back checker up to
the midpoint, as in your position) should take about one day, smaller
ones would be much faster.

With it, I get :

    1. Cubeful 2-ply    13/6                         Eq.: -2.201
       0.000 0.000 0.000 - 1.000 0.913 0.000
        2-ply cubeful prune [world class]
    2. Cubeful 2-ply    8/2 7/6                      Eq.: -2.204 (-0.003)
       0.000 0.000 0.000 - 1.000 0.915 0.000
        2-ply cubeful prune [world class]
    3. Cubeful 2-ply    7/6 7/1                      Eq.: -2.213 (-0.013)
       0.000 0.000 0.000 - 1.000 0.923 0.000
        2-ply cubeful prune [world class]
    4. Cubeful 2-ply    13/12 8/2                    Eq.: -2.234 (-0.034)
       0.000 0.000 0.000 - 1.000 0.939 0.000
        2-ply cubeful prune [world class]
    5. Cubeful 2-ply    13/12 7/1                    Eq.: -2.240 (-0.039)
       0.000 0.000 0.000 - 1.000 0.943 0.000
        2-ply cubeful prune [world class]
    6. Cubeful 2-ply    13/7 6/5                     Eq.: -2.294 (-0.093)
       0.000 0.000 0.000 - 1.000 0.984 0.000
        2-ply cubeful prune [world class]

It may seem radical to use a 1.5 Gb database just for this but, besides
the disk space needed, it is essentially free. The file is not read, it
is merely mapped into memory and the relevant blocks are read only when
needed.

_______________________________________________
Bug-gnubg mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/bug-gnubg