Re: [Bug-gnubg] pubeval benchmark

[Nikos Papahristou]

Interesting.
For comparison reasons with Palamedes and TD-Gammon, backgammons should
count as gammons. This changes the second performance metric to  0.6046ppg.

Nikos

On 20/1/2012 0:20 πμ, Joseph Heled wrote:
> Thanks for all of you who chipped in!
>
> Playing a 1 point match at 0 ply, gnubg won 7193 of 10000 matches,
> which is 0.4386 ppg in my book.
> Playing 10000 money sessions (gammons count, no cube) gnubg got 0.6296
> ppg (the breakdown is 4504, 2298, 285   2202, 676, 35 )
>
> -Joseph
>
> On 20 January 2012 06:16, Nikos Papahristou <address@hidden> wrote:
>> My evaluations of a starting roll 3-1 are exactly the same with Mark.
>> Joseph, my implementation of pubeval doesn't agree with the moves
>> selection in the games you sent.
>> In the first game, the first roll of pubeval (4-5) is played as 24/15,
>> and in the second game the first roll (52) is played as 13/8, 13/11.
>>
>> Nikos
>>
>> On 19/1/2012 17:14 μμ, Mark Higgins wrote:
>>> I get something different for a 3-1 off the starting board. Sorry for
>>> the non-standard format but hopefully it's intelligible. I'm printing
>>> out the layout and the pubeval value, ordered by descending value
>>> (based on my pubeval implementation). 8-5 6-5 is the best move.
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o *| * o|*
>>> *|o *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| o|*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| | *|*
>>> *| o| *|*
>>> *| o| *|*
>>> *|* o o| o *|*
>>> *|* o o| o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 10.3431*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o o *| * o|*
>>> *| *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| o|*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| o| *|*
>>> *| o| *|*
>>> *| o| o *|*
>>> *|* o| o *|*
>>> *|* o| o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 7.22287*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o *| * o|*
>>> *|o *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| |*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| o| *|*
>>> *| o| *|*
>>> *| o| o *|*
>>> *|* o| o *|*
>>> *|* o| o o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 6.73507*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *| o o *| * o|*
>>> *| *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| o|*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| o| *|*
>>> *| o| *|*
>>> *| o| o *|*
>>> *|* o| o *|*
>>> *|* o| o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 6.64798*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o o *| * o|*
>>> *| *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| |*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| o| *|*
>>> *| o| *|*
>>> *| o| o *|*
>>> *|* o| o *|*
>>> *|* o| o o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 6.61369*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o o *| * o|*
>>> *| *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| o|*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| | *|*
>>> *| o| *|*
>>> *| o| o *|*
>>> *|* o| o *|*
>>> *|* o o| o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 6.44695*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o o *| * o|*
>>> *| *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| o|*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| o| *|*
>>> *| o| *|*
>>> *| o| *|*
>>> *|* o| o *|*
>>> *|* o|o o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 6.17616*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o *| * o|*
>>> *|o *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| |*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| | *|*
>>> *| o| *|*
>>> *| o| o *|*
>>> *|* o| o *|*
>>> *|* o o| o o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 6.09006*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o *| * o|*
>>> *|o *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| |*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| o| *|*
>>> *| o| *|*
>>> *| o| *|*
>>> *|* o| o *|*
>>> *|* o|o o o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 5.81926*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o o *| * o|*
>>> *| *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| o|*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| o| *|*
>>> *| o| *|*
>>> *| o| *|*
>>> *|* o| o *|*
>>> *|* o o| o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 5.76592*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o *| * o|*
>>> *|o *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| o|*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| | *|*
>>> *| | *|*
>>> *| o| o *|*
>>> *|* o| o *|*
>>> *|* o o o| o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 5.65459*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o *| * o|*
>>> *|o *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| o|*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| o| *|*
>>> *| o| *|*
>>> *| o| *|*
>>> *|* o| o *|*
>>> *|* o o| o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 5.63356*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o o *| * o|*
>>> *| *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| o|*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| | *|*
>>> *| o| *|*
>>> *| o| o *|*
>>> *|* o| o *|*
>>> *|* o o| o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 5.55805*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o *| * o|*
>>> *|o *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| o|*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| | *|*
>>> *| o| *|*
>>> *| o| o *|*
>>> *|* o| o *|*
>>> *|* o o| o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 5.44036*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o *| * o|*
>>> *|o *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| o|*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| | *|*
>>> *| o| *|*
>>> *| o| *|*
>>> *|* o| o *|*
>>> *|* o o|o o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 4.76362*
>>>
>>> *|| | | | | ||| | | | | ||*
>>> *|o *| * o|*
>>> *|o *| * o|*
>>> *| *| * o|*
>>> *| *| o|*
>>> *| *| o|*
>>> *| | |*
>>> *| | |*
>>> *| | |*
>>> *| o| *|*
>>> *| o| *|*
>>> *| o| *|*
>>> *|* o| *|*
>>> *|* o o|o o *|*
>>> *|| | | | | ||| | | | | ||*
>>> *Value 3.78692*
>>>
>>>
>>> On Jan 19, 2012, at 9:57 AM, Joseph Heled wrote:
>>>
>>>> Here are all the possibilities for a 3,1 from the initial position. I
>>>> am worried that 8-5 6-5 sneaks at number 6
>>>>
>>>> score gnubg id position
>>>> 8.00384996831 4HPwATDgc/ABIQ (0, 1, 0, 0, 0, 1, -5, 0, -3, 0, 0, 0, 5,
>>>> -5, 0, 0, 0, 3, 0, 5, 0, 0, 0, 0, -2, 0)
>>>> 7.83777993917 4HPwATDEc/ABKA (0, 1, 1, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>> -5, 0, 0, 0, 3, 0, 4, 0, 0, 1, 0, -2, 0)
>>>> 7.81063996255 4HPwATDgc/ABEg (0, 0, 1, 0, 1, 0, -5, 0, -3, 0, 0, 0, 5,
>>>> -5, 0, 0, 0, 3, 0, 5, 0, 0, 0, 0, -2, 0)
>>>> 7.75730997324 4HPwATDgc+IBKA (0, 1, 1, 0, 0, 0, -5, 0, -3, 0, 0, 0, 4,
>>>> -5, 0, 1, 0, 3, 0, 5, 0, 0, 0, 0, -2, 0)
>>>> 7.43968993425 4HPwATDQZ/ABKA (0, 1, 1, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>> -5, 0, 0, 0, 2, 0, 5, 1, 0, 0, 0, -2, 0)
>>>> 6.84054994583 4HPwATCwZ/ABMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>> -5, 0, 0, 0, 2, 0, 4, 2, 0, 0, 0, -2, 0)
>>>> 6.81688992679 4HPwATDga/ABIg (0, 1, 0, 0, 1, 0, -5, 0, -3, 0, 0, 0, 5,
>>>> -5, 0, 0, 0, 2, 1, 5, 0, 0, 0, 0, -2, 0)
>>>> 6.67967991531 4HPwATDQc/ABIg (0, 1, 0, 0, 1, 0, -5, 0, -3, 0, 0, 0, 5,
>>>> -5, 0, 0, 0, 3, 0, 4, 1, 0, 0, 0, -2, 0)
>>>> 6.67962995172 4HPwATDCc/ABMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>> -5, 0, 0, 0, 3, 0, 4, 0, 0, 0, 1, -2, 0)
>>>> 6.24834996462 4HPwATDIZ/ABMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>> -5, 0, 0, 0, 2, 0, 5, 0, 1, 0, 0, -2, 0)
>>>> 6.10885995626 4HPwATDgc+EBMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 4,
>>>> -5, 0, 0, 1, 3, 0, 5, 0, 0, 0, 0, -2, 0)
>>>> 4.3719098568 4HPwATDEa/ABMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>> -5, 0, 0, 0, 2, 1, 4, 0, 0, 1, 0, -2, 0)
>>>> 4.29143989086 4HPwATDga+IBMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 4,
>>>> -5, 0, 1, 0, 2, 1, 5, 0, 0, 0, 0, -2, 0)
>>>> 4.23469984531 4HPwATCkc/ABMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>> -5, 0, 0, 0, 3, 0, 3, 1, 0, 1, 0, -2, 0)
>>>> 4.15422987938 4HPwATDQc+IBMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 4,
>>>> -5, 0, 1, 0, 3, 0, 4, 1, 0, 0, 0, -2, 0)
>>>> 3.22565984726 4HPwATDQV/ABMA (0, 2, 0, 0, 0, 0, -5, 0, -3, 0, 0, 0, 5,
>>>> -5, 0, 0, 0, 1, 1, 5, 1, 0, 0, 0, -2, 0)
>>>>
>>>>
>>>> On 20 January 2012 03:45, Mark Higgins <address@hidden
>>>> <mailto:address@hidden>> wrote:
>>>>> :) I'll give it a try and check the couple games you sent.
>>>>>
>>>>>
>>>>> On Jan 19, 2012, at 9:39 AM, Joseph Heled wrote:
>>>>>
>>>>>> I was asking for help, not for more work.
>>>>>>
>>>>>> For example, anyone who implemented play against pubeval can check a
>>>>>> few of the moves I sent.
>>>>>>
>>>>>> -Joseph
>>>>>>
>>>>>> On 20 January 2012 03:35, Mark Higgins <address@hidden
>>>>>> <mailto:address@hidden>> wrote:
>>>>>>> I used the following post to define pubeval; is this the proper
>>>>>>> definition still:
>>>>>>>
>>>>>>> http://www.bkgm.com/rgb/rgb.cgi?view+610
>>>>>>>
>>>>>>> ?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Jan 19, 2012, at 8:31 AM, Joseph Heled wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I just run 10000 1 point match games between gnubg and pubeval. I
>>>>>>>> think this is the most fair, since there are no gammons of cubes
>>>>>>>> involved.
>>>>>>>>
>>>>>>>> However, gnubg won 9840 of those, which makes me think something
>>>>>>>> is wrong.
>>>>>>>>
>>>>>>>> Can someone help me verify that pubeval makes the "right" moves?
>>>>>>>>
>>>>>>>> I can send a few games in oldmoves format, or can check specific
>>>>>>>> moves.
>>>>>>>>
>>>>>>>> I am attaching 2 games as examples.
>>>>>>>>
>>>>>>>> -Joseph
>>>>>>>> <game1.fibs><game2.fibs>