[gnugo-devel] Move generation and metamachine

[bump]

Arend proposed:

> 2. Rewrite strategical evaluation to a strict before move/after move
>    comparison. This has to use: moyo size, eye shape, esacpe values, ...

I've believed for a long time that this would be better. I am
skeptical that now is the best time to make such a change. But
leaving that aside I want to discuss the relevance of the metamachine
experiment, and to report its current performance.

This philosophy underlies metamachine, found in interface/gtp_examples.

Metamachine runs top_moves, and compares the top two. Specifically,
it tries them both, runs estimate_score and selects the one with the
higher score. Critical dragons are awarded to the opponent, since it
will be the opponent's move.

Naturally this is a little slow. Evaluate position is run 3 times
instead of once. Persistent caching helps. Gunnar's bugfix in
gunnar_3_2.8 causes the engine to be reset after the first undo
but Gunnar tells me that does not cause the persistent cache
to be purged. On a fast machine metamachine seems to play with
acceptable speed.

Since it runs evaluate_position in two future positions, metamachine
has information available to it that is not available to GNU Go after
running evaluate_position once. This could be mitigated by running
owl_threats. Currently owl_threats does not improve GNU Go's play,
and finding and fixing the problems with owl_threats might also be
useful.

If we want to consider such a move valuation process, I think the
metamachine results are instructive.

The results for reading.tst, owl.tst and so forth are
not valid, so I only report the results starting with blunder.tst.

In some testsuites such as trevora and blunder, we have a lot
of failures. But in others, we have more PASS than FAIL. Perhaps
fixing metamachine so that it plays well would be helpful.

There are 190 FAIL, 108 PASS. There are a few crashes, still.

Dan

./regress.sh . blunder.tst
1 unexpected FAIL: Correct '!E5', got 'E5'
2 unexpected FAIL: Correct 'C9|H9', got 'F9'
4 unexpected FAIL: Correct 'B5|C5|C4|D4|E4|E3|F3', got 'B7'
8 unexpected FAIL: Correct '!(A3|A5)', got 'A5'
11 unexpected FAIL: Correct 'D4|E4|E5|F4|C6', got 'B3'
17 unexpected FAIL: Correct 'R7', got 'S6'
18 unexpected FAIL: Correct 'F5', got 'E6'
./regress.sh . trevora.tst
140 unexpected FAIL: Correct 'G4', got 'F2'
210 unexpected FAIL: Correct 'J8', got 'B1'
230 unexpected FAIL: Correct 'G2', got 'F2'
250 unexpected FAIL: Correct 'C3', got 'B3'
300 unexpected FAIL: Correct 'B8', got 'B7'
320 unexpected FAIL: Correct 'F9', got 'F3'
380 unexpected FAIL: Correct 'G8|G7', got 'F8'
390 unexpected FAIL: Correct 'A2', got 'A3'
460 unexpected FAIL: Correct 'H8', got 'F7'
580 unexpected FAIL: Correct 'E6', got 'J7'
./regress.sh . nngs1.tst
8 unexpected FAIL: Correct 'M10|L9', got 'S14'
10 unexpected PASS!
11 unexpected FAIL: Correct 'T12', got 'K11'
17 unexpected FAIL: Correct 'L3|L4|J3', got 'M5'
18 unexpected FAIL: Correct '!L2', got 'L2'
19 unexpected FAIL: Correct 'Q14', got 'D5'
24 unexpected FAIL: Correct 'L7|J5|M6', got 'H4'
25 unexpected FAIL: Correct 'C9', got 'B10'
30 unexpected PASS!
33 unexpected FAIL: Correct '!S9', got 'S9'
37 unexpected FAIL: Correct 'H8|K14|L15|L14', got 'G8'
39 unexpected PASS!
40 unexpected FAIL: Correct 'R6', got 'Q17'
42 unexpected PASS!
54 unexpected FAIL: Correct 'Q14', got 'R5'
55 unexpected FAIL: Correct '!C7', got 'C7'
56 unexpected FAIL: Correct 'H4', got 'J7'
./regress.sh . strategy.tst
1 unexpected FAIL: Correct 'Q13', got 'R13'
8 unexpected FAIL: Correct 'N2|N1', got 'T7'
10 unexpected FAIL: Correct 'Q7', got 'Q8'
14 unexpected FAIL: Correct 'E9', got 'F13'
21 unexpected FAIL: Correct 'S13', got 'R10'
22 unexpected PASS!
29 unexpected FAIL: Correct 'R4', got 'D6'
134519896 unexpected FAIL: Correct '', got 'PASS'
43 unexpected FAIL: Correct 'B13|B14|A14', got 'C13'
45 unexpected PASS!
46 unexpected FAIL: Correct 'E1', got 'O13'
47 unexpected FAIL: Correct '!N17', got 'N17'
./regress.sh . endgame.tst
103 unexpected FAIL: Correct 'G9', got 'A7'
104 unexpected FAIL: Correct 'E8|D8|E7', got 'D9'
207 unexpected FAIL: Correct 'A11', got 'A6'
208 unexpected PASS!
211 unexpected PASS!
215 unexpected FAIL: Correct 'C13', got 'H1'
217 unexpected FAIL: Correct 'C12', got 'H1'
218 unexpected PASS!
301 unexpected PASS!
401 unexpected FAIL: Correct 'C4', got 'E5'
./regress.sh . heikki.tst
10 unexpected FAIL: Correct 'M18', got 'J5'
11 unexpected FAIL: Correct '1 J18', got '2 M18'
./regress.sh . neurogo.tst
1 unexpected FAIL: Correct 'Q3', got 'Q5'
2 unexpected FAIL: Correct 'R3', got 'F4'
11 unexpected PASS!
16 unexpected FAIL: Correct 'H17|K17|H15|L16', got 'L1'
21 unexpected FAIL: Correct 'S16', got 'N18'
./regress.sh . arb.tst
134519896 unexpected FAIL: Correct '', got 'PASS'
104 unexpected FAIL: Correct 'A7|B7', got 'B6'
231 unexpected FAIL: Correct 'F5', got 'D1'
./regress.sh . rosebud.tst
1 unexpected PASS!
./regress.sh . golife.tst
1 unexpected PASS!
2 unexpected FAIL: Correct 'H5', got 'J6'
7 unexpected FAIL: Correct 'F3', got 'G1'
./regress.sh . arion.tst
5 unexpected FAIL: Correct '!F15', got 'F15'
./regress.sh . viking.tst
13 unexpected PASS!
./regress.sh . ego.tst
1 unexpected FAIL: Correct 'K4|K3', got 'E17'
3 unexpected FAIL: Correct 'H18', got 'K18'
6 unexpected FAIL: Correct '!N5', got 'N5'
8 unexpected PASS!
./regress.sh . dniwog.tst
1 unexpected FAIL: Correct 'D15', got 'H14'
4 unexpected FAIL: Correct 'D7', got 'B9'
./regress.sh . lazarus.tst
2 unexpected FAIL: Correct 'M4', got 'C7'
6 unexpected FAIL: Correct 'H3', got 'T5'
14 unexpected PASS!
./regress.sh . handtalk.tst
Possible crash!: expected 21 results, got 10. Last succesful test was 10.
1 unexpected FAIL: Correct 'R12|R13|R14|R8|R7|R6', got 'G7'
3 unexpected FAIL: Correct '!F18', got 'F18'
4 unexpected FAIL: Correct 'Q2', got 'B13'
5 unexpected FAIL: Correct '!P15', got 'P15'
7 unexpected FAIL: Correct 'R4', got 'K4'
9 unexpected FAIL: Correct 'J17', got 'H17'
10 unexpected FAIL: Correct 'E9|F8|D8', got 'L5'
./regress.sh . trevorb.tst
110 unexpected PASS!
130 unexpected FAIL: Correct '!J4', got 'J4'
200 unexpected PASS!
220 unexpected FAIL: Correct 'J3', got 'G1'
230 unexpected PASS!
240 unexpected FAIL: Correct '!K6', got 'K6'
250 unexpected FAIL: Correct 'C2', got 'L8'
270 unexpected PASS!
280 unexpected FAIL: Correct '!A4', got 'A4'
340 unexpected FAIL: Correct 'M11', got 'F3'
360 unexpected PASS!
430 unexpected PASS!
440 unexpected FAIL: Correct 'E6', got 'J6'
480 unexpected FAIL: Correct 'M2', got 'L2'
500 unexpected FAIL: Correct '!J11', got 'J11'
570 unexpected FAIL: Correct 'N9', got 'L3'
600 unexpected FAIL: Correct 'F2|F3', got 'N7'
610 unexpected FAIL: Correct 'D7', got 'N7'
670 unexpected FAIL: Correct 'L5', got 'A9'
700 unexpected PASS!
760 unexpected FAIL: Correct 'E11', got 'D13'
770 unexpected FAIL: Correct 'E11', got 'D13'
780 unexpected FAIL: Correct 'J9', got 'A6'
820 unexpected PASS!
900 unexpected PASS!
910 unexpected FAIL: Correct 'H11', got 'G9'
920 unexpected PASS!
950 unexpected PASS!
./regress.sh . strategy2.tst
64 unexpected FAIL: Correct 'B13|C13|C12|E14|D14', got 'P12'
65 unexpected PASS!
134519896 unexpected FAIL: Correct '', got 'PASS'
72 unexpected PASS!
73 unexpected PASS!
75 unexpected FAIL: Correct 'Q11', got 'F7'
76 unexpected FAIL: Correct 'Q11', got 'F7'
78 unexpected PASS!
79 unexpected FAIL: Correct 'N3', got 'C11'
134519896 unexpected FAIL: Correct '', got 'PASS'
90 unexpected PASS!
./regress.sh . nicklas1.tst
502 unexpected FAIL: Correct 'G5', got 'H7'
811 unexpected FAIL: Correct 'K14|K15', got 'J15'
134519896 unexpected FAIL: Correct '', got 'PASS'
134519896 unexpected FAIL: Correct '', got 'PASS'
134519896 unexpected FAIL: Correct '', got 'PASS'
1107 unexpected PASS!
1111 unexpected FAIL: Correct 'N18', got 'N19'
1216 unexpected PASS!
134519896 unexpected FAIL: Correct '', got 'PASS'
1502 unexpected FAIL: Correct 'D14', got 'O4'
134519896 unexpected FAIL: Correct '', got 'PASS'
134519896 unexpected FAIL: Correct '', got 'PASS'
1604 unexpected FAIL: Correct 'J2', got 'G3'
134519896 unexpected FAIL: Correct '', got 'PASS'
134519896 unexpected FAIL: Correct '', got 'PASS'
./regress.sh . nicklas2.tst
134519896 unexpected FAIL: Correct '', got 'PASS'
134519896 unexpected FAIL: Correct '', got 'PASS'
2201 unexpected FAIL: Correct 'C2', got 'D1'
2401 unexpected PASS!
./regress.sh . nicklas3.tst
602 unexpected PASS!
1403 unexpected PASS!
./regress.sh . nicklas4.tst
1103 unexpected FAIL: Correct 'C2|C3', got 'D2'
./regress.sh . nicklas5.tst
1202 unexpected FAIL: Correct '!L17', got 'L17'
1203 unexpected FAIL: Correct 'K18|N18|N19|M19', got 'L19'
./regress.sh . manyfaces.tst
1 unexpected FAIL: Correct 'S13', got 'D18'
8 unexpected FAIL: Correct 'T2', got 'P4'
./regress.sh . niki.tst
./regress.sh . trevor.tst
Possible crash!: expected 92 results, got 2. Last succesful test was 2.
./regress.sh . tactics.tst
5 unexpected PASS!
./regress.sh . buzco.tst
./regress.sh . nngs.tst
160 unexpected PASS!
170 unexpected PASS!
200 unexpected PASS!
250 unexpected FAIL: Correct '!S15', got 'S15'
260 unexpected PASS!
290 unexpected PASS!
300 unexpected PASS!
310 unexpected FAIL: Correct 'B9', got 'N17'
330 unexpected PASS!
380 unexpected PASS!
390 unexpected PASS!
400 unexpected PASS!
420 unexpected PASS!
440 unexpected FAIL: Correct 'O2', got 'D13'
490 unexpected PASS!
500 unexpected FAIL: Correct 'J17', got 'H15'
510 unexpected FAIL: Correct 'G14', got 'H14'
570 unexpected PASS!
620 unexpected FAIL: Correct 'P16', got 'R8'
660 unexpected PASS!
850 unexpected PASS!
870 unexpected PASS!
134519896 unexpected FAIL: Correct '', got 'PASS'
880 unexpected PASS!
890 unexpected PASS!
910 unexpected PASS!
960 unexpected FAIL: Correct 'R16|R17|S16', got 'S17'
1010 unexpected PASS!
1110 unexpected FAIL: Correct 'J16', got 'K3'
1160 unexpected FAIL: Correct '!A8', got 'A8'
1180 unexpected PASS!
1260 unexpected PASS!
1300 unexpected PASS!
1310 unexpected FAIL: Correct 'Q15|Q14', got 'K16'
1420 unexpected PASS!
1440 unexpected FAIL: Correct 'H10', got 'L6'
1480 unexpected PASS!
1500 unexpected FAIL: Correct 'R16', got 'Q17'
1560 unexpected PASS!
1700 unexpected PASS!
1750 unexpected FAIL: Correct 'L8', got 'J9'
1910 unexpected PASS!
2040 unexpected PASS!
2060 unexpected FAIL: Correct 'C4|C1|F1', got 'C9'
./regress.sh . trevorc.tst
134519896 unexpected FAIL: Correct '', got 'PASS'
280 unexpected PASS!
310 unexpected FAIL: Correct 'H12', got 'H13'
350 unexpected FAIL: Correct 'E3', got 'E4'
360 unexpected FAIL: Correct 'H11', got 'J12'
370 unexpected FAIL: Correct 'F13|D1', got 'G8'
410 unexpected FAIL: Correct '!H13', got 'H13'
460 unexpected FAIL: Correct 'H8|K11', got 'H7'
470 unexpected FAIL: Correct 'K11', got 'H5'
500 unexpected PASS!
530 unexpected PASS!
580 unexpected PASS!
590 unexpected PASS!
670 unexpected FAIL: Correct '!G2', got 'G2'
720 unexpected PASS!
750 unexpected FAIL: Correct '!H1', got 'H1'
890 unexpected FAIL: Correct 'J10', got 'G12'
910 unexpected FAIL: Correct 'H13', got 'E1'
930 unexpected FAIL: Correct 'K10', got 'A13'
1030 unexpected FAIL: Correct 'G3', got 'K3'
1050 unexpected PASS!
1080 unexpected FAIL: Correct '!L8|K10|K9', got 'L8'
1090 unexpected FAIL: Correct '!E1', got 'E1'
1100 unexpected FAIL: Correct '!J8|H8', got 'J8'
1120 unexpected PASS!
1130 unexpected PASS!
134519896 unexpected FAIL: Correct '', got 'PASS'
1180 unexpected PASS!
1210 unexpected FAIL: Correct 'E5', got 'C9'
1370 unexpected FAIL: Correct 'B9', got 'B10'
1390 unexpected FAIL: Correct 'K9', got 'G4'
1420 unexpected FAIL: Correct 'J4', got 'G3'
134519896 unexpected FAIL: Correct '', got 'PASS'
1540 unexpected PASS!
1570 unexpected PASS!
1610 unexpected FAIL: Correct '!D1', got 'D1'
1680 unexpected FAIL: Correct 'K13', got 'E1'
./regress.sh . strategy3.tst
105 unexpected PASS!
113 unexpected FAIL: Correct 'P1', got 'T4'
115 unexpected FAIL: Correct 'G2', got 'C9'
135 unexpected PASS!
139 unexpected FAIL: Correct 'M18', got 'P17'
./regress.sh . capture.tst
./regress.sh . connect.tst
./regress.sh . global.tst
1 unexpected PASS!
5 unexpected PASS!
7 unexpected FAIL: Correct 'A5', got 'G9'
9 unexpected PASS!
22 unexpected FAIL: Correct 'F2', got 'N11'
28 unexpected FAIL: Correct 'N11', got 'N10'
31 unexpected PASS!
36 unexpected FAIL: Correct 'B6', got 'A6'
38 unexpected PASS!
./regress.sh . vie.tst
Possible crash!: expected 53 results, got 49. Last succesful test was 49.
./regress.sh . arend.tst
Possible crash!: expected 36 results, got 5. Last succesful test was 5.
3 unexpected PASS!
4 unexpected FAIL: Correct 'R14|S15|R17', got 'S16'
./regress.sh . 13x13.tst
Possible crash!: expected 91 results, got 0. Crash in first test.
./regress.sh . semeai.tst
./regress.sh . trevord.tst
190 unexpected FAIL: Correct 'T10', got 'P14'
230 unexpected FAIL: Correct 'G6', got 'L9'
280 unexpected PASS!
290 unexpected PASS!
310 unexpected PASS!
340 unexpected PASS!
360 unexpected FAIL: Correct 'K3|K4', got 'K8'
430 unexpected PASS!
470 unexpected FAIL: Correct 'J18', got 'D3'
480 unexpected PASS!
490 unexpected FAIL: Correct 'D5', got 'G4'
520 unexpected FAIL: Correct 'D15', got 'G16'
570 unexpected PASS!
610 unexpected FAIL: Correct 'D2', got 'C1'
620 unexpected PASS!
630 unexpected FAIL: Correct 'E2', got 'D1'
640 unexpected FAIL: Correct 'H2', got 'G1'
730 unexpected FAIL: Correct 'H3', got 'P15'
760 unexpected FAIL: Correct 'H6', got 'K8'
860 unexpected FAIL: Correct 'Q5', got 'J15'
930 unexpected FAIL: Correct '!P19', got 'P19'
940 unexpected PASS!
950 unexpected PASS!
1050 unexpected PASS!
1060 unexpected PASS!
1080 unexpected FAIL: Correct '!O16', got 'O16'
1100 unexpected FAIL: Correct 'N5', got 'O6'
1120 unexpected FAIL: Correct 'Q8', got 'D1'
./regress.sh . strategy4.tst
151 unexpected PASS!
152 unexpected FAIL: Correct 'S8', got 'R6'
153 unexpected FAIL: Correct 'Q9', got 'O11'
156 unexpected FAIL: Correct 'B14|B16', got 'K11'
157 unexpected PASS!
167 unexpected FAIL: Correct 'D4', got 'E5'
173 unexpected FAIL: Correct 'A9', got 'T17'
177 unexpected FAIL: Correct 'E2', got 'N17'
179 unexpected FAIL: Correct 'P3|O2|P2|O4|E2', got 'N17'
183 unexpected FAIL: Correct 'P10|H9', got 'G17'
191 unexpected FAIL: Correct 'C12', got 'A13'
199 unexpected PASS!
202 unexpected PASS!
209 unexpected PASS!
210 unexpected FAIL: Correct 'H7', got 'H8'
216 unexpected PASS!
217 unexpected PASS!
218 unexpected PASS!
219 unexpected PASS!