Polyglot not doing what it is supposed to - book compile

[JonP01]

Hi,

I am compiling a very small book of high quality moves and am using Polyglot as the compilation tool. Because the book is quite small (all positions are being hand checked), I have been doing a lot of manual checking of the resulting book - both using SCID as well as running the book in my actual application (results in my application and SCID are indentical).

I was noticing during my checking process that a number of moves were missing from my compiled book, and even entire games. I have re-read the Polyglot documentation thoroughly and have even tried variations on my line commands, all to no avail.

Below is the contents of a test PGN file (2 games only) that illustrate the two problems I am having


[Event "Novy Vremya tourney-4 1899-1900"]
[Site "corr"]
[Date "1899.??.??"]
[Round "?"]
[White "Chigorin, Mikhail Ivanovich"]
[Black "Tabunshchikov, Vasily Ivanovich"]
[Result "1-0"]
[ECO "C37"]
[EventDate "1899.??.??"]
[EventType "corr"]
[Source "ChessBase"]

1. e4 e5 2. f4 exf4 3. Nf3 g5 4. Bc4 g4 5. O-O gxf3 6. Qxf3 Qf6 7. d3 Bc5+ 8.
Kh1 Nc6 9. Nc3 Nge7 10. Bxf4 d6 11. Nb5 Bb6 12. Qg3 Qg6 1-0

[Event "USA-USSR tele-match"]
[Site "?"]
[Date "1955.??.??"]
[Round "?"]
[White "Byrne, Robert (GM"]
[Black "Keres, Paul (EST)"]
[Result "0-1"]
[ECO "C39"]
[EventDate "1955.??.??"]
[EventType "corr"]
[Source "ChessBase"]

1. e4 e5 2. f4 exf4 3. Nf3 g5 4. h4 g4 5. Ne5 Nf6 6. Bc4 d5 7. exd5 Bg7 8. d4
Nh5 9. O-O Qxh4 10. Qe1 Qxe1 11. Rxe1 O-O 12. Nc3 Nd7 0-1


I am using the following line command in an attempt to successfully compile the two games above into the test book:

polyglot make-book -pgn test.pgn -bin test.bin -min-game 1 -uniform

After the test book compiles, it only contains the first 7 ply of the first game, and none at all of the second game. I have tried various ways of getting around the problem, including using the entire original games, using the -max-ply parameter, duplicating the games three times over and getting rid of the min-game 1 parameter and getting rid of the uniform parameter.

Nothing works and I am continually left with an incomplete book. These issues are magnified in the proper book, with lines missing that I know are there, plus other openings suddenly ending at an earlier ply than they should.

Can anyone tell me what is going wrong here please?

[F. Bluemers]

Hi,

I am compiling a very small book of high quality moves and am using Polyglot as the compilation tool. Because the book is quite small (all positions are being hand checked), I have been doing a lot of manual checking of the resulting book - both using SCID as well as running the book in my actual application (results in my application and SCID are indentical).

I was noticing during my checking process that a number of moves were missing from my compiled book, and even entire games. I have re-read the Polyglot documentation thoroughly and have even tried variations on my line commands, all to no avail.

Below is the contents of a test PGN file (2 games only) that illustrate the two problems I am having


[Event "Novy Vremya tourney-4 1899-1900"]
[Site "corr"]
[Date "1899.??.??"]
[Round "?"]
[White "Chigorin, Mikhail Ivanovich"]
[Black "Tabunshchikov, Vasily Ivanovich"]
[Result "1-0"]
[ECO "C37"]
[EventDate "1899.??.??"]
[EventType "corr"]
[Source "ChessBase"]

1. e4 e5 2. f4 exf4 3. Nf3 g5 4. Bc4 g4 5. O-O gxf3 6. Qxf3 Qf6 7. d3 Bc5+ 8.
Kh1 Nc6 9. Nc3 Nge7 10. Bxf4 d6 11. Nb5 Bb6 12. Qg3 Qg6 1-0

[Event "USA-USSR tele-match"]
[Site "?"]
[Date "1955.??.??"]
[Round "?"]
[White "Byrne, Robert (GM"]
[Black "Keres, Paul (EST)"]
[Result "0-1"]
[ECO "C39"]
[EventDate "1955.??.??"]
[EventType "corr"]
[Source "ChessBase"]

1. e4 e5 2. f4 exf4 3. Nf3 g5 4. h4 g4 5. Ne5 Nf6 6. Bc4 d5 7. exd5 Bg7 8. d4
Nh5 9. O-O Qxh4 10. Qe1 Qxe1 11. Rxe1 O-O 12. Nc3 Nd7 0-1


I am using the following line command in an attempt to successfully compile the two games above into the test book:

polyglot make-book -pgn test.pgn -bin test.bin -min-game 1 -uniform

After the test book compiles, it only contains the first 7 ply of the first game, and none at all of the second game. I have tried various ways of getting around the problem, including using the entire original games, using the -max-ply parameter, duplicating the games three times over and getting rid of the min-game 1 parameter and getting rid of the uniform parameter.

Nothing works and I am continually left with an incomplete book. These issues are magnified in the proper book, with lines missing that I know are there, plus other openings suddenly ending at an earlier ply than they should.

Can anyone tell me what is going wrong here please?

Polyglot throws the second game out because white has better (4.Bc4 according to the first game)
you can test that by removing the result tags in your example.
I guess you'll have to split the pgn file in one for white and one for black.
Best
Fonzy

[JonP01]

Fonzy!! Thank you so much!!! You solved it for me I have been tearing my hair out for three days solid in front of my PC trying to work out why my opening book was simply not turning out as I had expected. I was staring down the barrel of three months hard work on my project stymied by what was going to be a very poor openings book. The answer was indeed to remove the result tags as you say - I just tested it and all the moves are there now. I am sure glad I now know that it takes these tags into account, because that would have meant it was trying to judge my openings book when I have already spent months of my time doing that already.

So I guess now the question is: what is the easiest way to proceed from here? I have far too many games to manually get rid of the tags, but I suppose I could do it in Microsoft Excel at a pinch and then convert back to chessbase format and then produce another "result tagless" PGN. But is there an off the shelf utility around that removes those tags in bulk, or alternatively should I just split my database into white wins, black wins and draws? And then what would I do with the draws? Just merge the white and black bins together and then merge the draws into those merged wins?

Thanks

[F. Bluemers]

I am no pg book expert, maybe you can still use the pgn as is and build two bin files from it.
One build with -white-only and one with -black-only and later combine these with -merge-book?

[JonP01]

OK, thanks. Will give that a go.

[JonP01]

I've just found a very easy way to solve it, so I will explain it in case anyone else has the same problem that I had.

I simply used PGNscanner to change the result tag to an asterisk instead of a win for either colour.

So the actual command to use after opening the PGN in PGNscanner would be:

reptxt result="*"

This replaces the whole result tag.

Then you can use the output PGN file generated rom PGNscanner as the input file to Polyglot. Then Polyglot will not discriminate based upon the game result. Doing it this way there is no need for separate white and black compilations.

To me, this is the most effective way when you are doing a small book where each position in every single game that you use is manually checked. Afterall, just because games are won or lost does not mean the losing side did not have a good position out of the opening.

[F. Bluemers]

Going over the source code,i wondered why the second game would not be in the book?
"polyglot make-book -pgn test.pgn -bin test.bin -min-game 1 -uniform" works just fine.
If you check it with scid,open the book window and enter 4.h4 for white,you will see that 4. ..g4 is in the book!.
Best
Fonzy

[JonP01]

I can't explain that I do know that after testing changing the result tag to an asterisk, the size of the compiled books has been nearly double what I was getting before. You'd think if all the moves were still there previously that it would not be half the size it is now.

But in all my testing over the last day, nothing changed my result except for either removing the result tag completely or changing it to an asterisk.

[F. Bluemers]

I can't explain that I do know that after testing changing the result tag to an asterisk, the size of the compiled books has been nearly double what I was getting before. You'd think if all the moves were still there previously that it would not be half the size it is now.

But in all my testing over the last day, nothing changed my result except for either removing the result tag completely or changing it to an asterisk.

Its seems only normal that polyglot filters the moves.From the example pgn,it only needs the black moves as this line is bad for white.

Best
Fonzy