Test of Polyglot and Glaurung books

[Volker Pittlik]

I've made a test of different books with Glaurung. The games have been played at bullet time controls. Having this:



in mind I think ~200 games per engine are a good compromise between test duration and accuracy. Therefore I let each tested book play 220 games with Glaurung 2 epsilon/4 in gauntlets. Glaurung (whatever book) did not play against itself.

The new book by Salvo Spitaleri (Sal) was used as Glaurung's own book. All other books were used as Polyglot books.
The books were:

- Dann Corbit's large book(DCl) downloaded from Marc Lacrosse's site,
- Performance.bin (perf) by Marc himself
- Guenther Simon's "GS_medium" (GSm) which was send to me by him by mail and is dated 07/June/13
- Salvo's book (Sal) downloaded from Tord's site
- one of my own creations (VP1)

3.5% of the games were lost on time (TimeControl: 60) and have been replayed until no game was lost for that reason anymore.

The other engines in the test group were: Ruffian 2.1.0, Fruit (Toga) 1.2.1a, Scorpio 1.91, Spike 1.2 Turin, Shredder Classic 1.3, Jonny 2.83, Yace Paderborn, Crafty-21.5, Zappa 1.1, Arasan 9.5, Hermann 2.0. Ponder and all learning were off.

The result of the different books are:

Code: Select all

Rank Name                   Elo    +    - games score oppo. draws
   1 Glaurung2 e4 VP1       116   42   41   220   69%   -39   16%
   2 Glaurung2 e4 GSm       109   41   40   220   69%   -39   18%
   3 Glaurung2 e4 perf       91   40   39   220   67%   -39   23%
   4 Glaurung2 e4 DCl        79   41   40   220   65%   -39   17%
   5 Glaurung2 e4 Sal        35   39   39   220   60%   -39   19%

I have not seen a bad line in Guenther's book. Dann's book got a result similar to former (not published) tests. I don't trust the outcome of my own creation therefore I'm going to play another test with slower time controls. This will take approx. 10 days. I also don't trust the result of Salvo's book but I noticed that it is out of book as white after 1.e4 d5. A Norwegian engine and no idea of Scandinavian opening? Maybe there is a difference between the book usage of Glaurung and Polyglot although they use the same book format.

Comments appreciated

Volker

[Tord Romstad]

Hello Volker,

Thanks for the interesting results. The results with the various books differ more than I would expect, although the differences are still within the margins of error.

You are right that Salvo's book doesn't seem to contain the Scandinavian defence at all. I have also noticed that there are a few other lines which end too early, e.g. Glaurung is immediately out of book in the modern main line of the Queen's Indian (1. d4 Nf6 2. c4 e6 3. Nf3 b6 4. g3 Ba6).

Salvo sent me a new and updated book today; perhaps it is better and more complete than the previous book (I haven't had the chance to try the new book yet, but at least it is bigger). I still haven't decided whether I will put this new book on my web site, or wait a bit longer. Too frequent book updates, just like too frequent engine updates, can be annoying to testers.

Tord

[Roger Brown]

Hello Volker,

[Snip]

Salvo sent me a new and updated book today; perhaps it is better and more complete than the previous book (I haven't had the chance to try the new book yet, but at least it is bigger). I still haven't decided whether I will put this new book on my web site, or wait a bit longer. Too frequent book updates, just like too frequent engine updates, can be annoying to testers.

Tord

Hello Tord,

I hear the concern for testers but for the rest of us, releases which reflect improvements or changes in playing style or (too many or's already) widening of books are appreciated.

Let the testers decide what to do about it. Release away.



Later.

[Dann Corbit]

I guess that Salvo's books are better than the ones that I make.
In head to head tests, his have shown better in the contests that I have seen. I know that he puts a tremendous amount of effort into book writing. My bookmaking is mostly simple, mechanical steps. So I suspect that the overall results from his books will be better.

For your own peace of mind, try a contest of at least 100 games with the old book and the new book (no book would also be good as a control and also a sanity check). If it tests better or within the standard error as being the same, then I think you can release it with a fair amount of confidence.

[Tord Romstad]

Dann and Roger,

Thanks for your replies. Apparently Salvo simply made a mistake while preparing the first book: He used "the wrong option with pgn-extract" (whatever that means - I have no idea what pgn-extract does) according to his latest e-mail to me. If I understand correctly, the book file is simply broken (although it proved to be more than good enough in Lodz).

I'll update my web site this weekend with the new book file, a new version of my GUI, and a long tournament report from Lodz.

Tord

[Volker Pittlik]

...I have no idea what pgn-extract does...

There is a link to it at the bottom of the main page. It is a great tool to manipulate pgn files. It offers lots of options. It comes with source code. I guess it compiles at Macs too.

... If I understand correctly, the book file is simply broken ...

If he has managed to make something like that he has created a very good test tool. I tried to make one but was not successful.

To Dann's suggestion of a test without book: I have done that some time ago and if I remember correctly Glaurung plays better with book (also a very short one) than without.

I leave tommorrow for a short vacation and will check the forum only one or two times daily. I'll report the outcome of the test at slower time controls after I return.

Volker

[Volker Pittlik]

...I'm going to play another test with slower time controls. This will take approx. 10 days....

The games are finished now. Time control was 5 minutes initially and an increment of 1 second per move. Everything else was unchanged.

Results:

Code: Select all

Rank Name                   Elo    +    - games score oppo. draws
   1 Glaurung2 e4 VP1        92   39   38   220   66%   -30   30%
   2 Glaurung2 e4 perf       88   39   38   220   66%   -30   30%
   3 Glaurung2 e4 DCl        66   39   38   220   63%   -30   26%
   4 Glaurung2 e4 GSm        51   38   37   220   61%   -30   30%
   5 Glaurung2 e4 Sal        27   38   37   220   58%   -30   30%


I am not unhappy my book came out first again although the distances are very small. It seems almost everything is within the error margins of part 1 except Guenther's book.

More games could make the error margins smaller but that seems to take to much time for me alone. If some people are interested a test with more games could be done.

I'm not sure if I should publish my book. I think it could be improved before. It is really simple to make:

I have a big pgn collection with ~2 million games. I filtered that pool for all games not older than fifteen years. Both player must have at least 2300 elo and it should be games at standard time controls under normal conditions (no blind chess for example).

The problem with a book based on such a collection is that there are much more games played by 2300-2400 elo players than 2600+ players. Also there are much more old than new games. The resulting books are not bad but also nothing special.

Most likely newer games played by strongest possible players contain the best openings. Therefore I sorted the games by strength and age and copied the newer and stronger games so often until they outweigh the older medium range games. In addition I add some stuff from CCRL and CEGT. The book seems not to bad but it is much bigger than the others (~10MB).

Regards

Volker

[Peter Fendrich]

Volker,
With learning on I think the results will be even closer.
if a book have "holes" with missing lines like you describe the engine will block those lines. To leave the book isn't necessarily bad for the engine but when it is the line will be blocked...

/Peter

[Guenther Simon]

Hi Volker,

Could you please send me the game of my used
Polyglot book?
Thanks for the test!

Best regards,
Guenther

[Volker Pittlik]

Volker,
With learning on I think the results will be even closer...

Possibly yes. But at the moment I believe learning does not work so well. Maybe after 10000 games or so. Another great area for testing.

Thanks

Volker

[Volker Pittlik]

Hi Volker,

Could you please send me the game ...

Done.

Volker

[Dann Corbit]

This is an excellent result for your book.

Is there any chance I can get the raw PGN? I would like to study it.

I am doing some occasional work on improving my book building skills.

[Salvo Spitaleri]

Hi Volker ,

to be fair with my book, You can download the working version on the Glaurung's pages. The book that you tested has been corrupt during the compression, the right vesrsion was uploaded only in date 13.07.2007. Thanks .

Ciao
Salvo

[Volker Pittlik]

This is an excellent result for your book....

Thank you, but it still may be a random result.

...Is there any chance I can get the raw PGN? I would like to study it...

I'm sure you already have most of it. I have to notice that I didn't finished what I planned to do, but here are the files and instructions how to create the book from them. You have also to download the CCRL 40/40 and CEGT 40/20 games if you don't have them already. In addition you need pgn-extract and SCID. Because you need it anyway and I want to save bandwidth I decided to create tar.bz2-ed SCID databases for my files.

Download the archive http://www.vpittlik.org/chess/vp1-pgn.tar.bz2 (~9 MB). You will find 3 SCID databases in it: 2400-2500, 2500-2600 and 2600+. There are games inside from players within these ELO ranges.

1. Create a new pgn file and export the pgn of all ~50000 games from the 2400-2500 database to it. Do the same with the ~32000 games from the 2500-2600 database but do it 2 times. Do the same with the 2600+ games but do it 4 times. At the end there should be ~179000 games in the new pgn file.

2. Download the CEGT and CCRL games and free them from dupes with SCID. Pgn-extract finds more than SCID, but I didn't use it. Because there have been new games added since I downloaded them you may shrink the CEGT and CCRL databases to ~154000 and ~57000 games. After that add these games to the pgn file you created in step 1.

3. (You may possibly skip this step but I didn't). Clean the new created pgn with pgn-extract. I used:
./pgn-extract -7 -N -C -V -llog -o<output.filename> <input.filename>. Do not use -d or -D option because the duplicate games are necessary in the book. Rename the outputfile to whatever you like and move it to the polyglot directory.

4. Create a simple polyglot book:

./polyglot make-book -min-score 50 -pgn <input.filename>.pgn -bin vp1.bin

IIRC min-score 50 is default anyway.

Now you have the book I used.

I guess there is some room for improvement. I think to give the newer games of high rated player more weight is a good idea.
Also I'm going to use more game collections next time.

If someone is interested in the ready made book only he can download it here (~10 MB).

Volker

[Volker Pittlik]

Hi Volker ,

to be fair with my book, You can download the working version on the Glaurung's pages....the right vesrsion was uploaded only in date 13.07.2007. Thanks

I don't want to be unfair with your book at all. As I wrote in one of the previous posts I think this corrupted book is still a good tool for testing! If someons claims he has created a good book he can test it by comparing the performance with the corrupted book. It must be better. I was not successful to create something like that intentionally.

If I create a new version of my book I'm going to use your new version to compare with.

Regards

Volker

[Dann Corbit]

Thanks for an excellent and detailed "how to" guide.