Playing without opening book (a 1900 games experiment)

[Marc Lacrosse]

A frequently asked question is:

What is the value of playing blitz without opening book as compared with playing with a good general opening book.

I just held a 1900 games tournament aiming at approaching an answer to this question.

I selected the four best free 32-bits engines with an established rating in the CCRL 40/4 list (as of 16.03.2007) :

Code: Select all

R - Rybka 1.0 Beta 32-bit   2900 +/-19
T - Toga II 1.3x4           2883 +/-23
S - Spike 1.2 Turin         2825 +/-08
N - Naum 2.0 32-bit         2799 +/-10

mean Elo : 2852


Each engine entered the tournament five times with the following Polyglot books :

Code: Select all

Books :
00 - no book
  no explanation required :-)
01 - Performance
 The 1.5 MB book which was built for Fruit 2.2.1 and is now freely available (see my site)
02 - DClarge
 A very good 6.2 MB general book built by Dann Corbit in july 2005 (available at my site)
03 - Best-03
 A recent attempt of mine at improving over performance.bin (1.4 MB ;private)
04 - Monster
 An enormous private polyglot book (165 MB; private)


So there were 20 players in the tournament (4 engines x 5 books).

The tournament was a ten rounds all-play-all one (1900 games).
PIV 3.0 Ghz 128MB noTB noEGBB ponder off.
1 min + 1sec per move

RESULTS

Grouping the games by engine :

Code: Select all

Rank Name    Elo    +    - games score  draws
   1 R-any    67   16   16   950   60%   33%
   2 T-any    19   16   16   950   53%   37%
   3 S-any   -27   16   16   950   46%   34%
   4 N-any   -59   16   16   950   41%   38%


This is very similar to the CCRL rating list, proving that the tournament is not biased :

Code: Select all

                        CCRL 40/4     Present test
Rybka 1.0 Beta 32-bit   2900 +/- 19   2919 +/- 16
Toga II 1.3x4           2883 +/- 23   2871 +/- 16
Spike 1.2 Turin         2825 +/- 08   2825 +/- 16
Naum 2.0 32-bit         2799 +/- 10   2793 +/- 16


Grouping the games by opening book :

Code: Select all

     Book          Elo    +    -  games score  draws
   1 Monster       2868  18   18   760   53%    34%
   2 Best-03       2866  18   18   760   52%    38%
   3 Performance   2864  18   18   760   52%    37%
   4 DClarge       2860  18   18   760   51%    38%
   5 No Book       2802  19   19   760   42%    30%


So here is the answer : a general opening book is worth 65 Elo points in this experiment as compared with playing without opening book at all.

The difference between No-Book and the best suiting book varies according to the chosen engine :

Code: Select all

Engine           Book       delta vs no book :
Rybka 1.0 Beta   Monster    +82  (2947 vs 2865)
Toga II 1.3x4    Monster    +61  (2897 vs 2836)
Spike 1.2 Turin  Monster    +69  (2846 vs 2777)
Naum 2.0         DClarge    +102 (2822 vs 2720)


So it seems that Naum and Rybka gain a little more from playing with an opening book than Spike and Toga.

Rybka without book is still stronger than Spike and Naum playing with their best suited opening book.

As compared with no book at all, playing with a general book is worth 65 elo points.
This is in the same order of magnitude as searching one-ply further, doubling the clock frequency or playing on dual processsor as compared with single processor : not that high but not insignificant.

These results (and more) will soon be available on my site.

Comments ?

Marc

[Volker Pittlik]

...
Comments ?
...

Very interesting. I was planing to make a similar test myself, but it seems you have saved some of my time .

I assumed so far that the differences between the different opening books are not so big as sometimes claimed. In fact the differences between the small and very big books in your experiment are in the error of measurement. I assume to have a book which isn't created to be lousy is enough.

There could be a difference to books created for an specific opponent, especially if you have access to its book.


Volker

[Tony Thomas]

Does your test answers how much weaker does an engine plays with a general book as opposed to a tuned book? I have heard that Zapchess and Junior plays about 100 points stronger with their own books compared to playing with general books, but I have yet to see the same rating list testing either of those engines using general books and the own book at the same time. I would have tested it for myself if I had either one of those chessbase programs, unfortunately I dont.

[Marc Lacrosse]

...
Comments ?
...

(..)
I assumed so far that the differences between the different opening books are not so big as sometimes claimed. In fact the differences between the small and very big books in your experiment are in the error of measurement. I assume to have a book which isn't created to be lousy is enough.

Yes indeed.

The most important factor is the quality of the underlying database.

I played a lot combining books with the "merge" feature after having created elementary books with various parameters ("min-game", "min-score") but this does not really affect overall performances of the resulting books.

Polyglot heuristics for book-building is very powerful "per se".

I have at least 5 or 6 "general" books whose performances are very close to each other when considered globally with several engines.

An interesting point though is the fact that when you consider one single engine, then the results with these different books can vary significantly : two books that are "globally" equivalent when tested with several engines can prove to suit a given engine very differently.

There could be a difference to books created for an specific opponent, especially if you have access to its book.
Volker

Sure!
This is a completely different matter.

I tested the same engines with the book I had "cooked" for Fruit at CCTblitz 2005 (Fruit won ex aequo with junior ahead of shredder, hiarcs, ...), playing against the same engines associated with my newer general books.

Results with the CCT book were disastrous for Spike, Rybka and Naum (closer to those with no book than to those with any general recent book!).

Only Toga had good results with it (this shows that this book was probably well tuned for the Fruit family of engines).
But even Toga had (slightly) better tests with the best-suited
recent general book than with this already old hand-made book...



Marc