Simple Tool to generate opening books?

[Peter Obermair]

Hello!

Is there a simple tool that helps to generate opening books (from pgn) to Polyglot?

Thank you

Peter

[Dann Corbit]

Hello!

Is there a simple tool that helps to generate opening books (from pgn) to Polyglot?

Thank you

Peter

A simple tool to generate polyglot books is polyglot.
"make-book" is the command to do it.

This is from the help file:

Book Making
-----------

You can compile a PGN file into a binary book using PolyGlot on the
command line. At the moment, only a main (random) book is provided.
It is not yet possible to control opening lines manually. I am
working on it though.

Usage: "polyglot make-book <options>".

"make-book" options are:

- "-pgn"

Name of the input PGN file. PolyGlot should support any
standard-conforming file. Let me know if you encounter a problem.

- "-bin"

Name of the output binary file. I suggest ".bin" as the extension but in fact PolyGlot does not care.

- "-max-ply" (default: infinite)

How many plies (half moves) to read for each game. E.g. if set to
"20", only the first 10 full moves of each game will be scanned.

- "-min-game" (default: 3)

How many times must a move be played to be kept in the book. In other words, moves that were played too rarely will be left out. If you scan full games "2" seems a minimum, but if you selected lines
manually "1" will make sense.

- "-only-white" *** NEW ***

Save only white moves. This allows to use different parameters for
white and black books, and merge them into a single file with the
"merge-book" command, see below.

- "-only-black" *** NEW ***

Same for black moves.

- "-uniform" *** NEW ***

By default, a probability is calculated by PolyGlot for each move
depending on how popular it is (how often it was playing in the
provided PGN file) and how much it "scored". This option bypasses the default mechanism and affects equal probability to all moves. This allows more variety of play.

This option is normally used only with hand-selected lines (e.g. "user books").

---

Example: "polyglot make-book -pgn games.pgn -bin book.bin -max-ply 30".

Building a book is usually very fast (a few minutes at most). Note
however that a lot of memory may be required. To reduce memory usage, select a ply limit.


Book Merging
------------

*** NEW ***

Usage: "polyglot merge-book -in1 <file1> -in2 <file2> -out <file>"

Merge two bin files into a single one. <file1> has "priority"; this
means that if a position is present in both input books, data from
<file2> will be ignored for this position.

The two main applications are:

1) combine a white book and a black book (in which case priority does not matter)

2) combine a "user book" of manually-selected lines with a broader one from a large game set

What follows is an admitedly complicated example of how this can be used. DO NOT MAILBOMB ME IF YOU DO NOT UNDERSTAND!

My hope is that at least one advanced user will get what I mean and writes a better explanation on a web page or forum thread (yes, that's YOU, thanks by the way) ...

---

Imagine that we've got 4 PGN files as follows:

w1.pgn: fixed white lines, all moves manually checked
w2.pgn: selected games (for random book as with PolyGlot 1.3)

b1.pgn and b2.pgn: same for black

The first step is to build 4 .bin files with appropriate options.
Lines starting with "> " indicate what is typed on the command line.

> polyglot make-book -min-game 1 -uniform -only-white -pgn w1.pgn -bin w1.bin

I added "-uniform" because it allows randomness in the fixed lines
(e.g. d4+e4 at 50%). It has no effect if lines are deterministic
(only one move for a given position).

"-min-game 1" is characteristic for user books. All moves are supposed to be safe so there is no reason to filter them with other heuristics.

> polyglot make-book -min-score 50 -only-white -pgn w2.pgn -bin w2.bin

This shows how min-score can actually be different for white and black (as with multiple books). I don't use "max-ply" because "min-game" default value of 3 will limit depth somewhat. You are of course free to use it.

Same for black:

> polyglot make-book -min-game 1 -uniform -only-black -pgn b1.pgn -bin b1.bin
> polyglot make-book -min-score 40 -only-black -pgn b2.pgn -bin b2.bin

At this point we have 4 .bin files. Notice that different parameters
were used for white and for black (not to mention that different PGN files can be used).

---

Let's now merge the white books.

> polyglot merge-book -in1 w1.bin -in2 w2.bin -out w.bin

Input files are not symmetrical, "in1" has priority over "in2".

"skipped xxx entries." message from PolyGlot means there were some position conflicts. This is normal since we want to overwrite some random moves with fixed lines instead.

Same for black:

> polyglot merge-book -in1 b1.bin -in2 b2.bin -out b.bin

Now we can finally merge the white and black books.

> polyglot merge-book -in1 w.bin -in2 b.bin -out book.bin

It's important to check that there are no conflicts, otherwise
something went wrong.

Note that this last operation was only made possible thanks to colour filtering, otherwise nearly all positions would lead to conflicts.
For this reason, it does not make much sense to mix old .bin files
(which contain moves for both colours).

All these command lines might seem numerous and complicated but they can be put together into batch files.