PolyGlot and opening book

[Fabien Letouzey]

Hi all,

As promised in the readme file, I am adding book support to PolyGlot.

For it to be usable, I need to provide commonly-used heuristics.
Experienced "book makers" can probably help me here.

In this post I want to focus on common random books built from game
collections, not hand-tuned ones yet. I also have to point out I am
considering here the PGN-extracting process, not how to choose a move
at run time (probabilities, which for now are simply the popularity).

In other words: given a PGN file, how does one usually decide whether
a move should be included in book or not.

For now I only have:

- minimum number of games; what's a typical value: 3-5?

Other common heuristics might be:

- at least one win (or draw)?
- minimum average score (e.g. 25% of the points)?
- minimum proportion (e.g. played 10% of the time)?

Maybe some others?

- ?

I would like to provide at least the most useful ones, but also with
decent default values.

Thanks for your help,

Fabien.

[Fabien Letouzey]

I forgot to mention another common heuristic:

- only keep the first n plies of each game; typical values: 20-30?

Fabien.

[Pallav Nawani]

Some more heuristics:
- ELO Ranking of the players
- Game length. If a game is a quick loss, then its likely that a player made a mistake in opening.
- Draws (I think draws are good, but some people exclude draws.

[Fabien Letouzey]

Some more heuristics:
- ELO Ranking of the players
- Game length. If a game is a quick loss, then its likely that a player made a mistake in opening.
- Draws (I think draws are good, but some people exclude draws.

Hi!

Hm ... IMO game selection should be done before submitting the PGN for book making. Or is any of these features missing in chess GUIs?

Fabien.

[Peter Fendrich]

Some more heuristics:
- ELO Ranking of the players
- Game length. If a game is a quick loss, then its likely that a player made a mistake in opening.
- Draws (I think draws are good, but some people exclude draws.

Hi!

Hm ... IMO game selection should be done before submitting the PGN for book making. Or is any of these features missing in chess GUIs?

Fabien.

I think you're right.
Another thing:
Reserve space (x bits) for each move with information that the engine can put to and get from.
This enables book-learning for those interested.
Probably best to have an on/off feature for engines without learning that want to save space...

/Peter

[Dann Corbit]

Hi all,

As promised in the readme file, I am adding book support to PolyGlot.

For it to be usable, I need to provide commonly-used heuristics.
Experienced "book makers" can probably help me here.

In this post I want to focus on common random books built from game
collections, not hand-tuned ones yet. I also have to point out I am
considering here the PGN-extracting process, not how to choose a move
at run time (probabilities, which for now are simply the popularity).

In other words: given a PGN file, how does one usually decide whether
a move should be included in book or not.

For now I only have:

- minimum number of games; what's a typical value: 3-5?

Other common heuristics might be:

- at least one win (or draw)?
- minimum average score (e.g. 25% of the points)?
- minimum proportion (e.g. played 10% of the time)?

Maybe some others?

- ?

I would like to provide at least the most useful ones, but also with
decent default values.

Thanks for your help,

Fabien.

You can use SCID or ChessAssistant (among other things) to filter the games for book creation.

My own favorite choices are:
1. Correpsondence games between players in the top 2%
2. Tournament level games between players of 2625 Elo or higher (both)
3. SSDF games on 1200 MHz machines between highly rated opponents

Things that are nice to add to the book are W/L/D numbers, a 2 byte slot for centipawn evaluation and a 1 byte entry for depth of that evaluation.
A slot for nags from each of: BCO/ECO/MCO/NCO
A counter for recent wins/losses/draws by your chess engine at blitz time control and a second counter set for your chess engine at standard time control.

Instead of a custom format, why not use FastDB:
http://www.garret.ru/~knizhnik/fastdb.html

Then, you can pull statistical data with SQL queries instead of writing custom programs. Far less tedious and error prone.

[Fabien Letouzey]

I think you're right.
Another thing:
Reserve space (x bits) for each move with information that the engine can put to and get from.
This enables book-learning for those interested.
Probably best to have an on/off feature for engines without learning that want to save space...

Hi Peter,

It is a PolyGlot book. Think of it as a GUI one if you want. The UCI engine is not aware of anything!

Learning will be handled by PolyGlot (optional of course).

Fabien.

[Peter Fendrich]

I think you're right.
Another thing:
Reserve space (x bits) for each move with information that the engine can put to and get from.
This enables book-learning for those interested.
Probably best to have an on/off feature for engines without learning that want to save space...

Hi Peter,
It is a PolyGlot book. Think of it as a GUI one if you want. The UCI engine is not aware of anything!
Learning will be handled by PolyGlot (optional of course).
Fabien.

Sure it is,
but that doesn't mean it must to be like that.
I would prefer my own learning alg's
/Peter