Polyglot opening book specification

[Michel]

I have written out the Polyglot opening book specification. You can find it on my Toga website

http://alpha.uhasselt.be/Research/Algebra/Toga/

When I find some more time I will provide tests and sample code.

In any case feedback is appreciated.

Regards,
Michel

[Harald Lüßen]

"row" and "file" are counted from 0 to 7. "kind_of_piece" is encoded as follows

Could you explain the square mapping please. Which row and file is A1, H1 A8, H8?

If the opponent has pushed a pawn which can be taken en passant then "enpassant" is the entry from RandomEnPassant whose offset is the file of the pushed pawn (counted from 0 to 7).

Is it coded always or only when a second pawn exists to do the ep move?

Could you write explicitely that the zobrist value of castle and enpassant is 0
if the other values do not apply. Dito the whole zobrist value calculation
starts with 0, right?

Is this algorithm free to use in every engine? I have my own keys and algorithm today
but I could use this one. Otherwise there would be two tables for engines
that want to use polyglot books.

Do you have any idea how 'good' these values are? Collisions? Hamming distance?

Harald

[Teemu Pudas]

"row" and "file" are counted from 0 to 7. "kind_of_piece" is encoded as follows

Could you explain the square mapping please. Which row and file is A1, H1 A8, H8?

A1 = 0, H1 = 7, A8 = 56, H8 = 63

File = <file> - 'A', row = <rank> - '1'.

If the opponent has pushed a pawn which can be taken en passant then "enpassant" is the entry from RandomEnPassant whose offset is the file of the pushed pawn (counted from 0 to 7).

Is it coded always or only when a second pawn exists to do the ep move?

And what if there's a pawn but no legal ep capture? Ditto for castling rights: are these two positions the same?
[diag]4k3/8/8/8/8/4q3/8/4K2R w K -[/diag]
4k3/8/8/8/8/4q3/8/4K2R w K -
4k3/8/8/8/8/4q3/8/4K2R w - -

[Zach Wegner]

4k3/8/8/8/8/4q3/8/4K2R w K -
4k3/8/8/8/8/4q3/8/4K2R w - -

I seriously doubt any program differentiates between these two positions. The program would have to generate all moves and see if there are any that preserve its castling rights. So yes, there is a difference.

As for e.p., I don't know the specifics of the PG book format, but it likely follows the FEN standard, which is that the e.p. square is set iff a pawn is next to the pawn just moved, but it doesn't necessarily have to be legal to capture.

[Michel]

Thanks for the feedback so far. I will try to reply and change the document accordingly.

Could you explain the square mapping please. Which row and file is A1, H1 A8, H8?

As already indicated A is file 0, row 1 in the usual chess notation is encoded as 0 here.

Is it coded always or only when a second pawn exists to do the ep move?

It is coded only if it can there is a pawn in the correct position to take it, but the capture need not be legal (see Zach's reply)(in other words: your second possibilty).

Could you write explicitely that the zobrist value of castle and enpassant is 0
if the other values do not apply. Dito the whole zobrist value calculation
starts with 0, right?

I will. Thanks.

Is this algorithm free to use in every engine?

I think this algorithm is free to use even for closed source engines.

As you know Polyglot is GPL but the GPL only covers actual code and
not algorithms (the latter can be covered by patents, which luckily have
not appeared for chess engines).

The array Random64 is course taken from the Polyglot source code
but I am pretty sure that a table of random numbers is never covered by
copyright.

A GPL'ed engine can of course take the code directly from polyglot. Others
have to reimplement the algorithm.

Do you have any idea how 'good' these values are? Collisions? Hamming distance?

Not formally. Basically a Zobrist hash function behaves like a random mapping
to a set with 2**64=16x10**18 elements.

If you have an opening book with 10**5 positions then the chance that an individual
position matches one from the opening book is one in 16x10**13 which is very low.

Personally I have never observed a hash collision with a Polyglot opening book.

If you check moves for legality then the chance of a collision is even lower.

If you want to use this hash function as the main hash function of the engine then
you have to take the Birthday paradox into account but this only comes into
play at around 2**32 positions.


Teemy Pudas: your question about castling is a good one. I suspect that in
your position white would still have the "can castle short" flag but I am not
a hundred percent sure. I will check.

[Michel]

Zach Wegner wrote

As for e.p., I don't know the specifics of the PG book format, but it likely follows the FEN standard, which is that the e.p. square is set iff a pawn is next to the pawn just moved, but it doesn't necessarily have to be legal to capture.

Actually are you sure this is the fen standard? This is not what I am reading here

http://www.drpribut.com/sports/standard.txt

A quick test with xboard seems to indicate that after a double pawn push there is always an ep target square even if there is no pawn to do the capturing.

So this is different from Polyglot.

(( I think this might be a small bug in Polyglot. If I read the source correctly then while importing a fen Polyglot follows the fen standard but in normal gameplay it does not. See fen.cpp and move_do.cpp ))

[Zach Wegner]

Yeah, my bad. I was confusing the FEN and ZCT standards.

[Teemu Pudas]

4k3/8/8/8/8/4q3/8/4K2R w K -
4k3/8/8/8/8/4q3/8/4K2R w - -

I seriously doubt any program differentiates between these two positions. The program would have to generate all moves and see if there are any that preserve its castling rights.

I only do it when I'm in check.

Besides, I don't see how a GUI can claim to have repetition detection if it doesn't do it this way.

[Marc Lacrosse]

I have written out the Polyglot opening book specification. You can find it on my Toga website

http://alpha.uhasselt.be/Research/Algebra/Toga/

When I find some more time I will provide tests and sample code.

In any case feedback is appreciated.

Regards,
Michel

A small request from one who uses Polyglot books a lot (and even helped define the book-making functionalities through dozens of mail exchanges with Fabien) but is no programmer at all :
I deeply miss the possibility to add a candidate-move for a given position with 0% probability of play, without changing any other present probability in the book.
Ideally this should be a windows command-line utility with a syntax similar to this:

Code: Select all

PGmoveAdd.exe SomeBook.Bin someFenOrEPD SomeMoveInAlgebraicNotation ModifiedBook.bin

Ideally this utility should check for the inexistence of the given move and issue an erormessage in case the given move already exists for this position in the book (but this is not important).

Could one of those who master the PG book format have a look at this request?

Such a small utility could really change the way all PG bookmakers do work.

Thanks in advance

Marc

[Michel]

I have posted a new version of the specification here

http://alpha.uhasselt.be/Research/Algebra/Toga/

There is also some test data.

Thanks for the people that provided feedback. This has allowed me to remove a number
of ambiguities.

Michel

[Michel]

Hi Marc.

Thanks for your great work on polyglot opening books!

I plan to write some sample code anyway to verify that I have made no mistakes in the specification. A utility to insert a FEN in a PG book would be a good exercise.
Or rather a delete/insert/modify.

Currently I am a bit busy but I will do it soon.

Michel

[Marc Lacrosse]

A utility to insert a FEN in a PG book would be a good exercise.
Or rather a delete/insert/modify.

Michel

Nice !

In the present state, modifying the weights is easily done in recent versions of the SCID database by P. George. But a command line utility would allow to automate things and would be welcomed indeed.

The "delete" function would also be very nice (AFAIK no utility allows to do that in the present state).
In case you plan to add a "delete" (or weight put to zero) would you please consider something like a switch for recursively deleting any sons and grandsons of the deleted position (and so on) as far as the considered positions are not reachable by other pathways in the book ?

Thanks again for your interest in PG books !

Marc

[Michel]

I have posted a new version as I had forgotten to indicate how castling moves are represented.

See here

http://alpha.uhasselt.be/Research/Algebra/Toga/

Michel

[Michel]

Once again a new version. There are no changes in the specs but one
of the test keys was wrong (a copying error).

I have now independently verified them.

[Michel]

I have now included some sample code in the specification

* pg_key.c: This little utility computes the PG key of a FEN. Note that it does ZERO error checking on the legality of the FEN and hence will segfault if something is wrong.
* pg_show.c: This utility looks up a PG key in a PG book and prints out the moves and their (relative) weights.

This code is released in the public domain.

As always, look here

http://alpha.uhasselt.be/Research/Algebra/Toga/

Regards,
Michel

[Michel]

Do you know how to compile things on Windows (e.g. using Cygwin or MinGW)? I normally use Linux and testing if things work properly on Windows takes some time (a reboot for starters!).

So it would be easier if I can just post sources of utilities.

Regards,
Michel

[Denis P. Mendoza]

Do you know how to compile things on Windows (e.g. using Cygwin or MinGW)? I normally use Linux and testing if things work properly on Windows takes some time (a reboot for starters!).

So it would be easier if I can just post sources of utilities.

Regards,
Michel

Try this (MSVC6) windows compile:

http://computerchessengines.mylivepage. ... ileid=3993

It's a good start for this polyglot project!

Many thanks Michel!

Denis

[Marc Lacrosse]

[
Try this (MSVC6) windows compile:

http://computerchessengines.mylivepage. ... ileid=3993

It's a good start for this polyglot project!

Many thanks Michel!

Denis

It compile works fine under W32 :

Code: Select all

C:\ch\0 work>pg_key "rnbqkbnr/ppp1pppp/8/3p4/3P4/8/PPP1PPPP/RNBQKBNR w KQkq d6"
06649ba69b8c9ff8

C:\ch\0 work>pg_show performance.bin 06649ba69b8c9ff8
move=c2c4 weight=76.03%
move=g1f3 weight=20.66%
move=c1g5 weight= 3.31%

Thanks to both !

Marc

[Michel]

Thanks Dennis and Marc,

However the two utilities were just "sample code" meant as a quick illustration of the specification (to help programmers). Not serious tools. Especially pg_key does not do any error checking. So it is not an example of good programming. I am pleased anyway that they work out of the box on windows without any testing.

I will now write the utility Marc asked me (insert a FEN with zero weight). It seems everybody knows how to compile on windows, so I can just post the source code.

Recursive delete is slightly more involved since it requires knowing about moves. I could take code from Polyglot for that but perhaps that is overkill.

Regards,
Michel

[Michel]

Hi Marc,

PGmoveAdd.exe SomeBook.Bin someFenOrEPD SomeMoveInAlgebraicNotation ModifiedBook.bin

I coded this utility. See

http://alpha.uhasselt.be/Research/Algeb ... 0.1.tar.gz

It is called pg_add and has the parameters you requested.

Code: Select all

$ fen="rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1"
$ pg_query performance.bin "$fen"
move=e2e4 weight=33.33%
move=d2d4 weight=33.33%
move=c2c4 weight=33.33%
$ pg_add performance.bin "$fen" a2a4 more_performance.bin
$ ./pg_query more_performance.bin "$fen"
move=e2e4 weight=33.33%
move=d2d4 weight=33.33%
move=c2c4 weight=33.33%
move=a2a4 weight= 0.00%


I have not tested it on Windows but it pure ANSI C so it should work.

I did some testing, but of course you should be careful that possible bugs do not corrupt
your books.

If there is a problem then I will check it.

Regards,
Michel

EDIT: If the FEN is not already in the book then currently pg_add prints an error message
and exits. Is that what you want?