Glass Opening Book Manager 1.0 - Released!

[Edmund]

Together with Pawel Koziol I am developing the Glass Chess Engine. This engine has its own Opening Book Format: Glass Opening Book (GOB).

For several reasons we decided to make the Opening Book Format and a tool to manage such opening books freely available for anybody to use. This can help engine developers who want to add this opening book support to their program, opening book designers and the people using the chess engines.

Of cause you will argue that a lot of other opening books exist, so why use this new format. First off all, the main characteristic of this opening book is that it is very simple. It only stores 12byte per entry (Hash, Move, Value, Manually edited Value, Weight). It is very memory friendly, as the Zobrist keys only require 392bytes. The opening book uses symmetric keys (ie. 1.e3 e5 2. e4 transposes into 1.e4 e5) - thanks to Gerd Isenberg for sharing his ideas on this topic. For each move one can select two different modes (tournament and normal mode) which can have different probabilities for a move. GOB uses the king captures rook castling rules and is thus capable of complying with the Ches960 rules.

The second important reason why you might want to try the GOB format, is because of its Opening Book Manager. It is an easy to use GUI that can be used to manipulate Glass Opening Books. One can entry new moves either right through the program itself, through a PGN importer, an algebraic notation file, or a binary import file. Another nice feature is its ability to load any UCI engine. One can then directly assign engine analysis values to certain positions (this feature is currently only possible with Glass Chess Engine). Furthermore I added the MiniMax function. This algorithm propagates certain values from nodes to the parent nodes and can be easily used to find problematic lines in the opening book.

So if you are interested in trying it, below you find a link to a page where you can download the Glass Opening Book Manger. In the zip file there is also a specification of the Opening Book Format and an explanation to the Program itself. Furthermore Pawel Koziol was so kind to design an example opening book, in this file you can see the effect of the MiniMax function.

http://www.marittima.pl/glass

[Rohan Padhye]

I liked it at first look. Didn't try it with the engine, just the book manager. Couple of suggestions/observations:

(1) It would be nice if your manager (not .gob files) could identify openings by their names/ECO codes. That would make the interface much friendlier.

(2) When selecting a possible move from the right side list avoid opening the context menu for left-click. Context menus are better off left for right-clicks. Besdies, all those options can be set by left-clicking on the item and changing the value below as well. So yeah, left click should only select, not open a menu.

I also noticed a couple of possible bugs but I'm not sure if this is the right place to point them out...

(1)When changing settings from the bottom-right panel, the changes don't appear on the list immediately. Is this a bug or is it supposed to work like that? For example I assign an annotation ?? to a move. Only when I select some other move and change it's property does the annotation appear on the previous move.

(2) After choosing export mode if I want to go back I 'Cancel' the file dialog and this causes a runtime error and the program exits.

[Edmund]

I liked it at first look. Didn't try it with the engine, just the book manager. Couple of suggestions/observations:

(1) It would be nice if your manager (not .gob files) could identify openings by their names/ECO codes. That would make the interface much friendlier.

(2) When selecting a possible move from the right side list avoid opening the context menu for left-click. Context menus are better off left for right-clicks. Besdies, all those options can be set by left-clicking on the item and changing the value below as well. So yeah, left click should only select, not open a menu.

I also noticed a couple of possible bugs but I'm not sure if this is the right place to point them out...

(1)When changing settings from the bottom-right panel, the changes don't appear on the list immediately. Is this a bug or is it supposed to work like that? For example I assign an annotation ?? to a move. Only when I select some other move and change it's property does the annotation appear on the previous move.

(2) After choosing export mode if I want to go back I 'Cancel' the file dialog and this causes a runtime error and the program exits.

Thanks for the comments!

I can try to implement the eco classifications in the next release.

I already tried the thing about the right-click, but it didn't really work with the standard functions I am using here in Visual Basic.

Concerning the bottom-right-panel, the information is about the move just played. So in the starting position you won't be able to set anything because there hasn't been any move played. So eg. if you play e2e4, then change the comment to "!" and then go back to the startposition, you will see the "e2e4!" in the list.

Thanks for the info concerning the bug with the export. I just fixed that and it will be alright in the next release too.

regards,
Edmund

[Dave Gomboc]

The opening book uses symmetric keys (ie. 1.e3 e5 2. e4 transposes into 1.e4 e5) - thanks to Gerd Isenberg for sharing his ideas on this topic.

I'd like to read more about the symmetric keys. Could you or Gerd please provide some links?

Thanks,
Dave

[Edmund]

The opening book uses symmetric keys (ie. 1.e3 e5 2. e4 transposes into 1.e4 e5) - thanks to Gerd Isenberg for sharing his ideas on this topic.

I'd like to read more about the symmetric keys. Could you or Gerd please provide some links?

Thanks,
Dave

Gerd's post: http://www.talkchess.com/forum/viewtopic.php?t=22274&postdays=0&postorder=asc&topic_view=flat&start=14

In the specification provided together with the Opening Book Manager I am describing my Hash-calculation in detail.

The main idea is to use Zobrist keys for black that are dependent on white's keys
So that in the end if a symmetric position is on the board the Zobrist keys are symmetrical as well. In Gerd's approach, symmetrical means: zob[black] == bswap(zob[white])

[crystalclear]

I took a look at the glass opening book format.

http://www.koziol.home.pl/marittima/gla ... istrib.zip

I'm still having teething trouble with it, but I think I'm getting there.

I could see that something strange appeared to be happening with castling flags and I didn't know whether 0 meant castling allowed or denied.
In the end I decided that the correct implementation and an easy test (albeit of an invalid chess position) is that the hash key of an empty board
with full castling rights is zero.

Then I added a king on H1 and got a hashcode that I could visibly see in the Zobrist pseudo random number table.


8/8/8/8/8/8/8/8 w KQkq - 0 1: 0000000000000000
8/8/8/8/8/8/8/7K w KQkq - 99 1: 456A96C1912CD954
8/8/8/8/8/8/8/7k w KQkq - 99 1: 6E4957D637E1E79F
StartPos: 95518ECECE8E5195 // Position hashcode
StartPos e2e4: DA638ECECE8E5195 // Bookmove hashcode

The 64 bit hashcodes that the book uses have the lower 48 bits for hashing the position and the upper bits for hashing the move.
These are stored in little-endian format in the file. Polyglot uses big-endian format and that confused me at first.

When I looked at a Glass book file with a hex editor, the entries appeared to be order in the same way that a polyglot book would be ordered.
But the keys are written backwards with respect to polglot keys.
That meant that instead of comparing keys I had to compare the keys with reversed byte order to locate and entry in the file.

Another complication was that I was reading polyglot moves directly out of the polyglot opening books using a positions hashkey and obtaining the moves from the book.

With the glass opening book, I have to first run my move generator and then verify that the hashed "position and move pair" is in the book.

I don't like the extra work, but I will apply it back to my polyglot opening book code, as in theory I could try to play an invalid move if I had a hash clash.
A hash clash with the glass opening book will always result in a valid move being played, even if it is hopelessly bad.

My remaining problems are probably a niggling little bug, but if it's anything significant I will edit my posting here.

Hmm: this is the edit I promised now!
The Glass documentation had this in it ...

Code: Select all

//in case it is black to move, the bytes are swapped:
if (stm) position = byteSwap(position);


I deactivated the software I'd used that implemented the lines above in my program.
I thought my opening book software had pretty much cracked things when it played e4, straight out of a Glass opening book.
But then it didn't play black moves. After a little detective work, I found that my hash codes for black positions were byte reversed.
Initially I thought that my byte swap software wasn't working. (I'd replaced a procedure call that didn't exist in my library with a gcc compiler built-in byte reversal.)
What was actually happening - I believe - is that bytes were being swapped back!
Removing the "if (stm) position = byteSwap(position);" line gave me correct hashcodes.

Code: Select all

U64 getZobristPcsq(U8 pc, U8 color, U8 sq) {
   if (color)       sq ^= 56;
   if (color^stm)   return byteSwap(*(U64 *) (random + pc*64 + sq));
   else          return *(U64 *) (random + pc*64 + sq);
}


I haven't understood the symmetrical hash codes that Glass uses. I'm merely trying to get things working.
My problem was maybe related to this procedure above. I implemented it in my software by having a 4th parameter with side to move information in it.

There are a few point in the Glass documentation where I was uncertain what was right.

Code: Select all

Then one has to generate the key for the move. This is done by xoring the key of placing the piece on the originating square with the key of placing it on the target square. No additional keys are added in case of a capture, ep move or similar. (eg. White Pawn moves e2e4)

U64 move = getZobristPcsq(0, 5, 11) ^ getZobristPcsq(0, 5, 27);


Clearly there is a mistake and color should be 0 and piece 5, not the other way round.
I suspect that the Glass documentation was written before that bit of their software was stable.

Anyway, my chess engine can now play a sequence of moves from a Glass opening book.

I really liked that the Glass website provided a tool that can read a FEN and display its hash code.
That was a great help in getting my software working. I'd like to thank the Glass guys for making
their stuff public and providing nice tools.

Edit:-
I just found that my castling stuff wasn't right. I had a glass opening book that was quite deep - a few megabytes of Ruy Lopez Breyer variation and yet my program would stop finding moves after playing about 5 against itself. It effectively stopped at a castling move. My castling rights were wrong and again the problem was byte swapping.
I think the Glass documentation needs to be more precise about the byte swapping, eg mentioning whether things are stored little-endian or big-endian in files, etc.

============

Any news anywhere on 6 men Gaviota tablebases?
I don't have the disk space I'd need, but I'd like to think that they exist or will exist sometime.