[Programming] Opening book

[Davy Jones]

Hello Winboard forum

I'm making a chess engine for an university project.
I got a problem with the opening books. I dont exactly get how it works. I know Winboard has a default_book.bin which is a opening book database. But how do I search in the database and give control to the book? Do I have to write source code to give control forth and back? Or is it allready integrated in Winboard (and if yes, how do i activate it?)

I was hoping i could tell Winboard to use it's default opening book (default_book.bin) when my chess engine starts and then it would do it automaticly until it goes out of book and then give control back to my chess engine.... does Winboard work that way?`Is it possible to do?

All help welcome

Regards Davy

[H.G.Muller]

Yes, that is how it works. WinBoard plays moves from the book as long as it finds them, and feeds them to the engine(s). Only when there are nobookmoves for the current position itgives the engines the 'go' command to set them thinking.

The .bin book file is just data. The code to probe it is part of WinBoard (in the source file book.c).

[Davy Jones]

Thx for the answer..
tho im still a bit confused.
How do i make my chess engine use the winboard book at start? (atm the searching algorithm start at first move).
I cant find any commands/help in the winboard protocol

Regards Davy

[H.G.Muller]

If your engineobeys the 'force' command, then it won't search from move 1. Because that is how it works. The GUI sends you 'force', and then a dozen of white and black moves (which it retreived from I sees that the game is out of book, it sends 'go' to the engine to start it searching.

[Davy Jones]

Thanks alot.. Made my chess engine use the opening book
In the Winboard.ini it says

"fmax 22" /fd="..\Fairy-Max" /firstXBook

What is the last part of the line (firstXBook)? Is it the opening book file? If yes, why cant i find it anywhere? Shouldnt i be able to edit the line to

"fmax 22" /fd="..\Fairy-Max" /default_book

to use the winboard default book?

Regards Davy

[H.G.Muller]

No, /firstXBook is an option that tells the WinBoard the first engine should use the GUI book. It is equivalent to unticking the 'Has own book' option for that engine in the Common Engine Options dialog, but including it with the engine makes sure that will happen automatically when you load that engine. To specify a book you would have to use /polyglotBook=default_book.bin, but it is in general less useful to specify such options (that would also affect the other engine with the engine) with the engine. Unless, perhaps, it is an engine that plays only atomic chess, so that you can be surethe opponent will also play that, and you would want the default book to be replaced by an atomic book.

[Davy Jones]

So if I write

"fmax 22" /fd="..\Fairy-Max" /firstXBook

It will use the default_book.bin (am i right?) Can I download another opening book and make that the default GUI book?

Regards Davy

[H.G.Muller]

Sure, you can specify the book to be used as GUI book in the Common Engine Options dialog, behind 'Use book:'

[Davy Jones]

Ok, got everything working now...
The last thing i need to know is what Winboard sends to the gui so the engine knows how the board looks like.

Does Winboard send the board when it goes out of book? Or does it sends the moves it do in the opening book one by one?

Regards Davy

[H.G.Muller]

It sends the moves, after putting the engine in force mode. When it is the engines turn, and there is no book move in the current position,i t sends the engine a 'go' commad to set it thinking. This does not preclude that you can get back in book later; before sending the opponent move that brings you back in the book, WB would then send 'force', the opponent move, and the engine's own (book) move.

[Davy Jones]

Im apparently stuck again
You say my engine has to obey force mode. I cant find much information about force mode in the protocol or any other places on the web.
Any chance you can explain me what to write/receive from Winboard to go into force mode? And what exactly does it do?

Below is what i think is right, can you confirm it?
- Winboard sends "force" to the engine on startup, right? What does my engine has to reply to enter force mode?
- When the opening book database goes out of book Winboard sends "go", right?

Regards Davy

[H.G.Muller]

The engine goes in force mode when it receives the command 'force'. The engine should not reply anything, it should just remember it is not playing for any side. At startup WB sends 'new', and the engine should understand it plays black after that, i.e. think up a move every time it is backs turn. If that is what you want, and you don't use the book, there would be no reason to send 'force'. WB can just send the move of the white player to the engine, and it will reply with a black move. When you want the engine to play white, WB sends it 'go', which means the engine should start playing the color that now has the move, i.e. white after the 'new' comand.

[Davy Jones]

The engine goes in force mode when it receives the command 'force'.

Okay, but i got no clue how i should make that source code for my engine.
How would the programming code look like? 'Do you have any pieces of code to make me understand?

Once again, thank you for your patience

Regards Davy

[H.G.Muller]

Yes, the 'Chess Programming Lessons' section of this forum contains code for a model WB protocol driver.

[crystalclear]

I've recently had a problem with my software using a polyglot opening book.

I use the chess position's hashkey to find entries in the opening book and then read the polyglot entries corresponding to that hashkey.
In there, there is move information that contains
from file
from rank
to file
to rank
promotion piece.

I was setting promotion piece in my software based on what was in in the opening book, and adjusting for different values for KNIGHT, QUEEN etc used by polyglot and my own program.
My program stumbled on what was effectively a move in the opening book that was Pawn h2-h3=Q or something like that.
Since a piece going from h2 to h3 cannot be a promotion, I should set my move to not be a promotion. Since the opening book contained a promotion piece I'd just copied that info and ended with an incorrect move in my program. Things got a bit nasty as the resulting thing my program stored was a different move to the one in the opening book, but nevertheless a legal move for reasons I will not go into.

I can't say polyglot is wrong, as I didn't write it and haven't seen a spec. And there is still a chance I have misunderstood events.
I am downloading a copy of polyglot now to see if I can find the bit I don't like in its source and modify it.
I have added some safeguards into my own software so that it can deal with existing polyglot opening books without having to adjust them to my way of thinking which is
promotion piece=NONE if a move is not a promotion.

I just mention things here in case anyone else has a niggling little opening book problem!

[H.G.Muller]

It sounds like an error in the book, because, as you say, h2-h3 could never be a promotion. In Chess this would simply be an invalid code in the move field, and Polyglot should never generate it. (Note that for variants with larger boards these codes can be used, but it could only get into the book when you edit it in such a variant with WinBoard, as Polyglot does not play variants.)

Which book did you find this in? Do you know the position where it occurred, or only the hash key?

[crystalclear]

I just went through the opening books I've downloaded with various engines in the past.
There are lots of files called book.bin and so I tend to rename them by appending a CRC32 of the file.
That helps me identify duplicate opening books.

I looked for things like this ...

Code: Select all

Example found: entry 3 of 257471 in C:\Chess\OpeningBooks\Alf_book109_c5f38322.bin
 Weird  Move              = b1e7
Example found: entry 1 of 368401 in C:\Users\protheric\Chess\OpeningBooks\v16_book\Book_7372bdbc.bin
 pg_to_file        = 1
 pg_to_rank        = 0
 pg_from_file      = 0
 pg_from_rank      = 4
 pg_promotion_piece= 3
 Weird  Move              = a5b1r


but when I eliminated opening books that were in a proprietary format (by assuming any real opening book would contain at least one move from StartPos) I was left with nothing of interest.

Code: Select all

No hashkey entry for StartPos found in C:\Chess\OpeningBooks\Alf_book109_c5f38322.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\bookc_e4d34958.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\book_4368d721.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\book_b8596ed9.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\glaurung_book_387bd364.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\Lime_binbook_d0df3e2b.bin. Not a polyglot opening book?
Book C:\Chess\OpeningBooks\performance_25150b06.bin is fine.
Book C:\Chess\OpeningBooks\scid_Elo2400_d0755235.bin is fine.
Book C:\Chess\OpeningBooks\scid_gm2600_96f04367.bin is fine.
Book C:\Chess\OpeningBooks\scid_varied_a3e0f52b.bin is fine.
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\v18_book\BOOK_c05aa8a6.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\v16_book\Book_7372bdbc.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\v14-v15_book\Book_a447d93f.bin. Not a polyglot opening book?

No hashkey entry for StartPos found in C:\Chess\OpeningBooks\GS_Alaric_11f_f378e15a.book. Not a polyglot opening book?
Book C:\Chess\OpeningBooks\GS_medium091230_dd8bcb2d.bin is fine.
Book C:\Chess\OpeningBooks\HannibalBookBig_53f28191.bin is fine.
Book C:\Chess\OpeningBooks\HannibalBookSmall_d82f5ec2.bin is fine.
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\ph-book2008_b506dcb2.bin. Not a polyglot opening book?
Book C:\Chess\OpeningBooks\Rybka_0a115dd7.bin is fine.
Book C:\Chess\OpeningBooks\TDDBII_f5e9c60b.bin is fine.
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\ThinkerX20_091230_e36cf79f.dat. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\ThinkerX40_091230_ae98205a.dat. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\wbqbook\WBQBOOK_3d35484c.bin. Not a polyglot opening book?
Book C:\Chess\OpeningBooks\Titans\Titans_ce6b7215.bin is fine.
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\TJbook\TJbook_252ebf61.bin. Not a polyglot opening book?
Book C:\Chess\OpeningBooks\T-X\T-X_2c720cad.bin is fine.
Book C:\Chess\OpeningBooks\book_small_cc4eab8b.bin is fine.
Book C:\Chess\OpeningBooks\twistedbook_fd814776.bin is fine.
Book C:\Chess\OpeningBooks\Book_f3a6cf29.bin is fine.

No hashkey entry for StartPos found in C:\Chess\OpeningBooks\NotGlot\book109_c5f38322.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\NotGlot\gavibook-small_98107f60.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\NotGlot\mainbook_a0434119.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\NotGlot\random1_ef9cd417.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\NotGlot\random2_796dd5af.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\NotGlot\tourbook_b33a316a.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\NotGlot\ATTACK_c5e9df87.BIN. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\NotGlot\book_aa977329.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\NotGlot\books_2f94ef23.bin. Not a polyglot opening book?
No hashkey entry for StartPos found in C:\Chess\OpeningBooks\NotGlot\position_ab088cee.bin. Not a polyglot opening book?

So it seems whatever opening book problems I've got are not a general problem.

It occurred to me that I really ought to check book moves much better. At the moment my program just trusts the book and plays the moves stored in there.
But since a hashkey can correspond to more than one position, with different possible moves, I ought to check that a retrieved move matches one of the moves from my move generator and only play a move that the move generator has approved.

I know that a hash clash is a low probability event and not likely to happen in practice. If an opening book is loaded by an engine that is installed on many computers and if these run tournaments of rapid play then eventually there would be an erroneous book move played. That would be worse than a hash clash in the hash table, where the likely consequences are that a non-PV is correctly rejected for the wrong reasons.

I still haven't got to the bottom of what has been happening in my opening book with my old version of polyglot, but I haven't found a similar problem in any opening books off the internet.

(Edit: 29th May 2012. I added some more analysis of opening books I downloaded from the internet. Everything that seemed to be in polyglot format was okay. The newer ones are in the list below the line break.)
(Edit: 15th June 2012. Added more opening books.)