Polyglot opening book specification

[Denis P. Mendoza]

This is becoming interesting Michel!

From Ilari's 'omniperft' project, I was forced to use GCC as compiler, as you already had a makefile. Here are the GCC builds!

http://computerchessengines.mylivepage. ... ileid=3996

What do you think Marc?

[Michel]

I just noticed that the 0.1 version does not operate correctly when compiled with Visual Studio (Cygwin is fine). The problem is that Windows expects fopen to be used with
the "b" flag whereas on Linux IO is always binary.

I have posted a version 0.1,1 which seems to work fine.

[Denis P. Mendoza]

Here's the updated GCC builds:

http://computerchessengines.mylivepage. ... ileid=3997

[ulver]

Such a small utility could really change the way all PG bookmakers do work.
[/quote] That sounds very interesting-is this tool works like allow manual adding in .ctg book?

[Thomas McBurney]

I have written out the Polyglot opening book specification. You can find it on my Toga website

http://alpha.uhasselt.be/Research/Algebra/Toga/

Thanks Michel,

I have started implementing it in FreeBASIC which should make it easier for authors to implement Polyglot books to other chess engine written in BASIC. I will make the source code available on my web site once I have finished the file reading part of the program. Hopefully I'll have this done next week.

Output from FreeBASIC program using your test data...

Code: Select all

FEN=rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1
Key=463B96181691FC9C

FENrnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 0
Key=823C9B50FD114196

FENrnbqkbnr/ppp1pppp/8/3p4/4P3/8/PPPP1PPP/RNBQKBNR w KQkq d6 0 2
Key=756B94461C50FB0

FENrnbqkbnr/ppp1pppp/8/3pP3/8/8/PPPP1PPP/RNBQKBNR b KQkq - 0 2
Key=662FAFB965DB29D4

FENrnbqkbnr/ppp1p1pp/8/3pPp2/8/8/PPPP1PPP/RNBQKBNR w KQkq f6 0 3
Key=22A48B5A8E47FF78

FENrnbqkbnr/ppp1p1pp/8/3pPp2/8/8/PPPPKPPP/RNBQ1BNR b kq - 0 3
Key=652A607CA3F242C1

FENrnbq1bnr/ppp1pkpp/8/3pPp2/8/8/PPPPKPPP/RNBQ1BNR w - - 0 4
Key=FDD303C946BDD9

FENrnbqkbnr/p1pppppp/8/8/PpP4P/8/1P1PPPP1/RNBQKBNR b KQkq c3 0 3
Key=3C8123EA7B067637

FENrnbqkbnr/p1pppppp/8/8/P6P/R1p5/1P1PPPP1/1NBQKBNR b Kkq - 0 4
Key=5C3F9B829B279560



Cheers,
Tom.

[Michel]

Hi Denis,

Here's the updated GCC builds:

Unfortunately they don't work (at least not in wine). I suspect there is something going on with the 64bit types of gcc on Windows. I get

$ wine pg_key.exe "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1"
000000001691fc9c

which is wrong.

The Cygwin gcc build works fine but then you need to include the cygwin dll. VS
also works.

I will try to debug this but today I have something else to do.

Regards,
Michel

[Michel]

I have started implementing it in FreeBASIC which should make it easier for authors to implement Polyglot books to other chess engine written in BASIC.

Thanks! The more independent implementations the better.

Regards,
Michel

[Marc Lacrosse]

That sounds very interesting-is this tool works like allow manual adding in .ctg book?

Yes, it's the same function, but in a much more "rough" style.

So far with SCID you could modify the probability that a move will be played (including setting it to 0% which is equivalemt to deleting it) but you could not add new moves to a book.

With this one you may add new moves.

So let's imagine you are examining a polyglot book in SCID.
You see that in a given position you would wish to add some move in the list of candidate moves.

Now this is possible :
1. you copy the epd of the position
2. You use Michel's utility from the command line to add the move
3. You reopen the modified book in SCID and adjust the probability

This is still more complicated than with ctg books, but it's nevertheless a great progress !

I do not doubt that we will soon see some kind of dedicated GUI for editing/tuning polyglot books.

Marc

[Michel]

I do not doubt that we will soon see some kind of dedicated GUI for editing/tuning polyglot books.

Yes that's what I hope also. I have no time to do GUI programming but I hope the sample code will reduce the threshold for somebody starting to build a GUI.

[Marc Lacrosse]

I do not doubt that we will soon see some kind of dedicated GUI for editing/tuning polyglot books.

Yes that's what I hope also. I have no time to do GUI programming but I hope the sample code will reduce the threshold for somebody starting to build a GUI.

I just sent a mail to Pascal George to ask him to consider adding a "Add-New-Move-in-a-PG-book" to his PG books editing functions in SCID.
I already discussed this topic with him at spring 2007 and at that time he told me that he was not interested in adding this feature...
But maybe he could change his mind ...
... and Michel's work could make this task easier ?

I will let you know.

Marc

[Michel]

I already discussed this topic with him at spring 2007 and at that time he told me that he was not interested in adding this feature...

I wonder why. It should be a trivial addition and it is obviously useful.

Michel

[Michel]

$ wine pg_key.exe "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1"
000000001691fc9c

Ok I fixed that.

It turned out that it was simply a problem with the printf function of gcc
under Windows (gcc uses the Windows printf function which is not standard compliant).

The other utilities pg_query and pg_add were not affected.

[Denis P. Mendoza]

Copy mate! Made some MingW for a change.

http://computerchessengines.mylivepage. ... ileid=3998

I hope Pascal George reconsider to make this project 'fruitful'. It will surely improve the polyglot 'Book Tuning' features in SCID. In the first place, how can he refuse. His initials fit the project !

Maybe Fonzy could add these utilities too in polyglot console commands.

Keep it up and thanks.

Denis

[Marc Lacrosse]

I just sent a mail to Pascal George to ask him to consider adding a "Add-New-Move-in-a-PG-book" to his PG books editing functions in SCID.

Unfortunately Pascal answered that he is too busy to work on SCID these days.
He advised to use Michel's utility for adding moves
He also told me that he finds Michel's work interesting and that a good wealth of info is already to be found in SCID source code ...

So if someone is able to have a look at SCID sources ...

Marc

[Michel]

The code that handles the book seems to be

tcl/windows/book.tcl

in the scid source code.

As you can see it is written in Tcl/Tk. It is not very long.

I can try to think about it, but it has been a long time since I did any tcl/tk programming.
So probably no quick results

Perhaps other people with tcl/tk experience can give it a try also.

BTW I noticed that scid contains a copy of Polyglot and just calls polyglot for its
book handling code.

[Michel]

In my spare time I am implementing a button to add moves in the book tuning window in scid. It will be a hacky solution but it will work.

Michel

PS. One thing I find strange about PG books is that there does not have to be a path
from the root node (the starting position) to every node in the book. In other words you can never know what is exactly in the book, unless you made the book yourself, and even then you may have forgotten.

I am thinking of implementing a utility that will transform a book into an equivalent
one in in which there is a path from the root node to every node by inserting moves
with zero weights. It requires some thought how this should be implemented.

There should be a converse utility which deletes all moves with zero weight (that is trivial
of course).

[Michel]

In my spare time I am implementing a button to add moves in the book tuning window in scid. It will be a hacky solution but it will work.

Ok I have done that. Unfortunately I have no idea how to compile scid under windows.
I have asked Pascal George to explain.

Here is a source tar ball.

http://alpha.uhasselt.be/Research/Algeb ... .26.tar.gz

Perhaps more useful is the patch

http://alpha.uhasselt.be/Research/Algeb ... oves.patch

because it tells you what has changed.

Regards,
Michel

[Michel]

Unfortunately I have no idea how to compile scid under windows.

I have succeeded in compiling Scid in Mingw. Here is a tarball which contains an executable which can add moves to Polyglot books.

http://alpha.uhasselt.be/Research/Algeb ... exp.tar.gz

(look in the directory scid_exp/bin )

IT ONLY WORKS IN ENGLISH SO FAR.

This addition will be in the next version of Scid. Pascal Georges has said he will fix the translation stuff.

Regards,
Michel

[Marc Lacrosse]

I have succeeded in compiling Scid in Mingw. Here is a tarball which contains an executable which can add moves to Polyglot books.

This seems to work perfectly here.

Thanks a lot Michel !

Marc

[matematiko]

Hello guys,

I'd like to thank you for all the hard work you are putting into making tuning polyglot books a nicier and easier experience. I am very impressed with your results. You started with a small utility and ended up creating a modified executable for the whole SCID GUI....WoW!!!!

Now of course, time to ask a question. I've been making polyglot books for a while now and I play all the time in Yahoo Chess servers against other engine users, my speciality and only interest is 1/0 games. My technique is to create a book for whites and a book for blacks then merge them into one book. Due to some hardware problems and a motherboard upgrade I lost all my databases and my bin books with the exception of some that I e-mailed to a friend. So I ended up with a book that is good for whites and a book that is good for blacks but I dont have the originals (white only and black only to merge them). Anyway, I decided that instead of making new books every month is time to start learning to hand tune polyglot books. So, today I played few games as blacks using the book I know is good for whites but not so good for blacks. One of the games left me in a very bad position and I said to my self "hurrayyyyyy, now I have some material to start making changes to this book", you know analize and find the blunder and make corrections accordingly.

But as soon as I started I found something very odd. My opponet's first move was e4 and my book responded with Nf6. The weird thing is that the weight for Nf6 is 0%. The way I understand this is that my book should had never make this move if the weight is 0%. So, how is this possible? What am I missing? What is the point of telling SCID to change the book in a way to not make a certain move in a certain position and still the book makes it?

Thanks in advance for you help.

Change of subject, my speciality is perl (intermidiate programmer), packet sniffing, protocol hacker and such. It had crossed my mind many times the idea of creating a "middle man"program to connect to PCC using Winboard and polyglot instead of the chessbase GUIs. The journey will be long but not impossible. What do you think? There will be and advantage of doing something like this? Or it will be just a waste of time? I'd really like to see Rybka playing in PCC 1/0 games without having to use ctg books and without dealing with all that cpu load that using a chesbase GUI represents. Any comments?

Cheers,