.ctg Books in Winboard via Aquarium Book Adapter

[Marc Lacrosse]

I have a question for the book experts:
(...)
Now I know zilch about book building.
(...))

Hi HG

Hmm really your post leads me to think that you know close to nothing regarding the actual building,use and optimisation of commonly used engine opening books.

Before explaining a few things, I would strongly say I do not see almost any interest in a ctg-to-polyglot converter now that the newly released aquarium book manager allows to directly use any ctg book in any uci-compliant interface.

A book is simply a collection of positions, candidate move(s) from each of these positions, and relative probabilities for each of these moves to be selected for actual play when a recorded position is met. The probability of being played is the weight of each move.

In the case of polyglot, the weight of a move is initially established when building the book from a collection of selected games according to a formula that is well explained on Michel's site (and -but this is another problem - I am sure that the formula could be improved! ).
Then in a second stage polyglot move weights can be modified either through the book-merge feature or through direct editing of the weights, as is possible with the SCID database software.
Then the ctg case.
Here things are much more complicated. From what I understand (I am far from being a ctg expert) first step of book building is similar to polyglot's way : you provide a selection of games and the book is built. Never was published any information regarding chessbase's formula for turning games into weights.
Clearly their formula overweights number of games over winning statistics.
Nobody knows precisely how they deal with relative amount of draws.
This initial weighting of the moves after building of the book is permanently recorded in the book (maybe it can be altered in some way afterwards).
Then the user of the book in CB interfaces will have the choice of using the book in two different modes : "tournament" mode and "normal" mode. I suppose that in "normal" mode the initial weighting is used (and this must be one of the two flags you are referring to for each move).
Then there is the "tournament" mode. In this mode other weights are used for candidate moves (and these modified weights probably correspond to the second flag you are referring to). These modified weights may result mainly of 1) manual hand-tuning of the book (with "red" and "green" flags for candidate moves, discarding or forcing certain moves) and 2) automated learning through playing with the book from within a CB interface (local games at home or online games on the Playchess server). There is even a third way to tune weights through a kind of selective import of lines that should be favored for white or black.
So a ctg user ends with a double-faced book.
Once again there is no precise published information regarding any aspect of these mechanisms.
So if you were to translate ctg weights into polyglot ones I would suggest to identify how are recorded the "tournament" weights.
These are the real weights that the book publisher wishes to be played.
They are supposed to be stronger and are the ones that any CB customer would actually use when trying to get optimal results.

Hope this could help ...

Marc

[H.G.Muller]

Yes, this is very helpful, thnks.

Hmm really your post leads me to think that you know close to nothing regarding the actual building,use and optimisation of commonly used engine opening books.

I guess this is pretty accurate assessment...

Before explaining a few things, I would strongly say I do not see almost any interest in a ctg-to-polyglot converter now that the newly released aquarium book manager allows to directly use any ctg book in any uci-compliant interface.

This could be. My initial incentive for looking into this is that I disliked the ide of having to play engines through a multitude of adapters, and was not sure how well the aquarium book supports WinBoard engines. It would be a nuisance having to play through Polyglot + Book adapter + WB2UCI. I already have written and used some book-converting software before, and I figured adapting that to another source format would be a trivial job. Working on the probem furthermore aroused my academic curiosity.

A book is simply a collection of positions, candidate move(s) from each of these positions, and relative probabilities for each of these moves to be selected for actual play when a recorded position is met. The probability of being played is the weight of each move.

OK, fully agreed. This is in the end what it all boils down to: for every (position, move) pair the book has to define a probability for the move to be played. Polyglot books directly record this probability, as the move weight. I would call that a "cooked" book, as opposed to a book format that stores other information, from which the engine would have to derive the probabilities at probe time, by performing some calculation on the data stored in the book. The latter I would call a "raw" book.

In the case of polyglot, the weight of a move is initially established when building the book from a collection of selected games according to a formula that is well explained on Michel's site (and -but this is another problem - I am sure that the formula could be improved! ).

OK, I am familiar with the Polyglot format because I ported Michel's probing code for it to WinBoard. I just checked how Polyglot derives the weights from the raw PGN statistics during book-making; this is indeed an area that is candidate for improvement. It now uses the playing frequency of the move, weighted by result (2*wins + 1*draws + 0*losses), without paying attention to other paths leading to the same position as the move leads to. (E.g. a move could have been played only twice, both in losing games, but lead to a position that had occured a dozen times, 8 times in wins, but it would still get weight 0.) But that is a bit off-topic here; perhaps we should devote another thread to that.

Then in a second stage polyglot move weights can be modified either through the book-merge feature or through direct editing of the weights, as is possible with the SCID database software.
Then the ctg case.
Here things are much more complicated. From what I understand (I am far from being a ctg expert) first step of book building is similar to polyglot's way : you provide a selection of games and the book is built. Never was published any information regarding chessbase's formula for turning games into weights.
Clearly their formula overweights number of games over winning statistics.
Nobody knows precisely how they deal with relative amount of draws.
This initial weighting of the moves after building of the book is permanently recorded in the book (maybe it can be altered in some way afterwards).

I used the description of the ctg format from the link given in one of the earlier post, and it seems ctg books are "raw" books. There are no weights stored in them. In stead there are fields storing the total number of games, nr of wins, draws and losses, and average rating. The latter is stored as total number of games and sum of all ratings, (so basically as a rational number, giving enumerator and denominator as integers). The total number of games is thus stored twice, while it seems redundant. (I can imagine that the rating would leave out unrated games, although I am not sure ow relevant such an "average" rating would be, but could the total nr of games be different from wins+draws+losses???)

Then the user of the book in CB interfaces will have the choice of using the book in two different modes : "tournament" mode and "normal" mode. I suppose that in "normal" mode the initial weighting is used (and this must be one of the two flags you are referring to for each move).
Then there is the "tournament" mode. In this mode other weights are used for candidate moves (and these modified weights probably correspond to the second flag you are referring to). These modified weights may result mainly of 1) manual hand-tuning of the book (with "red" and "green" flags for candidate moves, discarding or forcing certain moves) and 2) automated learning through playing with the book from within a CB interface (local games at home or online games on the Playchess server). There is even a third way to tune weights through a kind of selective import of lines that should be favored for white or black.

Wel, there are no weights stored in the book, so they have to be calculated at probe time, and I guess this rules out any manual tuning. The only thing it can do is use another formula to derive the weights from the raw book data when you select tournament mode. I could imagine that in fact it would apply the same formula as in "normal" mode, except that it ignores all moves leading to positions that are marked as "non-tournament". Selecting those positions would most likely be part of the hand-tuning preparatory process. I am not sure how a book builder would use this flag; it cannot be based on the position being very good or bad, because if it is good for white, black should avoid it in tournaments, but white of course should seek it. So perhaps it should be used for unclear positions, or perhap positions that are in theory equal (as shown by PGN statistics) but which the engine is known (or suspected) to handle poorly for both black and white.

So a ctg user ends with a double-faced book.
Once again there is no precise published information regarding any aspect of these mechanisms.
So if you were to translate ctg weights into polyglot ones I would suggest to identify how are recorded the "tournament" weights.
These are the real weights that the book publisher wishes to be played.
They are supposed to be stronger and are the ones that any CB customer would actually use when trying to get optimal results.

Yes, I guess we really would have to know what formula the probing code applies to convert WDL statistics on positions into move weights.

[Marc Lacrosse]

A book is simply a collection of positions, candidate move(s) from each of these positions, and relative probabilities for each of these moves to be selected for actual play when a recorded position is met. The probability of being played is the weight of each move.

OK, fully agreed. This is in the end what it all boils down to: for every (position, move) pair the book has to define a probability for the move to be played. Polyglot books directly record this probability, as the move weight. I would call that a "cooked" book, as opposed to a book format that stores other information, from which the engine would have to derive the probabilities at probe time, by performing some calculation on the data stored in the book. The latter I would call a "raw" book.

This definition is clear.
For what regards myself I like the principle of the "cooked book" better because things are more clear : on one side you produce, tune , modify, adapt a book with any tools of your liking. Then in the second stage the book you have issued at stage one is simply interpreted by Polyglot working as a pure interpreter. For this the polyglot book format is close to perfect as it unambiguously records (position,move,weight) triplets.
The fact that polyglot itself has functions for building the book is completely independent and for me these book-building functions could (or even should) well be isolated in an independent utility.

In the case of polyglot, the weight of a move is initially established when building the book from a collection of selected games according to a formula that is well explained on Michel's site (and -but this is another problem - I am sure that the formula could be improved! ).

OK, I am familiar with the Polyglot format because I ported Michel's probing code for it to WinBoard. I just checked how Polyglot derives the weights from the raw PGN statistics during book-making; this is indeed an area that is candidate for improvement. It now uses the playing frequency of the move, weighted by result (2*wins + 1*draws + 0*losses), without paying attention to other paths leading to the same position as the move leads to. (E.g. a move could have been played only twice, both in losing games, but lead to a position that had occured a dozen times, 8 times in wins, but it would still get weight 0.) But that is a bit off-topic here; perhaps we should devote another thread to that.

The point you stress is a very important one for transpositional openings. Ideally one could imagine some kind of minimaxing of the proximal part of the tree originating from a move (be it simply on a few plies) so that alternate move orders leading to positions within the tree would be taken into account for the weighting process.

But the weighting formula used by Polyglot for book-making is itself a problem. Being 2*wins + 1*draws + 0*losses this means that if you have two candidate moves with respectively A:+10 = 10 -10 and B:+10 =10 -100 results in the underlying database, both moves get the same score in the book-making process although actual success rate was 50% for A (15/30) and 12.5% for B (15/120)!
I have always been surprised to see how polyglot books are efficient while the book-making formula is so far from perfect ...

Going back to the first part of the discussion, this is also why I support the principle of "cooked book" (to take your definition) : I simply wish we could develop more sophisticated tools for the making and tuning of books. But these tools should simply produce the very same kind of .bin books as used and made by polyglot itself.

Marc

[H.G.Muller]

But the weighting formula used by Polyglot for book-making is itself a problem. Being 2*wins + 1*draws + 0*losses this means that if you have two candidate moves with respectively A:+10 = 10 -10 and B:+10 =10 -100 results in the underlying database, both moves get the same score in the book-making process although actual success rate was 50% for A (15/30) and 12.5% for B (15/120)!

Indeed, this sounds very bad... I started a new thread on discussing Polyglot weights.

Even with a "cooked book" like Polyglot format there would still be enough redundant information to do (or undo) part of the cooking at probe time. The standard probe code uses Probbility_i = weight_i / SUM_k{weight_k}. This is invariant w.r.t. an over-all scaling of all the weights. So if the weghts are somehow chosen in proposrtion to the total number of games on which they are based, it would still be possible to change the minimum number of games on which weights should be based at probe, by ignoring all weights below some value. The variability vs quality trade-off could be changed by using a power of the weights.

[Charles Browne]

In the Aquarium adapter the "Depth" feature is really nice.

Do any of the current polyglots have a similar way to control the polyglot book moves depth in an actual game?


other than just creating a book with the number of moves/plys wanted.

[Marc Lacrosse]

In the Aquarium adapter the "Depth" feature is really nice.

Do any of the current polyglots have a similar way to control the polyglot book moves depth in an actual game?


other than just creating a book with the number of moves/plys wanted.

No, book moves depth cannot be determined at play, it is fixed at the book creation.
It could surely be an interesting option to add to polyglot itself (= the ability to stop using the book after ply number "n" even though there are longer lines in the book).

Strangely enough, if the book is directly managed by the cutechess command line interface then there is a cutechess option for limiting book depth. But unfortunately the present state of cutechess is only able to use the same book for both opponents (see http://www.talkchess.com/forum/viewtopi ... =cutechess).

Marc

[H.G.Muller]

But unfortunately the present state of cutechess is only able to use the same book for both opponents (see http://www.talkchess.com/forum/viewtopi ... =cutechess).

Same in WinBoard. I am in constant doubt about this. On the one hand there is the WinBoard philosophy that every engine should be fully self-sufficient, and thus also manage its own book. I nevertheless added the GUI book as a means to generate random test positions when testing engines without books or randomness. A GUI book provides a way to make two engines use the same book, something that might not be easiy done in any other way. It would be very easy to have each engine use its own book. But this would invite engine authors to lobotomize their engines by completely relying on the GUI to provide this functionality for them. I don't think this is very desirable in the long run. On the other hand, this is a very patronizing attitude. And in on-line tournaments they can use a GUI book like it was the engine book already.

In future WB versions I do want to provide more user control on GUI-book probing. This would definitely include a way to limit the depth of the book. And it would also include tunining the variation of the book, by setting an arbitrary power N to exponentiate the weights to before converting them to probabilities. So N=0 would give you uniform move choice, N=1 would pick moves in proportionality to their weights, N=infinity would always pick the best move. Other controls are conceivable. (E.g. excluding all move with a weight that has a zero in the least-significant bit, so that this bit can be used for tweaking the book.)

[Michel]

Do any of the current polyglots have a similar way to control the polyglot book moves depth in an actual game?

I posted a new version which has a parameter BookDepth which allows you to control this.

http://alpha.uhasselt.be/Research/Algeb ... t-release/

[F.Huber]

I posted a new version which has a parameter BookDepth which allows you to control this.

WOW, your PG is really future-proof: it even contains options for features which haven't yet been implemented (and maybe never will)!

But now slowly this PG-settings section in WB becomes quite large - maybe you should try the UCI engine Fruit 2.3.1 and look at its settings window in WB.
I guess I'll soon have to connect my notebook to a 40" TV screen to be able to use UCI engines in WB ...

[Charles Browne]

Do any of the current polyglots have a similar way to control the polyglot book moves depth in an actual game?

I posted a new version which has a parameter BookDepth which allows you to control this.

http://alpha.uhasselt.be/Research/Algeb ... t-release/

With the first use of your new version of polyglot here, the BookDepth option looks to be working very well , even in a match with two engines connected by polyglot and using the same polyglot book at different depths.

Your change log reads:

=========1.4.37b================
- Two new options: BookDepth and BookTreshold. BookDepth limits the number of moves the book is used and BookTreshold sets the minimum weight a book move should have.

At this time I really don't know much about opening books so I left the BookTreshold option out of the polyglot.ini file for now.

BTW, Thank You.

[H.G.Muller]

I posted a new version which has a parameter BookDepth which allows you to control this.

WOW, your PG is really future-proof: it even contains options for features which haven't yet been implemented (and maybe never will)!

But now slowly this PG-settings section in WB becomes quite large - maybe you should try the UCI engine Fruit 2.3.1 and look at its settings window in WB.
I guess I'll soon have to connect my notebook to a 40" TV screen to be able to use UCI engines in WB ...

Perhaps we should rethink if it is really necessary to have all Polyglot options setable from the WinBoard menu. More is not always better. Some option seem obsolete, because they duplicate functions implemented by the GUI (while older GUIs that don't support them cannot set them interactively anyway). E.g. UseNice, and the Kibitz options. I am also not sure what Chess960 is good for. (The engine presumably tells Polyglot if it supports Chess960, and WinBoard will tell Polyglot through the varant command when the engine has to play Chess960. I don't see how user interference could be helpful here at all.) ShowPonder seems another one that wouldn't be very useful. And work-around options would be set once and for all for an engine that needs them. Although having to hand-edit polyglot.ini files is objectionable, I would not mind so much if this has to occur on behalf of non-compliant engines.

Options not in the dialog can still be set through the polyglot.ini file, it is not like they would be completele removed. But the dialog should be better reserved for options that the user really must change often. A "lean and mean" Polyglot, as it were.

[Michel]

if it is really necessary to have all Polyglot options setable from the WinBoard menu.

I agree. That is why I prefixed the name of the PG options by Polyglot so that they are recognizable by the GUI as such.

Some options can be set directly by the GUI such as UseNice (UseNice sets the priority of the engine which
is different from WB setting the priority of PG).

[H.G.Muller]

The point is that WinBoard does not ever have to set the UseNice option of Polyglot, as it sets the nice itself. I am not sure how this works in Windows, but in Unix it would mean that the engine, which is a child of Polyglot, inherits the nice value of ts parent. I guess we would want Polyglot on Windows to behave the same as Polyglot on Linux. So if this inheritance of priority is not automatic in Windows, Polyglot should somehow figure out its own priority, and pass it to the engine process it creates.

So I guess UseNice should be considered a deprecated option.

What point would there be in having Polyglot send such options to the GUI, if all it does is require special code in the GUI to ignore it?

[Michel]

So I guess UseNice should be considered a deprecated option.

Don't be so fast. Your argument only works for the initial priority (on Linux). Not if you want to set it afterwards.

[H.G.Muller]

Can Polyglot change the nice of its child process (the engine) after it forked off this process??? You should teach me that trick, then I could do that in WinBoard too (for WB engines)!

[F.Huber]

Perhaps we should rethink if it is really necessary to have all Polyglot options setable from the WinBoard menu. More is not always better.

Well, what about the following solution:
This settings window could be made of 2 (or more) pages, which are selected by tabs - one for the engine options and one for Polyglot.
And for engines (like e.g. Fruit 2.3.1) with a huge number of options even these could be divided into separate pages, so a single options page could be small enough to make no problems even on small notebook screens.

But I don't know how complicated it would be to program this in your programming language!?

Here is an example what I mean (taken from Arena):

[H.G.Muller]

Well, there are no examples of that anywhere in WinBoard, so I have no idea how to do it.

And even if we could host an infinity of options by stacks of tabs each with 2D scroll bars, etc... If that was only to display a large number of completely useless options, I still would not consider it an improvement. A good design only shows people what they need.

[Gábor Szots]

Does this mean only UCI engines can use a common book under WB?

[H.G.Muller]

No, what gave you that idea? (I mean, what does the "this" in "Does this mean" refer to?)

The GUI book works or all engines. But currently the GUI book cannot be limited at probe time to a user-given depth, like the book used by Polyglot now can because of Michel's freshly made enhancement. And the variability vs quality probe-time adjustment I have planned for WinBoard is not implemented yet either.

[F.Huber]

And even if we could host an infinity of options by stacks of tabs each with 2D scroll bars, etc... If that was only to display a large number of completely useless options, I still would not consider it an improvement. A good design only shows people what they need.

That's a bit overweening, isn't it?

IMO it's not you to decide whether an engine option is 'completely useless' or not - it's the decision of the engine author (or at least of the user if he needs and want to use this option at all).