Separate Opening Books for White & Black

[Norm Pollock]

It is my opinion that chess engines will have the potential to do better
if they use separate opening books for White and Black.

Statistics have consistently shown that, at higher levels of chess
competition, White scores about 55% to Black's 45%. This happens in
both human games and in chess engine games.

With the statistical reality in mind, it makes sense for Black not to use
the same opening book as White because he will be led down the path of
55-45. Black should prefer a smaller opening book that concentrates on
equalizing and getting out of book sooner. White should prefer a bigger
opening book that concentrates on aggressiveness.

Let me use a hypothetical example to explain further.
Suppose I want to build an opening book based only on Fischer's games
as White and Karpov's games as Black. My motivation is that
Fischer was famous for his aggressive play, and Karpov is famous for
his defensive play. I want to play aggressively as White, and defensively
as Black.

Suppose I have a pgn file that contained only Fischer's games as White,
and another pgn file that contained only Karpov's games as
Black. If I must make a single opening book from these two pgn files,
then I would have to concatenate the pgns sometime during the process.

Thereafter the White moves of Karpov's opponents will be mixed
with Fischer's White moves. Also the Black moves of Fischer's opponents
will be mixed with Karpov's Black moves. The opening book would not be
what I originally wanted.

Now if I could make a separate "White" book based only on Fischer's White
games, and a separate "Black" book based only on Karpov's Black games then
I would have what I wanted.

Some engines already allow for separate White and Black books, but
they are in the minority. Those that do include: Aristarch, The Baron,
Francesca Mad, Petir, and TwistedLogic. I do not know of any guis that
allow for separate White and Black opening books.

By the way, the above example is not really a good idea for bookmaking
because opening books need to be based on thousands of games with a
large variety of openings. Perhaps a more realistic example would be a
White Book based on players with 2500+ elo and a Black Book based on
players with 2600+ elo.

[Richard Pijl]

It is my opinion that chess engines will have the potential to do better
if they use separate opening books for White and Black.

Whether the white and black book are physical in on file is not relevant. They do not overlap, as in the white book, white is to move, and in the black book, black is to move.
Baron has, in its old format, two files as it was convenient at the time.
In my private version, I changed the book format into one single book (although it is composed of several files, there is no separation between files for the black and files for the white side). It is very easy for me to convert the separate white and black files that I used to have with an equivalent unified book.

What you describe as a method for book authoring is valid though. When writing a book, you should basically care about a select number of white moves (mostly only one), and anticipate all (regular) black moves, where for the black side you look at a select number of black moves and you'll have to anticipate all (regular) white moves. This however has no real consequences for a (binary) book format.
Richard.

[Marc Lacrosse]

It is my opinion that chess engines will have the potential to do better
if they use separate opening books for White and Black.

Did you find Colombo's egg ?

Every book-making program I know of for years allows for separate white and black books ...

It's true for the most common chessbase books, shredder books, chesspartner books, and it is even more evident for polyglot 1.4 ones ...

Is this _really_ anything like a new idea ???

I can't understand your post ...

Sorry if i missed something...

Marc

[Norm Pollock]

What I'm trying to express is that I feel that there will be an advantage if you can use separate pgn files for generating the white moves and black moves of an opening book. For example, a pgn of Fischer's white moves for the white moves of the opening book, and a pgn of Karpov's black moves for the black moves of the opening book. Whether the book is physically in 1 or 2 files is not my point. .

[Marc Lacrosse]

What I'm trying to express is that I feel that there will be an advantage if you can use separate pgn files for generating the white moves and black moves of an opening book. For example, a pgn of Fischer's white moves for the white moves of the opening book, and a pgn of Karpov's black moves for the black moves of the opening book. Whether the book is physically in 1 or 2 files is not my point. .

But you can do it very easily in Polyglot and in chessbase, for example (these are the two ones that I really know ; AFAIK the same can be done for any other major GUI books).
So what is the point?
Every bookmaker does it... (we do select some lines and fill any potential holes with a careful selection of _separate_ white and black databases - or separate games selection criteria- incorporated into the book as well...).

Marc

[Norm Pollock]

What I'm trying to express is that I feel that there will be an advantage if you can use separate pgn files for generating the white moves and black moves of an opening book. For example, a pgn of Fischer's white moves for the white moves of the opening book, and a pgn of Karpov's black moves for the black moves of the opening book. Whether the book is physically in 1 or 2 files is not my point. .

But you can do it very easily in Polyglot and in chessbase, for example (these are the two ones that I really know ; AFAIK the same can be done for any other major GUI books).
So what is the point?
Every bookmaker does it... (we do select some lines and fill any potential holes with a careful selection of _separate_ white and black databases - or separate games selection criteria- incorporated into the book as well...).

Marc

I'm only currently familiar with making an opening book through fritz9.

Suppose you import games from WhiteMoves.pgn and BlackMoves.pgn into your opening book. Won't the white moves from BlackMoves.pgn get mixed with the white moves from WhiteMoves.pgn? Likewise for the black moves. I believe that they will and that will change the probabilities of the moves in the opening book.

How could I go about ensuring that ONLY the white moves from WhiteMoves.pgn go into the opening book? Likewise for the black moves from BlackMoves.pgn. One way I can see this happening is if there are two different opening books. A white opening book made solely from WhiteMoves.pgn, and a black opening book made solely from BlackMoves.pgn.

It could also happen if the book making program asks for two import files, one for white moves and one for black moves. Or alternatively, "import games for white moves" and "import games for black moves". I do not see thess type of choices in the fritz9 "Edit/ Opening Books" menu.

[Marc Lacrosse]

I'm only currently familiar with making an opening book through fritz9.

Suppose you import games from WhiteMoves.pgn and BlackMoves.pgn into your opening book. Won't the white moves from BlackMoves.pgn get mixed with the white moves from WhiteMoves.pgn? Likewise for the black moves. I believe that they will and that will change the probabilities of the moves in the opening book.

How could I go about ensuring that ONLY the white moves from WhiteMoves.pgn go into the opening book? Likewise for the black moves from BlackMoves.pgn. One way I can see this happening is if there are two different opening books. A white opening book made solely from WhiteMoves.pgn, and a black opening book made solely from BlackMoves.pgn.

It could also happen if the book making program asks for two import files, one for white moves and one for black moves. Or alternatively, "import games for white moves" and "import games for black moves". I do not see thess type of choices in the fritz9 "Edit/ Opening Books" menu.

OK.
I am no CB GUI expert but here is the basic information:

1. You start with either an empty book or a neutral one made with a large number of good quality games.
2. you select this book in the GUI (through right click in the book pane)
3. You build a GKwhite.cbh (or .PGN) game file with games played by Kasparov as white
4. you select "Edit/Opening books/Add priority analysis" and select GKwhite.cbh
5. in the next screen : "repertoire white"
6. you repeat the procedure with your Karpov black games and select "repertoire black"

Then you go in "Edit/Opening books/Book options"

You select "Use book" and "Tournament book" and adjust other parameters.

That's it ...

If you do not own the Chessbase database program, creating an empty ctg book file and creating the Kasparov and Karpov game files is difficult (if not impossible). I can put the files on my site if you wish.

Marc

[Uri Blass]

It is my opinion that chess engines will have the potential to do better
if they use separate opening books for White and Black.

Statistics have consistently shown that, at higher levels of chess
competition, White scores about 55% to Black's 45%. This happens in
both human games and in chess engine games.

With the statistical reality in mind, it makes sense for Black not to use
the same opening book as White because he will be led down the path of
55-45. Black should prefer a smaller opening book that concentrates on
equalizing and getting out of book sooner. White should prefer a bigger
opening book that concentrates on aggressiveness.

Let me use a hypothetical example to explain further.
Suppose I want to build an opening book based only on Fischer's games
as White and Karpov's games as Black. My motivation is that
Fischer was famous for his aggressive play, and Karpov is famous for
his defensive play. I want to play aggressively as White, and defensively
as Black.

Suppose I have a pgn file that contained only Fischer's games as White,
and another pgn file that contained only Karpov's games as
Black. If I must make a single opening book from these two pgn files,
then I would have to concatenate the pgns sometime during the process.

Thereafter the White moves of Karpov's opponents will be mixed
with Fischer's White moves. Also the Black moves of Fischer's opponents
will be mixed with Karpov's Black moves. The opening book would not be
what I originally wanted.

Now if I could make a separate "White" book based only on Fischer's White
games, and a separate "Black" book based only on Karpov's Black games then
I would have what I wanted.

Some engines already allow for separate White and Black books, but
they are in the minority. Those that do include: Aristarch, The Baron,
Francesca Mad, Petir, and TwistedLogic. I do not know of any guis that
allow for separate White and Black opening books.

By the way, the above example is not really a good idea for bookmaking
because opening books need to be based on thousands of games with a
large variety of openings. Perhaps a more realistic example would be a
White Book based on players with 2500+ elo and a Black Book based on
players with 2600+ elo.

You can add movei as a program that allows seperate books for white and black

You basically needs 2 pgn files in order to start to create a book for movei.

Movei has a program that translate them to .bin file.

public movei come only with the .bin file but there are instructions to create book in the readme of version 295 and the instructions are not changed in later versions

Here are the instructions for version 295 from the readme:

Instructions to create book at console mode
1)have 2 pgn one for white moves and one for black moves
2)Rename the pgn for white to gameswhite.pgn
3)Rename the book for black to gamesblack.pgn
4)Run the program at console mode and type createbook
5)type numbers for maximal ply and minimal frequency when movei asks for it.
6)Wait until movei type the words book created.
7)type quit

Uri

[Norm Pollock]

OK.
I am no CB GUI expert but here is the basic information:

1. You start with either an empty book or a neutral one made with a large number of good quality games.
2. you select this book in the GUI (through right click in the book pane)
3. You build a GKwhite.cbh (or .PGN) game file with games played by Kasparov as white
4. you select "Edit/Opening books/Add priority analysis" and select GKwhite.cbh
5. in the next screen : "repertoire white"
6. you repeat the procedure with your Karpov black games and select "repertoire black"

Then you go in "Edit/Opening books/Book options"

You select "Use book" and "Tournament book" and adjust other parameters.

That's it ...

If you do not own the Chessbase database program, creating an empty ctg book file and creating the Kasparov and Karpov game files is difficult (if not impossible). I can put the files on my site if you wish.

Marc

First, thanks for the information on using CB gui with the separate white and black pgns being entered.

Second, as you mentioned, Polyglot (latest version) allows for separate white and black pgns being entered.

Third, I intend to fix the "difficult if not impossible" problem you mentioned. I intend to write a utility program that filters a large pgn for a particular player and optional color. It will be easy to use, for example:

newutility filename.pgn Karpov black
OR
newutility filename.pgn "Karpov,A" black

where "the player's name starts with the substring "Karpov,A"

It will create a file extracting Karpov's black games from filename.pgn

I do foresee some problems:
there can be more than one "Karpov" in the large pgn; his name can be spelled different ways therefore some games will not be extracted; users will not realize that the parameters are "case-sensitive"; users will have to use pgn-extract (for example) to remove duplicates.

[Marc Lacrosse]

First, thanks for the information on using CB gui with the separate white and black pgns being entered.

Second, as you mentioned, Polyglot (latest version) allows for separate white and black pgns being entered.

Third, I intend to fix the "difficult if not impossible" problem you mentioned. I intend to write a utility program that filters a large pgn for a particular player and optional color. It will be easy to use, for example:

newutility filename.pgn Karpov black
OR
newutility filename.pgn "Karpov Ana" black

where "the player's name starts with the substring "Karpov Ana"

It will create a file extracting Karpov's black games from filename.pgn

I do foresee some problems:
there can be more than one "Karpov" in the large pgn; his name can be spelled different ways therefore some games will not be extracted; users will not realize that the parameters are "case-sensitive"; users will have to use pgn-extract (for example) to remove duplicates.

No need to write something (unless you wish ) :

- the excellent PGNextract by D Barnes will extract whatever you wish. You find it at http://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/ (if you are under winXP take Jim Ablett's executable).

- the free SCID database (at http://scid.sourceforge.net)reads PGN, has nice name spellings standardisation features and will also easily extract what you need.


The problem could well be to create an empty CTG.
I am not sure, but I fear this is only possible in Chessbase database program, not in your Fritz playing GUI (but maybe I am wrong).

Marc

[Norm Pollock]

Creating an empty file is very easy in the F9 gui. Just "new/openings book/"empty"/open

True, pgn-extract and Scid do it also. But each requires a masters degree in order to use properly!

Off topic- there is one "bug" in Scid that I find annoying. Scid creates pgn files with a format error- it fails to leave a space after the period. For example, "4.Nf3" instead of the correct "4. Nf3". This falsely lets the user think Scid saves space and it can cause problems when using the pgn file. pgn-extract corrects pgn files with this "bug".