How do I create my own opening book for the Thinker chess en

[Uri Blass]

Geschrieben von:/Posted by: Uri Blass at 05 May 2004 22:04:05:

I read in the thinker page the answer but I do not understand it.
step 3:Run the makebook.exe program but I find no makebook.exe inspite of the fact that I downloaded thinker.
I also find no information about the algorithm that is used to create the book.
I only understood from Lance perkins reply to Dann Corbit that it is using statistics to create the book but there is no single way to use statistics
and I do not know exactly what it does.
Uri

[Dann Corbit]

Geschrieben von:/Posted by: Dann Corbit at 05 May 2004 23:00:35:
Als Antwort auf:/In reply to: How do I create my own opening book for the Thinker chess engine? geschrieben von:/posted by: Uri Blass at 05 May 2004 22:04:05:

I read in the thinker page the answer but I do not understand it.
step 3:Run the makebook.exe program but I find no makebook.exe inspite of the fact that I downloaded thinker.
I also find no information about the algorithm that is used to create the book.
I only understood from Lance perkins reply to Dann Corbit that it is using statistics to create the book but there is no single way to use statistics
and I do not know exactly what it does.

Install the thinkerboard thing (which has all the accessories in it)
Next, you will have to preprocess the PGN file. SCID is a nice thing to use but other things will work also. You must remove every game that does not end in a win a loss or a draw (*=unfinished confuse it)
For these next tasks, I like PGN-Extract by Barnes, but other things can do it:
You must remove all tags from the header except the 7 standard tags.
You must remove all comments from the games.
Then, run the program makebook and give it the PGN file.
You can reduce the number of maximum plies it traces with
-
e.g.
makebook -40 book.pgn
and you will have an output file called Thinker.dat



my ftp site {remove http:// unless you like error messages}

[Uri Blass]

Geschrieben von:/Posted by: Uri Blass at 05 May 2004 23:52:24:
Als Antwort auf:/In reply to: Re: How do I create my own opening book for the Thinker chess engine? geschrieben von:/posted by: Dann Corbit at 05 May 2004 23:00:35:

I read in the thinker page the answer but I do not understand it.
step 3:Run the makebook.exe program but I find no makebook.exe inspite of the fact that I downloaded thinker.
I also find no information about the algorithm that is used to create the book.
I only understood from Lance perkins reply to Dann Corbit that it is using statistics to create the book but there is no single way to use statistics
and I do not know exactly what it does.

Install the thinkerboard thing (which has all the accessories in it)

Thanks, I missed this part that generate the relevant program.
I still do not understand what it does.
I used a small pgn of 6.93 kbytes to test it and it created a small book of 347 bytes.
If every hash key that is in this book is 8 bytes when every move is 2 bytes then the book can have at most 34 positions and it is even without statistics so it is clear that most of the games are not in the book and I do not need to limit the number of moves because even without limiting the number of moves it does not remember all the games in book and maybe it remembers only positions that happened at most twice.
Here is the pgn that I used that is based on some games of old movei against bestia.

[Event "Computer chess game"]
[Site "URI-PC"]
[Date "2002.02.15"]
[Round "-"]
[White "Bestia_08"]
[Black "movei0041"]
[Result "1-0"]
[TimeControl "40/60"]
1. Nc3 Nc6 2. e4 e5 3. Nf3 Nf6 4. Bb5 Bb4 5. Qe2 d6 6. Qc4 Bxc3 7. Bxc6+
bxc6 8. dxc3 c5 9. Bg5 O-O 10. O-O-O Be6 11. Qe2 h6 12. Bh4 g5 13. Bg3 Qe8
14. Nd2 Bxa2 15. b3 h5 16. h4 gxh4 17. Bxh4 Qe6 18. Bxf6 Qxf6 19. Qxh5 Rfe8
20. Kb2 Bxb3 21. cxb3 Qg7 22. Qh6 Qxh6 23. Rxh6 Kg7 24. Rdh1 Re6 25. Rxe6
fxe6 26. Kc2 d5 27. f3 Kg6 28. g3 c6 29. Kd3 Rd8 30. Rh2 Kg5 31. Ke3 Kg6
32. Rh4 Rd7 33. Rh8 Kf6 34. Rc8 Rd6 35. Rc7 a6 36. Ra7 d4+ 37. Kd3 Rd8 38.
Rxa6 Rb8 39. Rxc6 Rb5 40. cxd4 cxd4 41. f4 exf4 42. gxf4 Kf7 43. Kc4 Rb7
44. Nf3 d3 45. Ne5+ Kf6 46. Nxd3 Re7 47. Nc5 Re8 48. Rxe6+ Rxe6 49. Nxe6
Kxe6 50. Kc5 Ke7 51. f5 Kf6 52. Kd6 Kf7 53. e5 Kg7 54. f6+ Kf7 55. b4 Kg6
56. b5 Kf7 57. b6 Kg6 58. b7 Kf5 59. b8=Q Ke4 60. f7 Kd3 61. f8=Q Kc3 62.
Qf3+ Kc4 63. e6 Kd4 64. Qb4#
{Bestia mates} 1-0
[Event "Computer chess game"]
[Site "URI-PC"]
[Date "2002.02.15"]
[Round "-"]
[White "Bestia_08"]
[Black "movei0041"]
[Result "1/2-1/2"]
[TimeControl "40/120"]
1. Nc3 Nc6 2. e4 e5 3. Nf3 Nf6 4. Bb5 Nd4 5. Ba4 Nxf3+ 6. Qxf3 Bc5 7. d3 c6
8. Bg5 d6 9. Nd5 Qa5+ 10. Bd2 Qxa4 11. Nc7+ Ke7 12. Nxa8 Bg4 13. Qg3 Rxa8
14. c3 Kf8 15. b4 Bb6 16. Bg5 Qc2 17. O-O Be2 18. Bxf6 gxf6 19. Rfc1 Qxd3
20. Qh4 Kg7 21. a4 Bd8 22. Qg3+ Qxg3 23. hxg3 Bd3 24. Rd1 Bxe4 25. Rxd6 Bc7
26. Rd7 Rc8 27. c4 Bf5 28. Rd2 a5 29. bxa5 Bxa5 30. Rdd1 Bc3 31. Ra3 Bd4
32. a5 Kg6 33. Rd2 Be6 34. Rc2 Bc5 35. Rb3 Ra8 36. Rxb7 Rxa5 37. Rc7 Ra6
38. Kh2 Bd6 39. Rb7 c5 40. Rb3 Rc6 41. Kg1 Bf5 42. Rcb2 Ra6 43. Rb6 Ra1+
44. Kh2 Rd1 45. Ra2 Be6 46. Raa6 Be7 47. Ra4 Kf5 48. Rb8 Rd4 49. Rh8 Bxc4
50. Rxh7 Bd6 51. Ra7 Bd5 52. f3 c4 53. Rh5+ Ke6 54. Rh8 Rd3 55. Re8+ Kf5
56. Rd8 Be6 57. Ra6 Bc5 58. Raa8 Bd4 59. Rh8 Be3 60. Rh5+ Bg5 61. Ra4 Bd5
62. Rh8 Rc3 63. Ra5 Ke6 64. Rd8 Bc6 65. Rc5 Bb7 66. Rc7 Bd5 67. Rcd7 Bc6
68. Rc7 Bb5 69. Rf8 Ra3 70. Rfc8 Rd3 71. Rb7 Rb3 72. Rc5 Bd7 73. Rbc7 Rb8
74. Ra5 Bb5 75. f4 exf4 76. Rc5 Bd7 77. Rxc4 Rh8+ 78. Kg1 fxg3 79. Re4+ Kd6
80. Ra3 Bc6 81. Rd3+ Kc5 82. Rc3+ Kd5 83. Re1 Bb5 84. Rxg3 Bh4 85. Rd1+ Ke4
86. Re1+ Kf5 87. Rf3+ Kg6 88. Re7 f5 89. Rb7 Be2 90. Ra3 f4 91. Rc3 Bg3 92.
Rd7 Rb8 93. Rc1 Kf6 94. Rd2 Bc4 95. Rdd1 Be6 96. Rc2 Rb6 97. Rcd2 Rc6 98.
Rb1 Bc4 99. Rd7 Ke6 100. Rd2 f5 101. Rbd1 Ke5 102. Rb2 Rd6 103. Rxd6 Kxd6
104. Rb7 Kc6 105. Ra7 Kd5 106. Ra8 Ke4 107. Re8+ Kd4 108. Rd8+ Bd5 109. Rd7
Ke5 110. Re7+ Kf6 111. Ra7 Be4 112. Kf1 Bd3+ 113. Kg1 Ke5 114. Rg7 Ke6 115.
Ra7 Be2 116. Rb7 Bc4 117. Rb6+ Ke5 118. Rb7 Kd4 119. Rb4 Kc5 120. Rb8 Bb5
121. Rb7 Bc6 122. Ra7 Kd4 123. Ra6 Bb5 124. Rd6+ Ke4 125. Re6+ Kd5 126. Re7
Kd6 127. Ra7 Ke6 128. Ra8 Kd5 129. Ra5 Kc5 130. Kh1 Kb4 131. Ra7 Kc3 132.
Rf7 Bd3 133. Kg1 Kd4 134. Rb7 Kc5 135. Rd7 Bc4 136. Rc7+ Kd5 137. Rd7+ Ke6
138. Rc7 Bb5 139. Rh7 Ke5 140. Rc7 Ke4 141. Rb7 Ba6 142. Re7+ Kd5 143. Rd7+
Kc6 144. Rd8 Bb5 145. Ra8 Kc5 146. Ra3 Kd4 147. Ra7 Bc4 148. Ra4 Be1 149.
Kh2 Bh4 150. Kg1 Bf6 151. Kf2 f3 152. Kxf3 Bg5 153. g3 Bh6 154. g4 f4 155.
g5 Bxg5 156. Ra5 Bd5+ 157. Kg4 Bh6 158. Ra6 Bg7 159. Kxf4 Be5+ 160. Kf5 Bb3
161. Ra8 Bc2+ 162. Ke6 Ke4 163. Rc8 Bb3+ 164. Ke7 Kd4 165. Ra8 Bc7 166. Rc8
Bf4 167. Ra8 Bg5+ 168. Ke8 Bc1 169. Rb8 Be6 170. Ra8 Be3 171. Rb8 Kd5 172.
Ra8 Ke4 173. Rb8 Ke5 174. Ra8 Bd2 175. Rb8 Bc4 176. Ra8 Ke6 177. Rb8 Be3
178. Ra8 Bd4 179. Rb8 Bd5 180. Rc8 Bb2 181. Rb8 Bc6+ 182. Kd8 Bc3 183. Kc7
Kd5 184. Rc8 Bf6 185. Rd8+ Kc5 186. Rb8 Be5+ 187. Kc8 Be4 188. Rb7 Bf5+
189. Kd8 Bf6+ 190. Ke8 Be6 191. Rb8 Kc6 192. Ra8 Kb6 193. Rb8+ Kc5 194. Ra8
Bc3 195. Rb8 Be5 196. Ra8 Kd4 197. Rd8+ Ke4 198. Ra8 Bc3 199. Rb8 Ke5 200.
Ra8 Bb4 201. Rb8 Ba3 202. Ra8 Bb2 203. Rb8 Bd4 204. Ra8 Bc3 205. Rb8 Ke4
206. Ra8 Bc4 207. Rb8 Bd4 208. Ra8 Bd5 209. Rb8
{Draw by fifty move rule} 1/2-1/2
[Event "Computer chess game"]
[Site "URI-PC"]
[Date "2002.02.15"]
[Round "-"]
[White "Bestia_08"]
[Black "movei0041"]
[Result "1/2-1/2"]
[TimeControl "40/180"]
1. Nc3 Nc6 2. e4 e5 3. Nf3 Nf6 4. Bb5 Nd4 5. Ba4 Nxf3+ 6. Qxf3 Bc5 7. d3 c6
8. Bg5 d6 9. Nd5 Qa5+ 10. Bd2 Qxa4 11. Nc7+ Ke7 12. Nxa8 Bg4 13. Qg3 Rxa8
14. h3 Be6 15. b3 Qa3 16. O-O Kf8 17. Rfc1 Kg8 18. Bh6 g6 19. Be3 Qb2 20.
Bxc5 dxc5 21. Rab1 Qc3 22. Qh4 Nd7 23. a4 Re8 24. Ra1 b6 25. Qh6 f5 26. Qg5
Qd4 27. a5 fxe4 28. axb6 axb6 29. Ra4 c4 30. dxc4 Rf8 31. Rf1 Qc3 32. Ra7
Re8 33. Rc1 b5 34. cxb5 cxb5 35. Raa1 Rf8 36. Qe7 Nc5 37. Ra7 Bf7 38. Rc7
Qd2 39. Rf1 Ne6 40. Rb7 Qd5 41. Rd7 Qc6 42. Rd6 Qc8 43. Qf6 Nd4 44. Qxe5
Ne2+ 45. Kh1 Qxc2 46. b4 Bc4 47. Rf6 Rd8 48. Re6 Nc3 49. Re8+ Rxe8 50.
Qxe8+ Kg7 51. Qe7+ Bf7 52. Qe5+ Kf8 53. Ra1 Na2 54. Qd6+ Ke8 55. Qe5+ Kd7
56. Qxb5+ Ke6 57. f3 Kf6 58. fxe4 Qxe4 59. Qf1+ Kg7 60. Rxa2 Bxa2 61. Qa1+
Kh6 62. Qxa2 Qxb4 63. Qf7 Qe4 64. Kh2 Qe5+ 65. g3 Qb2+ 66. Kg1 Qe2 67. Qf4+
Kg7 68. g4 h6 69. Qd4+ Kf7 70. Qf4+ Kg7 71. Qd4+ Kf7 72. Qf4+ Kg7
{Repetition} 1/2-1/2
[Event "Computer chess game"]
[Site "URI-PC"]
[Date "2002.02.15"]
[Round "-"]
[White "Bestia_08"]
[Black "movei0041"]
[Result "1-0"]
[TimeControl "40/240"]
1. Nc3 Nc6 2. e4 e5 3. Nf3 Nf6 4. Bb5 Nd4 5. Ba4 Nxf3+ 6. Qxf3 Bc5 7. d3 c6
8. Bg5 d6 9. Nd5 Qa5+ 10. Bd2 Qxa4 11. Nc7+ Ke7 12. Nxa8 Bg4 13. Qg3 Rxa8
14. c3 Kf8 15. b3 Qa3 16. Bg5 Be6 17. Bc1 Qa6 18. b4 Bb6 19. a4 d5 20. a5
Bc7 21. Bg5 b6 22. Bxf6 gxf6 23. Qf3 Ke7 24. exd5 Bxd5 25. Qh3 Rh8 26. c4
Be6 27. Qe3 bxa5 28. O-O a4 29. Qc1 Bd6 30. Qa3 Bf5 31. Rfd1 Rd8 32. Kf1
Ke6 33. Qxa4 Qxa4 34. Rxa4 c5 35. Ra6 Rc8 36. bxc5 Rxc5 37. d4 Rxc4 38.
dxe5 Kxe5 39. Raxd6 Be6 40. Ra6 Rc7 41. Ra5+ Ke4 42. Rh5 Rc2 43. f3+ Ke3
44. Re1+ Kd3 45. Rxh7 Ra2 46. h4 Bc4 47. g4 Kd4+ 48. Kg1 Bd5 49. Rd1+ Kc4
50. Rf1 Be6 51. Rf2 Ra5 52. Kg2 f5 53. g5 f4 54. Re2 Kd3 55. Kf2 Bc4 56.
Rh6 Kc3 57. Rc6 Kb3 58. Re1 Ra2+ 59. Kg1 Ra4 60. Rf6 Bd5 61. Rd1 Ra5 62.
Rxf4 Kc2 63. Rdd4 Bb3 64. Kf2 Kc3 65. Ke3 Re5+ 66. Rfe4 Rc5 67. f4 Kc2 68.
Rd2+ Kc3 69. Rd3+ Kc2 70. Re5 Rc8 71. h5 Bc4 72. Rd7 a6 73. f5 Kc3 74. h6
Ra8 75. h7 Rh8 76. g6 fxg6 77. fxg6 Ba2 78. Rb7 Bc4 79. Rc7 Rd8 80. g7 Rd3+
81. Kf4 Kb3 82. g8=Q Rd4+ 83. Re4 Rxe4+ 84. Kxe4 Bxg8 85. hxg8=Q+ Kb2 86.
Qb8+ Ka2 87. Rc4 Ka3 88. Ke5 a5 89. Qb5 Ka2 90. Ra4#
{Bestia mates} 1-0
[Event "Computer chess game"]
[Site "URI-PC"]
[Date "2002.02.15"]
[Round "-"]
[White "Bestia_08"]
[Black "movei0041"]
[Result "0-1"]
[TimeControl "40/300"]
1. e4 Nc6 2. d4 d5 3. exd5 Qxd5 4. Nf3 Bg4 5. Nc3 Bxf3 6. Nxd5 Bxd1 7.
Nxc7+ Kd8 8. Nxa8 Bxc2 9. Be3 e6 10. Kd2 Bf5 11. f3 Nf6 12. g4 Bb4+ 13. Ke2
Bc2 14. a3 Bd6 15. Kd2 Nd5 16. Bc4 Nxe3 17. Kxe3 Kd7 18. Rac1 Bg6 19. h4 h5
20. Rhg1 hxg4 21. h5 Bxh5 22. Rh1 gxf3 23. Rcf1 g6 24. Rxf3 f5 25. Nb6+
axb6 26. Rfh3 Ke7 27. Be2 f4+ 28. Ke4 f3 29. Bxf3 Bxf3+ 30. Kxf3 Nxd4+ 31.
Ke4 Rxh3 32. Rxh3 Nc6 33. Rb3 Bc7 34. Rh3 e5 35. Rh6 Kf6 36. a4 Bd6 37. Rh7
Ne7 38. Rh2 Ke6 39. Rh7 Bc5 40. Rh6 Bb4 41. Rh4 Nf5 42. Rh7 Nd6+ 43. Kf3 g5
44. Kg4 Bd2 45. Rh2 Bf4 46. Rh7 e4 47. Rc7 e3 48. Kf3 Nf5 49. Rc4 Kd5 50.
Rc8 Nd4+ 51. Kg2 e2 52. Re8 g4 53. Kf2 g3+ 54. Ke1 g2
{Bestia resigns} 0-1

[Dann Corbit]

Geschrieben von:/Posted by: Dann Corbit at 06 May 2004 01:26:44:
Als Antwort auf:/In reply to: Re: How do I create my own opening book for the Thinker chess engine? geschrieben von:/posted by: Uri Blass at 05 May 2004 23:52:24:

I read in the thinker page the answer but I do not understand it.
step 3:Run the makebook.exe program but I find no makebook.exe inspite of the fact that I downloaded thinker.
I also find no information about the algorithm that is used to create the book.
I only understood from Lance perkins reply to Dann Corbit that it is using statistics to create the book but there is no single way to use statistics
and I do not know exactly what it does.

Install the thinkerboard thing (which has all the accessories in it)

Thanks, I missed this part that generate the relevant program.
I still do not understand what it does.
I used a small pgn of 6.93 kbytes to test it and it created a small book of 347 bytes.
If every hash key that is in this book is 8 bytes when every move is 2 bytes then the book can have at most 34 positions and it is even without statistics so it is clear that most of the games are not in the book and I do not need to limit the number of moves because even without limiting the number of moves it does not remember all the games in book and maybe it remembers only positions that happened at most twice.
Here is the pgn that I used that is based on some games of old movei against bestia.

Perhaps the hash is not stored.
I remember that there is an Amiga program called Pigbase that stores with huge compression. It stores one byte for each move, by simply storing the tree and by storing each move as the move number generated between 1 and 255.
So with 1000 moves, perhaps 600 are repeated in the tree and so 400 bytes stores the whole thing.
But I have no idea how his book format works.
A thing to do in experiment is to have your engine say the first move to analyze that it receives in a log. Then you can find out where the book ends by playing through moves in your PGN file.
A way to do this that would be automated is to create a crafty book using the same PGN file (with 500 plies deep, only one instance of the move) and have the two programs play a blitz in one minute match of 1000 games. Then you will get a clear picture of the bookthinker book's actual storage of the moves.


my ftp site {remove http:// unless you like error messages}

[Lance Perkins]

Geschrieben von:/Posted by: Lance Perkins at 06 May 2004 02:41:12:
Als Antwort auf:/In reply to: Re: How do I create my own opening book for the Thinker chess engine? geschrieben von:/posted by: Dann Corbit at 06 May 2004 01:26:44:

I have posted before how makebook and bookthinker works.
Pls see: http://www.f11.parsimony.net/forum16635 ... /61762.htm
The book format does not store positions. Instead, it stores the actual games. Each game is preceeded by a count of how many times that game has occured in the input PGN and what is the outcome of that game (win/loss for which side). It is always possible for a game to occur more than once because only the first few moves are stored for all the games. These first few moves will be the same for many games.
Here's an example:
If your input PGN has the following games, and you only wanted to store 3 plies:
1. c2c4 e7e5 2. b1c3 g8f6
1. c2c4 e7e5 2. b1c3 f8b4
1. c2c4 e7e5 2. e2e3 g8f6
Then, this will be reduced by MakeBook into:
(count=2) c2c4 e7e5 b1c3
(count=1) c2c4 e7e5 e2e3
Furthermore, in the book file, this will end-up as:
(count=2) c2c4 e7e5 b1c3
(count=1) (same as 2 moves from above) e2e3
The book is also heavily compressed (LZ + huffman coding) by MakeBook. The default Thinker book for example is about 751K. But when uncompressed, it is more than 3MB. At the start of a game, BookThinker and Thinker expands the book file - this is the reason why it takes a while for Thinker to initialize at startup.
To convince Uri that such a small file can pack so much information, I will be posting a tool to convert any Thinker.dat back to PGN/text. This way, one can verify that all the games from the input PGN file made it to the book file.

I read in the thinker page the answer but I do not understand it.
step 3:Run the makebook.exe program but I find no makebook.exe inspite of the fact that I downloaded thinker.
I also find no information about the algorithm that is used to create the book.
I only understood from Lance perkins reply to Dann Corbit that it is using statistics to create the book but there is no single way to use statistics
and I do not know exactly what it does.

Install the thinkerboard thing (which has all the accessories in it)

Thanks, I missed this part that generate the relevant program.
I still do not understand what it does.
I used a small pgn of 6.93 kbytes to test it and it created a small book of 347 bytes.
If every hash key that is in this book is 8 bytes when every move is 2 bytes then the book can have at most 34 positions and it is even without statistics so it is clear that most of the games are not in the book and I do not need to limit the number of moves because even without limiting the number of moves it does not remember all the games in book and maybe it remembers only positions that happened at most twice.
Here is the pgn that I used that is based on some games of old movei against bestia.

Perhaps the hash is not stored.
I remember that there is an Amiga program called Pigbase that stores with huge compression. It stores one byte for each move, by simply storing the tree and by storing each move as the move number generated between 1 and 255.
So with 1000 moves, perhaps 600 are repeated in the tree and so 400 bytes stores the whole thing.
But I have no idea how his book format works.
A thing to do in experiment is to have your engine say the first move to analyze that it receives in a log. Then you can find out where the book ends by playing through moves in your PGN file.
A way to do this that would be automated is to create a crafty book using the same PGN file (with 500 plies deep, only one instance of the move) and have the two programs play a blitz in one minute match of 1000 games. Then you will get a clear picture of the bookthinker book's actual storage of the moves.

[Uri Blass]

Geschrieben von:/Posted by: Uri Blass at 06 May 2004 08:46:15:
Als Antwort auf:/In reply to: Re: How do I create my own opening book for the Thinker chess engine? geschrieben von:/posted by: Lance Perkins at 06 May 2004 02:41:12:

I have posted before how makebook and bookthinker works.
Pls see: http://www.f11.parsimony.net/forum16635 ... /61762.htm
The book format does not store positions. Instead, it stores the actual games. Each game is preceeded by a count of how many times that game has occured in the input PGN and what is the outcome of that game (win/loss for which side). It is always possible for a game to occur more than once because only the first few moves are stored for all the games. These first few moves will be the same for many games.
Here's an example:
If your input PGN has the following games, and you only wanted to store 3 plies:
1. c2c4 e7e5 2. b1c3 g8f6
1. c2c4 e7e5 2. b1c3 f8b4
1. c2c4 e7e5 2. e2e3 g8f6
Then, this will be reduced by MakeBook into:
(count=2) c2c4 e7e5 b1c3
(count=1) c2c4 e7e5 e2e3
Furthermore, in the book file, this will end-up as:
(count=2) c2c4 e7e5 b1c3
(count=1) (same as 2 moves from above) e2e3
The book is also heavily compressed (LZ + huffman coding) by MakeBook. The default Thinker book for example is about 751K. But when uncompressed, it is more than 3MB. At the start of a game, BookThinker and Thinker expands the book file - this is the reason why it takes a while for Thinker to initialize at startup.
To convince Uri that such a small file can pack so much information, I will be posting a tool to convert any Thinker.dat back to PGN/text. This way, one can verify that all the games from the input PGN file made it to the book file.

I believe you that it is possible to compress the book file but I do not see the advantage that you get by compressing it.
Memory is very cheap so I do not care if a book is 751Kbytes or 3Mbytes and the only save in memory is in time that I do not use the book.
The disadvantage of having to wait for thinker to expand the file when it starts seems to me bigger disadvantage.
If you care about time of downloading then it is only a reason to have small default book and it is not a reason to release the program that compress the book.
Uri

[Günther Simon]

Geschrieben von:/Posted by: Günther Simon at 07 May 2004 10:57:33:
Als Antwort auf:/In reply to: Re: How do I create my own opening book for the Thinker chess engine? geschrieben von:/posted by: Lance Perkins at 06 May 2004 02:41:12:

I have posted before how makebook and bookthinker works.
Pls see: http://www.f11.parsimony.net/forum16635 ... /61762.htm
The book format does not store positions. Instead, it stores the actual games. Each game is preceeded by a count of how many times that game has occured in the input PGN and what is the outcome of that game (win/loss for which side). It is always possible for a game to occur more than once because only the first few moves are stored for all the games. These first few moves will be the same for many games.
Here's an example:
If your input PGN has the following games, and you only wanted to store 3 plies:
1. c2c4 e7e5 2. b1c3 g8f6
1. c2c4 e7e5 2. b1c3 f8b4
1. c2c4 e7e5 2. e2e3 g8f6
Then, this will be reduced by MakeBook into:
(count=2) c2c4 e7e5 b1c3
(count=1) c2c4 e7e5 e2e3
Furthermore, in the book file, this will end-up as:
(count=2) c2c4 e7e5 b1c3
(count=1) (same as 2 moves from above) e2e3
The book is also heavily compressed (LZ + huffman coding) by MakeBook. The default Thinker book for example is about 751K. But when uncompressed, it is more than 3MB. At the start of a game, BookThinker and Thinker expands the book file - this is the reason why it takes a while for Thinker to initialize at startup.
To convince Uri that such a small file can pack so much information, I will be posting a tool to convert any Thinker.dat back to PGN/text. This way, one can verify that all the games from the input PGN file made it to the book file.

I am interested in that tool for other reasons Lance. It would help
me in hand tuning some lines (and experimenting) faster I guess.
So if you decided not to post your tool after Uris answer, I would
still appreciate if you would send it to me, at least.
Best regards,
Günther
P.S. would you mind about taking a look at that post+answer too?
(It's about an existing problem with BookThinker)
http://f11.parsimony.net/forum16635/messages/66024.htm