Winboard Forum

I played some games with a few engines. Opponent was Fruit in all cases using the Marc's "performance" book.

Each engine played 2*100 games at bullet time control (1+1). One series with all learning disabled then the same with all learning enabled. In the "learn off" series of Romi its learn file was deleted after every game.

Here are the results which are a bit surprising for me:

Code: Select all

No. Name                      Win Draw Loss Unf.  Score Games
-------------------------------------------------------------
  1 Fruit 2.1                +629 =194 -177   *0  726.0  1000
  2 Naum 2.0 learn off        +44  =27  -29   *0   57.5   100
  3 Naum 2.0 learn on         +37  =37  -26   *0   55.5   100
  4 Crafty-20.14 learn off    +25  =26  -49   *0   38.0   100
  5 Crafty-20.14 learn on     +21  =30  -49   *0   36.0   100
  6 Aristarch 4.50 learn off  +22  =26  -52   *0   35.0   100
  7 Aristarch 4.50 learn on   +20  =26  -54   *0   33.0   100
  8 RomiBeta1bP3j learn off    +3   =8  -89   *0    7.0   100
  9 RomiBeta1bP3j learn on     +3   =8  -89   *0    7.0   100
 10 GreKo 3.3a learn on        +2   =3  -95   *0    3.5   100
 11 GreKo 3.3a learn off       +0   =3  -97   *0    1.5   100


It seems there wasn't any effect of learning within 100 games.

Volker

Volker Pittlik wrote:I played some games with a few engines. Opponent was Fruit in all cases using the Marc's "performance" book.

Each engine played 2*100 games at bullet time control (1+1). One series with all learning disabled then the same with all learning enabled. In the "learn off" series of Romi its learn file was deleted after every game.

Here are the results which are a bit surprising for me:

Code: Select all

No. Name                      Win Draw Loss Unf.  Score Games
-------------------------------------------------------------
  1 Fruit 2.1                +629 =194 -177   *0  726.0  1000
  2 Naum 2.0 learn off        +44  =27  -29   *0   57.5   100
  3 Naum 2.0 learn on         +37  =37  -26   *0   55.5   100
  4 Crafty-20.14 learn off    +25  =26  -49   *0   38.0   100
  5 Crafty-20.14 learn on     +21  =30  -49   *0   36.0   100
  6 Aristarch 4.50 learn off  +22  =26  -52   *0   35.0   100
  7 Aristarch 4.50 learn on   +20  =26  -54   *0   33.0   100
  8 RomiBeta1bP3j learn off    +3   =8  -89   *0    7.0   100
  9 RomiBeta1bP3j learn on     +3   =8  -89   *0    7.0   100
 10 GreKo 3.3a learn on        +2   =3  -95   *0    3.5   100
 11 GreKo 3.3a learn off       +0   =3  -97   *0    1.5   100


It seems there wasn't any effect of learning within 100 games.

Volker

Hi Volker,

Did you think it was an easy subject? Learning main obstacle has always been the great number of variations involved from the standard starting position. I could talk you about an experiment I made some months ago: two learning engines (Romichess and The Baron) without any opening book, against Crafty 20.14 and Crafty Cito, both using Crafty excellent opening book. After 1000 r/robins, learning wasn't showing any effect. That's why I built Manhattan opening book (meaning it was like the project leading to the atom bomb). I'm pretty sure, you DIDN'T give it to any learning engine to use in games against Fruit.
I think, if you run a 1000 games match Romichess (with Manhattan book) against Fruit (with any book of your choice) you'll see the effects. If you're not eager to run it, I'll run it for you!

Bye,

Rodolfo Leoni wrote:... After 1000 r/robins, learning wasn't showing any effect. That's why I built Manhattan opening book (meaning it was like the project leading to the atom bomb). I'm pretty sure, you DIDN'T give it to any learning engine to use in games against Fruit. ...

After a while the test wasn't specific to Romi or your book anymore.

I made a book from the pgn you send me for one of the engines (and I used settings to build it properly). I cannot use an Arena book on Linux because it crashes here. The book is very good, but not so good that it could change my opinion.

I think you described the problem very well: there was no effect after 1000 games!

Another one suggested me not to give a book to the opponent engine. That setup would be unrealistic. Opponents come with a book and in normal tournaments much less games with different opponents are played.

I think I'll playing around with other things in the next time.

Volker

Volker Pittlik wrote:After a while the test wasn't specific to Romi or your book anymore.

I made a book from the pgn you send me for one of the engines (and I used settings to build it properly). I cannot use an Arena book on Linux because it crashes here. The book is very good, but not so good that it could change my opinion.

I think you described the problem very well: there was no effect after 1000 games!

Another one suggested me not to give a book to the opponent engine. That setup would be unrealistic. Opponents come with a book and in normal tournaments much less games with different opponents are played.

I think I'll playing around with other things in the next time.

Volker

Thank you for saying Manhattan is very good. But I think you didn't give it to learning engines.
I described you there were no learning effects because learning engines had no book! We can have those effects now, because we can give them a book!
The suggestion you received, it's a nonsense. Manhattan book project has the goal to help learning engine in tournament. It'll help Romichess on next WBEC, and maybe on next ChessWar, tournaments. We'll just have to wait for results.
Meantime, everybody can run his own tests with Romichess and Manhattan. It'd be nice to hear some neutral voices.

Bye,

Volker Pittlik wrote:I played some games with a few engines. Opponent was Fruit in all cases using the Marc's "performance" book.

Each engine played 2*100 games at bullet time control (1+1). One series with all learning disabled then the same with all learning enabled. In the "learn off" series of Romi its learn file was deleted after every game.

It seems there wasn't any effect of learning within 100 games.

I do not think this is very surprising. In most of the test matches I received from Rodolfo, the learning effect started to kick in only after a few hundred games. In some games it may even give a negative effect, as it might try the next best move instead of the best move when the score is dropping. In the next game it will find out that the next best move is really worse than the best move, and select the best move again.

If you play with a book for both engines as well, the number out-of-book positions that the engine has to deal with is also quite big. This also reduces the chances that it will learn something good. Rodolfo's Manhattan book solves half this problem by giving the learning engine just one choice while in book.

Therefor I think that the main strength of (position) learning is its use during analysis, moving back and forth in a game, while using the results that were learned earlier.
This, and dealing with repeaters on chess servers of course
Richard.

Richard Pijl wrote:

Volker Pittlik wrote:I played some games with a few engines. Opponent was Fruit in all cases using the Marc's "performance" book.

Each engine played 2*100 games at bullet time control (1+1). One series with all learning disabled then the same with all learning enabled. In the "learn off" series of Romi its learn file was deleted after every game.

It seems there wasn't any effect of learning within 100 games.

I do not think this is very surprising. In most of the test matches I received from Rodolfo, the learning effect started to kick in only after a few hundred games. In some games it may even give a negative effect, as it might try the next best move instead of the best move when the score is dropping. In the next game it will find out that the next best move is really worse than the best move, and select the best move again.

If you play with a book for both engines as well, the number out-of-book positions that the engine has to deal with is also quite big. This also reduces the chances that it will learn something good. Rodolfo's Manhattan book solves half this problem by giving the learning engine just one choice while in book.

Therefor I think that the main strength of (position) learning is its use during analysis, moving back and forth in a game, while using the results that were learned earlier.
This, and dealing with repeaters on chess servers of course
Richard.

Hi Richard,

you always have good and valuable opinions about anything. I'm the first to admit, learning has some limits. But it has some good applications. When doing analysis, of course. But now, thanks to this first very beta Manhattan book, it could have effects on tournament too. That's something worth to try, anyway.

I tell you the project I'm working with Romichess. I'm building a very large learn data file with Romichess, to be used as Manhattan book extension in next WBEC tournament. Up to now, Romi stored about 5000 games, all played with Manhattan book against several engines. Thank to Romi merge command, I can import into data file games played by other strong engines too, as well as by human GMs. Target is to have at least 50.000 learned games as extension of Manhattan book. Then, we'll check effects of all this work on WBEC Romi tournament (4th division). Just an experiment. If it works, that'd mean my work isn't useless.

Maybe I'm a bit emotional about all of that. But, just consider that work took several months of attempts. All positions were Rybka-checked. Not only for blunders check, but to find the best move too. Each of 7.125 book positions took me time. When Rybka didn't suffice, I used combined analysis of Glaurung and Spike, making moves and taking them back, until I felt good about any single choice. I don't assume I always found the best moves, as my own playing strenght is only CM (I miss a dozen italian ELO points to CM). But my effort was great. And my tests tell me my work was worth of such efforts.

Bye,

P.S.
My book don't solve half of problem. When playing only 1.c4 for white, my book solve over 95% of problem... Going deeper, 99,99% of problem is solved.

Volker Pittlik wrote:I played some games with a few engines. Opponent was Fruit in all cases using the Marc's "performance" book.

Each engine played 2*100 games at bullet time control (1+1). One series with all learning disabled then the same with all learning enabled. In the "learn off" series of Romi its learn file was deleted after every game.

Here are the results which are a bit surprising for me:

Code: Select all

No. Name                      Win Draw Loss Unf.  Score Games
-------------------------------------------------------------
  1 Fruit 2.1                +629 =194 -177   *0  726.0  1000
  2 Naum 2.0 learn off        +44  =27  -29   *0   57.5   100
  3 Naum 2.0 learn on         +37  =37  -26   *0   55.5   100
  4 Crafty-20.14 learn off    +25  =26  -49   *0   38.0   100
  5 Crafty-20.14 learn on     +21  =30  -49   *0   36.0   100
  6 Aristarch 4.50 learn off  +22  =26  -52   *0   35.0   100
  7 Aristarch 4.50 learn on   +20  =26  -54   *0   33.0   100
  8 RomiBeta1bP3j learn off    +3   =8  -89   *0    7.0   100
  9 RomiBeta1bP3j learn on     +3   =8  -89   *0    7.0   100
 10 GreKo 3.3a learn on        +2   =3  -95   *0    3.5   100
 11 GreKo 3.3a learn off       +0   =3  -97   *0    1.5   100


It seems there wasn't any effect of learning within 100 games.

Volker

Hi Volker,

Thank you for the test. I offer some observations on learning though. If the book that was used for Fruit has more than 100 end leaf positions for white and black then there is a possibility that no variation was played more than once. In this case learning was done, but, never utilized. A position needs to be encountered more when the gap in playing strength between the engines is greater. The match up between RomiChess and Fruit is extreamly biased against learning unless you are willing to play thousands of games to get an accurate read. OTOH learning for RomiChess against an engine that is only 100 to 200 points stronger than Romi will show a strong effect in far fewer games. To answer your question as to what it is good for--I thought that you read the thread between Bob and myself and my rather lengthly explaination of its only intended use and that computer test were just that, test. Testing for my original intended and stated purpose is better conducted against an engine with no book at all inorder to simulate testing vs. a human wanting to study a specific variation. Having said this though, engine vs. engine compitition testing is also very interesting.

So far in my RomiChess vs. Spike test--also highly biased against the testing of Romi's learning ability. First 1000 games gave a 9% score for Romi. Games 1001 - 1371 so far have Romi at 11% for those games. Spike is also a very good learning engine. Romi is learning Spikes internal book of 400000 moves and adding to it and perfecting it for her own use that is what it is good for concerning engine vs. engine compitition! However, this may not interest everyone.

Mike

Rodolfo Leoni wrote:P.S.
My book don't solve half of problem. When playing only 1.c4 for white, my book solve over 95% of problem... Going deeper, 99,99% of problem is solved.

Oh yes, it is just how you count. White side is a problem, black side is a problem -> white side has no choices anymore: Half the problem is gone

Of course, by eliminating all choices for one side, you're only left with the square root of all problems (assuming just 2 decent choices for every position)

Your book is a big step in increasing the usability of position learning in tournaments, and I think you already proved it to be a small, but high quality book. I just do not know if that is enough to bring results in 'small' tournaments like the one Volker just played. And once your book is public, it is also relatively easy to prepare for it ...
Richard.

Richard Pijl wrote:

Rodolfo Leoni wrote:P.S.
My book don't solve half of problem. When playing only 1.c4 for white, my book solve over 95% of problem... Going deeper, 99,99% of problem is solved.

Oh yes, it is just how you count. White side is a problem, black side is a problem -> white side has no choices anymore: Half the problem is gone

Of course, by eliminating all choices for one side, you're only left with the square root of all problems (assuming just 2 decent choices for every position)

Your book is a big step in increasing the usability of position learning in tournaments, and I think you already proved it to be a small, but high quality book. I just do not know if that is enough to bring results in 'small' tournaments like the one Volker just played. And once your book is public, it is also relatively easy to prepare for it ...
Richard.

Hi Richard,

I just thought at 4 Opening Encyclopedia eliminated (and most of volume A too) when giving this % value.
For opponent, book has the best opponent choices. Sometimes, they are half a dozen. But book is growing, as I'll use all games in progress to make a better work.
You're right, book is public. This one is public! I'd invite opponents to carefully prepare for it.
I think future is for engines like The Baron, Romichess, and mayby Spike.

Bye,