Test with open source engines

[Volker Pittlik]

I run some testgames with open source engines. I guess I'll limit myself to these engines because

- that limit the number of engines to look at
- I can optimize the executables to my system
- all compilations are of the same (most likely bad) quality
- hard times for clones.

Also the engines should be under active developement. If the author gives them a break for more than six months or so they also will have a break in testing.

In addition the engines should have clear instructions how to compile the sources (a Makefile is best) and the build should be possible without installing dozens of additional packages.

I report the result here and not in the tournamnt forum, because this will not be a continious list. I don't claim to determine the real strength of the engines.

I play the games with ponder on on a single cpu system. I never noticed a difference in the resulting rating list if ponder on or ponder off is used (if all engines can ponder). I also didn't notice an unfair usage of the ressources by the engines. If both engines are pondering they use 49+ % of the cpu each. If an engine can't ponder then it is in bad luck. The ability to ponder is an advantage. Why handicapping the engines which can do?

To prevent my harddisk from possible damage I use only 4-piece TBs in a ramdisk. This makes also a faster access to the TBs possible. I'm thinking about to put all 5-piece TBs on two of these 4 GB USB memory sticks but at the moment they are still a bit to expensive. I also don't know if the TBs will fit on them because the amount of storage is only possible with a compression software on these sticks.

Time control of the games was 5 minutes initially and 5 seconds per move. It came out GreKo lose all games with black under these conditions and was therefore taken out (Vladimir is informed already). Here are the results:

Code: Select all

Ranking-060712
XP2000+, PonderOn, 32 MB hash, 4 piece TBs, 5+5, VPittlik, 2006.07.12
                           Score     Toga Scor Craf Craf Glau Aras EXch King xpdn HoiC Prop GiuC
-------------------------------------------------------------------------------------------------
 1: Toga 1.2.1           36.5 / 44   XXXX ==01 1100 11=1 =0== 1=11 1111 1111 1111 1111 1111 1111
 2: Scorpio 1.8          35.5 / 44   ==10 XXXX =10= 0011 111= 0101 1111 1111 1111 1111 1111 1111
 3: Crafty-Cito 1.5      33.0 / 44   0011 =01= XXXX 001= 111= 1010 11== 1110 1111 1111 1111 1111
 4: Crafty-20.14         33.0 / 44   00=0 1100 110= XXXX 0=1= 1011 =11= 1111 1111 1111 1111 1111
 5: Glaurung 1.2         32.5 / 44   =1== 000= 000= 1=0= XXXX 1111 111= 1=11 1111 1111 1111 1111
 6: Arasan 9.2           27.0 / 44   0=00 1010 0101 0100 0000 XXXX 0=11 1111 1101 1111 1111 1111
 7: EXchess-5.01-beta    22.5 / 44   0000 0000 00== =00= 000= 1=00 XXXX 1111 1011 1=11 1111 1111
 8: King's Out 0.2.41    15.5 / 44   0000 0000 0001 0000 0=00 0000 0000 XXXX 1011 111= 11=1 1111
 9: xpdnt_060602         14.0 / 44   0000 0000 0000 0000 0000 0010 0100 0100 XXXX 1110 1111 1111
10: HoiChess 0.4.3       10.0 / 44   0000 0000 0000 0000 0000 0000 0=00 000= 0001 XXXX 1111 1111
11: Prophet v2.0 Beta 3   4.5 / 44   0000 0000 0000 0000 0000 0000 0000 00=0 0000 0000 XXXX 1111
12: GiuChess-1.0beta2     0.0 / 44   0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 XXXX
-------------------------------------------------------------------------------------------------
264 games: +128 =24 -112



As you know Toga is based on Fruit.

Scorpio came in 2nd ahead of the two Crafty versions.

I don't see Crafty-Cito dozens of phantasy rating points ahead of the original but I also don't see it behind it.

Glaurung's antecessor did better in the previous (unpublished) test. But as we all know these results are all like coin tossing if not hundrets or thousands of games have been played.

Arasan lost a game on time (at move 140). I don't expect any losses on time with a 5 seconds increment. It possibly have to do with pondering. I'll check that with the new version of the engine.

EXchess worked without any problem.

There is an unclear situation between Kings Out and xpdnt. Xpdnt won on time although it checkmated Kings Out. I can't read its logfile good enough if possibly the RESULT command from the engine was missing.

HoiChess and Prophet did well. Prophet didn't lose on time what happend in the previous (unpublished) test very often.

GiuChess didn't get any point and lost 8 of its game on time. I guess I'll wait for a new version before testing it again.

Comments welcome.

Regards

Volker

The games are here: http://www.volker-pittlik.name/temp/Ranking-060712.pgn.bz2

[ShaunBrewer]

I look forward to watching these with interest. What book/books are being used?

Rather than USB sticks have you also looked at compact flash with a reader - might be cheaper and you can get 8gb compact flash - (although these larger ones are still be a little expensive).

EDIT: USB sticks may be cheaper I had a quick look and I had perhaps miss remembered some of the prices for compact flash???

I would be interested to see the performance of flash v disk can anyone point me at a comparison.

Something I might need to look at too - as TB access does hammer the disks.

Shaun

[Volker Pittlik]

I look forward to watching these with interest. What book/books are being used?...

If available the book that comes with the engine.

Cito-Crafty got the same book as Crafty (in another directory of course).

Togo used "performance.bin" by Marc Lacrosse.

If an engine comes without book (but can use one) I created one for them. I use a collection from high rated standard games by humans (from chesslib.no and TWIC) and of high quality computer games (CEGT, CCRL, WBEC...).

Rather than USB sticks have you also looked at compact flash with a reader - might be cheaper and you can get 8gb compact flash - (although these larger ones are still be a little expensive).

EDIT: USB sticks may be cheaper I had a quick look and I had perhaps miss remembered some of the prices for compact flash???

I would be interested to see the performance of flash v disk can anyone point me at a comparison.

Something I might need to look at too - as TB access does hammer the disks.

Shaun

Maybe I can think of a setting to see the differences of speed if the TBs are read from harddisk, ramdisk and USB/flash. I guess harddisk will always be the slowest.

However, the good engines play the endgame well also without TBs (for example Crafty) or can't use them at all (for example Glaurung).

Regards

Volker

[Daniel Shawul]

I recently have a disk crash. At first I thought it was a virus, and tried latest Norton and AVG but got nothing. Then some one told me it could be dust collected on the processor,fan. I cleaned it nothing happened. Then I found out that this is probably caused by repeated generation/ compression of bitbases that i did in the past four months. Now i can't even generate a single 5 piece bitbase without the pc shutting down. Fortunately, it does not crush when scorpio uses the 5 bit bitbases during actual games.

Thanks for the games.

regards,
Daniel

[Uri Blass]

<snipped>

I play the games with ponder on on a single cpu system. I never noticed a difference in the resulting rating list if ponder on or ponder off is used (if all engines can ponder). I also didn't notice an unfair usage of the ressources by the engines. If both engines are pondering they use 49+ % of the cpu each. If an engine can't ponder then it is in bad luck. The ability to ponder is an advantage. Why handicapping the engines which can do?

The games are here: http://www.volker-pittlik.name/temp/Ranking-060712.pgn.bz2

By playing single cpu with ponder on you handicap engine that cannot ponder or ponder only in part of the time.
The problem is that in case that they do not ponder the opponent get 100% of the cpu time instead of 50%

The situation in real games with ponder on is better for engines that cannot ponder because at least they do not cause their opponents to calculate faster.

Uri

[Volker Pittlik]

...
By playing single cpu with ponder on you handicap engine that cannot ponder or ponder only in part of the time.
The problem is that in case that they do not ponder the opponent get 100% of the cpu time instead of 50%...

There is some truth in that. OTOH the engines which can't ponder are a minority. I doubt that in the group I've tested the engines which can't ponder would do much better if permanent brain is off.

Volker

[Bernd Nürnberger]

There is an unclear situation between Kings Out and xpdnt. Xpdnt won on time although it checkmated Kings Out. I can't read its logfile good enough if possibly the RESULT command from the engine was missing.

Hi Volker,
the missing result command is a bug in King's Out 0.2.41, fixed in version 0.2.42... btw. King's Out had really a good day against xpdnt..
Regards,
Bernd

[Volker Pittlik]

...the missing result command is a bug in King's Out 0.2.41, fixed in version 0.2.42... ...

Aarrgh! Overseen by me. I'm going to test the new version as soon as possible.

BTW: I'm studying Java at the moment. Of course your engine is very interesting for me .

Volker

[Bernd Nürnberger]

...the missing result command is a bug in King's Out 0.2.41, fixed in version 0.2.42... ...

Aarrgh! Overseen by me. I'm going to test the new version as soon as possible.

BTW: I'm studying Java at the moment. Of course your engine is very interesting for me .
Volker

But note that this bugfix and another minor fix are the only changes done to 0.2.41 (.41 und .42 differing only in three characters )...

Don't put your nose to close to my code... many parts are not done well (especially the huge board class). I plan a complete rewrite in respect to a better code...

[Marc Lacrosse]

(...)

Toga used "performance.bin" by Marc Lacrosse.


Volker

Nice to see that this already somewhat old book can still lead to good results for the Fruit family of engines

By the way I tried hard to produce something better for Toga a few months ago but I could not succeed.

Does someone know of a better "general" polyglot book for Toga (or for another strong engine ) ?

I would be much interested in testing and studying such a book.

Regards

Marc