Ultrafast UCI engines tournaments : a freeware tool
Posted: 05 Feb 2008, 14:05Vasik Rajlich and Larry Kaufman recently revealed on the Rybka forum that they test slight modifications of Rybka through the running of a very large number of ultrafast games : 80,000 three-seconds-total-duration (!) games played on two quads each night. This allows to detect improvements/worsenings of their engine in the one elo point range ...
For those who would like to go on this way toward ultrafast testing, no software I know of (free or commercial) is able to manage these kinds of tournaments and games.
In fact Rajlich and Kaufman designed a specific tool for this but they do not intend to make it publicly available.
So I tried to put together something of my own. This is not completely trivial under windows because as you go faster you have to deal with aborted or uncorrectly finished threads and processes after each game.
The resulting little tool is a kind of preprocessor : you define parameters in a small text file, the utility takes it and produces an enormous batch file. You launch the batch and the tournament is run. The process makes use of WinboardX (Tim Mann / Alessandro Scotti), Polyglot 1.4 (Fabien Letouzey) and of several freeware utilities dealing with threads and processes. I only tested it under XP64 but I have been told it also works under 32bits XP.
I have put together all required utilities in a package available at http://chessbazaar.mlweb.info.
The fastest I could go with it is a little more than four complete games per minute. At this rate more than 97% of games are played successfully with most top range engines (Zappa Mexico 1 being an exception with a 25% failure rate).
If you stay more reasonably at a duration of 30 seconds per game the success rate is close to 100% with all engines.
This is still far of the stunning testing scheme that Larry Kaufman uses but not too bad yet IMHO.
Last word : this is an amateur non-polished package for those who are not afraid to deal with winboard commands and parameters. It comes with a kind of manual and example config files but I will not be able to give more detailed explanations.
Marc Lacrosse
For those who would like to go on this way toward ultrafast testing, no software I know of (free or commercial) is able to manage these kinds of tournaments and games.
In fact Rajlich and Kaufman designed a specific tool for this but they do not intend to make it publicly available.
So I tried to put together something of my own. This is not completely trivial under windows because as you go faster you have to deal with aborted or uncorrectly finished threads and processes after each game.
The resulting little tool is a kind of preprocessor : you define parameters in a small text file, the utility takes it and produces an enormous batch file. You launch the batch and the tournament is run. The process makes use of WinboardX (Tim Mann / Alessandro Scotti), Polyglot 1.4 (Fabien Letouzey) and of several freeware utilities dealing with threads and processes. I only tested it under XP64 but I have been told it also works under 32bits XP.
I have put together all required utilities in a package available at http://chessbazaar.mlweb.info.
The fastest I could go with it is a little more than four complete games per minute. At this rate more than 97% of games are played successfully with most top range engines (Zappa Mexico 1 being an exception with a 25% failure rate).
If you stay more reasonably at a duration of 30 seconds per game the success rate is close to 100% with all engines.
This is still far of the stunning testing scheme that Larry Kaufman uses but not too bad yet IMHO.
Last word : this is an amateur non-polished package for those who are not afraid to deal with winboard commands and parameters. It comes with a kind of manual and example config files but I will not be able to give more detailed explanations.
Marc Lacrosse