Command line tool to play engine matches

[Alessandro Scotti]

Hi folks,
I would like to write a command line tool to play engine matches, as currently it's not easy to do that on non-Windows platforms.
For a start, I was thinking of something quite simple with just a few basic features:
- runs only a single match, then exits (with the proper code for interfacing with scripts);
- can start from a predefined FEN position;
- can monitor the game and adjudicate it;
- saves one or more log files with engine data (protocol messages, PV's and so on);
- initially supports only Winboard version 2 (*) and "conventional" time controls.

Project is open source (GPL) and probably written in Java.

I would already be quite happy with the above... what do you think?

(*) Design will allow the program to be easily extended with more protocols, however I'm not familiar with UCI that's why I would like to start with Winboard.

[Martin Danielsson]

Great idea!

I have been thinking about writing a similar tool
myself but never gotten around to do it.

One of the things I miss the most when working in Linux is a way to run
tournaments. Some easy way to set up for example a round-robin tournament
would be nice.



Regards,
Martin

[Rémi Coulom]

I would like to write a command line tool to play engine matches, as currently it's not easy to do that on non-Windows platforms.

What platform do you have in mind ? MacOS and Linux are probably the most popular alternatives, and I expect xboard to work well on both.

R?mi

[Klaus Friedel]

Hello,

if you are looking for some code, have a look at this nice
Java GUI:

http://jose-chess.sourceforge.net

Maybe you can use the engine or database code ?

Regards,
Klaus

[Dann Corbit]

You can use the stuff you find here:
ftp://cap.connx.com/pub/tournament_software/

You will find among other things:
swiss & round robin tournament automation algorithms
elo calculation algorithms (USCF and also Royal C. Jones methods)

HTH

[Tord Romstad]

I would like to write a command line tool to play engine matches, as currently it's not easy to do that on non-Windows platforms.

What platform do you have in mind ? MacOS and Linux are probably the most popular alternatives, and I expect xboard to work well on both.

Xboard works on both platforms, but in my opinion it does not really work well on any platform. There are just too many limitations. It does not support tournaments with more than two engines, it cannot run matches from a pre-defined set of positions, time handicap matches are not possible, it cannot run test suites, and it does not save evaluations, PVs or time information (except in form of the debug log file, which is difficult to parse).

It is possible to work around some of these limitations by using scripts, but this introduces another problem: Because xboard must often be restarted between every single game, it is not possible to keep it hidden or minimized. It pops up on the top of the desktop every 10 minutes or so, which is so annoying that it makes it impossible for me to work on the computer while tests are running. The problem is even worse on Mac OS X, because I have to start X11 just in order to be able to run tests with my engine.

A simple and unobtrusive console utility for running matches, tournaments and test suites would be vastly preferable, at least for my purposes. I will have to think a bit longer in order to make a list of the most important features I would like to see included. Unfortunately I am not sure that what Allessandro describes is exactly what I want. I would prefer a C program rather than a Java program, and UCI support has a higher priority than xboard support to me.

Tord

[eric_oldre]

Hi folks,
I would like to write a command line tool to play engine matches, as currently it's not easy to do that on non-Windows platforms.
For a start, I was thinking of something quite simple with just a few basic features:
- runs only a single match, then exits (with the proper code for interfacing with scripts);
- can start from a predefined FEN position;
- can monitor the game and adjudicate it;
- saves one or more log files with engine data (protocol messages, PV's and so on);
- initially supports only Winboard version 2 (*) and "conventional" time controls.

Project is open source (GPL) and probably written in Java.

I would already be quite happy with the above... what do you think?

(*) Design will allow the program to be easily extended with more protocols, however I'm not familiar with UCI that's why I would like to start with Winboard.

I think this would be a great idea!!!

my personal desire would just be to have some program that
I can use to play games. and let me have a "wrapper" program
to display whatever I want.

I actually started writing a (gasp!!!) c# library where you would create
a game object and it would handle starting the engines and
would raise events when things happened in the game such as a
move being played or a result.

however a command line program to play games would accomplish
the same thing for me as long as it gave updates during the game
and didn't simply print a result at the end.

[Fabien Letouzey]

Hi all,

Since the quote did not work last time I tried, let me use a more conventional mail-style reply.

> For a start, I was thinking of something quite simple with just a few basic features:

Yes, start as simple as possible.

> - runs only a single match, then exits (with the proper code for interfacing with scripts);

Fine with me. In fact I don't see in which way special code is needed. Just take parameters for match options and save the game as PGN (e.g. appended to a file).

Something more important is how you intend to handle engines: all data passed on the command line or some engine-specific configuration file?

- can start from a predefined FEN position;

Yes, however consider it as a short-term solution. In the long run, PGN is better because of move history.

- can monitor the game and adjudicate it;

Not necessary, you can add this later. Testing move legality and draw claims is already more than XBoard is doing (it knows only part of the rules).

- saves one or more log files with engine data (protocol messages, PV's and so on);

Yes. Start with just protocol messages (all mixed is fine), the rest is cosmetic.

- initially supports only Winboard version 2 (*) and "conventional" time controls.

Fine with me. Make sure you leave room for separate white/black time control later.

---

Don't try to kill the engine processes, except possibly after a time out. When the game is over, wait for the engines to quit before quitting yourself (because of broken pipes).

Fabien.

[Alessandro Scotti]

Hi,
thanks for the feedback and the suggestions! I think the initial set of goals must be changed a little now:
- can start from a PGN game *or* a FEN position (PGN would still allow that but there are cases when simple FEN is probably easier to use);
- allow different time controls for black and white (d'oh, didn't thought about it before);
- game adjudication and other minor things can be deferred a little to make room for the above;
- UCI is very important of course, it will be implemented as soon as there is a sufficiently stable base of code.

Considering that I already have code for reading and writing PGN in Kiwi, and some more, I probably could save some time by coding the thing in C++ although in this case I won't be able to test it on Mac OS X initially (...until my shiny new MiniMAC arrives! ).

As for tournaments and ratings, I'm very interested in those too, but I think they should be implemented as separate tools so that this one can be left as simple as possible.

Please keep the suggestions coming and tell me if you're not happy about something, it's much better to hear everything in advance and try to design things accordingly.

[Tord Romstad]

Hi,
thanks for the feedback and the suggestions! I think the initial set of goals must be changed a little now:
- can start from a PGN game *or* a FEN position (PGN would still allow that but there are cases when simple FEN is probably easier to use);
- allow different time controls for black and white (d'oh, didn't thought about it before);
- game adjudication and other minor things can be deferred a little to make room for the above;
- UCI is very important of course, it will be implemented as soon as there is a sufficiently stable base of code.

This sounds great! One more feature which I would like to see, and which should be relatively easy to implement:
- ability to run an engine through a test suite and print the results to a file.
Statistics about the number of solutions found would also be nice, but this could be tricky to implement for xboard engines (because the protocol does not specify the format of the PV).

Considering that I already have code for reading and writing PGN in Kiwi, and some more, I probably could save some time by coding the thing in C++ although in this case I won't be able to test it on Mac OS X initially (...until my shiny new MiniMAC arrives! ).

I am almost sure Mac OS X will not be a problem. And if you want some early tests on OS X, I can of course do it for you.

I hope you will be happy with your MiniMac, and that I will one day have Kiwi running on my Mac.

As for tournaments and ratings, I'm very interested in those too, but I think they should be implemented as separate tools so that this one can be left as simple as possible.

Agreed.

Tord

[Alessandro Scotti]

One more feature which I would like to see, and which should be relatively easy to implement:
- ability to run an engine through a test suite and print the results to a file.
Statistics about the number of solutions found would also be nice, but this could be tricky to implement for xboard engines (because the protocol does not specify the format of the PV).

Hi Tord,
I think that would be best implemented as a separate tool, but maybe there's already something out there:
- GradualTest by Odd Gunnar Malin (http://home.online.no/%7Emalin/sjakk/);
- EPD2WB by Bruce Moreland (http://www.seanet.com/%7Ebrucemo/gerbil/gerbil.htm).
Both come with source code. (I have tried neither though.)

[Fabien Letouzey]

Hi all,

- can start from a PGN game *or* a FEN position (PGN would still allow that but there are cases when simple FEN is probably easier to use);

FEN is fine for a start. I just wanted to make sure you leave room for PGN later; it is a lot more work.

- UCI is very important of course, it will be implemented as soon as there is a sufficiently stable base of code.

Don't hesitate to ask questions.

Considering that I already have code for reading and writing PGN in Kiwi, and some more, I probably could save some time by coding the thing in C++ although in this case I won't be able to test it on Mac OS X initially (...until my shiny new MiniMAC arrives! 8-)).

C++ will be better. As for Mac OS X compatibility, you cannot go wrong by following the POSIX standard. Applications can be ported to Windows using Cygnus GCC.

As for tournaments and ratings, I'm very interested in those too, but I think they should be implemented as separate tools so that this one can be left as simple as possible.

Agreed, this is even the essence of Unix. Plus I think what's needed for development is actually much more simple (e.g. no swiss pairing => "rounds" don't depend on previous results).

I see your tool as a command-line replacement for XBoard, it should not do much more (e.g. so that the other tools can be used with XBoard too).

How about we continue this in the programming forum? I think mostly programmers are going to be interested, at least before it's available.

BTW, you haven't answered about how you intended to handle engine parameters (directory, etc ...).

Fabien.

[Fabien Letouzey]

This sounds great! One more feature which I would like to see, and which should be relatively easy to implement:
- ability to run an engine through a test suite and print the results to a file.
Statistics about the number of solutions found would also be nice, but this could be tricky to implement for xboard engines (because the protocol does not specify the format of the PV).

PolyGlot will have this for UCI engines in a few weeks.

Fabien.

[Alessandro Scotti]

BTW, you haven't answered about how you intended to handle engine parameters (directory, etc ...).

Hi Fabien,
I'm still pondering this part and of course any input is welcome. What I have in mind is this: the tool uses an external "database" (a text file actually) that contains all the parameters needed to run an engine: executable location, command line options and so on. This will allow the following usage:

Code: Select all

match Glaurung_0.1.7 Fruit_2.0

(plus the other options related to time controls and so on) that I find much easier to read. Also the database could easily accomodate all older versions of an engine, of which one only has to remember the name.
However, all the options specified in the database can be overridden by a corresponding command line switch. So, the tool can also be used with no database at all.
I think this is not much different from how XBoard works.

[Dann Corbit]

This sounds great! One more feature which I would like to see, and which should be relatively easy to implement:
- ability to run an engine through a test suite and print the results to a file.
Statistics about the number of solutions found would also be nice, but this could be tricky to implement for xboard engines (because the protocol does not specify the format of the PV).

PolyGlot will have this for UCI engines in a few weeks.

Fabien.

Have you guys seen Bruce Moreland's tool Epd2WB?
http://www.brucemo.com/compchess/gerbil/index.htm

It runs an engine against an EPD test suite.

There is also gradualtest by O.G.M.:
http://home.online.no/~malin/sjakk/

Both come with source code.

[Peter Fendrich]

Have you guys seen Bruce Moreland's tool Epd2WB?
http://www.brucemo.com/compchess/gerbil/index.htm

It runs an engine against an EPD test suite.

There is also gradualtest by O.G.M.:
http://home.online.no/~malin/sjakk/

Both come with source code.

I like the GradualTest feature that writes two files.
- Right.epd
- Wrong.epd
With info about scores, prefered move, PV etc.
Taking a look at Wrong.epd is very educating at least!
/Peter

[eric_oldre]

Alessandro,
I haven't heard any updates on this idea in a while, Is this still a project you are considering?

If so I would be glad to help out in any way I can, since I'd be very interested in the final product. I can do beta testing, help with the design, can try to fix bugs, or whatever else you'd like help with.

Eric Oldre

[Alessandro Scotti]

Alessandro,
I haven't heard any updates on this idea in a while, Is this still a project you are considering?

If so I would be glad to help out in any way I can, since I'd be very interested in the final product. I can do beta testing, help with the design, can try to fix bugs, or whatever else you'd like help with.

Eric Oldre

Hi Eric,
thanks for offering help! Yes of course I'm still behind this project, but I don't think I will start to code something before a week or two, as I'm too busy in this period. In the meanwhile I'll keep working on the design and I've setup an environment that uses xboard to play matches. Hopefully, this will help me understand better how things are supposed to work, and suggest improvements. Actually, one of the things I'm considering is working on the xboard code itself and "simply" add the missing features...
Anyway, as soon as I have something to show I'll definitely post a message here! BTW, if you have a spare Unix machine I can send you the scripts I'm currently using to play matches, get statistics and so on... very useful for testing new ideas!

[Dann Corbit]

Alessandro,
I haven't heard any updates on this idea in a while, Is this still a project you are considering?

If so I would be glad to help out in any way I can, since I'd be very interested in the final product. I can do beta testing, help with the design, can try to fix bugs, or whatever else you'd like help with.

Eric Oldre

Hi Eric,
thanks for offering help! Yes of course I'm still behind this project, but I don't think I will start to code something before a week or two, as I'm too busy in this period. In the meanwhile I'll keep working on the design and I've setup an environment that uses xboard to play matches. Hopefully, this will help me understand better how things are supposed to work, and suggest improvements. Actually, one of the things I'm considering is working on the xboard code itself and "simply" add the missing features...
Anyway, as soon as I have something to show I'll definitely post a message here! BTW, if you have a spare Unix machine I can send you the scripts I'm currently using to play matches, get statistics and so on... very useful for testing new ideas!

Since Winboard/Xboard is not a public project, hosted at SourceForge, anyone can make improvements.