It's a plague of Winboard engines

[Kirill Kryukov]

I tried to run some matches with GNU Chess 5.07, and noticed that it has the same bug with number of other Winboard engines: It does not count the opening moves from external book as part of time control. This is really getting annoying. So many engines are hit by this bug. The result - they have to move instantly from move 49 to 80 in my tourney, also from move 89 to 120 if they survive that long, etc. (See here for details and discussion). Needless to say that this hurts the performance badly. So instead I have to run such engines without any book at all, which also hurts the performance and does not allow to accurately estimate their strength. The source of the problem - the horrible Winboard protocol which leaves important details unspecified.

So the list of engines bugged with this same problem (known to me) (Last edited 2011-02-20):

Open source:
GNU Chess 5.07 => Fixed by Michel in version 5.07.2b (source, Jim Ablett's compile).
Resp 0.19
Exchess 5.02 beta

Closed source:
Bodo 0.2b
Betsy 6.51
Grizzly 1.40.1b
La Dame Blanche 2.0c
Laurifer 1.0
Matant 5.04
Mustang 4.97
Requiem 0.53
Storm 0.6

If at least the open source engines could be fixed - that would solve half of the problem. The closed source engines would still remain impossible to test properly.

Is there anything that can be done with this issue on the interface level?

Or is there anyone who could try and patch the affected open source engine? It's a protocol issue after all, not some deep logic.

In the meantime I'll test GNU Chess both with and without book (both handicapped in different ways), and the one which turns out stronger will be listed in my tourney.

Best,
Kirill

[H.G.Muller]

Indeed, many engines are plagued by this. The only solution I found for it was to play incremental controls.

The source of the problem is not so much WB protocol, as well the protocol specs that were very unclear. (Especially the phrase "clock that always belongs to the engine, even if the engine changes color this clock remains with the engine" seems to cause confusion. Many people seem to interpret this as "time remains with the engine", which then leads to the logical conclusion that an engine times only its own moves, which leads to this bug. (I fell for that too, when I first adapted my engine to WB protocol.) What was meant is that the time command is always referring to the clock that the engine is playing. But the clocks remain with their color, whatever the engine does.

The current version of the protocol specs addresses this point unambiguously:

The clocks in fact always remain with the color. Which clock reading is relayed with "time", and which by "otim", is determined by which side the engine plays. Note that the way the clocks operate and receive extra time (in accordance with the selected time control) is not affected in any way by which moves are made by the engine, which by the opponent, and which were forced.

So I think I did solve the root cause, but that of course does no good for existing engines. Filing bug reports withe the authors is the only solution.

[Kirill Kryukov]

Hi H.G.,

It's great that you improved the protocol specs! Hopefully new engines will not have this problem. (Still I stand that a stateful protocol is inherently full of pitfalls).

I mailed a bug report to bug-gnu-chess@gnu.org, but I'm not holding my breath. In my experience the longer an engine was not updated - the smaller is the chance that it will ever get updated.

Hoping for the best.

Regards,
Kirill

[Olivier Deville]

Hi Kirill

I have the email address of Chua Kong Sian. I can forward him the bug report if you want.

Olivier

[Kirill Kryukov]

Hi Kirill

I have the email address of Chua Kong Sian. I can forward him the bug report if you want.

Olivier

Hi Oliver,

That would be nice! I emailed you a copy of the bug report. (You can also attach your comments to it if you like)

Thanks,
Kirill

[H.G.Muller]

(Still I stand that a stateful protocol is inherently full of pitfalls).

That might be, but I don't think that in this case staeless vs stateful has anything to do with it in this case. The problem is that on forcing in a move the state is not consistently updated: the move is accepted and the position altered, but the move counter is not incremented. Exactly the same could happen in an UCI engine: there you force in moves for every move the engine has to make. If an engine would not increment the move counter then, you would get similar problems...

[plattyaj]

So, looking at the code for Schola I see I would fall prey to this gotcha too. If I understand correctly, what I need to do is keep move counts for each color, update them regardless of whether the engine or the controller "play" the move and that way if I get a series of moves while in force, the move counter is correct when I receive the "go" command?

Andy.

[H.G.Muller]

That sounds like it would do it. You could also just count plies from the beginning of the game, and if you play M minutes for N moves, you should expect to get new time when the ply counter hits 2*N (and 4*N, etc.). It is not clear what you should do if you start from a FEN with a move number larger than 1. Fortunately, no known GUI seems to send such FENs. But to prepare for the future, you should take the move number contained in the FEN seriously. I.e. if it says 39, and you play 40/5, expect to get 5 more minutes after the next move.

[Michel]

I tried to run some matches with GNU Chess 5.07, and noticed that it has the same bug with number of other Winboard engines: It does not count the opening moves from external book as part of time control.

Are you sure about this? I had a quick look at the source code of GNU Chess and the ply counter (called GameCnt) seems to be updated correctly during force mode.

[Kirill Kryukov]

I tried to run some matches with GNU Chess 5.07, and noticed that it has the same bug with number of other Winboard engines: It does not count the opening moves from external book as part of time control.

Are you sure about this? I had a quick look at the source code of GNU Chess and the ply counter (called GameCnt) seems to be updated correctly during force mode.

Yes I'm sure. This is how it behaves in my tournament, in every game. For example you can download the already finished GNU Chess 5.07 games from KCEC. (Only one 32-game match with Madeleine 0.2 so far). The time usage pattern is exactly same with other engines suffering from this bug, and it's repeating in every game.

Details: I am running GNU Chess 5.07 64-bit JA, through WB2UCU adapter, under ChessProgram8 GUI by Chessbase. The command I use is "gnuchess_x64.exe -s 2097152" to adjust the hash.

May be I should try it with different GUIs to make sure it's not the GUI or adapter problem. Although countless other Winboard engines play without this problem. Only these few have this issue.

[Teemu Pudas]

(Still I stand that a stateful protocol is inherently full of pitfalls).

That might be, but I don't think that in this case staeless vs stateful has anything to do with it in this case. The problem is that on forcing in a move the state is not consistently updated: the move is accepted and the position altered, but the move counter is not incremented. Exactly the same could happen in an UCI engine: there you force in moves for every move the engine has to make. If an engine would not increment the move counter then, you would get similar problems...

Not true. Why would a UCI engine even have a move counter when the GUI sends the number of remaining moves with the go command?

[H.G.Muller]

Oh, does it? Well, then you would get an error when the engine would decrement that remaining-move-count for every move it receives.

It is always possible to make an erroneous implementation (by design, not mistake) if you don't properly understand what the counter is supposed to represent.

[Teemu Pudas]

Oh, does it? Well, then you would get an error when the engine would decrement that remaining-move-count for every move it receives.

The engine won't receive any moves during a search. Even when pondering, it's either stop->position(moves)->go or just "ponderhit".

[Zach Wegner]

It's great that you improved the protocol specs! Hopefully new engines will not have this problem. (Still I stand that a stateful protocol is inherently full of pitfalls).

UCI is not fully stateless, and even a stateless protocol could theoretically have the same problem. UCI just "holds the engine's hand" throughout the game. The bug is primarily an engine problem, and less so an xboard protocol problem since it naively assumes that engines will be able to handle this without bugs. Xboard could easily add new commands that give this information for every move (as it does with time/otim), but that wouldn't help these old engines. Plus, if you were to go to the effort of adding support for this command, you might as well just fix the bug.

[Kirill Kryukov]

Back to GNU Chess 5.07. Here are details for one typical game showing this bug:

• PGN file
• Evaluation profile and time usage in this game (with my comments)
• WB2UCI log of GNU Chess

GNU Chess was doing well till move 41. Then it used up all its time on moves 41-48. After that it had about 5 seconds for remaining 32 moves, so has to move instantly. It gradually lost its advantage and eventually lost the game too.

How to read the log file. Lines with "S>" - What engine sent to the WB2UCI adapter. Lines with "S<" - what adapter sent to engine.

Since I test in blitz, normally I disable all logging. This is just one demonstration game, not a part of my tournament.

[H.G.Muller]

Well, it is clear that GNU Chess counts erroneously. It ives its moves in the non-compliant "NR... MOVE" format, and it starts by saying: "1... e1g1". So it thinks the castling is the wrong move.

It seems that the "white" command resets its move count. I played it from the command line forcing in a few moves, and after you type "go" it produces a move with the correct number. But if you type "white" first (when white is to move, so that it should be a no-op), then the next move is number 1 again.

Proposed cure: Use InBetween to filter all "black" and "white" commands out of the GUI -> engine communcation.

[Kirill Kryukov]

Interesting. Did you try it? Could you post a complete configuration? A workaround like this would be great.

[H.G.Muller]

I did not try it, and I cannot guarantee it works for other engines. I only used InBetween once, for swapping the commands white and black when playing GNU Shogi under WinBoard. The InBetween.ini file I used for that was:

Code: Select all

[InBetween]
; You can set program option here if its not possible in the server program
;
; The priority flag is if you want the server app. to run with lower priority.
; This could be when you want use an engine to analyze games/positions and
; you also want to use the computer to other work. If this is not set it will
; try to see what the client wants and set the server to this priority.
; Don't use the high option is you don't know what you do.
;Priority := low, normal or high
;
;CommandLine := f:\cygwin\home\chess\GNShogi\gnushogi.exe
;
; The debug switch is if you want to see the command flow in a window when the
; server and client talks. 1 means viewpoint is on the client interface, 2 means
; viewpoint is on the server interface, and finaly 3 means that the viewpont is
; after the translation (input to client and input to server).
;Debug := 0, 1, 2 or 3
Debug := 0
;
; A try to pass any control signal.
;Ctrl := False or True
;
; The logfile is if you want that InBetween log all communications.
;Log := logfile.log

[Client2Server]
; Set the translation of client command to server here.
; Format:
;    clientword := serverword
black := zzzzz
white := black
zzzzz := white
[Server2Client]
; Set the translation of server response to the client here.
; Format:
;    serverword := clientword


So my guess is that

Code: Select all

white :=
black :=


in the Client2Server section is all you need. (The swapping actually was tricky, as there is apparently no guarantee in which order InBetween performs the substitutions.)

[Kirill Kryukov]

Thanks H.G. Yes, I tried it, and also I tried

Code: Select all

white := <delete>
black := <delete>

in client to server part. Both did not have any effect on time usage pattern.

[F. Bluemers]

Jim Ablett made several GNUchess compilations.
He might be able to fix it.