It's a plague of Winboard engines

[Kirill Kryukov]

I found one more engine with this bug: Laurifer 1.0. It is starting to play in my tourney. Unfortunately it looks like its performance will be severely affected by the bug. I added it to the list in the first post.

EDIT: It's a complete disaster. At move 49 when it realizes it has 5 seconds for the next 32 moves, it goes into panic mode and starts moving at depth 1. This means losing almost every game.

[Kirill Kryukov]

And here is another one: Requiem 0.53.

[Jonas Thiem]

My engine is going to have the same problem.

It's also a design issue though: my engine is only tracking "time" (not "otim") and the moves it did itself, not the ones for the opponent. Now force mode has it that the engine can't know which colour it is actually supposed to play and cannot properly track the moves. But as the CECP specs are pretty clear here I probably need to fix this...

Edit:

The engine should be able to issue a feature request something like

feature movecounts=1

(Or whatever name makes sense.) Of course, if an older WinBoard/xboard does not understand this feature request, it will reject it, and the engine can either decide to pout and die, or it can muddle along the best it can with the existing protocol. Otherwise, WinBoard will accept the request and subsequently send all "time" and "otim" commands in this format:

time <centiseconds> <remaining_moves_in_period>
otim <centiseconds> <remaining_moves_in_period>

This would actually be of great help with this!! Can't we have that feature please?? It removes the need to track both times but one can simply just track the own time and moves and as soon as force mode is entered and left again, that extended "time" command will resync the time and moves left again. This would actually be very helpful for me, I'd really want to have this feature.

[Kirill Kryukov]

Alright, here is one more engine with this very popular bug: Matant 5.04.

[Kirill Kryukov]

I found another engine that is a victim of the plague: Mustang 4.97. Another author fell into one of the many pits of the Winboard protocol. I hope that every author choosing to implement Winboard protocol will visit this forum and see this thread as a warning. (And at the very least avoid this particular bug).

(Note that I am not checking all available engines, but simply reporting the cases when I see this bug in my own very small tournament. The real number of engines with this bug is probably much higher.)

[Roger Brown]

I found another engine that is a victim of the plague: Mustang 4.97. Another author fell into one of the many pits of the Winboard protocol. I hope that every author choosing to implement Winboard protocol will visit this forum and see this thread as a warning. (And at the very least avoid this particular bug).

(Note that I am not checking all available engines, but simply reporting the cases when I see this bug in my own very small tournament. The real number of engines with this bug is probably much higher.)

Hello Kirill Kryukov,

I will uncomfortably overlook the Winboard protocol commentary as I am not sufficiently knowledgeable to address it.

I am curious about the explicit inferiority that you seem to be attributing to the Winboard protocol.

What are the sources of this inferiority and/or what are the sources of the superiority of the UCI protocol in your estimation?

What could the protocol or the gui do to encourage you to promote the Winboard protocol to all corners of the Earth?

I am well aware that you are a serious person so this is not a frivolous query at all.

Later.

[Kirill Kryukov]

Hello Kirill Kryukov,

I will uncomfortably overlook the Winboard protocol commentary as I am not sufficiently knowledgeable to address it.

I am curious about the explicit inferiority that you seem to be attributing to the Winboard protocol.

What are the sources of this inferiority and/or what are the sources of the superiority of the UCI protocol in your estimation?

What could the protocol or the gui do to encourage you to promote the Winboard protocol to all corners of the Earth?

I am well aware that you are a serious person so this is not a frivolous query at all.

Later.

Hi Roger,

I'm feeling free to state explicitly that I consider Winboard protocol inferior to UCI. The main source of the inferiority, in my view, is that Winboard protocol is stateful, while UCI is stateless. (Actually, I am not sure it is intrinsic quality of each protocol, however this is certainly the most common usage pattern of each.) The responsibility to maintain the game state is on the engine side in case of Winboard. In UCI, the engine receives a fresh new state on every move. The task of maintaining the game state looks trivial to someone writing a chess engine, and indeed is much simpler than the chess-playing logic. However, as this thread shows, there are some traps in maintaining an accurate state.

The real tragedy is that on the first glance Winboard looks easier to implement. Just send the moves, and receive the moves, what can be simpler? Compared to parsing the whole game every move in UCI. So, many authors choose Winboard, and have bugs that can be undiscovered for years.

The particular bug this thread is about is already found in 9 different engines. Of course, one remote possibility is that everyone copied from GNU Chess, but I find it hard to believe. More likely scenario is that every author re-implemented this bug independently. I think this trap justifies a big bold red warning in the protocol specs.

What could the protocol or the gui do to encourage you to promote the Winboard protocol to all corners of the Earth?

The protocol has to be stateless, in my opinion. Someone may say that Winboard allows in fact stateless operation, but that's not enough. The ideal chess protocol, IMHO, should clearly disallow any stateful mode. Perhaps this can be achieved in the Winboard GUI, without modifying the protocol, I am not sure.

I can quote my tournament KCEC (stable link / live) as at least some support to my preference for UCI: Currently it has 148 participants, including 60 UCI and 84 Winboard engines. Among them there are 42 engines that I consider buggy, unstable or incomplete - 10 UCI and 32 Winboard engines. This gives the bug ratio of 10/60 = 16.7% for UCI, and 32/84 = 38.1% for Winboard. (This is even before adding Mustang). Thus, an author choosing Winboard has about 2.3 times greater chance to end up with a buggy engine, compared to if he has chosen UCI instead. This is statistics from my own tournament, perhaps it does not represent the whole field well, but it's important for me. Of course the numbers will change after I add more engines. (For example, after I'll add Mustang). In this tournament I am testing some engines that I consider to be of historical interest, without any bias for protocol or stability (I am trying to test buggy engines too, as long as they run at all).

All above is just to clarify my point of view. Those who still prefer Winboard protocol are free to believe what they like.

[H.G.Muller]

I am very skeptical to your 'statistics' on bugginess. What kind of bugs are you talking about? Are they really protocol related? (And how do you determine that?) If older engines tend to have more bugs, that would strongly bias your observation, asUCI does not exist as long as WB protocol. In my experience it is mostly UCI engines that cause protocol-related stability problems, and Polyglot contains a number of work-around options (like sync-stop) to be able to run such engines.

In Xiangqi and Shogi the situation is even worse for the various UCI dialects. The UCI / UCCI / USI engines are an extremely non-compliant and buggy bunch, while the WinBoard engines in general work flawlessly. Send "go movestogo 40 wtime 60000 btime 60000" to PetitShogi, and it crashes unconditionally. Send "position startpos moves e3e4" to Cyclone, and it starts playing from the initial position. Send the time in msec to Jiaolong, which says "option usemillisec default true", and it thinks 1000x longer than you intended.

You are free toprefer what you like, of course, but the reasons you give just seem plain nonsense. The bug discussed here has nothing to do with staelessness, and tiny changes to both UCI and WB protocol, not affecting the statelessness in any way, would make that UCI engines had this problem, rather than WB engines. Not that I would encourage such a change: I think it is quite reasonable to require engines to correctly update a move counter. UCI engines must count reversible moves to know the value of the 50-move counter in the loaded position. If they mess that up, they will make 50-move-draw-related blunders. Is UCI therefore a poor protocol, and would it be a good protocol if every go command was also required to contain a 'reversiblemovestogo' keyword?

[Roger Brown]

Hi Roger,

I'm feeling free to state explicitly that I consider Winboard protocol inferior to UCI.

Hello Kirill,

No only are you free to say this, I would positively encourage comments like these here so that they can be examined and possible ideas generated. Please, do not wait on an invitation in the future. A number of persons will say X is better than Y when what they mean is that they cannot be bothered to use any gui but _________ . As I said, I know that your comments would be nothing if not well thought out.

The main source of the inferiority, in my view, is that Winboard protocol is stateful, while UCI is stateless. (Actually, I am not sure it is intrinsic quality of each protocol, however this is certainly the most common usage pattern of each.) The responsibility to maintain the game state is on the engine side in case of Winboard. In UCI, the engine receives a fresh new state on every move. The task of maintaining the game state looks trivial to someone writing a chess engine, and indeed is much simpler than the chess-playing logic. However, as this thread shows, there are some traps in maintaining an accurate state.

We are probably going to depart here. I believe the essential difference between UCI and Winboard protocols is a design philosophy. Do you regard the chess playing entity as the engine or the engine plus the gui? Winboard places the responsibility for playing chess squarely on the back of the engine.

The real tragedy is that on the first glance Winboard looks easier to implement. Just send the moves, and receive the moves, what can be simpler? Compared to parsing the whole game every move in UCI. So, many authors choose Winboard, and have bugs that can be undiscovered for years.

The particular bug this thread is about is already found in 9 different engines. Of course, one remote possibility is that everyone copied from GNU Chess, but I find it hard to believe. More likely scenario is that every author re-implemented this bug independently. I think this trap justifies a big bold red warning in the protocol specs.

I am not qualified to comment here, sorry. This needs H.G.

The protocol has to be stateless, in my opinion. Someone may say that Winboard allows in fact stateless operation, but that's not enough. The ideal chess protocol, IMHO, should clearly disallow any stateful mode. Perhaps this can be achieved in the Winboard GUI, without modifying the protocol, I am not sure.

Is it the treatment of the game state that is directly responsible for the bug you are observing? What I mean is, are you certain that if statelessness was to be implemented that the bug would disappear? I am not questioning the fact of your observations or your statistics but the cause of the effect you are observing.

[SNIP]

All above is just to clarify my point of view. Those who still prefer Winboard protocol are free to believe what they like.

No need for the defensive insert. As I said, comments are welcome here. Critical comments can only assist Winboard to go from strength to strength. I mean, there was a time the gui did not support UCI engines. Now, with the transparent use of that piece of genius called Polyglot, it can.

Who knows what the future may bring.

Later.

[H.G.Muller]

This bug is the inability of engines to properly count the number of moves that have been played in the game. Nothing more, nothing less.

It is true that the move counter is part of the game state, and that the UCI 'go' command happens to 'remind' the engine what the new value of this counter is. But this is just a design detail that has nothing to do with statelessness. There are many aspects of the game state that the engine is not reminded of in UCI. I already mentioned the value of the 50-move counter. An even more obvious one is the current position. UCI does _not_ remind the engine of the current position before every move. The design choice was to let the engine _construct_ the current position from the start position and the move list (and derive the current value of the 50-move counter of it as well). There is every room for the engine to bebuggy there (e.g. not recognize e.p. captures or perform castling or promotion wrong), as decoding the moves and perfofrming them is orders of magnitude more difficult than simply counting them.

The only difference with WB protocol in this respect is that the latter is designed to give the movestogo in advance, together with the initial position, in stead of after the list of moves. This is objectively speaking a _superior_ approach, as the initial 'movestogo' comes in the form of a 'level' command, which implies a promise for the next session. For instance, suppose you are playing 25 moves/50min, and you happened to stay in book until move 24. So now you have the full 50 min still on your clock, and 1 move to go. But after that, you will have to play 25 moves in 50 min. So what would be smarter:
1) think 50min about move 25, and then about 2 min on move 26-50 each?
2) think about 6 min on move 25-30 each, and then about 4 min on move 31-42 each, and then about 2 min on 43-50 each.
3) think about 4 min on move 25-50 each?
It turns out that (2) is better than (3) is _much_ better than (1), and a properly tuned WB engine will go for (2).
But a UCI engine of course has no choice but to do (1), because it has no idea how long the total session was, and thus how long the next session will be, or how many moves there would be in there. So UCI sucks in this respect. Just like UCI would have sucked if it would really have sent the current position (as FEN) rather than the moves leading up to it from the standard position: this would frustrate proper rep-draw detection, just as sending the current movestogo in stead of the initial moves-per-session (and letting the engine count). It is quite inconsistent that UCI design chose to put an appreciable burdon on the engine (performing all the moves) to avoid the rep-draw problem, while refraining from putting the much smaller burdon of counting the moves on the engine to allow proper time management.

So no matter what philosophy you prefer as a matter of taste, it is an objective fact that the way WinBoard protocol handles the move-counter issue is superior. Or, more accurately, that the UCI way of handling it is fundamentally _broken_, and not really acceptable. That counting moves proves too difficult a task for some engine authors is unfortunate, but cannot be held against the protocol. Obviously a more powerful protocol allowing you to do more, will also allow you to make new kinds of errors. But you cannot conclude from that that restricting the capabilities of a protocol is a good thing. I'd rather have a laptop than a pocket calculator, despite the fact that C programs can get trapped in infinite loops, and a pocket calculator can not. Simplicity is only a decisive factor between things of equal capability, but capability is the primary sort key. For me, at least.

[Kirill Kryukov]

Hi H.G.,

Thanks for joining, I always appreciate your insight.

I am very skeptical to your 'statistics' on bugginess. What kind of bugs are you talking about? Are they really protocol related? (And how do you determine that?)

My 'bugginess stats' is just my personal observaion, and I don't claim it to be anything more than that. Others may (and most probably will) observe a different bug ratio. I count all bugs that disrupt smooth testing, including time losses, crashes, illegal moves. (I don't count blunders, faulures to win a kqqk, etc). So, no, not all of them are protocol related. I also realize that correlation does not imply causation. Still my own observation is the primary factor that will determine my protocol preference.

If older engines tend to have more bugs, that would strongly bias your observation, asUCI does not exist as long as WB protocol.

I don't know if older engines have more bugs or not. I feel that I don't have enough data (in my tournament) to make judgement. I'd be curious if someone published any such statistics. (The fact that UCI was introduced later, but still gained significant popularity, indirectly supports the idea that it has at least some techincal advantage).

In my experience it is mostly UCI engines that cause protocol-related stability problems, and Polyglot contains a number of work-around options (like sync-stop) to be able to run such engines.

As far as I understand, Winboard itself tolerates a lot of non-compliant input from engines. WB2UCI has a number of work-arounds too, for buggy Winboard engines. Anyway, your and mine experiences are bound to differ. I wish I had more time to invest into documenting protocol issues, notifying the authors. A lot remains to be done in this area. Ideally we would create some ACID test for engines to pass, to enable fast testing of protocol compliance.

In Xiangqi and Shogi the situation is even worse for the various UCI dialects. The UCI / UCCI / USI engines are an extremely non-compliant and buggy bunch, while the WinBoard engines in general work flawlessly. Send "go movestogo 40 wtime 60000 btime 60000" to PetitShogi, and it crashes unconditionally. Send "position startpos moves e3e4" to Cyclone, and it starts playing from the initial position. Send the time in msec to Jiaolong, which says "option usemillisec default true", and it thinks 1000x longer than you intended.

Actually this is surprising to me, because Xiangqi and Shogi are relatively recent additions to "computer board games" scene, so I would imagine it would be done right from the start with them.

You are free toprefer what you like, of course, but the reasons you give just seem plain nonsense.

I believe my tournament is a legitimate use case for a chess engine. I'm disappointed with your rudeness.

The bug discussed here has nothing to do with staelessness, and tiny changes to both UCI and WB protocol, not affecting the statelessness in any way, would make that UCI engines had this problem, rather than WB engines.

In case of UCI such problems are easier to detect, and therefore they get fixed sooner. If an engine can't parse the input correctly, it won't play properly under any conditions. In Winboard, on the other hand, a bug like this can appear only in some tournament conditions. UCI control flow is much simpler than in case of Winboard. This particular bug shows up only when an engine is playing with an external book under N/M time control. If a UCI engine did not update the move counter, the bug would show up under much wider set of conditions.

Not that I would encourage such a change: I think it is quite reasonable to require engines to correctly update a move counter. UCI engines must count reversible moves to know the value of the 50-move counter in the loaded position. If they mess that up, they will make 50-move-draw-related blunders. Is UCI therefore a poor protocol, and would it be a good protocol if every go command was also required to contain a 'reversiblemovestogo' keyword?

A 50-move draw blunder is much less serious problem (IMHO), than being left with 5 seconds for the remaining 32 moves and subsequently lose on time every other game.

I know we discussed this before, and that in most cases the older engines can't be fixed. I am curious though. When someone writes a new engine, is there any kind of warning about the possibility of this bug in the recent Winboard protocol specs?

[Kirill Kryukov]

Hi Roger,

I appreciate your constructive approach to the discussion.

We are probably going to depart here. I believe the essential difference between UCI and Winboard protocols is a design philosophy. Do you regard the chess playing entity as the engine or the engine plus the gui? Winboard places the responsibility for playing chess squarely on the back of the engine.

People often use engines as analysis tools. I am curious how well different engines can perform as such tool. Therefore I include both UCI and Winboard engines into comparison.

Is it the treatment of the game state that is directly responsible for the bug you are observing? What I mean is, are you certain that if statelessness was to be implemented that the bug would disappear? I am not questioning the fact of your observations or your statistics but the cause of the effect you are observing.

This particular bug is well understood now. Please see this thread for detailed discussion. Yes, this bug can happen in stateful mode too, but there it would be noticed and fixed immediately.

[Kirill Kryukov]

H.G., thanks for the detailed comparison. Yes, Winboard way of setting time control looks superior to UCI. It's unfortunate that we can't have the best from both protocols in one.

(skipped)

So no matter what philosophy you prefer as a matter of taste, it is an objective fact that the way WinBoard protocol handles the move-counter issue is superior. Or, more accurately, that the UCI way of handling it is fundamentally _broken_, and not really acceptable. That counting moves proves too difficult a task for some engine authors is unfortunate, but cannot be held against the protocol. Obviously a more powerful protocol allowing you to do more, will also allow you to make new kinds of errors. But you cannot conclude from that that restricting the capabilities of a protocol is a good thing. I'd rather have a laptop than a pocket calculator, despite the fact that C programs can get trapped in infinite loops, and a pocket calculator can not. Simplicity is only a decisive factor between things of equal capability, but capability is the primary sort key. For me, at least.

I think it's not that simple (just difficult task to count the moves). With Winboard approach the author has to imagine many different scenarios that may happen. With UCI, there is only one scenario, basically. Yes, in UCI you have to parse the game, but almost any bug there will be quickly noticed (and fixed). Winboard protocol may be more powerful, but humans write engines. The unintended consequence of choosing a more powerful protocol may be that in real life testing the engine underperforms by hundreds of points. Again, it's OK if the author is active and can fix the engine, but most of engines I am using are abandoned. I guess I should consider myself unfortunate.

[H.G.Muller]

So, no, not all of them are protocol related. I also realize that correlation does not imply causation. Still my own observation is the primary factor that will determine my protocol preference.

That sounds very contradictory. Why would you allow bugs that are not protocol-related to contaminate your preference for a protocol? Not all time losses are bugs, some (many) engines simply are mis-tuned use too much time for the last move of a session. Both UCI and WB send the time for the last move just before the engine starts thinking on this move, so I don't see any room for protocol influence there. Crashes and illegal moves are almost never protocol related, but due to search bugs (not propery undoing moves, lack of resistance to illegal hash moves resulting from hash-key collisions).

As far as I understand, Winboard itself tolerates a lot of non-compliant input from engines. WB2UCI has a number of work-arounds too, for buggy Winboard engines. Anyway, your and mine experiences are bound to differ. I wish I had more time to invest into documenting protocol issues, notifying the authors. A lot remains to be done in this area. Ideally we would create some ACID test for engines to pass, to enable fast testing of protocol compliance.

Yes, this is true. Much to my chagrin, I can add. I think catering fornon-compliancy in the long run just makes things worse. It is like the speed limit in traffic. No matter what you make it, people will exceed it. So relaxing standards will just invite chaos.

Actually this is surprising to me, because Xiangqi and Shogi are relatively recent additions to "computer board games" scene, so I would imagine it would be done right from the start with them.

The problem is that the people making these engines often do not undrstand English very well. In the case of UCCI they almost all used the open-source protocol driver of Elephant eye (which is the reference implementation of UCCI), without really understanding the protocol. Another frequent source is adapting to a flakey GUI implementation.

I believe my tournament is a legitimate use case for a chess engine. I'm disappointed with your rudeness.

It was not my intenstion to be rude, I am sorry you perceive it as such. I just intended to be clear. The reasoning you presented was just not sound, because you indeed presented the case like the higher fraction of buggy engines amongst your WinBoard participants could be ascribed to the fact that they were using WB protocol, while (as we now established) you count non-protocol-related bugs as well.

In case of UCI such problems are easier to detect, and therefore they get fixed sooner. If an engine can't parse the input correctly, it won't play properly under any conditions. In Winboard, on the other hand, a bug like this can appear only in some tournament conditions. UCI control flow is much simpler than in case of Winboard. This particular bug shows up only when an engine is playing with an external book under N/M time control. If a UCI engine did not update the move counter, the bug would show up under much wider set of conditions.

This might happen to be true for this particular bug (because in a sense an UCI engine always plays with an external book, on every move), although even there engines sometimes have move bugs that only manifest themselves in rare conditions (e.g. underpromotions,check-detection after e.p. capture). It is not true that all UCI keywords are allways used in any game (although 'position' and 'go' obviously are). If 'go movestogo' works, it does no timply that 'winc' and 'binc' will work, that 'go infinite' will work, etc. Many UCI Xiangqi engines suffer from the problem that they crash after 128 moves, because their input buffer for the position overflows. Turns out the most popular GUI there always sends the FEN of the positon after the last irreversible move, so that the 50-move rule limits the length to always fit the buffer. This is something you would not notice so easily in UCI, and cannot hapen in WB protocol. Another example: engines are usually also hardly tested while playing from asetup postion. UCI engines are just as prone to undetected bugs in their FEN readers as WB engines.

A 50-move draw blunder is much less serious problem (IMHO), than being left with 5 seconds for the remaining 32 moves and subsequently lose on time every other game.

In principle 5 sec for 32 moves is not an excuse to lose on time for an engine. But it will not be good for the level of play, of course. So I don't deny this is a bad bug. I just blame the engine for it, not the protocol.

I know we discussed this before, and that in most cases the older engines can't be fixed. I am curious though. When someone writes a new engine, is there any kind of warning about the possibility of this bug in the recent Winboard protocol specs?

Unfortunately I haven't updated the specs for some time. I admit there is some very unclear talk there, on "clocks staying with the engine/user", which might very well be responsible for the high occurrence of this bug. I only recently understood what was meant by it: when the engine plays black (as it will do after 'new'), and receives 'time' followed by 'go', the 'time' command actually contained the time for white, although it was received while the engine was set to play black. I guess it would be more clear to express this as: "time specifies the time on the clock of the side the engine is going to move for". And of course I can explicitly state that "all moves have to be counted, no matter who plays them".

[Kirill Kryukov]

So, no, not all of them are protocol related. I also realize that correlation does not imply causation. Still my own observation is the primary factor that will determine my protocol preference.

That sounds very contradictory. Why would you allow bugs that are not protocol-related to contaminate your preference for a protocol? Not all time losses are bugs, some (many) engines simply are mis-tuned use too much time for the last move of a session. Both UCI and WB send the time for the last move just before the engine starts thinking on this move, so I don't see any room for protocol influence there. Crashes and illegal moves are almost never protocol related, but due to search bugs (not propery undoing moves, lack of resistance to illegal hash moves resulting from hash-key collisions).

OK, this deserves some more explanations, which I thought was not necessary.

I observe that Winboard engines give me troubles 2.3 times more often than UCI engines. Many different mechanisms and factors are contributing to the end outcome of some engines being buggy. Some mechanisms are clearly protocol related (for example, correctly updating the move counter with Winboard protocol needs much more careful and defensive thinking when designing an engine, compared to UCI). Some other mechanisms can be hidden and non-trivial. For example, people with particular types of personality may be more prone to choose a particular protocol, and the same personality may lead them to have more bugs. Or, engines produced within the same sub-community will depend on shared testing practice, which tends to leave some bugs undetected. Who knows. Many non-trivial ways and situations may combine to create severe bugs that are left undetected.

The complex history of computer chess ecosystem created a situation where Winboard protocol usage correlates with bugginess (under my testing conditions). I have no reason to believe that it will be any different for the future engines (those that I did not test yet). Yes, the mechanisms producing those bugs are often not undestood, but the same mechanisms may be still functioning on new engines created today. So I expect the Winboard/UCI bug ratio to remain about the same as I add new engines to my tournament.

Of course, there may be some correlation between release year and bugginess. This is yet to be seen.

I believe my tournament is a legitimate use case for a chess engine. I'm disappointed with your rudeness.

It was not my intenstion to be rude, I am sorry you perceive it as such. I just intended to be clear. The reasoning you presented was just not sound, because you indeed presented the case like the higher fraction of buggy engines amongst your WinBoard participants could be ascribed to the fact that they were using WB protocol, while (as we now established) you count non-protocol-related bugs as well.

There may be many non-trivial ways for the protocol choice to affect the bugginess. Only some ways are understood (like in case of the bug this thead is about).

Actually, it's quite possible, that if I substract this single bug, the remaining bugs will produce insignificant difference between Winboard and UCI. I did not check, because the total number of engines I use is probably too small for serious discussion.

Perhaps I should stress once again, that I quoted the bugginess stats from my tournament only to illustrate my personal protocol preference. It's specific to my testing conditions, and to my arbitrary choice of engines. I'm not trying to present it as the universal rule in computer chess.

In case of UCI such problems are easier to detect, and therefore they get fixed sooner. If an engine can't parse the input correctly, it won't play properly under any conditions. In Winboard, on the other hand, a bug like this can appear only in some tournament conditions. UCI control flow is much simpler than in case of Winboard. This particular bug shows up only when an engine is playing with an external book under N/M time control. If a UCI engine did not update the move counter, the bug would show up under much wider set of conditions.

This might happen to be true for this particular bug (because in a sense an UCI engine always plays with an external book, on every move), although even there engines sometimes have move bugs that only manifest themselves in rare conditions (e.g. underpromotions,check-detection after e.p. capture). It is not true that all UCI keywords are allways used in any game (although 'position' and 'go' obviously are). If 'go movestogo' works, it does no timply that 'winc' and 'binc' will work, that 'go infinite' will work, etc. Many UCI Xiangqi engines suffer from the problem that they crash after 128 moves, because their input buffer for the position overflows. Turns out the most popular GUI there always sends the FEN of the positon after the last irreversible move, so that the 50-move rule limits the length to always fit the buffer. This is something you would not notice so easily in UCI, and cannot hapen in WB protocol. Another example: engines are usually also hardly tested while playing from asetup postion. UCI engines are just as prone to undetected bugs in their FEN readers as WB engines.

Yes, there are many dark corners in UCI too. This only highlights the necessity to have a protocol compliance certification. Some set of tests, which will check that the engine behaves well on many special cases. This community should have developed such certification years ago, but I guess everyone is busy working on their own projects.

[Roger Brown]

[SNIP]

Yes, there are many dark corners in UCI too. This only highlights the necessity to have a protocol compliance certification. Some set of tests, which will check that the engine behaves well on many special cases. This community should have developed such certification years ago, but I guess everyone is busy working on their own projects.

Hello Kirill,

What you call a community is really a loosely knit group of hobbyists and enthusiasts who has an interest in writing chess engines and mostly giving them away - for free. So no-one was thinking of developing a unified engine theory really because there was no unified field.....

Then you have the gui writers - who are often themselves writers of chess engines - who are trying desperately to ensure that compatibility with previous, ah, legacy engines (W1) is maintained along with later developed W2, UCI, UCCI and other protocols. Then too, the protocol development has been marked by attempts by part-timers (H.G., Alessandro et al) to hammer away at parts in an improvement and cleanup drive.

I would have thought that as UCI is a newer (cleaner) protocol that all the dark corners would have been swept clean and lit.

It does not sound easy but the certification idea is certainly a sensible one. Do you have any ideas of what should be in it? Of course that is a natural question!



I take it then that an engine will be sent a stream of commands and based on the responses a user or tester or chess engine author would know where a potential - if not actual - protocol issue might reside.

Please contribute any ideas which you might have. Once the thing can be articulated, coding it becomes possible.

So speaks a non-coder!

Later.

[Kirill Kryukov]

It does not sound easy but the certification idea is certainly a sensible one. Do you have any ideas of what should be in it? Of course that is a natural question!



I take it then that an engine will be sent a stream of commands and based on the responses a user or tester or chess engine author would know where a potential - if not actual - protocol issue might reside.

Hi Roger,

I was imagening something resembling a epd test suite, just more general. A set of pairs "input - output". Output should be described as some sort of regular expression, to allow for some flexibility in engine output. Also, both input and output may contain timing data, not only the text to send and receive. Also "output" should include various other behaviour details, like "the engine allocated 1 GB of RAM", or "the engine is pondering using 2 cores". Timings in output should be given as acceptable range rather than exact point of time. To support all this the GUI should probably depend on some external tools (like 'ps' in Linux, and some sysinternals tools in Windows). Yes, the GUI is the best place to put this all in, because GUI already interacts with engines, so it's a good starting point. I realize that this probably a huge work to implement all this, but well, I can dream.

When there is a GUI that supports testing an engine against this kind of test suite, users (and authors) will gladly assemble such test suites, both for specific bug or specific engine, as well as general protocol compliance suite.

[Kirill Kryukov]

I found another victim of the Winboard plague: Grizzly 1.40.1b.

[Yannik Snoeckx]

Hello !

I have just thought about a workaround for this problem. It's quite ugly and not for all situations, but could be better than let those poor engines play 30 moves in 1 second.

If you start the game from a set of positions or a book always ending in a certain number of moves, I know many that sort exist, why not give the time control with the same number of moves.

For example, you have a set of positions all ending in 10 moves. Then choose a TC of 10/1 instead of 40/4 and you should trick the engine in using its time right since the 10 moves it forgets are then irrelevant.

What do you think about that ?

Best wishes.
YS

[Kirill Kryukov]

Yeah, probably this could work. I use 8 move books, so instead of 40/4 that I use normally I would use 8 moves / 48 sec TC. One problem is my GUI does not support TC set in seconds. Also I'm not sure I want to mix these results with 40/4 results in the same database. I guess there won't be too much difference in engine strenghts, still it's a different TC with different emphasis on time management.

More importantly, I'm not sure if I want to leave this horrible bug unpunished. And, if some author fixes this bug, he deserves a rating jump in my list.