Input thread unde polyglot

[Peter Fendrich]

I have this strange error that appears under polyglot but not Arena UCI.
I have an Input thread that very simplified looks like this:

Code: Select all

char sCommand[2048];
while (!bQuit){   // until we leave
    sCommand[0]='\0';
    rc=fgets(c,2047,stdin);
    bInputAvailable=true;
    WaitForSingleObject(hGetInputEvent,INFINITE);
}


For example if I run "Analyze" from Arena this happens
During the search this thread is blocked at fgets(...) waiting for input.
(I have checked that)
When I stop Analyze different things happens in Arena UCI and polyglot under Arena or Winboard.
1) Under Arena UCI 'stop' is read by the fgets and everything is ok.
2) Under Winboard/Polyglot or ArenaWB/Polyglot, polyglot sends the 'stop' command (due to the polyglot log) but this time the fgets doesn't kick in.

I'm completely lost in this
It only happens under Polyglot but it is of course something stupid I made myself but what is it.....?
BTW. the stdin is unbuffered.
/Peter

[Anonymous]

I have this strange error that appears under polyglot but not Arena UCI.
I have an Input thread that very simplified looks like this:

Code: Select all

char sCommand[2048];
while (!bQuit){   // until we leave
    sCommand[0]='\0';
    rc=fgets(c,2047,stdin);
    bInputAvailable=true;
    WaitForSingleObject(hGetInputEvent,INFINITE);
}


I don't know much about the multithreading approach. Proabably a typo, when you simplified and posted the code, but might be a bug:

instead of
rc=fgets(c,2047,stdin);

probaby you meant

rc=fgets(sCommand,2047,stdin);

BTW. you can use 2048 instead of 2047 here, or even better

rc=fgets(sCommand,sizeof sCommand,stdin);

Why do you initialize sCommand[0]? Just checking rc should be sufficient and better (I assume you left this out, to make the posted code snippet shorter).

A slightly related question: Doesn't work it (under Arena) without setting stdin to unbuffered?

For really long games your buffer will be too small. 2048 is just sufficient for about 400 half moves in the UCI protocol.

Cheers,
Dieter

[Peter Fendrich]

I have this strange error that appears under polyglot but not Arena UCI.
I have an Input thread that very simplified looks like this:

Code: Select all

char sCommand[2048];
while (!bQuit){   // until we leave
    sCommand[0]='\0';
    rc=fgets(c,2047,stdin);
    bInputAvailable=true;
    WaitForSingleObject(hGetInputEvent,INFINITE);
}


I don't know much about the multithreading approach. Proabably a typo, when you simplified and posted the code, but might be a bug:

instead of
rc=fgets(c,2047,stdin);

probaby you meant

rc=fgets(sCommand,2047,stdin);

BTW. you can use 2048 instead of 2047 here, or even better

rc=fgets(sCommand,sizeof sCommand,stdin);

Why do you initialize sCommand[0]? Just checking rc should be sufficient and better (I assume you left this out, to make the posted code snippet shorter).

Yes, you're right it is very simplified in the first place and it was a typo.
In fact sCommand is declared as string and I have an inner loop filling sCommand with sCommand.append(c) until a newline is there.
I'm quite sure that this isn't the problem because there is no loop needed in my example. I can see (by printing text before and after fgets) that it actually is standing there waiting for the fgets to run. I just can't see the difference between Arena and polyglot when both are sending the 'stop'.

A slightly related question: Doesn't work it (under Arena) without setting stdin to unbuffered?

I've never tried really. I've always used unbuffered stdin and stdout...

Thank you for your answer and good luck with the new 'I just forgot the name'!
/Peter

[Anonymous]

Another idea. Try to put Odd Gunnar's inbetween between you engine and polyglot, and investigate inbetween's log files (is stop there?).

Could it be a line end issue? (lf vs. cr+lf?). No idea, if polyglot or inbetween will log exactly enough, to see the actual line ends. Putting "tee" into the pipe (perhaps from the cygwin tools) might help.

Perhaps you should try the fgets() approach instead of the C++ string, to narrow the problem (but this might be too much work).

Good luck,
Dieter

PS. I started from scratch (not using a single line of code of Yace) for a new engine called Bursche. The name was suggested by Gabor. Move generator and some other "infrastructure" is finished - but I did not touch it anymore since quite a while.

[Peter Fendrich]

Another idea. Try to put Odd Gunnar's inbetween between you engine and polyglot, and investigate inbetween's log files (is stop there?).

Good idea!
I actually did that. I get the same error with analyze so I tested that.
I started analyze from the start position and stopped it after 10 seconds.
Arena's log tell me that 'exit' followed by 'force' is sent to polyglot.
Polyglots log never says anything about receiving exit or'force'
Finally InBetween's log tell me that no 'stop' is sent to Alaric from polyglot.

Does Arena send 'exit' and 'force' to polyglot? Yes!
I set InBetween between Arena and polyglot and Arena sent the two commands.
It seems to me that polyglot doesn't react on 'exit'.

/Peter

[Dann Corbit]

Another idea. Try to put Odd Gunnar's inbetween between you engine and polyglot, and investigate inbetween's log files (is stop there?).

Good idea!
I actually did that. I get the same error with analyze so I tested that.
I started analyze from the start position and stopped it after 10 seconds.
Arena's log tell me that 'exit' followed by 'force' is sent to polyglot.
Polyglots log never says anything about receiving exit or'force'
Finally InBetween's log tell me that no 'stop' is sent to Alaric from polyglot.

Does Arena send 'exit' and 'force' to polyglot? Yes!
I set InBetween between Arena and polyglot and Arena sent the two commands.
It seems to me that polyglot doesn't react on 'exit'.

/Peter

Here is what we see in Polyglot's reaction to exit in adapter.cpp and in xboard.cpp

xboard.cpp:

Code: Select all

   } else if (match(string,"exit")) {

      Computer[White] = false;
      Computer[Black] = false;

      Analyse = false;

      mess();



adapter.cpp:

Code: Select all

   } else if (match(string,"exit")) {

      Computer[White] = false;
      Computer[Black] = false;

      Analyse = false;

      mess();

/*
      if (option_get_int("UCIVersion") >= 2) {
         engine_send(Engine,"setoption name UCI_AnalyseMode value false");
      }
*/



[Peter Fendrich]

Here is what we see in Polyglot's reaction to exit in adapter.cpp and in xboard.cpp

xboard.cpp:

Code: Select all

   } else if (match(string,"exit")) {

      Computer[White] = false;
      Computer[Black] = false;

      Analyse = false;

      mess();



adapter.cpp:

Code: Select all

   } else if (match(string,"exit")) {

      Computer[White] = false;
      Computer[Black] = false;

      Analyse = false;

      mess();

/*
      if (option_get_int("UCIVersion") >= 2) {
         engine_send(Engine,"setoption name UCI_AnalyseMode value false");
      }
*/



I don't find the xboard.cpp in my the polyglot source (version 1.3).
Anyway the code in adapter.cpp looks allright. The mess() function does the stop_search() function that looks like:

Code: Select all

static void stop_search() {
   if (Uci->searching) {
      ASSERT(Uci->searching);
      ASSERT(Uci->pending_nb>=1);
      my_log("POLYGLOT STOP SEARCH\n");
      engine_send(Engine,"stop");
      Uci->searching = false;
   }
}

IIRC the 'POLYGLOT STOP SEARCH' is really appearing in the polyglot log but I can't find the "stop" (and neither can InBetween) that the engine should get.
Another thing, not importantant but the ASSERT(Uci->searching) looks strange to me! First "if (Uci->Searching)" and then look if it really was Uci->Searching...
Now, don't think that I ever done illogical things like that myself
/Peter

[Peter Fendrich]

I made some more tests both with Fruit and Alaric under polyglot, using InBetween to see what's going on between polyglot and the engine.
The two engines reacts a little bit different but the observation is the same.
It seems to be some kind of synchronisation error between polyglot and Arena (the same behaviour under Winboard). I sometimes can see the 'stop' command coming from polyglot to the engine delayed a couple of seconds or in some cases much more. It is easy to observe the phenomena with analyze but I have similar problem with pondering. The command to the engine is delayed and it is not always obvious during pondeing in longer games.

Is this only in my environment? Can someone verify this?
Just start analyze and stop it after 3 seconds and see if the engine also stops at the same time or if it is delayed some seconds.
/Peter

[Roman Hartmann]

Hi Peter,
I have some problems with Polyglot as well. Sometimes my engine will get stuck when running under winboard/polyglot and lose on time. Happens when running the engine in ChessPartner as well. It doesn't hapen under Arena, Fritz or Shredder Classic though. As the problem doesn't occur that often I have some troubles to track that problem down.

Anyway, I added some logging features to my engine and hope to find the problem that way. The logs of polyglot indicate only that my engine sends EOF after bestmove. Unfortunately my first attempt to log what's going on wasn't that succesful as I forgot to fflush() the data to the logfiles. The problem also occurs only very seldom (1-2% of played games), so it's a very timeconsuming task to get rid of that bug.

log from polyglot (last output of engine)

Code: Select all

< ENGINE bestmove d4c3
< ENGINE EOF


I'm pretty sure the problem is on my side somewhere but I wonder why this problem doesn't occur under other GUI's as well.

Roman

[Peter Fendrich]

Hi Peter,
I have some problems with Polyglot as well. Sometimes my engine will get stuck when running under winboard/polyglot and lose on time. Happens when running the engine in ChessPartner as well. It doesn't hapen under Arena, Fritz or Shredder Classic though. As the problem doesn't occur that often I have some troubles to track that problem down.

Anyway, I added some logging features to my engine and hope to find the problem that way. The logs of polyglot indicate only that my engine sends EOF after bestmove. Unfortunately my first attempt to log what's going on wasn't that succesful as I forgot to fflush() the data to the logfiles. The problem also occurs only very seldom (1-2% of played games), so it's a very timeconsuming task to get rid of that bug.

log from polyglot (last output of engine)

Code: Select all

< ENGINE bestmove d4c3
< ENGINE EOF


I'm pretty sure the problem is on my side somewhere but I wonder why this problem doesn't occur under other GUI's as well.

Roman

Hi Roman,
It seems to me that it is some problem with your engine.
InBetween is a wonderful tool for such problems.
You place it between polyglot and your engine. It's easy. Just put InBetween.exe as engine in polyglot.ini and put your engine in Inbetween.ini.
You will get a log telling you what's going on between polyglot and your engine. Good luck!
http://home.online.no/~malin/sjakk/download/InBetween.zip
/Peter

[Roman Hartmann]

Hi Roman,
It seems to me that it is some problem with your engine.
InBetween is a wonderful tool for such problems.
You place it between polyglot and your engine. It's easy. Just put InBetween.exe as engine in polyglot.ini and put your engine in Inbetween.ini.
You will get a log telling you what's going on between polyglot and your engine. Good luck!
http://home.online.no/~malin/sjakk/download/InBetween.zip
/Peter

Hi Peter,
I also think the problem is somewhere on my side. I had winboard/polyglot running over night and at some point it got stuck. These time I even have the logs of polyglot and the results of my own logging approach. Seems the problem is related to pondering somehow.

polyglot-log:

Code: Select all

POLYGLOT FEN rnbqkbnr/pp1ppppp/8/8/3pP3/2P5/PP3PPP/RNBQKBNR b KQkq - 0 3
> ENGINE position startpos moves e2e4 c7c5 d2d4 c5d4 c2c3
> ENGINE go wtime 179920 btime 179930 ponder
< ENGINE info depth 1 nodes 23 nps 1 score cp 199 time 0 pv d4c3
< ENGINE info depth 2 nodes 251 nps 1 score cp 50 time 0 pv d4c3 b1c3
< ENGINE info depth 3 nodes 1638 nps 1 score cp 110 time 0 pv g8f6 c3d4 f6e4
< ENGINE info depth 4 nodes 16379 score cp 110 nps 545967 currmove d4c3
< ENGINE info depth 4 nodes 20050 nps 668333 score cp -25 time 30 pv d4c3 b1c3 e7e5 d1d7
< ENGINE info depth 5 nodes 102982 nps 735586 score cp 155 time 140 pv d4c3 b2c3 d8b6 b1d2 b6f2
< XBOARD time 17984
< XBOARD otim 17984
< XBOARD usermove g1f3
POLYGLOT MOVE Nf3
POLYGLOT PONDER -> THINK (miss)
POLYGLOT STOP SEARCH
> ENGINE stop
POLYGLOT START SEARCH
POLYGLOT FEN rnbqkbnr/pp1ppppp/8/8/3pP3/5N2/PPP2PPP/RNBQKB1R b KQkq - 1 3
> ENGINE position startpos moves e2e4 c7c5 d2d4 c5d4 g1f3
> ENGINE go wtime 179840 btime 179840
< ENGINE info depth 6 nodes 135320 nps 676600 score cp 87 time 200 pv h7h6 e4e5 d8a5 f2f4 d4d3 f4f5
< ENGINE bestmove d4c3
< XBOARD force
POLYGLOT STOP SEARCH
> ENGINE stop
POLYGLOT WAIT
< XBOARD ping 108
> XBOARD pong 108
< XBOARD quit
POLYGLOT *** "quit" from XBoard ***
POLYGLOT *** QUIT ***
> ENGINE quit
< ENGINE EOF


The following lines are particulary interesting. c2c3 is replaced by g1f3. Seems winboard/polyglot sent a ponder move to roce. My engine doesn't support pondering yet and that seems to cause the problem most probably.

Code: Select all

> ENGINE position startpos moves e2e4 c7c5 d2d4 c5d4 c2c3


...

Code: Select all

> ENGINE position startpos moves e2e4 c7c5 d2d4 c5d4 g1f3


My engine delivered the move for the first position while polyglot already had sent the moves for a different position. So I might have at least still a timing problem as the engine doesn't respond quickly enough. The 2nd setpos command did not reach my engine as it doesn't show up in the log file of roce.

log of roce:

Code: Select all

[GUI-COMMAND] ucinewgame
[GUI-COMMAND] position startpos moves e2e4 c7c5 d2d4 c5d4 c2c3

8   {r}{n}{b}{q}{k}{b}{n}{r}

7   {p}{p} . {p}{p}{p}{p}{p}

6    .  .  .  .  .  .  .  .

5    .  .  .  .  .  .  .  .

4    .  .  . {p}[P] .  .  .

3    .  . [P] .  .  .  .  .

2   [P][P] .  .  . [P][P][P]

1   [R][N][B][Q][K][B][N][R]

     a  b  c  d  e  f  g  h

[GUI-COMMAND] go wtime 179920 btime 179930 ponder
{additional infos}
Timesettings:
calctime: 3598
search_time: 0
search_time 2 (search-start): 3598
white time: 179920
black time: 179930
white inc time: 0
black inc time: 0
[SENT TO GUI] depth 1 nodes 23 nps 1 score cp 199 time 0 pv d4c3
{additional infos} infos alpha beta: alpha: 129, beta: 269
[SENT TO GUI] depth 2 nodes 251 nps 1 score cp 50 time 0 pv d4c3 b1c3
{additional infos} infos alpha beta: alpha: -20, beta: 120
[SENT TO GUI] depth 3 nodes 1638 nps 1 score cp 110 time 0 pv g8f6 c3d4 f6e4
{additional infos} infos alpha beta: alpha: 40, beta: 180
[SENT TO GUI] depth 4 nodes 20050 nps 668333 score cp -25 time 30 pv d4c3 b1c3 e7e5 d1d7
{additional infos} infos alpha beta: alpha: -95, beta: 45
[SENT TO GUI] depth 5 nodes 102982 nps 735586 score cp 155 time 140 pv d4c3 b2c3 d8b6 b1d2 b6f2
{additional infos} infos alpha beta: alpha: 85, beta: 225
[SENT TO GUI] depth 6 nodes 135320 nps 676600 score cp 87 time 200 pv h7h6 e4e5 d8a5 f2f4 d4d3 f4f5
{additional infos} infos alpha beta: alpha: 17, beta: 157
[SENT TO GUI] bestmove d4c3


Meanwhile I also setup the inbetween.exe and started a batch of games. Now I have to wait till the bug shows up again and if it the bug is looking similar to this one.
Maybe inbetween.exe provides some additional infos which might give me some insights whats going wrong here. But I guess the problem is related to pondering.

I still Don't understand why Winboard/Polyglot is sending ponder moves when the engines run on the same cpu though...
I found 3 other ponder commands in the polyglot.log file (around 80 games were played and all of them logged).

Roman

[Peter Fendrich]

The following lines are particulary interesting. c2c3 is replaced by g1f3. Seems winboard/polyglot sent a ponder move to roce. My engine doesn't support pondering yet and that seems to cause the problem most probably.
Kod:
> ENGINE position startpos moves e2e4 c7c5 d2d4 c5d4 c2c3
...
Kod:
> ENGINE position startpos moves e2e4 c7c5 d2d4 c5d4 g1f3

You know these two lines are perfectly normal (when pondering).
The first line, followed by 'go ponder...', means that c2c3 is the ponder move. The second line is sent to you because the pondermove (c2c3 in this case) wasn't the right move and the opponent moved g1f3 instead. So now you will get the new position with the right move (g1f3) and a "real" go without ponder.
If you can't handle ponder this is no good for you!
I noticed that you return bestmove without the pondermove.
This should prevent pondering during the game but and I can't say why the first go you receive from the GUI is a 'go ponder...' .
Maybe you have ponder=on somewhere in the GUI?

/Peter

[Roman Hartmann]

You know these two lines are perfectly normal (when pondering).
The first line, followed by 'go ponder...', means that c2c3 is the ponder move. The second line is sent to you because the pondermove (c2c3 in this case) wasn't the right move and the opponent moved g1f3 instead. So now you will get the new position with the right move (g1f3) and a "real" go without ponder.
If you can't handle ponder this is no good for you!
I noticed that you return bestmove without the pondermove.
This should prevent pondering during the game but and I can't say why the first go you receive from the GUI is a 'go ponder...' .
Maybe you have ponder=on somewhere in the GUI?

/Peter

Yes, my engine doesn't support pondering yet. I didn't expect that the engine would hang when a pondermove ist sent though.
I implemented a workaround meanwhile which should fix this issue. The engine still doesn't support pondering, of course, but it shouldn't hang anymore now when the gui sends a move to ponder on.

Roman

[Fabien Letouzey]

Hi all,

I don't have time to read this thread now, but please remind me of the issue next week.

Is it clear what the problem is?

Fabien.

[Peter Fendrich]

Hi all,

I don't have time to read this thread now, but please remind me of the issue next week.

Is it clear what the problem is?

Fabien.

I think I have a fairly clear picture.
I also think that you have better things to do right now!
/Peter

[Fabien Letouzey]

I think I have a fairly clear picture.
I also think that you have better things to do right now!
/Peter

Hi Peter,

Please send me an email with your thoughts. I think I won't have the time to scan fora in the next few days. Bug fixes are always urgent and in fact I am supposed to update PolyGlot soon.

Fabien.

[Uri Blass]

I wonder what motivates Fabien to work so hard.

He has no customers who buy PolyGlot and he does not have to hurry to fix bugs.

I think that nobody is going to complain if he delay fixing bugs and read the forum(and people do not have moral right to complain if he decide to delay fixing bugs because of reading forums or becauuse of different reasons).

Uri

[Dann Corbit]

I wonder what motivates Fabien to work so hard.

I think he is just a nice guy.

He has no customers who buy PolyGlot and he does not have to hurry to fix bugs.

Sometimes, the intellectual persuit of correctness is its own reward.

I think that nobody is going to complain if he delay fixing bugs and read the forum(and people do not have moral right to complain if he decide to delay fixing bugs because of reading forums or becauuse of different reasons).

It is better not to look a gift horse in the mouth, but people do it anyway.

Uri

[Peter Fendrich]

I wonder what motivates Fabien to work so hard.

That goes for a lot of people, like Leo, Bob Hyatt, Olivier, Guenther, Dann, Thomas and Volker just to name a few...
Not all of them have a product to debug but still they work hard without any financial return.

He has no customers who buy PolyGlot and he does not have to hurry to fix bugs.

I think that nobody is going to complain if he delay fixing bugs and read the forum(and people do not have moral right to complain if he decide to delay fixing bugs because of reading forums or becauuse of different reasons).

Absolutely.
/Peter