TRACE 1.32 Released

[Ross Boyd]

Hi all,

TRACE 1.32 is available for download on Leo's site. Thankyou Leo!

http://wbec-ridderkerk.nl/

In our test gauntlets against a few engines it appears, overall, not as weak as 1.31 but who can know for sure

Have fun with it!

Ross & Tracey

Here are the release notes:

TRACE 1.32 - 15 Nov 2004 RELEASE
- Some small optimisations... 5% faster overall.
- Infamous trapped bishop code added.
- Mate threat and recapture extension fixes/tweaks.
- Backward pawn penalty increased.
- Bishop fianchetto code tweaks.
- All output sent via WIN32 WriteFile().
- Search is now aborted using C++ try/catch blocks.
- Updated the EGTB code to Eugene's latest... now the exe is huge...
- Bugfix: In extreme time trouble it could fail to find a move.
- Bugfix: Eval without EGTB returned draw for K vs K+R. (Thx to Mihail Croitor)
- Bugfix: Input parsing of promotions was broken. (Thx to my daughter Leah!)
- Bugfix: "st x" and "sd x" commands were broken.
- Bugfix: Analyze mode no longer researches when maxply reached...
- Bugfix: Makemove/takeback could corrupt the board
- Bugfix: Mate scores in WB/Arena corrected
- Bugfix: PV output could overflow into next field during long searches.
- WacNew on P3-450Mhz scores 298/300 @ 20 secs
- bench 7 on P3-450Mhz: nodes=500333, secs=4.46, nps=112232 score=81
pv=h2h4 f8d8 h4h5 g6f8 g5e4 d6e7

[Volker Böhm]

Why do you send any output via WIN32 WriteFile()? I use stdio.h - any possible improvements by WriteFile()???

Greetings Volker

[Ross Boyd]

Hi Volker,

Previous versions of TRACE used only printf() and fflush(stdout) for output. And during initialisation the io stream was set to unbuffered using setbuf() and setvbuf(). For all input I used fgets().

With WinXP I noticed that fgets() input could become garbled. You enter a winboard command and it would report "T" unknown command....

Then Alessandro Scotti released his excellent Threaded Input Library. There were some teething problems with the way XP handles ReadFile(). We were getting weird lockups. While trying to cure these problems I decided to use WriteFile() exclusively for output since I was using ReadFile() for all input.... hopefully to avoid any potential problems.

Possibly, using WriteFile() avoids some abstraction layers assuming the stdio library wraps the WriteFile() call. Maybe someone who understands the interaction between the C stdio library and Win32 can explain what is really happening. To tell the truth, I'm quite clueless on this.

But WriteFile() seems to work okay so I'll' keep using it for now.

BTW, congrats on Spike! Its really improving and fast!

Best wishes,

Ross

[Dan Honeycutt]

Hi Ross:

Did using WriteFile() solve the lockup problem with Alessandro's input thread?

Dan H.

[Ross Boyd]

Did using WriteFile() solve the lockup problem with Alessandro's input thread?

Maybe... maybe not.

It performed without a hiccup here in ~1000 games... but I don't test with pondering on.... doh.

Leo has sent me 2 debug logs from his test tourneys. I believe its a simple pondering bug but we shall see. I'll know for sure in a few hours.

OT: Man, its hot here. 7:30 am and about 32 C. Its going to be a stinker today.

I'll keep you posted ... (about the bugs)


Ross

[Ross Boyd]

Well, it seems there were some problems with TRACE's pondering.

I've fixed it and sent a tentative 1.33 version to Leo for testing. He uses a Dual processor PC and the threaded code doesn't appear to like it... so its now using plain vanilla input polling again.

I sure hope 1.33 works reliably... this is the stuff that insomnia is made of.

Cheers!

[Uri Blass]

Hi Ross,

You write:
"Then Alessandro Scotti released his excellent Threaded Input Library."
Why excellent?
If it does not work it is not excellent.

I am happy that I am not using that code after reading about all the problems with it.
I am not going even to think about using it unless I hear that people use it for a long time with no problems.

I am happy enough for now with no threads and checking for winboard command every 8192 nodes(or waiting passively for them when movei does not calculate).

Uri

[Ross Boyd]

Uri,

Obviously there are teething problems with the threaded input library. I believe, eventually, it will work without hassles... but first we must understand why Win32's ReadFile() is occasionally blocking for input when there is no input. That seems to be the root of the problem.

For the record, I don't recommend the library until the above problems are fixed. And when it is fixed I will gladly say again that it is an excellent piece of software.

But for now, it has excellent potential.

Cheers,

Ross

[Dan Honeycutt]

If it does not work it is not excellent.

Cold. Hard. Logical. Unfeeling.

[Volker Böhm]

Hi Ross,

there is no need to use import polling to get the engine to work with a dual processor. It is "just" a bid hard to tetect all race conditions. Spike had some problems (detected by leo) but now works fine with multiple threads - without using any library.

Greetings Volker

[Ross Boyd]

Dan stop it, I just spurted coffee over the keyboard...

there is no need to use import polling to get the engine to work with a dual processor. It is "just" a bid hard to tetect all race conditions. Spike had some problems (detected by leo) but now works fine with multiple threads - without using any library.

Volker, I agree that threaded input isn't necessary. However, Alessandro's library is useful for a few reasons. It bundles all the threaded IO in a class that makes it very easy to implement. Looking at the code, I don't think it suffers from a race condition. Perhaps the way it handles ReadFile() to get input is incorrect.

You have successfully implemented threaded input. Do you think you could have a very quick look at Alessandro's code - specifically, at the ReadFile() handling? Perhaps you will immediately recognise the problem?
I don't think Alessandro would mind. We both would like to get it working in our engines... and then the whole CC community can benefit.... and more importantly, I can again call the library EXCELLENT without copping logic bombs from Uri.

If you are interested I can email you the code. Its well-written and easy to read.

Best regards,
Ross

[Alessandro Scotti]

Hi folks,
first of all thanks a lot for the kind words on the input library!
A few days ago both Guenter and Leo reported a loss on time for Kiwi in the pre-tournament tests, and although eventually there were a couple of other bugs in the ponder code, it seems that polling for input made a real difference at least for Guenter. I've put the new code in the usual place:

http://www.ascotti.org/programming/chess/chess.htm

There is now a symbol to #define that enables polling when the library detects that input is a pipe on Windows. The library interface itself hasn't changed. Although I have run the input threaded code for months on several different systems without a problem, it's clear that there are situations where it might fail. I've been able to reproduce the problem myself, although only in a "artificial" environment. I've literally performed dozens and dozens of tests, and was able to get the wrong behavior even when the main thread is sleeping or waiting on some object. This would exclude a race condition because the input thread was basically running alone. I suspect that there may be a flaw in ReadFile itself... In my tests it always locked for an interval that was a multiple of 5 seconds, a unit of time that is nowhere in my code.
But... who really knows? I've found no useful info on the net and a post on the Microsoft newsgroup remained unanswered. In the meanwhile there is much to do on the engine, and for the moment I don't miss the hours of frustration I spent on the system library! At any rate, I will definitely come back to it when I have enough time... I have a debug version of Windows ready and a kernel debugger just in case...
BTW Ross idea of using WriteFile is very interesting... I'll take a shot at that asap!
For me, the main advantage of using this library is that it keeps all non-portable code nicely separated from the rest of the program, and in a class that is easy to use. It runs on Linux and Solaris, and I have compiled it (with some tweaks) on AIX and HP-UX. When I can do some serious tests, I'll update it with the code and build information for all these platforms, and probably more.
Wow... that was a long post, I better go now. Happy programming to all and good luck for the upcoming tournaments!

[Alessandro Scotti]

Hi all,

TRACE 1.32 is available for download on Leo's site. Thankyou Leo!

http://wbec-ridderkerk.nl/

TRACE 1.32 - 15 Nov 2004 RELEASE
...
- Search is now aborted using C++ try/catch blocks.
...

Hi Ross, congrats for the new Trace!
I have a question for you: when I leave C++ exceptions enabled, I always find they slow down my engine a bit. What's your experience? And, what compiler are you using?

[Ross Boyd]

Alessandro wrote:

Hi Ross, congrats for the new Trace!
I have a question for you: when I leave C++ exceptions enabled, I always find they slow down my engine a bit. What's your experience? And, what compiler are you using?

Thanks re TRACE! It seems to work on all but Leo's machine....
Ay carumba! I'm slowly ironing all the bugs out of ponder mode... nearly there.... releasing 1.33 is imminent...

AFAIK, using C++ exceptions didn't hurt performance... but I didn't pay much attention. So, the real answer is "I don't know". It seemed preferable to setjmp()/longjmp() which is what it was using before.

The compiler is the free MSVC++ 13.10.3052
These are the compile options used...
/Ox /Og /Ob2 /Oi /Ot /Oy /GT /GL /EHsc /GF /FD /MT /Zp16 /W3 /nologo /TP /D "WIN32"

Do you think I need to explicitly set a stack size for the linker? Hmmm... this post should be in the Programming section....

Best wishes,

Ross




[/b]

[Ross Boyd]

Hi again Alessandro,

I have a question for you: when I leave C++ exceptions enabled, I always find they slow down my engine a bit.

I can tell you with absolute certainty that the overhead of try/throw/catch exceptions was exactly NIL for me. I removed them all and benchmark timings were identical.

Hope that helps...

Ross (still debugging pondering)

[Volker Böhm]

Hi Ross,

if you talk about the System.cxx file in Kiwi_System.zip with win32 and posix thread handling, I have been taking a look.

I can?t find any problems, nor any possible race condition. It handles non finished lines and multiple lines read. Usually I wouldn?t switch a flag by two threads (inputAvailiable flag) but it is handled very well by synchronizing the threads.

Maybe you can check if *pInputAvailiable != true before setting *pInputAvailable = true; (this shouldn?t happen) and if it happens write a fatal error message to standard out. - Maybe this will detect anything I can?t imagine.

There could be some pitfalls by using the library. Perhaps if you check them (again) you will find a problem. (sorry if they are sometimes of the stype "did you plug it in").

1. Never use a copy of the variable "inputAvailable", thus I don?t think it is a good idea to return it by a method "getInputAvailiable" as it may return a copy of the variable without "volatile" flag. Thus better access it directly.

2. Never use the pointer returned by readInputLine after the call to acknowledgeInput.

Thus better to call
delete myStr; // declared as member and initialized by 0
char* c = readInputLine();
myStr = new[strlen(c) + 1];
strcpy(myStr, c);
accnowlegedgeInput();
processString(myStr);

or better if you have a String object:
String myStr = readInputLine();
accnowledgeInput();
processString(myStr);


3. be sure that you handle the case of search-finished while polling correctly.

Sorry that I can?t be of greater help.

Greetings Volker

[Alessandro Scotti]

Hi again Alessandro,

I have a question for you: when I leave C++ exceptions enabled, I always find they slow down my engine a bit.

I can tell you with absolute certainty that the overhead of try/throw/catch exceptions was exactly NIL for me. I removed them all and benchmark timings were identical.

Hope that helps...

Ross (still debugging pondering)

Hi Ross,
thanks for the answer and just one more question!
What I have observed is not directly related to the actual use of try/catch, but rather to enabling/disabling exception handling in the compiler.
In my case, the switch is called /GX I think. When exception handling is enabled, my program is slower no matter whether I actually use exceptions or not. The reason is that the compiler inserts special code in each function that is able to unwind the stack in case an exception is thrown (or an unhandled exception arrives "from below").
I am now trying to discover if the compiler you are using is smart enough to keep such code to a minimum, and to remove it from where it is not needed. In this case, exceptions will become very interesting for me too!
Can you confirm that you run your tests with different compile options?

[Ross Boyd]

Alessandro Scotti wrote:

Can you confirm that you run your tests with different compile options?

I sure can. In MSVC+ the compiler switch is /EHsc... when I tested performance (without it) I removed the /EHsc switch as well as the try/catch/throw code.

BTW, MSVC complains if you use try/catch/throw without specifying the /EH parameter...

I am still madly debugging pondering here.... up till 4:30am last night. I even wrote a buffered input handler/wrapper for ReadFile() which VERY NEARLY ALMOST worked...

Best wishes,
Ross




[/b]

[Ross Boyd]

Hi Volker,

Thankyou for looking at the Input code. I appreciate the time you spent on it and will investigate your comments as soon as time permits.
Right now, I have to fix a pondering bug... or miss out on WBEC.... this is what chess players call time pressure....

Best wishes,

Ross

[Alessandro Scotti]

Right now, I have to fix a pondering bug... or miss out on WBEC.... this is what chess players call time pressure....

Good luck Ross, I just came out from a week where I got very little sleep because of the pondering bugs found by Leo... you have all my sympathy!
And in the worst case, just disable pondering for good so you can start the WBEC and fix the bug later!

Thanks also for the compiler info and the tip on using exceptions!