How fast is slow enough? (Diagram inside)

[Volker Pittlik]

Which time control should be used has often been discussed. Some trust on blitz or even bullet time control, others only accept games at at the classical tournament speed. I made a test and - didn't find an answer.

However, maybe my findings can give us a little hint. I used Thomas Mayer's version of epd2wb and several Winboard and UCI -engines and compared how many solutions of the Arasan 5 testsuite are found after different amounts of time.

The Arasan testsuite was choosen because it is difficult enough (at my 2 GHz AMD) to differentiate between the engines. Other testsuites are to easy (WAC for example). Most engines find almost all solutions within a very few seconds.

See this diagram of the results:



I don't have a conclusion. But maybe you are interested.

The compressed Excel sheet with all results is here:

http://www.volker-pittlik.name/winboard/times/times.zip.

Regards

Volker

[Reinhard Scharnagl]

Hi Volker,

I have not found that testsuite to be downloadable somewhere. Could you provide a link?

Reinhard.

[Volker Pittlik]

... Could you provide a link?

http://www.arasanchess.org/tests.zip

Some other tests are included too.

Volker

[Robert Allgeuer]

When looking at your graphs at least one observation is that more or less those engines that are score high in short time controls do so also at long time controls. There is no real criss-crossing of the graphs, e.g. something of the type that one engine starts out at the end of the field but overtakes the others and is in the leading pack at longer time controls. The graphs are rather parallel. (I see that Shredder loses a bit in this specific test, but one would have to do this type of test with many more positions, excluding mate or possibly even tactical positions)

My personal opinion on long vs. short time controls: even when some engines do gain and others lose with longer time controls (usually <30 ELO points though), test results at short time controls are pretty representative of general strength. Prerequisite: a short time control with increment is used. Otherwise time control and losses on time start dominating test results, and then the measured strength may indeed differ dramatically.

Robert

[Reinhard Scharnagl]

Hi Volker,

thank you for the link.

Reinhard

[Ross Boyd]

Hi Volker,

Interesting test results.

What strikes me is the incredible solve rate for Shredder at the 1 to 10 second mark. It completely dominates the other engines in this area.

So, shredder's search must have some great extensions and/or clever pruning to sniff out the best moves so quickly. I guess this translates to an ability to spot tactics a move or two earlier than other engines in blitz games... very handy.

Cheers,
Ross

[Volker Pittlik]

...
My personal opinion on long vs. short time controls: even when some engines do gain and others lose with longer time controls (usually <30 ELO points though), test results at short time controls are pretty representative of general strength...

I tend to agree. If you compare your rating list with others at slower time control you will find that the engines are aranged in more or less the same ranking order.

...Prerequisite: a short time control with increment is used. Otherwise time control and losses on time start dominating test results, and then the measured strength may indeed differ dramatically...

Another critcal factor is the number of games of course. If I can choose between rating list based upon 100 games at blitz (if not run at a stupid speed of all moves in one minute or so) and 10 games at 40/2 hours I would choose the first. A completly different question is the quality of the games...

Regards

Volker

[Volker Pittlik]

...What strikes me is the incredible solve rate for Shredder at the 1 to 10 second mark. It completely dominates the other engines in this area...

Yes, it's amazing. I don't know if Shredder is even more superior in blitz than at slower time controls.

Volker

[Robert Allgeuer]

How many positions are there in the Arasan test-suite? Maybe Shredder's high solution percentage for very short time controls is also just a statistical phenomenon due to not enough positions tested.

Robert

[Volker Pittlik]

How many positions are there in the Arasan test-suite? Maybe Shredder's high solution percentage for very short time controls is also just a statistical phenomenon due to not enough positions tested.

Robert

There are 141 positions. BTW: there are no mate-in-n positions amoung them.

Volker

[Igor Gorelikov]

My personal opinion on long vs. short time controls: even when some engines do gain and others lose with longer time controls (usually <30 ELO points though), test results at short time controls are pretty representative of general strength.

Robert

In general you are right but there are some exceptions. Some engines behave different at different time controls.

You may look at two rating lists of my old IL (blitz and rapid).

http://freehosting.hostrave.com/p/iggor ... _final.txt
http://freehosting.hostrave.com/p/iggor ... _final.txt

Igor