How fast is slow enough? Opinions appreciated

[Volker Pittlik]

Geschrieben von:/Posted by: Volker Pittlik at 16 August 2004 12:39:26:

Time control has often been subject of discussions between computer chess
friends. Some prefer games at 40 moves in 2 hours (or even slower), others
use faster and even bullet time settings.
Both sides have good arguments. Supporters of slow time control argue at
tournament time controls argue we see "better games" and the engines are playing
closer to their "real strength". The opponents bring forward the argument
that they are able to play a lot more games and their results are more
meaningful in terms of "statistical significance".
This is of course a simplification of the debate but I think this are two
main topics. I'm not aware if it ever has been tested how fast time
controls are acceptable. If there is something around please let me know.
To find a rule of thumb in this matter I'm considering to make such a test
myself. I'm think of the following design:
Let's take an number of engines, commercial and freeware, some under active
development as well as dinosaurs, strong ones and some not so strong.
Books, tablebases and learning should be disabled.
Then take a number of randomnly choosen chess games and select the position
after 1/5, 2/5, 3/5 and 4/5 of the moves in these games.
In the first step let the engines calculate the next move (white and black
alternating) in these positions for three minutes each.
Then reduce the calculation time to 2 minutes, 1 minute, 30 seconds, 20
seconds and so on. After each reduction of time compare the suggested moves
with the moves calculated at 3 minutes per move. If the identicalness drops
below 95% the time control should be considered as to fast.
What do you think about that?
Regards
Volker

[Gábor Szots]

Geschrieben von:/Posted by: Gábor Szots at 16 August 2004 13:04:04:
Als Antwort auf:/In reply to: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Volker Pittlik at 16 August 2004 12:39:26:

Time control has often been subject of discussions between computer chess
friends. Some prefer games at 40 moves in 2 hours (or even slower), others
use faster and even bullet time settings.
Both sides have good arguments. Supporters of slow time control argue at
tournament time controls argue we see "better games" and the engines are playing
closer to their "real strength". The opponents bring forward the argument
that they are able to play a lot more games and their results are more
meaningful in terms of "statistical significance".
This is of course a simplification of the debate but I think this are two
main topics. I'm not aware if it ever has been tested how fast time
controls are acceptable. If there is something around please let me know.
To find a rule of thumb in this matter I'm considering to make such a test
myself. I'm think of the following design:
Let's take an number of engines, commercial and freeware, some under active
development as well as dinosaurs, strong ones and some not so strong.
Books, tablebases and learning should be disabled.
Then take a number of randomnly choosen chess games and select the position
after 1/5, 2/5, 3/5 and 4/5 of the moves in these games.
In the first step let the engines calculate the next move (white and black
alternating) in these positions for three minutes each.
Then reduce the calculation time to 2 minutes, 1 minute, 30 seconds, 20
seconds and so on. After each reduction of time compare the suggested moves
with the moves calculated at 3 minutes per move. If the identicalness drops
below 95% the time control should be considered as to fast.
What do you think about that?
Regards
Volker

Hello Volker,
This is in fact a very interesting idea that may lead to reducing time control in a sensible way, making possible the increase of the number of games for a given time period.
What I don't understand is why different positions of the same game should be used. It looks to me that for the purpose of the test any test suite will do.
I think the suggested method may be simplified by letting the engine analyze the position and taking notes of its suggested move at predefined times. A log file would be of great help, making it unnecessary to watch the analysis.
Best regards,
Gábor

[Volker Pittlik]

Geschrieben von:/Posted by: Volker Pittlik at 16 August 2004 13:32:00:
Als Antwort auf:/In reply to: Re: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Gábor Szots at 16 August 2004 13:04:04:

...

What I don't understand is why different positions of the same game should be used. It looks to me that for the purpose of the test any test suite will do.
I think the suggested method may be simplified by letting the engine analyze the position and taking notes of its suggested move at predefined times. A log file would be of great help, making it unnecessary to watch the analysis.

I would like to test different types of positions (opening, middlegame, positional problems and tactical struggle). If I choose say 100 games I'll get 400 positions and can hope no engine will be advantaged or disadvantaged by the postions.
The most test suite I know are mostly dealing with checkmating or winning a piece in n moves. I guess there will not bee much difference between an analysis in 3 or 1 minutes or even in 30 seconds. The positions in the Nolot test OTOH are still far to complicated for most engines as far as I know.

Of course! That will save me a lot of time .
Regards
Volker

[Andrew Fan]

Geschrieben von:/Posted by: Andrew Fan at 16 August 2004 13:59:45:
Als Antwort auf:/In reply to: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Volker Pittlik at 16 August 2004 12:39:26:

Time control has often been subject of discussions between computer chess
friends. Some prefer games at 40 moves in 2 hours (or even slower), others
use faster and even bullet time settings.
Both sides have good arguments. Supporters of slow time control argue at
tournament time controls argue we see "better games" and the engines are playing
closer to their "real strength". The opponents bring forward the argument
that they are able to play a lot more games and their results are more
meaningful in terms of "statistical significance".
This is of course a simplification of the debate but I think this are two
main topics. I'm not aware if it ever has been tested how fast time
controls are acceptable. If there is something around please let me know.
To find a rule of thumb in this matter I'm considering to make such a test
myself. I'm think of the following design:
Let's take an number of engines, commercial and freeware, some under active
development as well as dinosaurs, strong ones and some not so strong.
Books, tablebases and learning should be disabled.
Then take a number of randomnly choosen chess games and select the position
after 1/5, 2/5, 3/5 and 4/5 of the moves in these games.
In the first step let the engines calculate the next move (white and black
alternating) in these positions for three minutes each.
Then reduce the calculation time to 2 minutes, 1 minute, 30 seconds, 20
seconds and so on. After each reduction of time compare the suggested moves
with the moves calculated at 3 minutes per move. If the identicalness drops
below 95% the time control should be considered as to fast.
What do you think about that?
Regards
Volker

Remember to clear the hash tables (quit then reload the engine) between runs.
Do time increments instead of decrements.
Andrew.

[Jose Carlos]

Geschrieben von:/Posted by: Jose Carlos at 16 August 2004 15:35:14:
Als Antwort auf:/In reply to: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Volker Pittlik at 16 August 2004 12:39:26:

Time control has often been subject of discussions between computer chess
friends. Some prefer games at 40 moves in 2 hours (or even slower), others
use faster and even bullet time settings.
Both sides have good arguments. Supporters of slow time control argue at
tournament time controls argue we see "better games" and the engines are playing
closer to their "real strength". The opponents bring forward the argument
that they are able to play a lot more games and their results are more
meaningful in terms of "statistical significance".
This is of course a simplification of the debate but I think this are two
main topics. I'm not aware if it ever has been tested how fast time
controls are acceptable. If there is something around please let me know.
To find a rule of thumb in this matter I'm considering to make such a test
myself. I'm think of the following design:
Let's take an number of engines, commercial and freeware, some under active
development as well as dinosaurs, strong ones and some not so strong.
Books, tablebases and learning should be disabled.
Then take a number of randomnly choosen chess games and select the position
after 1/5, 2/5, 3/5 and 4/5 of the moves in these games.
In the first step let the engines calculate the next move (white and black
alternating) in these positions for three minutes each.
Then reduce the calculation time to 2 minutes, 1 minute, 30 seconds, 20
seconds and so on. After each reduction of time compare the suggested moves
with the moves calculated at 3 minutes per move. If the identicalness drops
below 95% the time control should be considered as to fast.
What do you think about that?
Regards
Volker

Hi Volker. Some time ago, I posted in CCC an idea for a different experiment regarding time controls comparison. It came from the idea that fast games are more likely to add random noise because of luck (for example, in playing a good move without really understanding the position, or not being able to finish tactically a winning attack, etc.) and thus, more fast games are needed for statistical significance than slow games. This argument might not be correct, but it looks reasonable and my experiment would prove or disprove it. The experiment also depends on the assumption that programs play at similar level at fast and slow time controls (which is the opinion of most top programmers), so buggy engines (like mine in this regard) should be avoided:
We play 10000 games at 1 0, program A vs program B. A wins x% of the points.
Now we start playing 5 0, and see how many games are needed to stabilize the result at A = x%, B = (100-x)%.
We can repeat the experiment for slower time controls.
My experiment has nothing to do with yours, except for the background idea of comparing meaningful results at different time controls. I expect the number of games decreasing as time increases, but I have no idea about the possible shape of the curve.
José C.

[Volker Boehm]

Geschrieben von:/Posted by: Volker Boehm at 16 August 2004 17:26:28:
Als Antwort auf:/In reply to: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Volker Pittlik at 16 August 2004 12:39:26:

Time control has often been subject of discussions between computer chess
friends. Some prefer games at 40 moves in 2 hours (or even slower), others
use faster and even bullet time settings.
Both sides have good arguments. Supporters of slow time control argue at
tournament time controls argue we see "better games" and the engines are playing
closer to their "real strength". The opponents bring forward the argument
that they are able to play a lot more games and their results are more
meaningful in terms of "statistical significance".
This is of course a simplification of the debate but I think this are two
main topics. I'm not aware if it ever has been tested how fast time
controls are acceptable. If there is something around please let me know.
To find a rule of thumb in this matter I'm considering to make such a test
myself. I'm think of the following design:
Let's take an number of engines, commercial and freeware, some under active
development as well as dinosaurs, strong ones and some not so strong.
Books, tablebases and learning should be disabled.
Then take a number of randomnly choosen chess games and select the position
after 1/5, 2/5, 3/5 and 4/5 of the moves in these games.
In the first step let the engines calculate the next move (white and black
alternating) in these positions for three minutes each.
Then reduce the calculation time to 2 minutes, 1 minute, 30 seconds, 20
seconds and so on. After each reduction of time compare the suggested moves
with the moves calculated at 3 minutes per move. If the identicalness drops
below 95% the time control should be considered as to fast.
What do you think about that?
Regards
Volker

Hi Volker,
I am not a tester, but I am testing our own engine a lot. One "primary" test is a set of 100 positions after move 10 that a new engine plays against a reference engine once with black and once with white. I play 5 min. per game to get a statistically large amount of results.
I am comparing not only the amount of results, but also the concrete "result pattern". A little change in the engine or the amount of time the engine uses changes the "result pattern" completely.
The consequence is, that a littel change in time control (even if you use 5:30 instead of 5 min) will give total different games, even with the same base position and learning and bibliothek switched off.
Another Point to consider:
What are 40 min in a game? A game with longer time control played on a Pentium with 90 MHZ is now a bullet game on an Athlon 64.
Again another Point:
Some engines will are weaker with short time controls (compared to other engines with same time-control), other engines does not change much.
The most relevant point:
Nobody really knows how time control or draws affect the relevance of the result. Thus you should suppose that winning/losing against a engine of same strength is like throwing a coin.
The statistic says that with a relevance of 95% you have to get more than 7 points to judge one engine stronger as another. A relevance of 95% means that there is a 5% chance that you are wrong, thus in 20 matches you have on engines that wins 7,5:2,5 against an engine of same strength.
If you have 50 games you only need 60% of points to have a relevant (less 5% error) information that one engine is stronger than the other.

Conclusion:
Make the time control as long as possible to get at least 50 games for every engine.
Greetings Volker

[Igor Korshunov]

Geschrieben von:/Posted by: Igor Korshunov at 16. August 2004 18:09:
Als Antwort auf:/In reply to: Re: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Jose Carlos at 16 August 2004 15:35:14:

Hi Volker. Some time ago, I posted in CCC an idea for a different experiment regarding time controls comparison. It came from the idea that fast games are more likely to add random noise because of luck (for example, in playing a good move without really understanding the position, or not being able to finish tactically a winning attack, etc.) and thus, more fast games are needed for statistical significance than slow games. This argument might not be correct, but it looks reasonable and my experiment would prove or disprove it. The experiment also depends on the assumption that programs play at similar level at fast and slow time controls (which is the opinion of most top programmers), so buggy engines (like mine in this regard) should be avoided:
We play 10000 games at 1 0, program A vs program B. A wins x% of the points.
Now we start playing 5 0, and see how many games are needed to stabilize the result at A = x%, B = (100-x)%.
We can repeat the experiment for slower time controls.
My experiment has nothing to do with yours, except for the background idea of comparing meaningful results at different time controls. I expect the number of games decreasing as time increases, but I have no idea about the possible shape of the curve.
José C.

IMHO, luck has the same importance at slower controls as at bullet.

[Dann Corbit]

Geschrieben von:/Posted by: Dann Corbit at 16 August 2004 22:10:08:
Als Antwort auf:/In reply to: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Volker Pittlik at 16 August 2004 12:39:26:

Time control has often been subject of discussions between computer chess
friends. Some prefer games at 40 moves in 2 hours (or even slower), others
use faster and even bullet time settings.
Both sides have good arguments. Supporters of slow time control argue at
tournament time controls argue we see "better games" and the engines are playing
closer to their "real strength". The opponents bring forward the argument
that they are able to play a lot more games and their results are more
meaningful in terms of "statistical significance".
This is of course a simplification of the debate but I think this are two
main topics. I'm not aware if it ever has been tested how fast time
controls are acceptable. If there is something around please let me know.
To find a rule of thumb in this matter I'm considering to make such a test
myself. I'm think of the following design:
Let's take an number of engines, commercial and freeware, some under active
development as well as dinosaurs, strong ones and some not so strong.
Books, tablebases and learning should be disabled.
Then take a number of randomnly choosen chess games and select the position
after 1/5, 2/5, 3/5 and 4/5 of the moves in these games.
In the first step let the engines calculate the next move (white and black
alternating) in these positions for three minutes each.
Then reduce the calculation time to 2 minutes, 1 minute, 30 seconds, 20
seconds and so on. After each reduction of time compare the suggested moves
with the moves calculated at 3 minutes per move. If the identicalness drops
below 95% the time control should be considered as to fast.
What do you think about that?
Regards
Volker

It depends on your goals. It takes a very long time to do a contest at 40moves/2hrs. This contest of 40 games took about 10 days running around the clock:

Program Elo + - Games Score Av.Op. Draws
1 Ruffian_105 : 2705 153 115 20 60.0 % 2635 40.0 %
2 Aristarch 4.50: 2703 153 133 20 60.0 % 2633 30.0 %
3 Smarthink-017a: 2622 95 162 20 45.0 % 2657 50.0 %
4 Crafty-1915 : 2569 119 145 20 35.0 % 2676 40.0 %

There is still not enough data to draw any sound conclusions.
But I like to look at the games. So I don't mind waiting so much.



my ftp site {remove http:// unless you like error messages}

[Volker Pittlik]

Geschrieben von:/Posted by: Volker Pittlik at 17 August 2004 08:48:18:
Als Antwort auf:/In reply to: Re: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Andrew Fan at 16 August 2004 13:59:45:

...

Remember to clear the hash tables (quit then reload the engine) between runs.
Do time increments instead of decrements.

I agree this is very important otherwise the results may be affected by previous searches.
Im thinking about it. I would like to choose a setting where I get the value
of the calculation at 3 minutes per move first. If it turns out that the
identicalness drops below 50% at a certain amount of time per move it seems
not reasonable to continue the experiment.
Best regards
Volker

[Volker Pittlik]

Geschrieben von:/Posted by: Volker Pittlik at 17 August 2004 09:13:09:
Als Antwort auf:/In reply to: Re: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Jose Carlos at 16 August 2004 15:35:14:

...

Hi Volker. Some time ago, I posted in CCC an idea for a different experiment regarding time controls comparison. It came from the idea that fast games are more likely to add random noise because of luck (for example, in playing a good move without really understanding the position, or not being able to finish tactically a winning attack, etc.) and thus, more fast games are needed for statistical significance than slow games. This argument might not be correct, but it looks reasonable and my experiment would prove or disprove it.

Interesting too. Let's assume there are -for example- 10 possible moves in a given position. One of them is a little bit better than the others what is found out by the engine at slow time control. Three others are not really bad and not really good and all others are a clearly bad.
At faster the time controls it becomes more and more unlikely that the one a little bit better move is found and the other three not clearly bad moves are choosen. I would assume that the engine is choosing now from the four not clearly bad moves with the same chance. But this is an assumption only.
Volker

[Volker Pittlik]

Geschrieben von:/Posted by: Volker Pittlik at 17 August 2004 09:30:10:
Als Antwort auf:/In reply to: Re: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Volker Boehm at 16 August 2004 17:26:28:

...

The consequence is, that a littel change in time control (even if you use 5:30 instead of 5 min) will give total different games, even with the same base position and learning and bibliothek switched off.
Another Point to consider:
What are 40 min in a game? A game with longer time control played on a Pentium with 90 MHZ is now a bullet game on an Athlon 64.
Again another Point:
Some engines will are weaker with short time controls (compared to other engines with same time-control), other engines does not change much.
The most relevant point:
Nobody really knows how time control or draws affect the relevance of the result. Thus you should suppose that winning/losing against a engine of same strength is like throwing a coin.
The statistic says that with a relevance of 95% you have to get more than 7 points to judge one engine stronger as another. A relevance of 95% means that there is a 5% chance that you are wrong, thus in 20 matches you have on engines that wins 7,5:2,5 against an engine of same strength.
If you have 50 games you only need 60% of points to have a relevant (less 5% error) information that one engine is stronger than the other.

Conclusion:
Make the time control as long as possible to get at least 50 games for every
Greetings Volker

Therefore I would like to test positions and not complete games (what was my first thought). I have observed too often that even games with the same engines and the same time control are running different also when books, learning and everything is disabled. This may be caused by other things running at the computer (virus scanner, mail program checking for mail, Windows services in the background and so on).

I tend to agree. OTOH it has been argued that some engines are behaving different depending on the absoulute amount of remaining time. I can't confirm or deny that.

This is a interesting point too. If I run this experiment it is possible that we also get some data for this. OTOH: Do you know any rating list or whatever where this difference is shown at the level of statistical significance? I'm not aware of something like that.

Interesting points too, but I'm not going to play games.
engine.
See above.
me 2.

[Volker Pittlik]

Geschrieben von:/Posted by: Volker Pittlik at 17 August 2004 09:45:11:
Als Antwort auf:/In reply to: Re: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Dann Corbit at 16 August 2004 22:10:08:

Time control has often been subject of discussions between computer chess
friends. Some prefer games at 40 moves in 2 hours (or even slower), others
use faster and even bullet time settings.
Both sides have good arguments.

It depends on your goals.
It takes a very long time to do a contest at 40moves/2hrs. This contest of 40 games took about 10 days running around the clock:
Program Elo + - Games Score Av.Op. Draws
1 Ruffian_105 : 2705 153 115 20 60.0 % 2635 40.0 %
2 Aristarch 4.50: 2703 153 133 20 60.0 % 2633 30.0 %
3 Smarthink-017a: 2622 95 162 20 45.0 % 2657 50.0 %
4 Crafty-1915 : 2569 119 145 20 35.0 % 2676 40.0 %
There is still not enough data to draw any sound conclusions.
But I like to look at the games. So I don't mind waiting so much.

...
The debates are sometimes very emotional. I would like to have something to argue based on facts.



I like to watch slow games too. At least they have to be that slow I can follow them. For test purposes I tend to play as slow as necessary and as fast as possible.
Regards
Volker

[Ingo Bauer]

Geschrieben von:/Posted by: Ingo Bauer at 17. August 2004 11:14:
Als Antwort auf:/In reply to: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Volker Pittlik at 16 August 2004 12:39:26:

Hello Volker
Even if I agree with your goal, to find a "resonable" time control, I think this approch has a little problem.
It does not matter what percentage you take (50, 90, 95 or 99) the remaining percentage will be the critical one where the engine could have found the best move that gives it the boost to dominate all other engines (exaggerated!)
At least that will be the arguments of the "longer is better" fraction. The problem is - they may be right!
Bye
Ingo Bauer

Time control has often been subject of discussions between computer chess
friends. Some prefer games at 40 moves in 2 hours (or even slower), others
use faster and even bullet time settings.
Both sides have good arguments. Supporters of slow time control argue at
tournament time controls argue we see "better games" and the engines are playing
closer to their "real strength". The opponents bring forward the argument
that they are able to play a lot more games and their results are more
meaningful in terms of "statistical significance".
This is of course a simplification of the debate but I think this are two
main topics. I'm not aware if it ever has been tested how fast time
controls are acceptable. If there is something around please let me know.
To find a rule of thumb in this matter I'm considering to make such a test
myself. I'm think of the following design:
Let's take an number of engines, commercial and freeware, some under active
development as well as dinosaurs, strong ones and some not so strong.
Books, tablebases and learning should be disabled.
Then take a number of randomnly choosen chess games and select the position
after 1/5, 2/5, 3/5 and 4/5 of the moves in these games.
In the first step let the engines calculate the next move (white and black
alternating) in these positions for three minutes each.
Then reduce the calculation time to 2 minutes, 1 minute, 30 seconds, 20
seconds and so on. After each reduction of time compare the suggested moves
with the moves calculated at 3 minutes per move. If the identicalness drops
below 95% the time control should be considered as to fast.
What do you think about that?
Regards
Volker

[Volker Pittlik]

Geschrieben von:/Posted by: Volker Pittlik at 17 August 2004 11:44:11:
Als Antwort auf:/In reply to: Re: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Ingo Bauer at 17. August 2004 11:14:

...

It does not matter what percentage you take (50, 90, 95 or 99) the remaining percentage will be the critical one where the engine could have found the best move that gives it the boost to dominate all other engines (exaggerated!)
At least that will be the arguments of the "longer is better" fraction. The problem is - they may be right!

I agree. I tend to use 95% because that margin is often used in social science. It is just convention. If someone is going to build an aircraft such an reliability seems to be a bit careless.

They might be or might be not. At the moment choosing the time control seems to be influenced by wild guessing. There are of course other good reasons the choose a certain time control. The amount of time a tester can spend in total is absolutely acceptable IMHO.
Meanwhile I believe that there isn't any chance to keep the number or intensity of flame flars about this and other topics down. But I'll try to use that chance .
Regards
Volker

[Ingo Bauer]

Geschrieben von:/Posted by: Ingo Bauer at 17. August 2004 12:17:
Als Antwort auf:/In reply to: Re: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Volker Pittlik at 17 August 2004 11:44:11:

Hi

At least that will be the arguments of the "longer is better" fraction. The problem is - they may be right!

They might be or might be not. At the moment choosing the time control seems to be influenced by wild guessing.
Meanwhile I believe that there isn't any chance to keep the number or intensity of flame flars about this and other topics down. But I'll try to use that chance .

The purists will argue that 40/120 + 20/60 + 60 or 90min + 30 sec / game is FIDE timecontrol. This is the convention and we use it, other time controlls will be insufficient. (Basta!)

I stop here!
Ingo

[Volker Pittlik]

Geschrieben von:/Posted by: Volker Pittlik at 17 August 2004 13:06:34:
Als Antwort auf:/In reply to: Re: How fast is slow enough? Opinions appreciated geschrieben von:/posted by: Ingo Bauer at 17. August 2004 12:17:

...

The purists will argue that 40/120 + 20/60 + 60 or 90min + 30 sec / game is FIDE timecontrol. This is the convention and we use it, other time controlls will be insufficient. (Basta!)
I stop here!

I love them! OTOH it is a great answer. The problem occurs when we are looking for the corresponding question .
...
42!
Volker