Ponder_on-ponder_off comparision

[Volker Pittlik]

Geschrieben von: / Posted by: Volker Pittlik at 18 June 2000 18:45:14:
I compare the performance of some programs with ponder=on and ponder=off. As most of us know this have been discussed very often. I would like to add some data. If I remember correctly pondering in particular for Crafty is a great advantage especially at fast time controls.
All games have been played on my dualboard (520 MHz) so each engine has a cpu for it's own when pondering is on. The fastest time control I can run (without to many games are lost on time) is one minute initially and an increment of two seconds for each move. The engines were using the books of their own. Learning was on, all 4- and some 5-piece tablebases were in use.
The engines were paired in 20 games matches, round robin so I got 1200 games for the first part of this test (ponder was on). The test will be continued with the same engines (same versions as well) and ponder=off. Results for part one:

Program Score % Av.Op. Elo + - Draws
1 Crafty1710 : 148.5/200 74.2 2268 2452 35 66 16.5 %
2 AnMon506 : 123.0/200 61.5 2306 2387 41 49 19.0 %
3 TCB0045 : 116.5/200 58.2 2292 2350 43 47 17.5 %
4 Amyf_000420 : 114.5/200 57.2 2308 2359 44 47 16.5 %
5 Gromit_30 : 111.0/200 55.5 2294 2332 45 46 15.0 %
6 Comet_B21 : 110.0/200 55.0 2282 2317 46 44 18.0 %
7 SOS_991103 : 107.0/200 53.5 2284 2308 47 45 15.0 %
8 Phalanx22 : 99.5/200 49.8 2307 2306 40 49 20.5 %
9 Bringer16 : 92.5/200 46.2 2298 2272 44 47 17.5 %
10 Arasanx_53 : 65.5/200 32.8 2319 2194 54 38 19.5 %
11 EXchess314 : 61.0/200 30.5 2322 2179 56 37 20.0 %
12 LambChop_710 : 51.0/200 25.5 2329 2143 69 35 14.0 %
---------------------------------------------------------------------
(Calculation has been made with elostat by Frank Schubert.)
{D:\WINBOARD\SPIELE\Ponder_on-Ponder_off>pgnres ponder_on /t
Results from file ponder_on.pgn:

No. Name Win Draw Loss Unf. Score Games % SOP
-------------------------------------------------------- -------
1 Crafty1710 +132 =33 -35 *0 148.5 200 74.2% 5280.0
2 AnMon506 +104 =38 -58 *0 123.0 200 61.5% 5055.0
3 TCB0045 +99 =35 -66 *0 116.5 200 58.2% 4075.0
4 Amyf_000420 +98 =33 -69 *0 114.5 200 57.2% 4500.0
5 Gromit_30 +96 =30 -74 *0 111.0 200 55.5% 5780.0
6 Comet_B21 +92 =36 -72 *0 110.0 200 55.0% 5545.0
7 SOS_991103 +92 =30 -78 *0 107.0 200 53.5% 6235.0
8 Phalanx22 +79 =41 -80 *0 99.5 200 49.8% 6530.0
9 Bringer16 +75 =35 -90 *0 92.5 200 46.2% 4540.0
10 Arasanx_53 +46 =39 -115 *0 65.5 200 32.8% 3405.0
11 EXchess314 +41 =40 -119 *0 61.0 200 30.5% 4750.0
12 LambChop_710 +37 =28 -135 *0 51.0 200 25.5% 4305.0

Total Games: 1200
White Wins: 548 (45.7%)
Black Wins: 443 (36.9%)
Draws: 209 (17.4%)
Unfinished: 0 (0.0%)}
(Calculation has been made with pgnres by Peter McKenzie.)
All games are available in my games archive (ponder_on.zip)
Volker
http://members.yoom.com/VolkerPi/gamesA ... rchive.htm
Please replace y by x in the URL above. Somehow the admins of Parsimony have included the word you got then have added to the global bad word list.
Wondering
Volker

[Thorsten Czub]

Geschrieben von: / Posted by: Thorsten Czub at 18 June 2000 22:14:41:
Als Antwort auf: / As an answer to: Ponder_on-ponder_off comparision geschrieben von: / posted by: Volker Pittlik at 18 June 2000 18:45:14:
the discusison about how ponder affects chessprograms,
cannot be made without differenciating the programs
you DO the test.
each program behaves different.
if you have very similar programs, the results of your
test will be different than having non-similar
programs.
very high percantages of playing the ponder move does not
mean the programs are similar strength, it can also
be a sign for the fact the programs are similar programmed.
if a program has completely different = low playing ponder
percantage, this does not mean it is weak.
its all very complicate.
the level is another parameter that confuses all.
programs behave different in using ponder time
if they run on blitz or tournament controls. they profit
different in longer time controls than in shorter.
i have often seen on championships that ponder is very
important.
often this is not seen in the game-score-file ,
but when you see it life, you see e.g. that nimzo
has a wrong ponder, gets a fail-high in the ponder
because the move it considers for the opponent is
shit.
when the opponent makes the different move, nimzo
has time problems in a difficult position and loses
due to the wrong ponder.
this is all not to be seen when you only look into the game-score.
you have to see the programs playing live and have to
keep an eye on the main-lines the whole game.
difficult to solve this question by quantifying it.

[Volker Pittlik]

Geschrieben von: / Posted by: Volker Pittlik at 19 June 2000 15:37:23:
Als Antwort auf: / As an answer to: ponder is how much elo difference ? geschrieben von: / posted by: Thorsten Czub at 18 June 2000 22:14:41:

ponder is how much elo difference ?
the discusison about how ponder affects chessprograms,
cannot be made without differenciating the programs
you DO the test.
very high percantages of playing the ponder move does not
mean the programs are similar strength, [...]
[...]it can also
be a sign for the fact the programs are similar programmed.
if a program has completely different = low playing ponder
percantage, this does not mean it is weak.
its all very complicate.
the level is another parameter that confuses all.
programs behave different in using ponder time
if they run on blitz or tournament controls. they profit
different in longer time controls than in shorter.
i have often seen on championships that ponder is very
important.
often this is not seen in the game-score-file ,
but when you see it life, you see e.g. that nimzo
has a wrong ponder, gets a fail-high in the ponder
because the move it considers for the opponent is
shit.
when the opponent makes the different move, nimzo
has time problems in a difficult position and loses
due to the wrong ponder.
this is all not to be seen when you only look into the game-score.
you have to see the programs playing live and have to
keep an eye on the main-lines the whole game.
difficult to solve this question by quantifying it.

I would like to help to answer this question.
Difference of what quality?
[...]
I'm not looking how often the programs get the move of the
opponent. I'm interested to compare the results of this tests
(for example: is Crafty winning the ponder-off round too?).
I'm not speculating about the design of the programs.
You are right. I can't remember to claim something like that.
Yes, so I try to keep the conditions for the test as simple as possible.
I suspect to get the largest differences at short time controls.
So I've choosen 60+2 the shortest time control I can run on my machine without getting to much games lost on time.
So let's try to determine *how* important it is.
WB-Nimzo has a time problem in general. It moves much to fast.
You're possibly right but finding of errors in move generation is not the goal of this test.
I hope my results are not randomly.
Volker