Test suites (particularly Alessandro Scotti)

[ShaunBrewer]

Anyone/Alessandro,

I just discovered the test suites on your site - are these verified, would they be suitable for my requirements or could anyone point me at a alternatives.

I am looking for a test suite/suites I can use to check a programs strength at different time controls as well as improvements between versions.

Ideally I would like to group the results by start, middle and end game.

I am planning to use EPD2WB (with a winboard / UCI adapter if teh engine is not wb compatible) to run the tests - if anyone knows of a better alternative (it must be command line) - please let me know

Thanks for your time

Shaun

[Alessandro Scotti]

Hi Shaun,
unfortunately the test suites on my site are not verified, I just found them on the net (not easily) and decided to put them in a safe place!
For UCI engines, the way to go is Polyglot (which you can download from the WBEC site linked on top of this page, not before having clicked a few ads though! ). For example:

Code: Select all

polyglot epd-test -epd wac.epd -max-time 5


other command line options are: -min-depth, -max-depth, -min-time, -depth-delta.

[ShaunBrewer]

Alessandro,

thanks for the info - mmm can find loads of EPD suites just not sure how good any of them are.

Thanks again for the polygot hint

Shaun

[ShaunBrewer]

Sorry Alessandro,

Can you point me at PolyGlot help files that mention this epd test suite processing?

I have it workign but want to know things like - does it just stop when the best move is found or does it stay for the duration incase the engine changes its mind.

Thanks

Shaun

[Alessandro Scotti]

Sorry Alessandro,

Can you point me at PolyGlot help files that mention this epd test suite processing?

I have it workign but want to know things like - does it just stop when the best move is found or does it stay for the duration incase the engine changes its mind.

Thanks

Shaun

Hello Shaun,
there's no official readme for these functions, you just have to take a look at the source files to see the various options.
I don't know the exact rules Polyglot is using, but it does seem to wait, unless the position is mate.

[Tord Romstad]

Sorry Alessandro,

Can you point me at PolyGlot help files that mention this epd test suite processing?

I have it workign but want to know things like - does it just stop when the best move is found or does it stay for the duration incase the engine changes its mind.

You can configure the behaviour yourself. The -min-time and -max-time parameters are used to set the minimum and maximum time for each position. The -depth-delta parameter is used to set how many iterations the engine has to keep the solution move as its best move before it is accepted (the default is 3, IIRC).

For instance:

Code: Select all

polyglot epd-test -epd wac.epd -min-time 3 -max-time 10 -depth-delta 5

This will make the program analyse each position in wac.epd for at least 3 and at most 10 seconds. The analysis of a position will be stopped before the 10 seconds are passed if the correct move has been found and kept for at least 5 iterations, and the program has been thinking for at least 3 seconds.

Tord

[ShaunBrewer]

Alessandro,

I will look at the source - in the test I ran lasdt night it did not wait (otherwise the test would not have finished)

Thanks again

Shaun

My bat file to run:

Code: Select all

copy GFruit-4bx.ini polyglot.ini
polyglot epd-test -epd Mats.epd -max-time 5 >Mats(5s-GFruit-4bx).txt
copy Rybka.ini polyglot.ini
polyglot epd-test -epd Mats.epd -max-time 5 >Mats(5s-Rybka).txt
copy GFruit-4bx.ini polyglot.ini
polyglot epd-test -epd Mats.epd -max-time 10 >Mats(10s-GFruit-4bx).txt
copy Rybka.ini polyglot.ini
polyglot epd-test -epd Mats.epd -max-time 10 >Mats(10s-Rybka).txt
copy GFruit-4bx.ini polyglot.ini
polyglot epd-test -epd Mats.epd -max-time 30 >Mats(30s-GFruit-4bx).txt
copy Rybka.ini polyglot.ini
polyglot epd-test -epd Mats.epd -max-time 30 >Mats(30s-Rybka).txt
copy GFruit-4bx.ini polyglot.ini
polyglot epd-test -epd Mats.epd -max-time 60 >Mats(60s-GFruit-4bx).txt
copy Rybka.ini polyglot.ini
polyglot epd-test -epd Mats.epd -max-time 60 >Mats(60s-Rybka).txt
copy GFruit-4bx.ini polyglot.ini
polyglot epd-test -epd Mats.epd -max-time 300 >Mats(5m-GFruit-4bx).txt
copy Rybka.ini polyglot.ini
polyglot epd-test -epd Mats.epd -max-time 300 >Mats(5m-Rybka).txt
copy GFruit-4bx.ini polyglot.ini
polyglot epd-test -epd Mats.epd -max-time 900 >Mats(15m-GFruit-4bx).txt
copy Rybka.ini polyglot.ini
polyglot epd-test -epd Mats.epd -max-time 900 >Mats(15m-Rybka).txt


Rybka.ini:

Code: Select all


[PolyGlot]

EngineDir = .
EngineCommand = Rybka.exe

Log = false
LogFile = xxx.log

[Engine]

Hash = 64


GFruit-4bx.ini :

Code: Select all

[PolyGlot]

EngineDir = .
EngineCommand = GFruit-4bx.exe

Log = false
LogFile = xxx.log

[Engine]

Hash = 64


Epd: (enhanced e.g. tab character and id syntax corrected on some rows)

Code: Select all

2r1k2r/pp2bp1p/1q3pp1/3P1b2/4N3/2Qp1N2/PP3PPP/R3R1K1 w k - bm d6; id MATS001;
r1b2rk1/2p1nppp/pp1q1n2/3p4/3P4/P1NBP3/1PQ1NPPP/R3K2R w KQ - bm e4; id MATS002;
2r1r1k1/1p1q1ppp/3p1b2/p2P4/3Q4/5N2/PP2RPPP/4R1K1 w - - bm Qg4; id MATS003;
2rr2k1/1b3ppp/pb2p3/1p2P3/1P2BPnq/P1N3P1/1B2Q2P/R4R1K b - - bm Rxc3; id MATS004;
8/pR4pk/1b6/2p5/N1p5/8/PP1r2PP/6K1 b - - bm Rxb2; id MATS005;
2r3k1/p2q1rpp/1p3pn1/3P4/4P2P/PQ4B1/6P1/2R2RK1 w - - bm d6; id MATS006;
3k4/8/3n2pp/pp1Pp3/P3P2P/3K2P1/8/3B4 b - - bm b4; id MATS007;
r1b2rk1/5ppp/p3p3/1pnpP3/5P2/2N3P1/PPP3BP/R4RK1 w - - bm Ne2; id MATS008;
2r2k2/1p3p1p/p1N3p1/3p4/8/1P2PP2/P5PP/2R2K2 w - - bm Rc5; id MATS009;
4rnk1/pp1q1ppp/2p4r/3p4/3P4/4P1NP/PPQ2PP1/R3R1K1 w - - bm b4; id MATS010;
r2q1rk1/pp2bppp/4pn2/3nN1B1/3P4/1B5Q/PP3PPP/3R1RK1 w - - bm f4; id MATS011;
2b1k2r/1p2q1pp/p1p5/P7/3r1P2/1Q1N4/1P1bP1BP/R4RK1 b k - bm Be6; id MATS012;
r1q2r1k/p2n2pp/1p1p1b2/2pP1p2/2P1pP2/1QB1N1P1/PP2P2P/R4R1K w - - bm Nxf5; id MATS013;
4r3/p1p2bk1/3p2p1/2p4p/4PR2/1P1B4/P1P3PP/6K1 b - - bm c4; id MATS014;
r2q1rk1/1p4pp/4p1n1/pNbpPp2/P1P5/3Q4/1P3PPP/R1B1R1K1 b - - bm Qh4; id MATS015;
b7/3knp2/p2p2pb/1p1Pp2p/1B2P1PP/1P1B1P2/P7/5NK1 b - - bm hxg4; id MATS016;
2n5/1b2k3/p2pB1p1/BpbPp1P1/4P3/1P4K1/P1N5/8 b - - bm Ba7; id MATS017;
4r1k1/pppb1pp1/1n6/3P4/2PN4/3B4/P1P4P/K5R1 b - - bm Na4; id MATS018;
4r1k1/1p3p1p/4b1p1/3p4/3P1P2/6P1/4B2P/2R3K1 b - - bm Bd7; id MATS019;
2r1k2r/5ppp/p1pRp3/8/8/4P3/PPP3PP/2KR4 b k - bm h5; id MATS020;
r3kb1r/2p3pp/p1n1p3/1pn1P3/8/1BPq4/P2N1PPP/R1BQ1RK1 w kq - bm Bc2; id MATS021;
2kr3r/pp1n4/2pb2q1/3pp2p/2P5/1B1P1R2/PP1NQ1PP/5RK1 b - - bm e4; id MATS022;
r1bq1rk1/ppp2pbp/6p1/8/4p3/BP2P3/P1P1BPPP/R2QK2R w KQ - am Bxf8; id MATS023;
r3k2r/1ppnqppp/p1pb4/4p3/4P3/P2P1P1P/1PPBNP2/R2QK2R b KQkq - bm Nc5; id MATS024;


Final results:

GFruit

Code: Select all

PolyGlot 1.4 by Fabien Letouzey
MATS001 1    1    1 -  6   0.08     44312  +0.91 d6 O-O dxe7 Rxc3 exf8=R+ Kxf8 bxc3 Bxe4 Rxe4 Qb2 Rd1 Qc2 Rd2 Qc1+ Re1 Qxc3
MATS002 1    2    2 -  7   0.38    195596  +0.68 e4 dxe4 Nxe4 Nxe4 Bxe4 Ra7 Bxh7+ Kh8 Bd3 Nc6 O-O Bg4 Rae1 Bxe2 Rxe2 Nxd4
MATS003 1    3    3 -  3   0.01      3570  +3.41 Qg4 Qb5 Qc4 Qd7 Qc7 Qb5 a4 Qxa4 b3 Rf8 Qxc8 Qxb3
MATS004 1    4    4 -  8   0.50    356706  +4.63 Rxc3 Bxb7 Rxg3 Rac1 Rh3 Rc2 Nf2+ Qxf2 Bxf2 Rcxf2 Rhd3 Rc2 Rd1
MATS005 1    5    5 - 20 389.95 341007708  +0.46 Rxb2 Nxb2 c3 Rxb6 c4 Rb4 a5 Nd3 cxd3 Rh4+ Kg6 Kf1 c2 Rc4 d2 Rc6+ Kf5 g4+ Kxg4 Rc4+ Kf5 Rc5+ Kg6 Rc6+ Kh7 Re6
MATS006 1    6    6 - 13  24.41  15774249  +1.95 d6 Rc5 Rxc5 bxc5 Rf5 Nf8 Rxc5 Ne6 Rb5 g6 Rb8+ Kg7 h5 gxh5 Qd5 Rf8 Rb7
MATS007 1    7    7 -  1   0.00         4  +0.30 b4 Bb3 Kc8 Bc4 Kd7 Bb3 Kc7 Bd1 Kb6 Bb3 Kc5 g4 Kb6 Bc4 Kc5
MATS008 0    7    8 - 15  35.28  27465646  +0.43 b4 Nd7 a4 bxa4 Nxa4 Ra7 Rf3 Rc7 Rc3 Rxc3 Nxc3 Bb7 Nd1 Rc8 Ne3 Rc3 Kf2 h6 Ke2 Bc6 Bf3 d4 Bxc6 Rxc6
MATS009 1    8    9 - 11   0.34    371266  +0.39 Rc5 bxc6 e4 dxe4 fxe4 Ke7 Ke2 Kd6 Ra5 Ra8 Kd3 Ke6 Ke3 f6
MATS010 0    8   10 - 14  78.67  48679998  -0.01 Rad1 a6 a3 Ne6 e4 Nf4 e5 f6 Re3 fxe5 dxe5 Rf8 Rd4 Kh8 Qc5 Qf7 Ne2 Nxe2+ Rxe2 Rg6
MATS011 1    9   11 - 13 198.65 117017026  +0.88 f4 Rc8 f5 Ne4 Bxe7 Qxe7 Rde1 Nef6 Kh1 h6 fxe6 Qxe6 Qg3 Rfd8 Bc4 Qb6 Bxd5 Rxd5
MATS012 1   10   12 -  7   0.20    122051  +1.37 Be6 Qc2 Be3+ Kh1 O-O Ra4 Qh4 Rxd4 Bxd4 Nc5 Bxc5 Qxc5 Rxf4
MATS013 1   11   13 -  6   0.09     58722  +0.42 Nxf5 Bxc3 Nxd6 Qb8 Nxe4 Bf6 Rab1 Qe8 Qf3 Rd8 e3 Qg6
MATS014 1   12   14 -  8   0.16    130967  +0.20 c4 bxc4 Re5 c3 Be6 g3 Ra5 Rf2 Ra3 e5 dxe5
MATS015 1   13   15 -  1   0.01       197  +0.78 Qh4 Qc2 Bb4 Bd2 Bxd2 Qxd2 Qxc4 Nd6 Qf4 Qxf4 Nxf4 Nxb7
MATS016 1   14   16 -  5   0.02      8860  -0.29 hxg4 fxg4 f5 g5 Bg7 Ne3 f4 Ng4 Ng8 Kf2 Bb7 Kf3
MATS017 1   15   17 - 17  16.88  15546294  -0.96 Ba7 Bg4 Bg1 Kf3 Bc5 Kg2 Ba7 Nb4 Bb6 Bxc8 Bxa5 Nc6+ Bxc6 dxc6 Bb6 Bxa6 b4 Kf3 Ke6 Bd3 d5 exd5+ Kxd5
MATS018 1   16   18 - 15  73.48  57210769  +0.63 Na4 Kb1 Nc5 Kb2 Re3 Kc3 Rh3 Rg2 Kf8 Kd2 g6 Kc3 Rh4 Nf3 Rh5 h4 b6
MATS019 0   16   19 - 21 508.77 445720637  +0.66 Bf5 Kf2 Rc8 Ra1 Rc2 Ke3 Rc3+ Kf2 Bd3 Bxd3 Rxd3 Ra4 Kg7 Rb4 Rd2+ Kf3 h5 h4 Rd3+ Kf2 b6 Kg2 b5 Rxb5 Rd2+ Kf3 Rxd4
MATS020 1   17   20 -  9   0.37    322229  -0.82 h5 R6d4 Rh6 Ra4 Ra8 Ra5 Ke7 Rg5 g6 Rc5 Rc8
MATS021 0   17   21 -  5   0.03     16509  +0.35 Qf3 Qxf3 Nxf3 Be7 Be3 Nxb3 axb3 Rf8 b4 g6 Rfd1 Rf5 Rdb1 Kf7 Rd1 Ke8 Ra2 Nxe5 Nxe5 Rxe5 Bf4 Rd5 Rxd5 exd5 Bxc7
MATS022 0   17   22 - 10   5.15   3107531  -0.35 Nb6 a4 dxc4 Nxc4 Nxc4 Bxc4 Kb8 a5 Rhf8 Rxf8 Rxf8 Rxf8+ Bxf8 Qxe5+ Bd6 Qe3 Qf5 a6 b5 Be6 Bc5 Bxf5 Bxe3+ Kf1
MATS023 1   18   23 -  7   0.07     53608  +0.08 O-O Qxd1 Raxd1 Re8 Bc5 Be6 f3 b6 Bd4 exf3 Rxf3 Rad8
MATS024 0   18   24 - 15 163.71  95871322  +0.56 g6 Bc3 Qh4 Qd2 O-O O-O-O Qxf2 f4 exf4 Rhf1 Qe3 Nxf4 Qxd2+ Kxd2 Rfe8 Kc1 Ne5 Bd4 f5


Rybka (latest 13d)

Code: Select all

PolyGlot 1.4 by Fabien Letouzey
MATS001 0    0    1 -  3   0.01       120  -0.39 Qxd3 O-O Qd4 Rfd8
MATS002 1    1    2 -  8   2.33    151688  +0.22 e4 dxe4 Nxe4 Nxe4 Bxe4 Nd5 O-O Re8 Bxh7+ Kf8 Be4
MATS003 1    2    3 -  3   0.01       256  +1.06 Qg4 Qb5 Qc4 Qd7 Qc7 Qb5 Rxe8+ Rxe8 Rxe8+ Qxe8 Qxb7 g6
MATS004 1    3    4 -  9   4.13    184680  +3.80 Rxc3 Bxb7 Rxg3 Rac1 Rgd3 Rc2 Ne3 Bg2
MATS005 1    4    5 - 18 653.09  98305632  +3.63 Rxb2 Re7 Rxa2 Nc3
MATS006 1    5    6 -  3   0.01       392  +1.65 d6 Nf8 Rc7 Rxc7 dxc7 Ne6 Rd1 Qc8
MATS007 1    6    7 -  3   0.00       200  +0.42 b4 g4 Kc7 Bc2 Kb6 Bd1 Kc5 h5 g5 Bb3 Ne8 Bd1
MATS008 0    6    8 -  3   0.01       200  +0.45 b4 Nd7 a4 bxa4 Nxa4 Ra7 c4 dxc4 Rfd1 Rc7 Rac1 g5
MATS009 1    7    9 - 13   1.85    573592  +0.41 Rc5 bxc6 Ke2 Ke7 Ra5 Ra8 Kd3 f5 b4 Kd6 Kd4 g5
MATS010 0    7   10 - 15 372.90  31839728  -0.09 Rad1 Rg6 Rd3 Ne6
MATS011 1    8   11 - 12 145.13   6291760  +0.73 f4 Qd6
MATS012 1    9   12 -  7   1.10     49472  +1.38 Be6 Qc2 Be3+ Kh1 O-O Qc3 Rc4
MATS013 0    9   13 - 16 156.26  18153432  +0.13 Qb5 Bxc3 bxc3 a6 Qc6 Qd8 a4 Rc8 Qb7 a5 Kg2 Rc7 Qa6 Qf6
MATS014 1   10   14 -  8   0.52     75088  +0.28 c4 bxc4 Re5 Rf1 Rc5 Rb1 Bxc4 a4
MATS015 1   11   15 -  3   0.02       600  +0.75 Qh4 Be3 Bb4 Re2 Nxe5
MATS016 1   12   16 -  3   0.00       160  -0.33 hxg4 fxg4 Bc1 g5 Bb2 Kf2 Bd4+ Kf3
MATS017 0   12   17 -  5   0.02      2208  -1.47 Bg1 Bg4 Nb6 Kg2 Bc5 b4 Nc4 bxc5 Nxa5 c6 Nxc6 dxc6 Bxc6 Bf3
MATS018 1   13   18 -  4   0.03      2200  +0.37 Na4 h4 Re5 Kb1 g6 Kc1 Nc5
MATS019 0   13   19 - 13   7.20   1374120  +1.08 Ra8 Kf2 Ra2 Ke3 Rb2 Rc5 Kg7 Ra5 Kf6 Rb5 Rxb5 Bxb5 Bf5 Kf3
MATS020 1   14   20 -  6   0.09     15120  -0.88 h5 Rd7 Rh6 b3 Rg6 g3 Rf6 Ra7
MATS021 0   14   21 -  3   0.01       352  +0.15 Qf3 Qxf3 Nxf3 Nxb3 axb3 Be7 Rd1 Kf7 Kf1 Rhd8 Rxd8 Bxd8 Be3 a5
MATS022 1   15   22 -  5   0.21      7488  -0.14 e4 dxe4 d4 Bc2 Rhg8
MATS023 1   16   23 -  3   0.01       248  +0.05 O-O Re8 Qxd8 Rxd8 Rad1 Be6 Be7 Re8
MATS024 1   17   24 - 17 1129.04  98560192  +0.61 Nc5 Be3 Qf6 Rg1 Ne6 Ng3 g6 Qe2 Nf4 Qf1 O-O-O O-O-O Ne6


Trying to see if the test suit is correct before I used it generally.

When I have any more answers I will post the syntax and confirmation regarding the test suite

Thanks again

Shaun

[ShaunBrewer]

-- duplicated post removed --

[ShaunBrewer]

Tord,

Just spotted your post - overlapped mine - no my typing is not that slow - I got sidetracked.

Anyway - thanks - I now understand how to use this correctly!

All the best

Shaun