Quicktest with WB. engines' results

[Mike Scheidl]

This is about the tactical strength of WinBoard engines and other engines. Tactics is still the main component of a chess program's strength & weaknesses profile. Nowadays - assuming that an engine has reached a certain level of that which I call "tactical basis" - it is not as decisive anymore as it was i.e. in the 1980ies (when chess progs were primitive, compared to today). But when an engine does not reach that kind of "mediocre" (at least) combinative speed, it cannot prevail, just for that reason. So, with a tactical test like the Quicktest ? M.Scheidl 2002 we may not be able to tell the top engines from the near-to-top engines, but we can tell if an engine has the tactical basis required for good to best general performance, or if there is concern that it may fail often due to combinative weakness even when knowledge and positional play are good.

I admit that I've often tested the UCI versions of many (WB.) engines.

My observation during testing many engines in the Quicktest is, that there is a "typical" number of solutions which many very good freeware engines, and also commercial oldies reach. On an AthlonXP@1200 MHz that is 15 or 16 solutions from a total of 24, IOW ~2/3. Engines which do not achieve 14 solutions, may still be quite good in general gameplay, but I'd say it are rare exceptions when they would reach the top-20. On the other end of the bandwidth, top freeware and commercial engines solved up to 21 on that hardware.

I have a rating system for the QT. which includes the solving times too, but I don't insist that it is perfect or even logical I'll give these ratings in brackets. I didn't test most of the current top commercial engines, i.e. none of the "8" versions. From all those I've tested, Hiarcs 9 performed best with 21(434). From the oldies, Nimzo 8 19(380) or even Hiarcs 7.32 19(366) have good results. The best freeware engine among my results so far, has Ruffian 1.0.0 (WB) with 19(373). For comparison, Tiger 14 achieved only 17(333) (not surprisingly, Gambit Tiger 2.0/aggressiv reached a better rating although solving one less, 16(363)). From the newer releases: Pro Deo 1.0 (Rebel stile) 18(368), or Pharon 3.1 15(302). Pharaon solved two more shortly after the time limit which is 1 minute per position.

An interesting result recingly was LiGo Revival 1.0.0.13 and the update ..14, when a forward pruning bug was fixed. That immediatly resulted in two more solutions and +26 QT. ratings points. So, a test like that can also be used to tell when the search has been improved (when anything else has remained more or less the same). Then, more and/faster solutions will happen while the "solving profile" in general is very similar - when the improvement was successfull.

The complete results are in an XLS file:

Quicktest results (48 kb zipped)

(People without Excel can download a viewer for free, from Microsoft's website)

1n1r1rk1/ppq2ppp/3p2b1/3B1NP1/4PB1R/bP2P2P/P1P5/3KQ1R1 w - - bm Qc3; id Quick-01;
1q6/r4pbk/1r1p2pp/B2Pn3/Q2NP3/1p3P2/6PP/1R3RK1 b - - bm Rxa5; id Quick-02;
3Q4/3p4/P2p4/N2b4/8/4P3/5p1p/5Kbk w - - bm Qa8; id Quick-03;
4qrk1/3nppb1/R1Np2p1/3P2P1/1Pr5/4B3/5Q1P/5R1K w - - bm Ra8; id Quick-04;
r3r3/2R2pk1/p2p1bpp/3P4/q2pQ3/5N1P/5PP1/1R4K1 w - - bm Rxf7+; id Quick-05;
r1b1Rbk1/pp3p2/2npN2p/2qp2p1/8/1QPB3P/PP3PPB/6K1 b - - bm Bxe6; id Quick-06;
r5k1/Rb4p1/2q2pBp/1pp5/1b4QN/1P2P2P/5PP1/6K1 w - - bm Rxb7; id Quick-07;
3R4/5r1p/5ppk/8/1Q3PPq/5P2/6K1/8 w - - bm Rg8; id Quick-08;
2kr3r/ppp3pp/2pbbn2/4N3/3Pp3/2P3Pq/PP1NQP1P/R1B2RK1 w - - am Nxe4; id Quick-09;
2r3k1/pp1bpp1p/3p1npQ/q1r5/4P1P1/2NR1P2/PPP1N3/2K4R w - - bm g5; id Quick-10;
r1b2rk1/pp3p2/2p2bpQ/8/1q1P4/2N2N2/Pn3PPP/1B1RR1K1 w - - bm Bxg6; id Quick-11;
r2qk2r/1p1bbp2/1P2p3/p2pPp2/n2N1N1p/3PB3/5QPP/R4RK1 w kq - bm Rxa4; id Quick-12;
3r1n1r/1p2q1k1/p1p1P1p1/3n4/5Pp1/P5N1/1P3QP1/1BR1R1K1 w - - bm Bxg6; id Quick-13;
r2q1rk1/p1p3pp/b2bp3/2pp4/6p1/2NPPN2/PPP2PP1/R1BQR1K1 w - - bm Ne5; id Quick-14;
r2qr1k1/p2b1ppp/5n2/2pp4/5b2/NP6/PBP1NPPP/R3QRK1 b - - bm Bxh2+; id Quick-15;
3k4/p7/K3BP2/8/7p/8/2P4P/8 w - - bm Kb7; id Quick-16;
rq4k1/pp1nrppp/4bn2/6R1/3QP3/P4PN1/4B1PP/2B2RK1 w - - bm Rxg7+; id Quick-17;
2r4k/pb2q2P/1p6/3Pp3/4p3/1P2R3/PBrQ2PP/5RK1 w - - bm Qb4; id Quick-18;
5k2/6p1/2p2p2/P7/1Q6/2P1pqPP/7K/8 b - - bm c5; id Quick-19;
rnbq1b1r/ppp1p1pp/1n1p2k1/4P1N1/8/5Q2/PPPP1PPP/RNB1K2R b KQ - bm Qe8; id Quick-20;
r1b1kb1r/2q2ppp/p2ppP2/1pn3P1/3NP3/2N2Q2/PPP4P/2KR1B1R w kq - bm Bxb5+; id Quick-21;
4r2k/3n3p/2q3p1/2p1p1Q1/1pP1P3/1P6/5PP1/R2B2K1 b - - am Qxe4; id Quick-22;
r3r1k1/1Bp1qppp/3p1n2/pNb5/2P5/PQ6/1P3PPP/R2R2K1 b - - bm Ng4; id Quick-23;
3B4/1R3p1k/2p4p/2Pp3r/3P4/4Q1K1/6P1/3b1q2 w - - bm Bf6; id Quick-24;

(two are avoid move pos.)

Quicktest, english description

[Klaus Friedel]

Maybe you should change Nr. 16 because e6d7 mates as well (Mate in <= 16)

Regards,

Klaus Friedel

[Anonymous]

1n1r1rk1/ppq2ppp/3p2b1/3B1NP1/4PB1R/bP2P2P/P1P5/3KQ1R1 w - - bm Qc3; id Quick-01;

I don't have much to discuss here. But one formal thing, you should quote the ids.

1n1r1rk1/ppq2ppp/3p2b1/3B1NP1/4PB1R/bP2P2P/P1P5/3KQ1R1 w - - bm Qc3; id "Quick-01";

Regards,
Dieter

[Mike Scheidl]

Maybe you should change Nr. 16 because e6d7 mates as well (Mate in <= 16)

Nope this is about finding the shortest mate. 2nd best moves are not interesting in this test, only the best.

Btw. the trick in No. 16 is to find the "cage" idea around Black's king.

[Uri Blass]

Finding the shortest mate faster in case of an obvious win is not important for playing strength so it is better to avoid tests like that if your target is to estimate the tactics as component in playing strength.

Programs may not find shortest mate faster because of some pruning that only happen in won position and irrelevant to playing strength.

Uri

[Mike Scheidl]

Finding the shortest mate faster in case of an obvious win is not important for playing strength

Sorry, I don't care. I want the best move.

P.S.: Don't you realise the cage idea??

[Ross Boyd]

3r1n1r/1p2q1k1/p1p1P1p1/3n4/5Pp1/P5N1/1P3QP1/1BR1R1K1 w - - bm Bxg6;id Quick-13;

Hi Mike,

I noticed Arena 1.092 misreads the bm tag for Quick-13...
It needs a space after "Bxg6;" like this...

3r1n1r/1p2q1k1/p1p1P1p1/3n4/5Pp1/P5N1/1P3QP1/1BR1R1K1 w - - bm Bxg6; id Quick-13;

BTW...

TRACE 1.30 scored 15/24 for a result of 286.
TRACE 1.31 scored 18/24 for a result of 335.
TRACE 1.32beta scored 19/24 for a result of 353.

Thanks for posting your test-suite,

Ross