Winboard Forum

Greetings all. It's been a few years! But, I've dusted off the old SharpChess code, and decided to dive back in

I've upgraded all the code to C# DotNet 4, and uploaded it to https://github.com/PeterHughes/SharpChess in the hope that I'll get a few open source developers interested in the project

Anyway, in the past, I'd tested SharpChess manually, and I guess also by entering the excellent tournaments at http://wbec-ridderkerk.nl/. This time around, one of the things I'm most excited about is creating unit tests to more scientifically measure improvements in playing strength.

To this end, I'm hoping to use standard, pre-created test positions, to test whether SharpChess is performing correctly, and to what level of competence.

My problem is, that despite my best efforts! , SharpChess is still a relatively weak engine, and usually only searches between 6-8 plys of depth in 15-30 second search . But, many of the computer test suites expect solutions that require deep searches, or are designed to be run over long periods.

What I really want is a set of more shallow chess tests, perhaps in the in the 6 ply range, which can be executed in a few seconds. I could then set these up as unit tests, and maybe run a dozen or so, to quickly test whether small tweaks to SharpChess engine are having the desired effect!

I wonder if anyone here knows of tests positions that would meet this criteria?

Many thanks
Peter Hughes

I just looked at you posting and thought about it a little.

I have some test positions that I regularly run after some changes, to ensure that I haven't broken the engine with a change. I run them from a batch file
and I have some test at increasing depth. When I reach a depth that starts to take long enough that I get impatient, I have PAUSE statements in the batch file to allow me to duck out there if I want to get on with development.

But my tests are either extremely shallow - eg
depth one: test that queening a pawn is prefered to taking a minor piece, etc
depth three: find king and queen knight fork, and pin and capture in endgame position, skewers, etc
or possibly deeper but very specific, eg a pawn needing to run immediately to queen, or a king needing to chase immeidately to prevent an enemy pawn queening.

The tests I have of the depth you mention are just a few checkmates that I have taken from my engines games, mate in 3, mate in 4, and mate in 5.
There are plenty of those around, they are easy to create by just taking finished games and going back a few moves, and I don't believe they are that useful for a couple of reasons. 1. They are usually from an already "clearly" won position. Since checkmate is actually possible I suppose most of the alpha-beta cutoffs are going one way in some sense.

I also have some sanity tests with AVOID MOVE (am) used in EPD files instead of BEST MOVE (bm).
They are on the lines of simply testing that pieces are not thrown away needlessly, eg king not stepping away from its pawn or rook in certain KPK and KRK endgame.

Anyway, for whatever reason, I have noticed that the checkmate tests, the basic sanity tests and the chess tactics test are complemetary. Some coding errors will leave all the checkmates working fine, but throw up errors in pawn race tests, and vice versa.

The reason I don't have many test around depth 6 is difficulty in seeing the chess logic for myself clearly - you soon get into depths with many combinations and you start wondering if there isn't a deeper insight that means the less obvious move is actually better. I have seen this with well known test positions (Bratco-Kopec, excuse errors if I've remembered the names incorrectly), and running Stockfish and Houdini on them. When these engines start disagreeing with some standard analysis at depth 19, who is really to say whether they have found a deeper insight, or merely failed to spot the solution.

What did come to mind thinking about your question was a file that I have downloaded called ECM.EPD I think. As I understand it, the positions are from a book called The Encyclopedia of chess middlegames. I then looked at the file and decided that many of the positions are actually deeper than what you are asking for. But there are nearly a thousand positions and you are not looking for that many, so chances are there are enough positions in there of the right depth.
So which are the right ones?

I played a little with a Houdinin and ECM.EPD and the first few positions seem to be a mixture of things Houdini spots in 5 ply, 7 ply and 9 ply. It does seem to be biased toward the deeper problems, but it seems there will be enough of the 5 or 7 ply type problems to satisfy you if you have the patience to click analyse with a good fast engine to classify a problem and edit it into your own specific file if it meets your criteria.

With nearly 1000 problems I don't intend to continue looking at them for too long but Houdini found positions 7 and 9 in 5 ply, and positions 2 and 3 in 7 ply searches.

Code: Select all

3rr1k1/pb3pp1/1p1q1b1p/1P2NQ2/3P4/P1NB4/3K1P1P/2R3R1 w - - bm Rxg7+; id "ECM.007";
3r4/1b2k3/1pq1pp2/p3n1pr/2P5/5PPN/PP1N1QP1/R2R2K1 b - - bm Rxh3; id "ECM.009";

7r/1p2k3/2bpp3/p3np2/P1PR4/2N2PP1/1P4K1/3B4 b - - bm Bxf3+; id "ECM.002";
4k3/p1P3p1/2q1np1p/3N4/8/1Q3PP1/6KP/8 w - - bm Qb5; id "ECM.003";

Nothing much seems to be found on an even ply search like 6. This is probably because at even depth the opponent is on the move. Sometimes a search does thro up the good move on an even ply during analysis. I guess this is where there are a few candidate moves and an opponent move refutes the current best move, allowing another to rise to the top.

I may find a few more positions in the ECM.EPD file that are in the 5-7 ply range later. It should be of use to me too. I like to have things ordered by depth of search as there is no point in trying a long deep search with a clearly broken engine. So I do the short search problems first.

I've now run the engine Houdini on the first 30 ECM positions to see at what depth it finds the best move. 4,5,6,7,8 respectively.
I'll not break up the text below labelling the test specifically, in case you want to copy past them.
And I have not tried any other engine on this yet, so it could be that search extensions etc in the one engine I used are distorting the depth at which the correct move is found.

Code: Select all

2q1r1k1/1ppb4/r2p1Pp1/p4n1p/2P1n3/5NPP/PP3Q1K/2BRRB2 w - - bm f7+; id "ECM.001";
r3kb1r/pp2pppp/3q4/3Pn3/6b1/2N1BN2/PP3PPP/R2QKB1R w KQkq - bm Nxe5; id "ECM.015";
r1b1k3/5p1p/p1p5/3np3/1b2N3/4B3/PPP1BPrP/2KR3R w q - bm Rxd5; id "ECM.017";
r1r3k1/p3qpp1/b1P4p/3p4/3Nn3/4P3/P1Q2PPP/1BR1K2R b K - bm Qb4+; id "ECM.022";
r1b5/4k3/p7/3p1n2/3Bp3/2P2r1P/PPBK1P2/4R2R w - - bm Bc5+; id "ECM.026";
r4rk1/1b3Npp/p7/1p3Q2/3P4/1B2q3/P5PP/3n1R1K b - - bm Bxg2+; id "ECM.027";

3rr1k1/pb3pp1/1p1q1b1p/1P2NQ2/3P4/P1NB4/3K1P1P/2R3R1 w - - bm Rxg7+; id "ECM.007";
3r4/1b2k3/1pq1pp2/p3n1pr/2P5/5PPN/PP1N1QP1/R2R2K1 b - - bm Rxh3; id "ECM.009";
rnb2rk1/pp2np1p/2p2q1b/8/2BPPN2/2P2Q2/PP4PP/R1B2RK1 w - - bm Nd5; id "ECM.013";
r2q1rk1/p3b1pp/2p5/1pn5/1n1Bp1b1/1P6/PQ1PPP2/2RNKBNR b K - bm Bxe2; id "ECM.030";

1rr1nbk1/5ppp/3p4/1q1PpN2/np2P3/5Q1P/P1BB1PP1/2R1R1K1 w - - bm Bxa4; id "ECM.020";

7r/1p2k3/2bpp3/p3np2/P1PR4/2N2PP1/1P4K1/3B4 b - - bm Bxf3+; id "ECM.002";
4k3/p1P3p1/2q1np1p/3N4/8/1Q3PP1/6KP/8 w - - bm Qb5; id "ECM.003";
r7/5kp1/2p1p2p/1p1n3P/2rP4/2P3R1/PK2RPP1/2B5 b - - bm Rxc3; id "ECM.021";
rn3rk1/4bppp/1q2p3/p2pP3/8/1PN2B1P/P4PP1/2RQ1RK1 w - - bm Bxd5; id "ECM.023";
3r2k1/1q1P1ppp/r2R2n1/p4Q2/1ppB2R1/6P1/PP3PP1/6K1 w - - bm Rgxg6; id "ECM.029";

r1b1r1k1/p1p3pp/2p2n2/2bp4/5P2/3BBQPq/PPPK3P/R4N1R b - - bm Bg4; id "ECM.008";
5rk1/7p/p1N5/3pNp2/2bPnqpQ/P7/1P3PPP/4R1K1 w - - bm Ne7+; id "ECM.012";


Now I've just tried my own engine on the positions that Houdini reportedly found solutions for in 4 moves.

The text below is from an output EPD file from my engine after running "pfts 4".

2q1r1k1/1ppb4/r2p1Pp1/p4n1p/2P1n3/5NPP/PP3Q1K/2BRRB2 w - - acn 109510; acs 1; bm f7+; ce -97; id "ECM.001"; pm Qc2;
r3kb1r/pp2pppp/3q4/3Pn3/6b1/2N1BN2/PP3PPP/R2QKB1R w KQkq - acn 17158; acs 1; bm Nxe5; ce 367; id "ECM.015"; pm Bb5+;
r1b1k3/5p1p/p1p5/3np3/1b2N3/4B3/PPP1BPrP/2KR3R w q - acn 44964; acs 1; bm Rxd5; ce 20; id "ECM.017"; pm Bd2;
r1r3k1/p3qpp1/b1P4p/3p4/3Nn3/4P3/P1Q2PPP/1BR1K2R b K - acn 17358; acs 1; bm Qb4+; ce -55; id "ECM.022"; pm Bc4;
r1b5/4k3/p7/3p1n2/3Bp3/2P2r1P/PPBK1P2/4R2R w - - acn 37902; acs 1; bm Bc5+; ce 188; id "ECM.026"; pm Bc5+;
r4rk1/1b3Npp/p7/1p3Q2/3P4/1B2q3/P5PP/3n1R1K b - - acn 117879; acs 1; bm Bxg2+; ce 265; id "ECM.027"; pm Bxg2+;

Searching 4 ply deep, my engine finds the right move (ie proposed move pm from the engine equals best move bm in the input EPD file) to just the last two positions.

A search 5 ply deep gives

2q1r1k1/1ppb4/r2p1Pp1/p4n1p/2P1n3/5NPP/PP3Q1K/2BRRB2 w - - acn 287684; acs 1; bm f7+; ce -64; id "ECM.001"; pm f7+;
r3kb1r/pp2pppp/3q4/3Pn3/6b1/2N1BN2/PP3PPP/R2QKB1R w KQkq - acn 508302; acs 2; bm Nxe5; ce 377; id "ECM.015"; pm Bb5+;
r1b1k3/5p1p/p1p5/3np3/1b2N3/4B3/PPP1BPrP/2KR3R w q - acn 250633; acs 1; bm Rxd5; ce 30; id "ECM.017"; pm Bd2;
r1r3k1/p3qpp1/b1P4p/3p4/3Nn3/4P3/P1Q2PPP/1BR1K2R b K - acn 305474; acs 1; bm Qb4+; ce -35; id "ECM.022"; pm Qb4+;
r1b5/4k3/p7/3p1n2/3Bp3/2P2r1P/PPBK1P2/4R2R w - - acn 181840; acs 1; bm Bc5+; ce 213; id "ECM.026"; pm Bc5+;
r4rk1/1b3Npp/p7/1p3Q2/3P4/1B2q3/P5PP/3n1R1K b - - acn 261868; acs 1; bm Bxg2+; ce 285; id "ECM.027"; pm Bxg2+;

So it solves ECM.001 and ECM.022 in addition to ECM.026 and ECM0.27

Skipping to 7 ply (because we only move alternate moves really) move engine gives ....

2q1r1k1/1ppb4/r2p1Pp1/p4n1p/2P1n3/5NPP/PP3Q1K/2BRRB2 w - - acn 2531144; acs 8; bm f7+; ce 216; id "ECM.001"; pm f7+;
r3kb1r/pp2pppp/3q4/3Pn3/6b1/2N1BN2/PP3PPP/R2QKB1R w KQkq - acn 32579517; acs 129; bm Nxe5; ce 712; id "ECM.015"; pm Nxe5;
r1b1k3/5p1p/p1p5/3np3/1b2N3/4B3/PPP1BPrP/2KR3R w q - acn 11673929; acs 43; bm Rxd5; ce 115; id "ECM.017"; pm Nf6+;
r1r3k1/p3qpp1/b1P4p/3p4/3Nn3/4P3/P1Q2PPP/1BR1K2R b K - acn 24940857; acs 87; bm Qb4+; ce 10; id "ECM.022"; pm Qb4+;
r1b5/4k3/p7/3p1n2/3Bp3/2P2r1P/PPBK1P2/4R2R w - - acn 1998104; acs 8; bm Bc5+; ce 223; id "ECM.026"; pm Bc5+;
r4rk1/1b3Npp/p7/1p3Q2/3P4/1B2q3/P5PP/3n1R1K b - - acn 2653242; acs 8; bm Bxg2+; ce 380; id "ECM.027"; pm Bxg2+;

So it has arguably found the right move to ECM.015 too, leaving just ECM.017 unsolved.

Thank you Crystalclear. That's really useful information. I'll get those tests a go and report back on how I get on.

@CystalClear

Here are the results the six tests in SharpChess.

Depth is the number of cycles of iterative deepening before the expected "best move" was proposed.
Node is the total number of positions searched.

Code: Select all

Test Depth  Nodes
01   2      1,251
15   6      288,261
17   7      482,422
22   2      398
26   2      462
27   3      5006


If you say that Houdini found the best move to all these positions @ depth < 4, then I need to do some work in positions 15 and 17.