FRC perft and GUI

[Volker Annuss]

12rkr3/5PP1/8/5Q2/5q2/8/5pp1/2RKR3 w KQkq - 0 1

the leading '1' seems to be an error.

Sorry, I just made a quick test without looking at the position and Hermann only writes an error message after the command "debug on".

[Jaime Benito de Valle]

A few more. Depth 7 this time

1/960 bbqnnrkr/pppppppp/8/8/8/8/PPPPPPPP/BBQNNRKR w KQkq - 3249555919 nodes
2/960 qbbnnrkr/pppppppp/8/8/8/8/PPPPPPPP/QBBNNRKR w KQkq - 3049768268 nodes
3/960 qbnnbrkr/pppppppp/8/8/8/8/PPPPPPPP/QBNNBRKR w KQkq - 2899201437 nodes
4/960 qbnnrkbr/pppppppp/8/8/8/8/PPPPPPPP/QBNNRKBR w KQkq - 3069206061 nodes
5/960 bqnbnrkr/pppppppp/8/8/8/8/PPPPPPPP/BQNBNRKR w KQkq - 3116904202 nodes
6/960 qnbbnrkr/pppppppp/8/8/8/8/PPPPPPPP/QNBBNRKR w KQkq - 3044936259 nodes
7/960 qnnbbrkr/pppppppp/8/8/8/8/PPPPPPPP/QNNBBRKR w KQkq - 2914861403 nodes
8/960 qnnbrkbr/pppppppp/8/8/8/8/PPPPPPPP/QNNBRKBR w KQkq - 2952652080 nodes
9/960 bqnnrbkr/pppppppp/8/8/8/8/PPPPPPPP/BQNNRBKR w KQkq - 3159359007 nodes
10/960 qnbnrbkr/pppppppp/8/8/8/8/PPPPPPPP/QNBNRBKR w KQkq - 3015788917 nodes
11/960 qnnrbbkr/pppppppp/8/8/8/8/PPPPPPPP/QNNRBBKR w KQkq - 3019911426 nodes

[Anonymous]

Hi, what hardware did you use for this smirf perft? Seems pretty fast without caching.

[Reinhard Scharnagl]

Hi Jan,

well, I have posted values with and without caching. The values without for sure could be faster, if not additionally calculating a detailed statistic. But indeed I am still satisfied with my (legal only) move generator. Thus I am actually working on other targets like improving my evaluation function.

It is all done using a P4 2.8 GHz under Windows XP pro.

Reinhard.

[Jaime Benito de Valle]

Hi Jan,

Hi, what hardware did you use for this smirf perft? Seems pretty fast without caching.

If Smirf uses a legal-only generator, it doesn't need to do/undo the leaf nodes in order to count them; just return the number of moves. Pseudo-legal generator based perfts have to try all moves in order to count the legal ones. This makes the pesudo-legal perft far slower because most of the total nodes are leaf nodes (5 - 30 times slower maybe?)
Although this is a great advantage in a perft count, in a normal search those moves have to be made anyway in order to evaluate the leaf positions, so it's not necessarily a clear advantage when it comes to play chess.

[Reinhard Scharnagl]

Hi Jaime,

... makes the pesudo-legal perft far slower ...

you are not right with that for Smirf because of two reasons:

a) in the following repeatedly shown example you easily could have noticed, that there is e.g. also an information, whether a move is a check (single or double threat) or not. This is a lot more of information than provided by a usual legal / pseudo-legal move generator during Perft.

b) generating legal moves only is simply a shifting of efforts instead of checking the legality of moves by expanding an additional ply to do comparable verifications during the generation process. Thus there is no kind of cheating doing things the way like I do in Smirf.

The conclusion from all that is that Smirf's Perft results are fully comparable without any restriction to results from pseudolegal move generators.

Code: Select all

FEN: rbbknnqr/pppppppp/8/8/8/8/PPPPPPPP/RBBKNNQR w KQkq - 0 1

  +-*--b--c--*--e--f--g--*-+ MS Vis. Studio C++ Vers. 13.10
8 |[r][b][b][k][n][n][q][r]| (Compilation: Jan 23 2005)
7 |[p][p][p][p][p][p][p][p]|
6 |   :::   :::   :::   :::| Perft Testseries
5 |:::   :::   :::   :::   |
4 |   :::   :::   :::   :::| (without caching)
3 |:::   :::   :::   :::   |
2 |<P><P><P><P><P><P><P><P>| Smirf Test No.: 00
1 |<R><B><B><K><N><N><Q><R>|
=>+-*--b--c--*--e--f--g--*-+ Break Time 10.0 Sec.

Ply     Nodes    all (x)   (e.p.)    all (+) all (++)    Promot.    Castl. Sec.
-------------------------------------------------------------------------------
1          20          0        0          0        0          0         0    0
2         400          0        0          0        0          0         0    0
3        9030         98        0          6        0          0         0    0
4      202966       3107        0        253        0          0         0    0
5     5044135     134425      256      22055        0          0         0  0.2
6   124381396    4165851     5278     637078        9          0         0  5.0
7  3323146824  145775591   322141   28718882      500          0       624  126
-------------------------------------------------------------------------------

Regards, Reinhard.

P.S.: what I had forgotten to mention is, that there are no additional "tricks" in making and unmaking the moves. That is, Smirf updates material balance values, castling rights, position keys for the transposition table etc. just like it would act during a regular calculation process. The routines are 100% identic and not somehow "purified" for the Perft calculation.

P.P.S.: you should also be aware of the fact, that Smirf is not running on a fixed 8x8 board geometry. That implies that Smirf has to dynamically adapt on a 8x8 or 10x8 board structure and must be able to handle all combinations of the pieces of the extended Capablanca piece set including Chancellor and Archbishop, which makes e.g. also the pieces values variables instead of constants. Overmore, for to cover a huge range of possible variants, Smirf supports positions, with as much pieces a board ever could hold, and it is generic supporting the Fischer castlings defined on variable places compared to the standard approach. But the size of Smirf's engine actually is just 48K (including about 1/3 test routines).

All that are a lot of details, which normally would slow down any move generation process. It would be more realistic for me to demand such properties from other chess engines, before their Perft results could be compared to Smirf at equal chances.

[Volker Annuss]

Hello Reinhard,

Thus there is no kind of cheating doing things the way like I do in Smirf.

in my opinion there is not even a competition in having fast perft commands. Its just a test command for verifying move generation and (un-)making moves. So cheating with perft is impossible.

Even if an engine would already know the perft values for some precalculated positions, I would not call that cheating. It just speeds up tests and this may make sense for some uninteresting branches in an interesting position.

Greetings,
Volker

[Reinhard Scharnagl]

Hi Volker,

... there is not even a competition in having fast perft commands ...

that is of course true. But I answered on

... If Smirf uses a legal-only generator, it doesn't need to do/undo the leaf nodes in order to count them; just return the number of moves. Pseudo-legal generator based perfts have to try all moves in order to count the legal ones. This makes the pesudo-legal perft far slower because most of the total nodes are leaf nodes (5 - 30 times slower maybe?) ...

which stated, that the values reached by Smirf would not be comparable to figures gained by other methods. But as I have explained, they really are.

(P.S.: Jaime obviously is forgetting, that securing the legality of the generated moves and providing additional information whether being a check threat or not, also needs a kind of internal expansion and testing.)

If I would be interested in creating Perft records, I would not have tried as I have done with Smirf, doing a lot of things, which are not at all necessary in a more traditional 8x8 approach. Nevertheless I do not like it at all, when Smirf's results would be made look smaller than they are.

Regards, Reinhard.

[Jaime Benito de Valle]

hi Reinhard,

I wasn't acusing you of cheating, using tricks or anything like that. This is not a competition. You're using a method which, regarless of its effectivity in general, and in my opinion, is very fast for perft calculations.
I wrote that post because I remember the first time I saw the perft of an engine with a legal-only move generator I almost gave up programming because it was a lot faster than mine, and I thought I was just useless because I could never make my code 10 times faster (or whatever the exact figure was). Then some people exaplained me what I just wrote above about comparing the perft of the two types of move generators (Carlos, Pepito's programmer was the fist one to tell me).

After that, I decided to change my move generator into a legal-only one, and my perft became far far much faster than the previous one. Unfortunately, the nps in real games dropped, and I decided to go back to the pseudo-legal ones. Movei, if I recall correctly, uses the legal moves only generator. I guess it works well for some people, but not for others. I'm sure there are lots of very strong engines out there using the legal-only system.

Without caching, you display 3323146824 nodes in 126 seconds. That is 26374 knps!!! I've talked to several people with strong fast engines using pesudo-legal generator, and for a processor like yours they get something in the range of 2000-5000 knps. Movei gets about 18000-20000 knps in my computer.

Again, I was just trying to give Jan a different perspective, so he doesn't get discouraged about programming (if that's what he wants), not critisizing you in any way.

I'm really posting here because I want to get and provide help in FRC & normal chess programming, so accept my apology is you felt insulted.

Best regards,

[Reinhard Scharnagl]

Hi Jaime,

there is no reason for placing an apology, but I think it is important to make some things clear. Indeed a legal move generator could be faster in Perft, if it would NOT additionally supply that move with information on being a single or double check threat. This information is absolutely unnecessary for Perft. You could moreover avoid an updating of material balance and hash entry keys, which would change the make and unmake of moves being remarkable faster. This point is independent from the legality of the created move - Smirf nevertheless has no special move procedures to link for Perft.

What remains as an asymmetry when comparing Perft results of legal-only and pseudo-legal move generators is the following: the difference of the costs to make and unmake the moves of the last ply compared to the effort of checking the move legality and check threats ad hoc.

I can only estimate that difference. So I remember to have made Perft runs making and unmaking the moves of the last ply (without any need). And the time required has about doubled. But then I internally have done a lot of work twice. So a serious estimate for making Perft results comparable would by using a factor between 1.0 and 1.33 - as far as I prosume.

But Smirf is working on an untypical structure covering 10x8 and 8x8 boards, up to 80 pieces, variable piece values, more pieces types, support for Fischer castlings etc.. So I am finally convinced the results are very comparable to other engines, playing traditional chess only.

I did not make such a fast generator because I would have believed by that to create a very fast chess program. Smirf actually is (in my point of view) very slow, because its evaluation functuion is very expensive. It is not working with piece / coordinate related weight tables, instead it trys to evaluate the interdependencies between all pieces on the board. My intention has been, to create a highly effective data structure, making me flexible for the rest of my chess programming time. As I have mentioned earlier that structure has nothing to do with bitboards. It could be regarded as a flat two dimensional view on a double linked sorted piece list having entries with bit encoded information of the piece gaits. That structure is the result of about twenty years work where I from time to time have thought on that problem.

So for me it is more important to be able to work with my own ideas and solutions instead of copying and patchworking from others. There is no need to be among the writers of the fastest move generators. Everybody who has done a good part of a chess program by his own could be proud on that. Only a few will have the chance to also write a highly performant chess engine. Well, I am trying with Smirf to create one, but the chances are not that big, facing a scene covered with very good chess engines.

I'm really posting here because I want to get and provide help in FRC & normal chess programming ...

That is highly welcomed. Indeed I see the future of chess playing in 8x8 Chess960 (FRC) and of chess programming in 10x8 CRC. That are wonderful new drosophilas for to test the engines themselfes, not huge opening libraries. The intelligence of a chess program might be:

answer quality / (used time * MB persistant data * Dhrystone)

Regards, Reinhard.

[Uri Blass]

Hi Reinhard,

For me making and unmaking the last move is expensive because I
need to update attack information.

I do not know about other people but when I started my move generator was more than 10 times slower than other people and I did not dare to continue it to a chess program because I decided that I need to improve it's speed.

The problem was that I calculated my attack table from scratch after every move and I also had no piece list.

I think that you have a good chance to get to the level of commercial programs but of course it will take time.

I think that opening books in chess is not so important because of the following reasons:
1)good engine can often find the right book moves by itself.
2)The difference between opening book that is generated automatically from games and the opening book of the best programs is not very big
and shredder can win tournaments against most engines even with an external book that is generated automatically from games.

Uri

[Reinhard Scharnagl]

Hi Uri,

For me making and unmaking the last move is expensive because I
need to update attack information.

well, I do not yet use such tables, thus moving is actally not that expensive in Smirf. And I am not sure, wether providing attack tables could help to speed up my evaluation function. But having a slow program has also advantages - I detected. You could see effects from shaping the search tree more quick. :-)

When I look at old move generating code of mine, there has been a lot of speed ups and a lot of useless attempts. But indeed today it runs about 20 times faster than in the beginning (after making slow processors comparable). You see, I had started with a poor design like others.

I think that you have a good chance to get to the level of commercial programs but of course it will take time

Indeed I want to release my Smirf soon, probably as shareware. But I think it would be rather accepted by its different lay out supporting 10x8 and 8x8 variants. http://www.chessbox.de/beta.html - Nevertheless Smirf should not start as the weakest one - it should have about Crafty level - we will see.

I think that opening books in chess is not so important ...

Indeed, with rising performance Smirf today is producing acceptable moves even without any book. Hope this will help it in Mainz http://www.chesstigers.org/en/Announcement.html .

Regards, Reinhard.

[tinkertink]

Is there a Chess 960 engine that has a "perft" command and also a "divide" command like ROCE (Roman's Own Chess Engine)?

EDIT: Foolish me -- I scoured all over the net except CCRL's FRC section before making my post. I looked at some of the free FRC engines, and apparently AICE has a "perft" command. You can use "setboard <fen string here>" to set the board.

I also realized that the *only* difference b/n FRC and regular chess is castling -- so I just kept rewriting and editing the castling code, and now all is good.

[Roman Hartmann]

Is there a Chess 960 engine that has a "perft" command and also a "divide" command like ROCE (Roman's Own Chess Engine)?

...

The version I'm currently working on has this features. Probably I'm going to release this version in the next few days. It won't be stronger but it has some new featurs and some bugs were fixed.

best regards
Roman

[Volker Annuss]

Is there a Chess 960 engine that has a "perft" command and also a "divide" command like ROCE (Roman's Own Chess Engine)?

Hermann now has a perft with divide.

Start Hermann from the command line.

Switch to uci mode
> uci

set up a position
> position fen ...

Do perft with divide
> perft depth <d> Hash <MB> divide

Hash and divide are optional.

Greetings
Volker

[tinkertink]

Is there a Chess 960 engine that has a "perft" command and also a "divide" command like ROCE (Roman's Own Chess Engine)?

Hermann now has a perft with divide.

Start Hermann from the command line.

Switch to uci mode
> uci

set up a position
> position fen ...

Do perft with divide
> perft depth <d> Hash <MB> divide

Hash and divide are optional.

Greetings
Volker

Thanks Volker. To clarify, the command to set up a position in Hermann is:
> position fen <fen string here, without quotes>
I just figured that out yesterday. For a long time I kept thinking somehow that Hermann was broken. I'm using Hermann 2.4.

Also, I'd like to point out that I tested all the positions in this thread and verified them with my (slow) perft function, but there was still a hidden bug in there. I came upon this thread http://www.open-aurec.com/wbforum/viewtopic.php?f=4&t=5585&p=27462&hilit=perft+and+speed#p27461, and namely, the following position (a derivative of the position discussed in that thread):
Qr1kqrb1/p1pppppp/1p4n1/8/8/1P6/PBPP1PPP/NR1K1RBN b KQkq - 0 1
The bug was that I still allowed black to castle qeenside O-O-O even though doing so would have resulted in leaving the king in check. Anyway, I'm quite confident that if anyone's FRC/chess960 move generator agrees with the results from the posts in this thread and that other one I just mentioned, that it is working without bugs.

[sanel]

it is very necessary, to establish FRC sets of verified numbers.historic Mainstreet Inn
Thus it is fine to post such results, even when not matching.

[crystalclear]

For my chess engine I am trying to improve my move generator's speed, but sometimes changing it might lead to illegal moves being made, or legal moves being missed.

When I read about perft tests, there seems to be a general idea of searching to considerable depth.
It is my opinion that this is not very productive.
What I am working on for myself is a file with vaious small test cases inside, where a move generator might make a mistake.
A shallow perft test is then enough to see if the genertor makes the well defined potential mistake.
Then, when a perft result is incorrect, one should have a good idea what is wrong.

8/8/8/1k6/1p6/8/2P5/1R1KB3 w - - 0 1

In this position, white can advance a pawn two squares.
Black has a pawn in the right position to capture en passent, but the pawn is pinned (by the rook) so the en passent is not allowed.
The black king can move to a position where his pawn is pinned by the white bishop.
If the white pawn then advances two squares, the pinned black pawn can recapture despite the pin, because it remains between the white bishop and black king.

I think it would constructive to have an EPD file with such positions, where a move generator is likely to fail.

A chess engine could then quickly run low depth perft tests to detect problems with its move genertor, instead of doing a very deep perft test examining many positions that are unlikely to fail.

Another position in my file is this one.

r3k2r/8/1N4N1/8/8/1n4n1/8/R3K2R w KQkq e6 0 1

There are no pawns, but kings and rooks to test castling.
The rooks can move to test loss of castling rights.
And there are knights that can interfere with castling by checking squares or blocking the kings path.

Part of the idea is that a perft failure on one position is a clear indication of an enpassant problem and that a perft failure in another position is a clear indication of a castling problem.

For a general test of enpassant,
a not too deep perft (3) test from here
4k3/pppppppp/8/8/PPPPPPPP/8/8/4K3 w - - 0 1
should test all possible black pawn en passent captures.
I have a similar position for white
4k3/8/8/pppppppp/8/8/PPPPPPPP/4K3 b - - 0 1

=

I think if a deep perft test fails from a complicated position, then it is often finding a tricky problem. At times you have to travel several moves forward from the perft root position to find the situation that confuses it. That's one reason that I'd like a collection of positions that test different things. Advance to the failing position, and remove the unnecessary pieces, and extra a FEN for the root cause of the move generator failure.

My engine can't do FRC. I do care a little about embarassing myself posting incorrect peft numbers. But the main reason that I'm not giving perft numbers for the positions above is that it would be counter productive if any are wrong.
My engine isn't stable at the moment.

Oh what the heck, let's make a fool of myself ...

Code: Select all

4k3/pppppppp/8/8/PPPPPPPP/8/8/4K3 w - - Perft 3178;
4k3/8/8/pppppppp/8/8/PPPPPPPP/4K3 b - - Perft 3178;
8/8/8/3k4/1p6/8/2P5/3K4 w - - Perft 54;
8/8/8/1k6/1p6/8/2P5/1R1KB3 w - - Perft 11581;
8/8/8/1k6/1pP5/8/8/1R1KB3 b - c3 Perft 9630;
r3k2r/8/1N4N1/8/8/1n4n1/8/R3K2R w KQkq e6 Perft 26117661;
rnbq1bnr/PPPPPPPP/3K4/8/8/3k4/pppppppp/RNBQ1BNR w - - Perft 3306;
r2Nk2r/pPp3P1/8/bP4B1/1p5b/6P1/PpP3p1/R3K2R b KQkq - Perft 966;


When my chess program does a test with an EPD file and sees the proprietary Perft opcode, it does perft tests at increasing depth until it reaches or exceeds the counter value. Then if the value is the same, it has passed the test and if the calculated value exceeds the one in the file, it has failed. So my engine can do perft tests from values in an EPD file without needing to know the depth used for the original calculation.

I have some Zobrist (polyglot) hash function values for simple positions too if they are of any use.

[H.G.Muller]

Actually the large depth is the main thing that makes it useful. With special test cases, like you propose, you would only catch problems that you did foresee. While most problems are unforeseen, as errors you foresee, you usually don't make...

[tinkertink]

Actually the large depth is the main thing that makes it useful. With special test cases, like you propose, you would only catch problems that you did foresee. While most problems are unforeseen, as errors you foresee, you usually don't make...

I fully agree. I once tried to create a test position to purposely trigger a known castling bug in my move generator, so that I only needed to search to a depth of 3 or 4... but failed.

It boils down to this question: can I test every single codepath combination in my move generator manually? The accepted answer is "no", which is why we have perft in the first place... If you're doing a perft with depth 1 or 2, then you're missing the whole point! You'd probably need at least several hundred (thousand?) test positions to get the same benefit of running perft to a depth of 6 on two or three positions...