Gaviota, bug fix release

[Miguel A. Ballicora]

Gaviota 0.74.12 has been released together with a new utility for TB builders (tbcheck 1.1).

http://sites.google.com/site/gaviotache ... e/releases

0.74.12 FINALLY! solved the problems to build Gaviota tablebases in Windows, with lower memory than 1 Gb, in both 32 and 64 bit systems. Thanks a lot to Gabor, Ray, and Volker who generated them in different systems. (win32, win64, Linux). Thanks to them, the generation for any other user will be very straightforward.

As a consequence of all this testing, I wrote tbcheck, which is an utility that checks if any of the files you generated is corrupted. You want to have this if you decide to generate the TBs.

Dann C. found two problems that were also fixed. In analysis mode now Gaviota ignores the book as it should, and an apparent communication problem with Arena. He also compiled a Win64 version.

If you downloaded any 0.74.xx version and did not experience any of these problems, maybe you can ignore this message . The strength is exactly the same and you can keep using that version.

Regarding TBs, now that the generation is correct the roadmap is
1) build a compressor for the tables (I do not believe is needed for performance but releasing > 30 Gb of the HD of testers will be a good thing)
2) Optimize for speed (generation, compression/decompression, & probing)
3) Probably release the code at this point
4) go for DTZ50
5) go for 6 pieces?

Just for the record v0.74.11 (unreleased) was used in the ACCA tournament using the TBs, who were critical to draw crafty in a BP vs R position with few seconds left in the clock.

Miguel

[Volker Pittlik]

...
Regarding TBs, now that the generation is correct the roadmap is
...
3) Probably release the code at this point

That would be great I think. From my point of view the Nalimov tables are still the "best". They are relatively small and deliver perfect knowledge about the endgame. Their huge disadvantage is that legality of use depends on permission of two persons. This has caused and is causing a lot of trouble.

The alternatives (except Thompson) only deliver if the position is won or not. Some doesn't work here for whatever reason (new Scorpio's) or don't work and are of a suspicious legal state (Robbolito).

To have an open type of egtbs (which also work so nice as yours) would be a big step forward.

vp

[Dann Corbit]

There are many reasons why an open TB system is a very good idea.

I know of people who have literally waited years to get permission to use Nalimov files (Eugene can be hard to contact for whatever reasons).

If you want to do research (e.g. Longest mate in N) right now it is only a few people that can even do this sort of thing because there is no open source code to compute large tablebase files.

The biggest problem (from my view) is that the proliferation of tablebase files eats disk like a madman.

I have 60 GB of Nalimov files, 350 MB Feg files, 378 MB egbb files, 350 MB egbb2 files, 220 MB Knightd3 egbb, 3.5 GB RetroChess egtb, 130 MB Booot egtb files, 30 MB edwards egtb files, (and some others).

Imagine five years in the future, with 100 different people writing their own versions of 6 & 7 man files. (Fine, if you want to do that, but please interface the MAB files as well, or your engine won't get to use EGTB on my system).

This seemed a possible alternative:
http://www.freesoft.org/software/hoffman/
But it is nothing more than an HTML interface over Eugene's files, and the license is not clear. Besides which, it will only build on a true POSIX system (Mingw, for instance, won't work).

HGM was working on a open source tablebase system but he seems to be distracted from that task for now, so I doubt it will be completed within the next year or so.

So, as I see it, MAB files are the only logical way to go. Once all of the kinks are worked out, I hope that everyone abandons every other format and uses this one instead.
Perhaps those that don't will face the same sort of ChessBase "We don't do UCI with our engines[1]"penalty and fade into obscurity.

[1] Of course, most formerly non-uci engine vendors have simply written their own UCI versions (Junior, Shredder, Hiarcs, etc.). Is Fritz the only hold-out?
P.S. Will Chessbase ever properly handle EPD as an output format?

[Michel]

Probably release the code at this point

That would be fantastic. But please if you do release the code (and if you intend it to be open source) please use an unambiguous GPL
compatible license.

The situation with Scorpio bitbases for example is very problematic. Although the author Daniel Shawul has said repeatedly
that he considers them open source and everyone is free to use them this is not written out explicitly in a license. As a result
legal reasons prevent Debian (and hence Ubuntu) to use them.

[Miguel A. Ballicora]

Probably release the code at this point

That would be fantastic. But please if you do release the code (and if you intend it to be open source) please use an unambiguous GPL
compatible license.

I am no expert on licences, but I think that BSD should do it. That is basically do whatever you want but give credit where credit is due and should be compatible with the stricter GPL. Right?

Miguel

The situation with Scorpio bitbases for example is very problematic. Although the author Daniel Shawul has said repeatedly
that he considers them open source and everyone is free to use them this is not written out explicitly in a license. As a result
legal reasons prevent Debian (and hence Ubuntu) to use them.

[Michel]

I am no expert on licences, but I think that BSD should do it.

BSD without the advertising clause is fine. See here for various free software licenses.

http://www.fsf.org/licensing/licenses/index_html

and here

http://www.opensource.org/licenses/alphabetical

[crystalclear]

I have 60 GB of Nalimov files, 350 MB Feg files, 378 MB egbb files, 350 MB egbb2 files, 220 MB Knightd3 egbb,
3.5 GB RetroChess egtb, <==============
130 MB Booot egtb files, 30 MB edwards egtb files, (and some others).

I discovered Retrochess endgame tablebases yesterday, so it's early days for me with their endgame table bases and file format.

I have Gaviota tablebases working with my engine, but I just have 5 men tablebases. I tried to contact the author (Miguel A. Ballicora) by email to see if there are 6 men Gaviota tablebases in existence or planned (with an offer to help if required but got not reply). I saw something about the Retrochess tablebases on the internet and downloaded their package out of curiosity. The reason was that their tablebase goes to 7 men in theory. It seems to come with four things packaged together.
1. a tablebase generator GUI - point and click to what you want to generate - with a useful pause button.
2. some small tablebases 3,4 men
3. a GUI to examine the tablebases visually
4. a DLL to examine them using software

The probing interface uses a FEN, which makes it very convenient to convert from one engine's format to another's, but probably slow. I prefer Miguel A. Ballicora's interface for probing from an engine during deep searching. However speed doesn't matter at all once you enter the final stages of a game. It matters for probing during a search, but not once the board position is within the tablebase. So it occurred to me that I could run my engine the way it is now, but if I happen to be at a board position within the scope of the Retrochess tablebases and if they are available on the computer, I could probe them for a move.

With the Gaviota tablebases, I'd had to convert my piece enumeration to Gaviota's and fill in arrays with the position of the pieces. The Retrochess tablebase's FEN interface seemed slower, but much easier to code up. I figured to test if it works, I'd need something like this ...

Code: Select all

    void* RetroChess;
    RetroChess = InitCHESS();

    short dtm;

    dtm = GetEvaluateFromFEN(RetroChess, "6Q1/8/5k2/8/8/8/8/K7 w - -");

    printf("Distance to mate is \n", dtm);


If it says some number meaning mate in 9 then it works and in theory I could :-
1. add software to my engine to interrogate the database when wanting to make a move,
2. create or download more tablebases
and maybe play some 7 men configurations perfectly when the engine reaches them.

It all looked rather easy. Pity I haven't got it working yet!
Has anyone else tried out Retrochess tablebases?
Is there a site to download them?
I generated a little bit of 6-man tablebase, with the intention that if my engine could play it properly then there is no reason why it shouldn't play 7 men properly if I can get my hands on the data later. There might also be occasions where a an exact probe of 7 men data would save a deeper search which would only reduce to 5 men after many moves. So the (presumably) slower probing code could pay off.

Another question ... (I picked up some CRC64 software. I use a 32 bit CRC in my software at the moment to identify different chess games by taking a CRC of the hash codes of the positions in the game, but I plan to switch to reduce the chance of a collision.) Is one 64 bit CRC generating polynomial more standard than another? I don't even know which one I'm using. But here are my CRCs for some of the Retrochess endgame tablebase files. I hope to generate and collect more over time.

Code: Select all

RetroChess Tablebase 64 bit CRCs
================================

7B25A62AC3434D95 Chess_Bases_6/6_KNN-KNN.tby
0D5A32E4A7E77A4E Chess_Bases_6/6_KNNN-KN.tby
60A204BB49ECA9C2 Chess_Bases_5/5_KNN-KB.tby
AC7075F8E6C6B6C8 Chess_Bases_5/5_KNN-KN.tby
6D9558843E900812 Chess_Bases_5/5_KNNN-K.tby
3A848736D1472EFB Chess_Bases_5/5_KBN-KN.tby
69592E12800AE721 Chess_Bases_5/5_KNN-KQ.tby
E11274F44E4BCC9D Chess_Bases_5/5_KNN-KR.tby
B6F8728371B9E1DE Chess_Bases_5/5_KRNN-K.tby
C4416782CC3340F6 Chess_Bases_5/5_KBBB-K.tby
1FE4CA9EE4F5452B Chess_Bases_5/5_KBBN-K.tby
C6126ACD2102DC0F Chess_Bases_5/5_KBNN-K.tby
DB77AB625625597C Chess_Bases_5/5_KBN-KB.tby
6CEC9137D25A4B7F Chess_Bases_5/5_KRR-KR.tby
AC799D0C942EE7E1 Chess_Bases_5/5_KRR-KB.tby
59B5AB00F17656E1 Chess_Bases_5/5_KRR-KN.tby
ED779DA51D99B88B Chess_Bases_5/5_KQQ-KR.tby
FA08EBB8F9FD5613 Chess_Bases_5/5_KRB-KR.tby
CFF54C6246687F6D Chess_Bases_5/5_KRB-KB.tby
28962137C379ED1F Chess_Bases_5/5_KRB-KN.tby
F5564C5AFEED5693 Chess_Bases_5/5_KRN-KQ.tby
DD3A8007364C45F1 Chess_Bases_5/5_KQP-KN.tby
83B80FBCED5863CE Chess_Bases_5/5_KRB-KQ.tby
6BA2E69D900AC729 Chess_Bases_5/5_KRN-KR.tby
76EB8FCFC6D0FBFA Chess_Bases_5/5_KRN-KB.tby
EDC03C5F48038C37 Chess_Bases_5/5_KBB-KQ.tby
4327A8238FD704F2 Chess_Bases_5/5_KBB-KR.tby
162554CDB530BAC9 Chess_Bases_5/5_KBB-KB.tby
80E504F9A4D7C4B8 Chess_Bases_5/5_KBB-KN.tby
A301C4C34C9395F7 Chess_Bases_5/5_KQP-KB.tby
195F4162AA515F39 Chess_Bases_5/5_KQR-KB.tby
E41CDEEDC1FE4890 Chess_Bases_5/5_KQR-KN.tby
AFCF3A16B8A25D77 Chess_Bases_5/5_KBN-KQ.tby
8A549AC9637CC6D7 Chess_Bases_5/5_KBN-KR.tby
8FD46AA832451B78 Chess_Bases_5/5_KQR-KP.tby
7604ABA67AE32BD5 Chess_Bases_5/5_KQQ-KP.tby
1C6AF3F2F38F1B1D Chess_Bases_5/5_KRN-KN.tby
84311EEDBD499245 Chess_Bases_5/5_KQQ-KQ.tby
4D22AD212831BEE9 Chess_Bases_5/5_KQQ-KB.tby
10E31E5081332F3B Chess_Bases_5/5_KQQ-KN.tby
35FA1368CF3FD018 Chess_Bases_5/5_KQR-KQ.tby
AE64F4B7DDC4868A Chess_Bases_5/5_KQR-KR.tby
0A197A729224BD94 Chess_Bases_4/4_KRN-K.tby
111275ADD66398FE Chess_Bases_4/4_KRP-K.tby
A8E11B01D7A93E2D Chess_Bases_4/4_KRR-K.tby
51B604AF47913EC5 Chess_Bases_4/4_KBB-K.tby
ACFC2CC3C5AF520B Chess_Bases_4/4_KB-KB.tby
35B9452186468C62 Chess_Bases_4/4_KB-KN.tby
EAF803E77CBE34F8 Chess_Bases_4/4_KB-KP.tby
1FA23345A7E4215D Chess_Bases_4/4_KBN-K.tby
D5CE15E09863885C Chess_Bases_4/4_KBP-K.tby
40EFD6A281EC93BD Chess_Bases_4/4_KN-KN.tby
9CCD780DE7918D5A Chess_Bases_4/4_KN-KP.tby
BDFD19EBAFE6AC34 Chess_Bases_4/4_KNN-K.tby
C9D67C71391E93A2 Chess_Bases_4/4_KNP-K.tby
246A7BBB7E51602B Chess_Bases_4/4_KP-KP.tby
E6E50F590A3E61D3 Chess_Bases_4/4_KPP-K.tby
0F2515823B7103A2 Chess_Bases_4/4_KQB-K.tby
83909D30F7EA3D1F Chess_Bases_4/4_KQ-KB.tby
71DFFB308D7F448E Chess_Bases_4/4_KQ-KN.tby
88EB4D71BB499C65 Chess_Bases_4/4_KQ-KP.tby
6C735B82E9E59710 Chess_Bases_4/4_KQ-KQ.tby
2355DAC13DEBF235 Chess_Bases_4/4_KQ-KR.tby
02AE480E023D93EE Chess_Bases_4/4_KQN-K.tby
F17257DF606215B6 Chess_Bases_4/4_KQP-K.tby
98FFEFD9310F4C0C Chess_Bases_4/4_KQQ-K.tby
24CC342A0CE86B17 Chess_Bases_4/4_KQR-K.tby
1471FF99DEAEDA81 Chess_Bases_4/4_KRB-K.tby
564F3ABFFF20F491 Chess_Bases_4/4_KR-KB.tby
D7DA7CCCB0AE72E8 Chess_Bases_4/4_KR-KN.tby
C28BC410F85A0D0F Chess_Bases_4/4_KR-KP.tby
C29EE1243E116C85 Chess_Bases_4/4_KR-KR.tby
E9C386B05A32CB54 Chess_Bases_3/3_KP-K.tby
9934B12E8533F07E Chess_Bases_3/3_KQ-K.tby
5B3D73B9E0F489E5 Chess_Bases_3/3_KR-K.tby
E88AEC0FA6D9DC39 Chess_Bases_3/3_KB-K.tby
E74A084199A02325 Chess_Bases_3/3_KN-K.tby
77 files


I'm seldom online but when I am, this is the directory where I'll store the files I've generated to try out.
http://www.crystalclear.co.nr/retrochess
(I'm not making a serious attempt to collect a complete set of anything at the moment.)

After generating a few of these, it seems to me that the generator program is slower than some other table base generators, and I guess the reason the program has been made available online is to get others to participate in generation of the tablebases. My gut feeling is that a faster program is needed. I don't feel it is worth clogging up my hard drive any further.
I haven't got the DLL working, but that is probably due to my total inexperience with DLLs and C++.