bitbase compression

[Daniel Shawul]

Hi
The sizes of all my bitbases for 3/4 piece endgames add up to 18MB.
With optimum indexing, the size can be reduced to 15MB.
i have heard compression/decompression is possible in memory, but i have no idea whatsoever. My bitbases size reduce to a remarkable 3.2MB
when compressed with winrar. I am assuming this is the equivalent of what i get when they are compressed in memory?? If so, how to do it?
best
daniel

[Jim Ablett]

http://datacompression.info/SourceCode.shtml

[Daniel Shawul]

Hi Andrew,Jim
Thanks for the answers.
Which compression algorithm is completely free? It doesn't have to be the best. All I need is to reduce the 18MB to some value which is affordable in most tournament conditions.
Daniel

[Dann Corbit]

Hi Andrew,Jim
Thanks for the answers.
Which compression algorithm is completely free? It doesn't have to be the best. All I need is to reduce the 18MB to some value which is affordable in most tournament conditions.
Daniel

There are many that are completely free. I think arithemetic encoding has a patent, but the LZW patent has expired so you can use that one.

Adaptive Huffman might be good enough. Try several of them and see which one you like best.

[Jim Ablett]

Hi Daniel,

For Rar compression >

It's cross-platform, free and open source and full Rar 2.0 compat.

http://www.unrarlib.org/

regards,
Jim.

[Dann Corbit]

Hi Daniel,

For Rar compression >

It's cross-platform, free and open source and full Rar 2.0 compat.

http://www.unrarlib.org/

regards,
Jim.

Isn't that one decompression only?

The 7-Zip collection has every sort of algorithm in it but some of them have stipulations on usage.

[Dann Corbit]

Zlib seems to have a license compatible with Scorpio:
http://www.zlib.net/

/* zlib.h -- interface of the 'zlib' general purpose compression library ↑
version 1.2.3, July 18th, 2005 █
█
Copyright (C) 1995-2005 Jean-loup Gailly and Mark Adler █
░
This software is provided 'as-is', without any express or implied ░
warranty. In no event will the authors be held liable for any damages ░
arising from the use of this software. ░
░
Permission is granted to anyone to use this software for any purpose, ░
including commercial applications, and to alter it and redistribute it ░
freely, subject to the following restrictions: ░
░
1. The origin of this software must not be misrepresented; you must not ░
claim that you wrote the original software. If you use this software ░
in a product, an acknowledgment in the product documentation would be ░
appreciated but is not required. ░
2. Altered source versions must be plainly marked as such, and must not be ░
misrepresented as being the original software. ░
3. This notice may not be removed or altered from any source distribution. ░
░
Jean-loup Gailly Mark Adler ░
jloup@gzip.org madler@alumni.caltech.edu ░
░
░
The data format used by the zlib library is described by RFCs (Request for ░
Comments) 1950 to 1952 in the files http://www.ietf.org/rfc/rfc1950.txt ░
(zlib format), rfc1951.txt (deflate format) and rfc1952.txt (gzip format). ░
*/ ░

[Tord Romstad]

Tony Werten mentioned an interesting idea during the tournament in Mainz: His engine XiniX uses lossy compression of bitbases. By some clever techniques which he did not explain in detail (or which I was too tired or drunk to understand, I'm not quite sure), he is able to determine whether any specific bit in the compressed bitbase can be trusted or not. In other words, probing the bitbase can return the values 0, 1 or "don't know".

Using this technique, he has been able to compress the KRP vs KR bitbase down to just 4 MB, with a "don't know" percentage of just 25% (IIRC).

Tord

[Tony van Roon-Werten]

Tony Werten mentioned an interesting idea during the tournament in Mainz: His engine XiniX uses lossy compression of bitbases. By some clever techniques which he did not explain in detail (or which I was too tired or drunk to understand, I'm not quite sure), he is able to determine whether any specific bit in the compressed bitbase can be trusted or not. In other words, probing the bitbase can return the values 0, 1 or "don't know".

Using this technique, he has been able to compress the KRP vs KR bitbase down to just 4 MB, with a "don't know" percentage of just 25% (IIRC).

Tord

Hi Tord,

you remembered quite correct. ( Also that I didn't give you the details

I've done some more testing now and am a bit disappointed. Altough the bitbases help, it's not as much as I hoped.

In KRPKR fe:

wtm: 75 % recognition of current position.
Other 25 %: say 16 moves and I recognize 12. If 1 of these 12 is a win, I can quit else I'll still have to search the remaining 4 because the recognition only gave a lower bound.

btm: again 75% recognition
other 25 % again chance that at least 4 moves will also not be recognized.

So although the branching factor goes down to 1.3 (guestimate) the tree still grows. I actually hoped for something below 1

My gues is that the recognition has to go over 85% for that. Problem is that these files get a lot bigger for these last 10% than they do for the first 10%

For most files that is, KPPKP fits in 10MB with 99% hitrate. Unfortunately that seems to be an exception together with KQNKQ and family. These latter basicly seems to compress to "return draw"

Maybe I'll try 6 pieces too I don't think they will work however, probably too big files for too little recognition. Besides that, these 5 piece take 24 hour each, I'm pretty sure 6 will be at least 10 times slower.

OTOH even only KPPKPP in memory would still be nice.

Tony