small searches for compression

[Daniel Shawul]

Is it good to do small searches to increase compression
of bitbases? How many plies should the search be in order to
come up with a good prediction? I dont like this approach because
of two things.
1. I think that a very good compressor may give the same result
without the need for this
2. The idea somehow contradicts the original aim of having endgame tables,which is to avoid search.

Ofcourse if the gain in compression ration is quite significant i might
do it.
So please poll yes/no
daniel

[Tord Romstad]

Hi Daniel,

Have you considered Tony Werten's idea of using lossy compression of bitbases? It seems very attractive to me. When probing a bitbase, you can get the possible values "won", "lost", "drawn" or "don't know" as results. According to Tony, it is possible to get very good compression while keeping the "don't know" percentage low. With some luck, most of the "don't know" positions can be resolved with one or two plies of search.

Tord

[Daniel Shawul]

Hi Tord,
I havent tried it. How lossy is it?
The bitbases that are causing trouble are the kxxkx types.
A real good predictor is required for this.
I will read that page again to get the idea.
DAniel

[Tord Romstad]

Hi Tord,
I havent tried it. How lossy is it?

Depends on the type of compression used and on the level of compression, I guess. Perhaps clever indexing schemes could also help.

The bitbases that are causing trouble are the kxxkx types.
A real good predictor is required for this.
I will read that page again to get the idea.

There is no page to read, I'm afraid (at least no page which I am aware of). It was just a conversation over a few beers in Mainz last year.

Tord

[Daniel Shawul]

I found his explanation in the archieves.
He mentioned krpkr could be compressed this way to 4mb
which is very good. BTW the knightdreamer team has
compressed all 5 pieces down to 230mb without any loss of info.
daniel

[Alessandro Scotti]

Hi Daniel,
one thing I sometimes wonder is how Delfi manages to keep its bitbases so small. I think one could approach the same level of compression by storing only bitbases for the stronger side (then possibly extending one ply) but didn't fully experiment with this.

[Daniel Shawul]

Hi Alessandro
That is also another possibility.
I generally do not like things that involve search to get better compression. I want to first exhaust what is possible with the best compressor. Unfortunately all the compressors i tried zip/datacomp have
not done much better than my rough implementation of rle+huffman.
One thing i noted is that predictors seems to work very well for the rle part.
By mistake i set the broken values to draws all the time, and the 5 piece were compressed to 1.2gb by datacomp, and mine gives like 2.11gb and then when i fix the bug datacomp gave 1.0gb while rle+huffman gives 1.64gb. So there is not much to gain from a good compressor by predicting the values. That is one reason i am looking for the best.

daniel