6-men Nalimov EGTB.

[Josu? Forte]

Hi all,

I have just implement Nalimov EGTB in my engine.
Runs fine whith 5-men tablebases.

I want to do some tests with 6-men tablebases.
Where can I find them? (in compressed format)

By the way, I would like to request Eugene Nalimov written permission to use his code.
I sent an email to "eugenen@microsoft.com" but it seems it is not working.

What is his current email address?

Thanks in advance for any help.

Regards,
Josu

[Ron Murawski]

Hi Josu?,

For a discussion of where/how to download the 6-man tablebases look here:
http://wbforum.volker-pittlik.name/viewtopic.php?t=3816

I emailed Eugene several times before receiving permission. Be patient.

BTW, clicking on your WWW address button results in:
"The system detected an Unresolved Host Name while attempting to retrieve the URL: http://www.matheuschess.hpg.ig.com.br/index.htm. DNS resolution failure encountered for the host 'www.matheuschess.hpg.ig.com.br'."

Best regards,
Ron

[Josu? Forte]

Hi Ron,

Thanks for the link.
I will send another email to Nalimov.

Yes, you are right about Matheus homepage. It is not working since one month ago. I will choose another host to get the page back.

Best regards,
Josu

[jshriver]

I'm hosting some of the 6men on my website.
http://www.olympuschess.com/tb6 I add new sets ever so often.

-Josh

[diepeveen]

Hi Ron,

Thanks for the link.
I will send another email to Nalimov.

Yes, you are right about Matheus homepage. It is not working since one month ago. I will choose another host to get the page back.

Best regards,
Josu?

Instead of spamming the guy, why not make your own EGTBs like many programmers have done by now?

Vincent

[Josu? Forte]

Hi Joshua,

Thanks for your tip!

First of all I would like to congratulate you for your nice and useful site.
I downloaded a pair of 6-men files and tested my engine. Worked fine with 6-men files. It is amazing.

Keep working like this. The computer chess community needs people like you.

Best regards,
Josu

[Josu? Forte]

Hi Vincent,

If I am not wrong you are the author of a strong engine called diep. Congrats for your creation!

Well, answering your question "why not make your own EGTBs like many programmers have done by now?".

I would say for the same reason so many skilled programmers have done this way up to now. To just mention one of them, that I most respect, Robert Hyatt. As Bob said one time in a forum discussion "why reinvent the whell?".

But you are right. I will not spam Nalimov a second time about this subject. Instead, a couple of weeks ago I started working on a bitbase code which seems to be superior to EGTB in terms of size and speed.
A coulple of EGTB files (kpk.nbw and kpk.nbb), with size of about 80 Kb each file, can be now converted to a mere compressed single file of 3600 bytes. And I can use it without permission.

Nevertheless, I respect very much Nalimov's work.

With my best regards,
Josu

[diepeveen]

Hi Vincent,

If I am not wrong you are the author of a strong engine called diep. Congrats for your creation!

Well, answering your question "why not make your own EGTBs like many programmers have done by now?".

I would say for the same reason so many skilled programmers have done this way up to now. To just mention one of them, that I most respect, Robert Hyatt. As Bob said one time in a forum discussion "why reinvent the whell?".

But you are right. I will not spam Nalimov a second time about this subject. Instead, a couple of weeks ago I started working on a bitbase code which seems to be superior to EGTB in terms of size and speed.
A coulple of EGTB files (kpk.nbw and kpk.nbb), with size of about 80 Kb each file, can be now converted to a mere compressed single file of 3600 bytes. And I can use it without permission.

Nevertheless, I respect very much Nalimov's work.

With my best regards,
Josu?.

You have it 100% right. He did do a good job objectively because he did make EGTBs and did do a lot of effort for them in a time that most simply didn't have the time or wanted to waste time to it.

However if we look objectively to what is POSSIBLE, then he is what we call 'one eyed in the land of the blind'.

Everyone with an engine considers WDL. Diep too.

Note tricky is debugging your own format. Very tricky. The debugging of all 6 men took 6 months of a quad machine some years ago. I still must do a second check now as i don't store automatically CRC information about files, so if a file is corrupt i have a problem. I'm busy with that process now, which nalimov of course got automatic from kadatch for free.

Even then i discovered an en passant bug not too long ago.

Note nalimov has one somewhere too which no one reported yet amazingly (at least not publicly), as i guess no one can use all his 6 men in search.

Advantages of WDL over DTM:
- DTM needs 2 bytes a position
- WDL stores 5 positions a byte.

Up to factor 10 difference.

RAM is expensive.

Of course the advantage of DTM is basically a single big advantage, that's that you can publish the maximin value of a position without using chess rules.

Another advantage:
- you cannot load in RAM all the decompression tables from
nalimov EGTBs, so effectively you cannot use them in search.
No one has all nalimov's in search.
- Nalimov has done a great job in writing that code already years ago, so perhaps it's not fair to judge him based upon 6 men, as probably his code was intended for 5 men; but for 6 men it is very slow code. My EGTB code starts instantly the program. Doesn't take 30 minutes loading time to index all EGTBs, and then for compressed nalimov's you'll need another 2 GB of ram to load them.

Windows in 32 bits simply has not enough indexation space for that. Majority of users has a 32 bits machine. We can argue long. An engine must work well in 32 bits simply for users. Frans Morsch and i agree there.

- indexation in cache goes O( n ) in nalimov. So the bigger your cache, the longer it takes to get a position out of RAM. In Diep it is garantueed not slower than O( log n ) and usually approaching O( 1 ) quickly.
- number of files in Nalimov is big. Perhaps the problem was not Eugene himself here but that Kadatch goes till 2^31 -1 bytes as a maximum size for files. Yet that huge number of files problem is really overwhelming.

It's easy to criticize and hard to make something yourself. If you would realize how much system time i put into generating (i wrote in total 6 generators now, my last one, a lightning speed fast generator not 100% finished yet as it has no en passant for the 7 men), verifying and rechecking, you will realize it is not an easy job.

Generating at 32 bits machine is for example extreme slow for what i do.

Yet it is handsdown possible to do it better than the solutions shown now.

I find it fundamentally wrong that in important part of game everyone uses code of someone else.

I know that about every commercial programmer finds this deep in his heart. Chessbase already years ago started talking to me they wanted their own EGTBs too.

It's a matter of time before everyone has his own format.

WDL is just too superior to DTM and as a dutchman i like to not copycat other persons source code.

Note that the influence at the game of EGTBs nowadays at top level matches (so not the online matches but in tournaments) is a lot less than it was in the past, because of the dynamic openings that get played.

So it's not for sure that having EGTBs will improve you elopoint wise in a world championship in the future.

However there is clear proof that it CAN save you several half points from world champs and big tournaments in the past.

Just look to games in past tournaments where EGTBs would have mattered.

We do not know the future there. What i do know is that all 5 men fit within 150MB in WDL if you supercompress them and in Nalimov that's 7+ GB.

Vincent

[H.G.Muller]

I can imagine that for probing during search WDL is usually good enough to know if you should go for the conversion or not. For actually playing an end-game it seems not sufficient to ensure progress, though: in (say) KbbKn evrything where you don't sacrifice a Bishop will remain won, and the bit-base data is basically useless because there are easier ways to check if you lost a Bishop.... But then probing a DTM at game level directly from disk is not too slow.

But how do you handle positions that can be won, but only in more than 50 moves? Do you store them as wins or as draws? In the former case it might fumble wins because it unnecessarily converts to a 'slow win' where 'fast wins' could have been forced, in the other case you might miss a swindle victory against opponents that did not have the TB.

Does this simply occur so infrequently that you don't need to worry about it? Many EGTBs of course have no 'slow wins' at all.

[Daniel Shawul]

can imagine that for probing during search WDL is usually good enough to know if you should go for the conversion or not. For actually playing an end-game it seems not sufficient to ensure progress, though: in (say) KbbKn evrything where you don't sacrifice a Bishop will remain won, and the bit-base data is basically useless because there are easier ways to check if you lost a Bishop.... But then probing a DTM at game level directly from disk is not too slow.

I had also doubts when i first tried it. Well it is possible to ensure progress with no special evaluation fucntion , atleast this works efficiently for 5 men. I still have doubts for 6men but I guess DIEP uses them also for 7men.
What i do to ensure progress
a) if we encounter a capture/promotion or a pawn move just return immediately the score. This moves zero the fifty move count and we are surely making progress. Also prefer egtbs with low number of pieces , ie 3men over 4 men.
b) for other moves , do not return a bitbase score in the first half of the search (IE iteration_depth / 2).
This combined with proper handling of bitbase scores in the hashtable have worked fine for me.

About 50 move rule both DTM and WDL suffer from it.
My hope is that that rule will be changed in the near future at least for comps

Daniel

[H.G.Muller]

Hmm, that really surprises me. I would have guessed that probing the bitbase would do more harm than good: every end leave will be a bitbase hit so if you return the same score for any won position, you are even blinding the engine for the heuristic progress measurements like King centralization/cornering. It would basically cause random diffusion through the sub-space of won states, until a mate accidentally came within the horizon. For 4-men like KbnK this seems very unlikely.

It seems that one should at least allow the ordinary evaluation to contribute to the score of a bit-base hit. I can imagine that using the bit-base probe as a (dominant) additive term in the normal evaluation could be very helpful in end-games where you might easily step into the draw sector (i.e. generally drawn), such as KrKb or KrKn. But I can't see how KbnK or KqKr would profit from it...

[Josu? Forte]

But how do you handle positions that can be won, but only in more than 50 moves? Do you store them as wins or as draws? In the former case it might fumble wins because it unnecessarily converts to a 'slow win' where 'fast wins' could have been forced, in the other case you might miss a swindle victory against opponents that did not have the TB.

Hi Muller,

I will be direct to your point.
50 moves rule can be easily handled in EGTB.
See the code that I use in Matheus. It works fine.

Code: Select all

/* EGTB Probing code. */
#ifdef EGTB
   if (egtb_limit >= total_pieces &&
      ((status[move_number].white_castle + status[move_number].black_castle) == 0) &&
      (!status[move_number].enpassant_sq)) {
      if (EgtbProbe(&score)) {
         int mate_distance = 0;

/* Mate scores have to be adjusted because here "score" represents mate in N moves
   from the root. We must store "score" as mate in N moves from this position. */

         alpha = score;
         if (alpha) {
            if (alpha > 0)
               score = alpha + ply;
            else
               score = alpha - ply;
            mate_distance = MATE_VALUE - abs(score);
         }

/* If 50 moves rule reached then position is a draw. */
         if ((status[move_number].fifty_moves_rule + mate_distance) > 100) {
            pv_length[ply] = 0;
            return(0);
         }

/* Record EGTB score in hashtables -> "always" and "depth" elements. */
         if (!follow_pv) {
            hash_entry->always.flag.bound = hash_entry->depth.flag.bound = EGTB_SCORE;
            hash_entry->always.move = hash_entry->depth.move = NO_MOVE;
            hash_entry->always.hashkey = hash_entry->depth.hashkey = hashkey;
            hash_entry->always.score = hash_entry->depth.score = score;
            hash_entry->always.depth_remaining = hash_entry->depth.depth_remaining = MAX_DEPTH;

            if (age != hash_entry->depth.flag.age) {
               hashtable_write++;
               hash_entry->depth.flag.age = (unsigned char)age;
            }
      
            pv_length[ply] = 0;
            return(alpha);
         }
      }
   }
#endif /* EGTB */


Best regards,
Josu

[Uri Blass]

But how do you handle positions that can be won, but only in more than 50 moves? Do you store them as wins or as draws? In the former case it might fumble wins because it unnecessarily converts to a 'slow win' where 'fast wins' could have been forced, in the other case you might miss a swindle victory against opponents that did not have the TB.

Does this simply occur so infrequently that you don't need to worry about it? Many EGTBs of course have no 'slow wins' at all.

My solution is simply to translate mate in more than 50 to number that is not win and not draw but advantage and stop the search at that point.


I have a table that translate every mate to number of pawns.

Uri

[bob]

That code won't work and has a _serious_ bug...

say we are in an ending like KBB vs KN, and we have played 40 moves toward a mate in 70. And your test says "draw" since the mate distance is beyond the 50 move rule counter. Unfortunately, between here and where the 50 move rule counter hits 100, we are going to make a capture (winning the knight) which resets the counter. But you have already said "draw" because you can't see the counter getting reset.

That's not worth the effort since it tries to hide a problem but introduces a potentially bigger problem...

[Josu? Forte]

That code won't work and has a _serious_ bug...

say we are in an ending like KBB vs KN, and we have played 40 moves toward a mate in 70. And your test says "draw" since the mate distance is beyond the 50 move rule counter. Unfortunately, between here and where the 50 move rule counter hits 100, we are going to make a capture (winning the knight) which resets the counter. But you have already said "draw" because you can't see the counter getting reset.

That's not worth the effort since it tries to hide a problem but introduces a potentially bigger problem...

Hi Bob,

You are simply 100% right!

I did not think about that.
But you have just given me a way to by-pass the problem.

If you get a mate score from EGTB you can get the mate_path that lead to that score, right?

So, instead of returning a draw score, as stated in my previous code, we simply need to go through the mate_path and verify if there is a capture or pawn_push before 50_moves_rule is reached.
See may code below.

This way we can definitely solve the problem. (at least this problem. )

Thanks for you comments.

My best regards,
Josu?.

Code: Select all

#ifdef EGTB
   if (egtb_limit >= total_pieces &&
      ((status[move_number].white_castle + status[move_number].black_castle) == 0) &&
      (!status[move_number].enpassant_sq)) {
      if (EgtbProbe(&score)) {
         int mate_distance = 0;

// Mate scores have to be adjusted because here "score" represents mate in N moves
// from the root. We must store "score" as mate in N moves from this position.

         alpha = score;
         if (alpha) {
            if (alpha > 0)
               score = alpha + ply;
            else
               score = alpha - ply;
            mate_distance = MATE_VALUE - abs(score);
         }

/* If 50 moves rule reached then position is a draw. */
         if ((status[move_number].fifty_moves_rule + mate_distance) > 100) {
            Retrieve_EGTB_Path(&egtb_line);
            if (no_capture or pawn_push in egtb_line before 50_moves_rule is reached) {
               pv_length[ply] = 0;
               return(0);
            }
         }


// Record EGTB score in hashtables -> "always" and "depth" elements.
         if (!follow_pv) {
            hash_entry->always.flag.bound = hash_entry->depth.flag.bound = EGTB_SCORE;
            hash_entry->always.move = hash_entry->depth.move = NO_MOVE;
            hash_entry->always.hashkey = hash_entry->depth.hashkey = hashkey;
            hash_entry->always.score = hash_entry->depth.score = score;
            hash_entry->always.depth_remaining = hash_entry->depth.depth_remaining = MAX_DEPTH;

            if (age != hash_entry->depth.flag.age) {
               hashtable_write++;
               hash_entry->depth.flag.age = (unsigned char)age;
            }
      
            pv_length[ply] = 0;
            return(alpha);
         }
      }
   }

#endif /* EGTB */


[Uri Blass]

That code won't work and has a _serious_ bug...

say we are in an ending like KBB vs KN, and we have played 40 moves toward a mate in 70. And your test says "draw" since the mate distance is beyond the 50 move rule counter. Unfortunately, between here and where the 50 move rule counter hits 100, we are going to make a capture (winning the knight) which resets the counter. But you have already said "draw" because you can't see the counter getting reset.

That's not worth the effort since it tries to hide a problem but introduces a potentially bigger problem...

Hi Bob,

You are simply 100% right!

I did not think about that.
But you have just given me a way to by-pass the problem.

If you get a mate score from EGTB you can get the mate_path that lead to that score, right?

Without looking at your code I can say that it also does not work.

The problem is that the mate path is not the correct path.
It is possible that the mate path based on the tablebases lead to draw by the 50 move rule but the winner can get longer mate when the capture happens earlier.

The only practical solution to solve the problem is simply to generate tablebases based on the 50 move rule.

Uri

[Uri Blass]

That code won't work and has a _serious_ bug...

say we are in an ending like KBB vs KN, and we have played 40 moves toward a mate in 70. And your test says "draw" since the mate distance is beyond the 50 move rule counter. Unfortunately, between here and where the 50 move rule counter hits 100, we are going to make a capture (winning the knight) which resets the counter. But you have already said "draw" because you can't see the counter getting reset.

That's not worth the effort since it tries to hide a problem but introduces a potentially bigger problem...

If you replied to my post then I did not say that I have a draw score for mate in more than 50 but positive score that is not mate.

For example let say that I translate mate in 70 to 5 pawns advantage

The point is that practically I do not know if mate in 70 is draw or not a draw so if I can see another line with +9 pawns score I can be sure about win more relative to mate in 70 but if I see a line with +1 pawns score the mate in 70 score is more convincing.

Of course this is not perfect solution and I only consider it as better than trusting the nalimov tablebases

Uri

[Josu? Forte]

The problem is that the mate path is not the correct path.
It is possible that the mate path based on the tablebases lead to draw by the 50 move rule but the winner can get longer mate when the capture happens earlier.

You are right in your point but still I don't see any problem to avoid what you and Bob have reported.
Instead of retrieving just one possible wrong mate_path we can seek all the paths, from that point, that lead to a mate and, in the same time, analyse each path just like we do in a tree searching.
In terms of efficiency this is not a problem as this kind of positions rarely happens and you can save the result in a hashtable for future probes, avoiding the work again...

The only practical solution to solve the problem is simply to generate tablebases based on the 50 move rule.

I just don't agree for the reasons I explained above.

Bob and Uri, thanks for your comments. They are making the discussion richer and clear.

Best regards,
Josu

[bob]

That code won't work and has a _serious_ bug...

say we are in an ending like KBB vs KN, and we have played 40 moves toward a mate in 70. And your test says "draw" since the mate distance is beyond the 50 move rule counter. Unfortunately, between here and where the 50 move rule counter hits 100, we are going to make a capture (winning the knight) which resets the counter. But you have already said "draw" because you can't see the counter getting reset.

That's not worth the effort since it tries to hide a problem but introduces a potentially bigger problem...

Hi Bob,

You are simply 100% right!

I did not think about that.
But you have just given me a way to by-pass the problem.

If you get a mate score from EGTB you can get the mate_path that lead to that score, right?

So, instead of returning a draw score, as stated in my previous code, we simply need to go through the mate_path and verify if there is a capture or pawn_push before 50_moves_rule is reached.
See may code below.

This way we can definitely solve the problem. (at least this problem. )

Thanks for you comments.

My best regards,
Josu?.

Code: Select all

#ifdef EGTB
   if (egtb_limit >= total_pieces &&
      ((status[move_number].white_castle + status[move_number].black_castle) == 0) &&
      (!status[move_number].enpassant_sq)) {
      if (EgtbProbe(&score)) {
         int mate_distance = 0;

// Mate scores have to be adjusted because here "score" represents mate in N moves
// from the root. We must store "score" as mate in N moves from this position.

         alpha = score;
         if (alpha) {
            if (alpha > 0)
               score = alpha + ply;
            else
               score = alpha - ply;
            mate_distance = MATE_VALUE - abs(score);
         }

/* If 50 moves rule reached then position is a draw. */
         if ((status[move_number].fifty_moves_rule + mate_distance) > 100) {
            Retrieve_EGTB_Path(&egtb_line);
            if (no_capture or pawn_push in egtb_line before 50_moves_rule is reached) {
               pv_length[ply] = 0;
               return(0);
            }
         }


// Record EGTB score in hashtables -> "always" and "depth" elements.
         if (!follow_pv) {
            hash_entry->always.flag.bound = hash_entry->depth.flag.bound = EGTB_SCORE;
            hash_entry->always.move = hash_entry->depth.move = NO_MOVE;
            hash_entry->always.hashkey = hash_entry->depth.hashkey = hashkey;
            hash_entry->always.score = hash_entry->depth.score = score;
            hash_entry->always.depth_remaining = hash_entry->depth.depth_remaining = MAX_DEPTH;

            if (age != hash_entry->depth.flag.age) {
               hashtable_write++;
               hash_entry->depth.flag.age = (unsigned char)age;
            }
      
            pv_length[ply] = 0;
            return(alpha);
         }
      }
   }

#endif /* EGTB */


The question is, what is that going to cost? one big I/O (to read a compressed block) is expensive. if you are 40 plies away, you have to do 40 reads to walk the EGTB PV to get to the ultimate mate to see if it is a draw or if it resets the 50 move counter with a capture. But the cost seems to be beyond acceptable. Even a single EGTB probe is expensive enough, this could be 50-100-200 times worse...

[bob]

what Uri is explaining is the high computational cost. Just walking the shortest-mate PV is too expensive, but you can't afford to do that. You must do a full minimax search of all tabelbase PVs from the current node to see if you can find a capture before the 50 move rule is reached, that still leads to a forced mate within 50 moves or to a capture that again resets the counter. That you can't do in a search unless you are going to just deal with 1 ply searches to actually play the game...