I ask for permission to copy glaurung code to support UCI.

[Uri Blass]

programmers do not need to invent everything and can use library functions that other people invented.

Do you think it would be usefull to have a free library with popular functions in chess programming, like bitboard move generation, pgn reading, san decoding, and other things? Is there any such library available?

I think that having free library should be only for non chess functions.

There should be no free library for bitboard move generation.

pgn reading and san decoding has nothing to do with playing strength so it is different.

free library for pgn reading may assume existing functions like function to generate all moves.

Uri

[Alessandro Scotti]

There should be no free library for bitboard move generation.

pgn reading and san decoding has nothing to do with playing strength so it is different.

I agree with the principle, but I have nothing against free libraries only their use in possibly non-free software.
BTW, I include EGTB probing code with the "playing strength" related stuff, so that's also something I'll have to do myself if needed (Nalimov can't be beaten of course, but I think there's some serious programming fun to be had in that area).
Anyway programs published under the GPL license, such as for example GNU chess, could also be considered sort of free libraries, which any GPL program can use. That already includes at least one bitboard implementation...

[Uri Blass]

I do not agree that the problem is only with using free libraries in non free
software.

I see no reason to make the life of the commercial programmers harder.

Nalimov tablebases are already used by a lot of programs including the commercial programs and there is no reason to limit them only to old programs.

They can be also used as debugging tool for evaluation rules about endgames but I decided to delay that use to the future because I do not like to have 2 different projects when one is only for debugging.

I also disagree with the claim that it is impossible to compete with them.

Uri

[Uri Blass]

Another point is that nalimov tablebases is about fixed knowledge and not about creativity.

They certainly can be beaten because they give wrong information and they can give mate in 200 for draw by the 50 move rule.

I will be forced to translate scores like mate in 80 to something like +2
pawns when I implement the nalimov tablebases(otherwise the program may miss a simple win and go for endgame that is drawn by the 50 move rules).

I even plan to translate mate in 20 to something like +10 pawns because I do not like to see programs play stupid sacrifices even if the sacrifice wins.

Uri

[Dan Honeycutt]

Another point is that nalimov tablebases is about fixed knowledge and not about creativity.

Hi Uri:

This does not seem to agree with your earlier statement:

There should be no free library for bitboard move generation

Both are deterministic. Both require no creativity. Both benefit from efficient programming. To me tablebase code and a bitboard move generator look pretty similar. Why do you see them different?

Best
Dan H.

[Roman Hartmann]

The idea of having a free available library of chess functions sounds nice at first sight. When I started to write a chess engine about a year ago I also first thought about recycling some parts of an existing chess program in order be able to work on the interesting parts of a chess engine (evaluation) as quickly as possible. But by doing it that way I would have troubles later to claim that I wrote my own chess engine so I decided to write it from scratch.
I'm not sure if I would have started to write a chess engine at all if I would have known in prior how long it would take to only have a working move generator. But when I compare the sources I'm working on now with the first sources from about a year ago I can see a lot of progress.

By taking debugged and perfect working sources of third parties I wouldn't be able to see this kind of evolution.
Also the understanding of the sources is probably better if you actually write all/most parts of your engine rather than putting some perfect working 'lego'-parts together and call it your engine. Personally I tend to stopp thinking about a problem once I saw a working solution and therefore I also avoid looking at other sources too closely. Taking (or looking at) third parties code kills my own creativity and I try to avoid that.
This is only my very personal opinion about this topic and I'm not blaming anyone for choosing a different way, of course.

best regards
Roman

[Uri Blass]

Another point is that nalimov tablebases is about fixed knowledge and not about creativity.

Hi Uri:

This does not seem to agree with your earlier statement:

There should be no free library for bitboard move generation

Both are deterministic. Both require no creativity. Both benefit from efficient programming. To me tablebase code and a bitboard move generator look pretty similar. Why do you see them different?

Best
Dan H.

I disagree here.

Move generator is not a deterministic job because the question what data you update during move generation is important and the question what data structure you choose is important.

I think also that tablebases can be about creativity but I was talking specifically about nalimov tablebases that have 2 disadvantages relative to other possibilities:

1)nalimov tablebases are wrong because they do not consider the 50 move rule.

2)it is also possible to be more efficient in compression(as far as I know it is the reason that chessmaster does not use them but use smaller tables).

Uri

[Uri Blass]

The idea of having a free available library of chess functions sounds nice at first sight. When I started to write a chess engine about a year ago I also first thought about recycling some parts of an existing chess program in order be able to work on the interesting parts of a chess engine (evaluation) as quickly as possible. But by doing it that way I would have troubles later to claim that I wrote my own chess engine so I decided to write it from scratch.
I'm not sure if I would have started to write a chess engine at all if I would have known in prior how long it would take to only have a working move generator. But when I compare the sources I'm working on now with the first sources from about a year ago I can see a lot of progress.

By taking debugged and perfect working sources of third parties I wouldn't be able to see this kind of evolution.
Also the understanding of the sources is probably better if you actually write all/most parts of your engine rather than putting some perfect working 'lego'-parts together and call it your engine. Personally I tend to stopp thinking about a problem once I saw a working solution and therefore I also avoid looking at other sources too closely. Taking (or looking at) third parties code kills my own creativity and I try to avoid that.
This is only my very personal opinion about this topic and I'm not blaming anyone for choosing a different way, of course.

best regards
Roman

Hi Roman,
I think that interesting part of chess engine are not only evaluation but also search and move generator.

I printed a lot of my move generator on paper in order to look at the code and think about defining exactly what I want to change.

The parts that I think that it is better if programmers avoid are:
1)reading pgn
2)understanding winboard(uci) commands

It is better to start from a code that is not working but there are some missing empty chess functions(descreption what they should do can be mentioned in the code) that if you write them the code can work when the programmer can think only about chess and not about winboard code or reading pgn code.

Uri

[Roman Hartmann]

Hi Uri,

Hi Roman,
I think that interesting part of chess engine are not only evaluation but also search and move generator.
Uri

I certainly agree on that now but about a year ago when I was starting to work on my engine I wanted to have a working chess engine as fast as possible. I guess this is true for every beginner, they want to see some progress.
Also in all the forums I had a glimpse at that time people were mainly talking about improving their engines so it seemed to me that creating a working chess engine is a piece of cake anyway and the main work is about improving the evaluation function later. A year later now I know that also working on the move generator can be already a lot of fun. At least it was/is for me.

I printed a lot of my move generator on paper in order to look at the code and think about defining exactly what I want to change.
Uri

I printed a list with all the definitions of my functions to keep track of them.

The parts that I think that it is better if programmers avoid are:
1)reading pgn
2)understanding winboard(uci) commands

It is better to start from a code that is not working but there are some missing empty chess functions(descreption what they should do can be mentioned in the code) that if you write them the code can work when the programmer can think only about chess and not about winboard code or reading pgn code.
Uri

I don't have any functions dealing with PGN yet but I guess I will write them from scratch when I need them. Implementing UCI/WB isn't as easy as I thought at first and my first try to load my engine as a WB-engine under Arena was not succesfull and only caused Arena to hang. But as my search and eval functions don't work yet I will work on that before I fix the WB interface ony my engine.

Roman

[Dan Honeycutt]

Move generator is not a deterministic job because the question what data you update during move generation is important and the question what data structure you choose is important.

I'm talking about a bitboard move generator like Crafty. All it does is produce moves. It's deterministic in that in a given position there are "exactly" these moves and the move generator better produce "exactly" those moves.

If you start adding scores to the moves or use the move generator to update other information then I agree it is not deterministic.

Dan H.

[Uri Blass]

Move generator is not a deterministic job because the question what data you update during move generation is important and the question what data structure you choose is important.

I'm talking about a bitboard move generator like Crafty. All it does is produce moves. It's deterministic in that in a given position there are "exactly" these moves and the move generator better produce "exactly" those moves.

If you start adding scores to the moves or use the move generator to update other information then I agree it is not deterministic.

Dan H.

My move generator also does not give exactly the same moves as Crafty because it gives only legal moves.

Uri

[Dann Corbit]

Another point is that nalimov tablebases is about fixed knowledge and not about creativity.

Hi Uri:

This does not seem to agree with your earlier statement:

There should be no free library for bitboard move generation

Both are deterministic. Both require no creativity. Both benefit from efficient programming. To me tablebase code and a bitboard move generator look pretty similar. Why do you see them different?

Best
Dan H.

I disagree here.

Move generator is not a deterministic job because the question what data you update during move generation is important and the question what data structure you choose is important.

I think also that tablebases can be about creativity but I was talking specifically about nalimov tablebases that have 2 disadvantages relative to other possibilities:

1)nalimov tablebases are wrong because they do not consider the 50 move rule.

2)it is also possible to be more efficient in compression(as far as I know it is the reason that chessmaster does not use them but use smaller tables).

Uri

If Nalimov tables considered the 50 move rule, they would be literally millions of times larger. I think it would be a very, very bad idea to put that into them. It is also very easy for the programmer to track that part.
Addition of castling rights would also make them 16 times larger with very little benefit. Most of the time, when you are down to 6 chessmen or less, you cannot castle anymore anyway.

I do not think that the size of Nalimov tablebase files is a problem. Eugene uses a very efficient compression scheme.
De Koning EDGB are smaller because they do not store win/loss/draw/broken but only win/not-win. The De Koning database cannot tell if a move loses or it draws. It can only see win or not.
Also, that ChessMaster format is not published so you cannot use it.

[Uri Blass]

Jos? C.: I'm so sorry I edited this message by Uri instead of replying to it. My big mistake. I'll ask Volker if this can be corrected. I'm afraid I'll be fired as a moderator for being so clumsy

I do not understand.

I think that by considering the 50 move rule the tablebases can be smaller.

The tablebase need to include only distance to conversion that can be
0-99 or draw and it is less bits than distance to mate that can be more than 200.

That wouldn't be any helpful. You can do it in your program with:

Code: Select all

if (EGTB_distance_to_mate + current_fifty_count > 50)
  return(DRAW)


But I think what he meant (and would make tables much bigger) is to consider captures and promotions inside the mating line, which is the point that makes long mates uncertain (will there by a capture before the fifty rule hits?). That would also need to consider actual fifty_count, so there should be 50x positions stored in the table, one for every value of fifty_count.

[Dan Honeycutt]

My move generator also does not give exactly the same moves as Crafty because it gives only legal moves.

When I wrote the word "exactly" I wondered "is Uri going to point out that it generates psudo-legal moves?" I thought "nahhh".

Dan H.

[José Carlos]

Sorry, I clicked "Edit" instead of "Quote" and modified Uri's post.
I'll ask Volker if this mistake can be corrected.
My apologies again.

[Anonymous]

Dann Corbit wrote:
>If Nalimov tables considered the 50 move rule, they would be literally
> millions of times larger.

Why would they be so much larger. My TBs (private at the moment) are only a few precent bigger than uncompressed Nalimov TBs. They do handle the 50 moves rule correctly. They don't always give the shortest mate, but this really is no problem.

> I think it would be a very, very bad idea to put that into them. It is
> also very easy for the programmer to track that part.

What do you have in mind, here?

>Addition of castling rights would also make them 16 times larger with
>very little benefit.

No, only a couple percents bigger for TBs with pawns. A bit more for pawnless TBs. Only when the white K is on E1, you ave to consider castling. Most positions in TBs have the white king not on E1. It is possible to encode all KK positions including castling rights in 12 bit. (The trivial 64 squares times 64 already needs 12 big, but Nalimov is of course a bit more efficient here).

> Most of the time, when you are down to 6 chessmen or less, you cannot
> castle anymore anyway.

I agree. Adding castling would be a total waste, even if it only adds little space. I did a search in your "crapbase", and there was not even one position with 6 men and castling rights. 50 moves rule is also somewhat accademic. But I happen to be interested to avoid this problem.

Regards,
Dieter

[Uri Blass]

Jos? C.: I'm so sorry I edited this message by Uri instead of replying to it. My big mistake. I'll ask Volker if this can be corrected. I'm afraid I'll be fired as a moderator for being so clumsy

I do not understand.

I think that by considering the 50 move rule the tablebases can be smaller.

The tablebase need to include only distance to conversion that can be
0-99 or draw and it is less bits than distance to mate that can be more than 200.

That wouldn't be any helpful. You can do it in your program with:

Code: Select all

if (EGTB_distance_to_mate + current_fifty_count > 50)
  return(DRAW)


But I think what he meant (and would make tables much bigger) is to consider captures and promotions inside the mating line, which is the point that makes long mates uncertain (will there by a capture before the fifty rule hits?). That would also need to consider actual fifty_count, so there should be 50x positions stored in the table, one for every value of fifty_count.

Hi Jose,
As dieter explained the point is not to have distance to mate but distance to conversion in the tables.

The program may miss the shortest mate but it is not important and it is important only that it will win won positions.

Uri

[Uri Blass]

Dann Corbit wrote:
>If Nalimov tables considered the 50 move rule, they would be literally
> millions of times larger.

Why would they be so much larger. My TBs (private at the moment) are only a few precent bigger than uncompressed Nalimov TBs. They do handle the 50 moves rule correctly. They don't always give the shortest mate, but this really is no problem.

> I think it would be a very, very bad idea to put that into them. It is
> also very easy for the programmer to track that part.

What do you have in mind, here?

>Addition of castling rights would also make them 16 times larger with
>very little benefit.

No, only a couple percents bigger for TBs with pawns. A bit more for pawnless TBs. Only when the white K is on E1, you ave to consider castling. Most positions in TBs have the white king not on E1. It is possible to encode all KK positions including castling rights in 12 bit. (The trivial 64 squares times 64 already needs 12 big, but Nalimov is of course a bit more efficient here).

> Most of the time, when you are down to 6 chessmen or less, you cannot
> castle anymore anyway.

I agree. Adding castling would be a total waste, even if it only adds little space. I did a search in your "crapbase", and there was not even one position with 6 men and castling rights. 50 moves rule is also somewhat accademic. But I happen to be interested to avoid this problem.

Regards,
Dieter

I also agree that adding castling is a total waste unlike adding 50 move rule.

I remember one example for a drawn game when programs went into tablebases for forced mate but there was a draw by the 50 move rule.

Uri

[José Carlos]

Yes, distance to conversion sounds like a good idea.
But I think it also has flaws: after the conversion, the position might be a mate in 51, so it should be considered draw, also before the conversion. It might also happen to need yet another conversion to get the mate.
Maybe all of this is solvable or very rare to be important...

[Uri Blass]

Distance to conversion is clearly enough if you start from the simple positions and later use your results for more complicated positions.

Conversion cannot lead to draw by the 50 move rule in that case because
the relevant position after the conversion will be considered as a draw by the tablebases so the search based on tablebases will not calculate it as a relevant conversion to win the game.

Uri