Glaurung 0.2.0

[Tord Romstad]

Hi folks,

The source code and Mac OS X binaries for Glaurung 0.2.0 are now ready for download. The big news this time is that Glaurung finally has an opening book. Make sure you download the correct book file: The opening book file for Mac OS X is different from the one for Linux and Windows. Other notable changes are an improved hash table replacement scheme, hashing in the qsearch, and slightly higher (but perhaps still too low) scores for passed pawns. My initial results with the new version are surprisingly good, and it is possible (but by no means sure) that it will turn out to be a significant improvement compared to 0.1.7. I think the difference will be most visible at slow time controls.

This will, I hope, be the last new public version for a while. From now on, I plan to concentrate on adding endgame knowledge, tuning eval weights and search parameters, and cleaning up the more messy parts of the source code.

Tord

[Alessandro Scotti]

Hi Tord,
great news... and you still give the source away... that's crazy! Anyway I've downloaded the Mac OS X executable... only 3 weeks or so before I can use it!

[Dann Corbit]

New binary on my ftp site.

Question:
Is the book generator code available?
I would be curious to see what you did.

[Dann Corbit]

I have some problems and I am not sure what causes it.

Most of the time, Glaurung loses on time.

Perhaps something is wrong with my book that I got from your site. I chose the one that said Linux and Windows.

If it assumes some particular struct alignment, then I am sure that is what is wrong.

It is definitely book related. If I rename the book to foo.bar it plays fine, but it thinks from the initial board position (obviously)

[Tord Romstad]

Hi Alessandro!

Hi Tord,
great news... and you still give the source away... that's crazy!

I've found that I rather like being an open source author. It is possible that it will stay that way.

Anyway I've downloaded the Mac OS X executable... only 3 weeks or so before I can use it!

Meanwhile, you can try the Linux version. I haven't yet compiled a Linux executable, but it should be very easy to build the program from the source code. Unpacking and typing 'make' should do the trick.

Tord

[Tord Romstad]

Hi Dann,

Thanks for your help! The code for building the book is not ready for download yet, but if people are interested I may make it available later. The book building process is done in two stages. First I take a big PGN file and runs it through a little utility called 'unpgnify'. The output is a text file where all PGN tags except the result is discarded, the moves are converted to coordinate notation, the games are truncated to 60 plies, and there is exactly one game per line. For instance, the game 1. e4 d5 2. exd5 Qxd5 3. Ke2 Qe4# would give the following output line:

Code: Select all

B e2e4 d7d5 e4d5 d8d5 e1e2 d5e4

The 'B' at the beginning of the line means that the game was won by black. I use 'W' and 'D' for white wins and draws.

I run this text file through a second program which produces the binary book file. It parses all games and stores all hash keys, moves and results in memory. When all games are processed, I collapse this data into a more compact format. For each (position, move) pair which occurs somewhere in the input file, I count the number of wins, draws and losses. Finally, I write the (position, move, wins, losses, draws) tuples to disk. Moves which have scored less than 40% or have been played only a couple of times are discarded.

Each entry in the book file consists of 16 bytes, which are read and written as two 64-bit unsigned ints. The first 64-bit int in the entry contains of 48 bits of the hash key and 16 bits for the move. The second 64-bit int contains the number of wins, losses and draws, 20 bits for each. I plan to use the remaining 4 bits for some sort of tournament book later (a bit which says "don't ever play this move" and another bit which says "always play this move in this position", for instance).

I have some problems and I am not sure what causes it.

Most of the time, Glaurung loses on time.

Perhaps something is wrong with my book that I got from your site. I chose the one that said Linux and Windows.

Very strange. I just tested in Linux, and it works fine ...

If it assumes some particular struct alignment, then I am sure that is what is wrong.

As far as I can see, no assumptions are made about struct alignment, but I do make assumptions about the order of the bytes in 64-bit ints. I thought this was always the same on little-endian processors, but perhaps I am wrong and this depends on the compiler.

If you or somebody else could compile the following little program, put it in the same directory as Glaurung's book.bin file, and run the program, we could perhaps understand the nature of the problem:

Code: Select all

#include <stdio.h>

int main(void) {
  unsigned long long x;
  int i;
  FILE *f;

  f = fopen("book.bin", "r");
  if(f == NULL)
    printf("book.bin not found!\n");
  else {
    for(i=0; i<10; i++) {
      fread(&x, sizeof(unsigned long long), 1, f);
      printf("%llx\n", x);
    }
    fclose(f);
  }
  return 0;
}


If everything works, the output should look like this:

Code: Select all

e78289e91994
30000000000
11f449fe80007
30000000000
124efde290113
30000100000
1afd30e893a76
30000000000
1b4ebed3518c0
30000000000


I hope we will be able to fix this before Monday.

Tord

[Jim Ablett]

Hi Tord,
I did as you asked. Here's the output >

Code: Select all

89e91994
0
9fe80007
0
de290113
100000
e893a76
0
ed3518c0
0


Jim.

[Tord Romstad]

Hi Jim,

Thanks a lot for your help! I think we are now much closer to understanding the problem. It looks like only 32 bits are read, despite the 'sizeof(unsigned long long)' in the call to fread(). What happens if you use the following program instead?

Code: Select all

#include <stdio.h>

int main(void) {
  union {
    unsigned long long a;
    unsigned b[2];
  } x;
  int i;
  FILE *f;

  f = fopen("book.bin", "r");
  if(f == NULL)
    printf("book.bin not found!\n");
  else {
    for(i=0; i<10; i++) {
      fread(&(x.b[0]), sizeof(unsigned), 1, f);
      fread(&(x.b[1]), sizeof(unsigned), 1, f);
      printf("%llx\n", x.a);
    }
    fclose(f);
  }
  return 0;
}


Tord

[Jim Ablett]

Same result.

Jim.

[Tord Romstad]

Same result.

Curiouser and curiouser.

How about this program?

Code: Select all

#include <stdio.h>

int main(void) {
  union {
    unsigned long long a;
    unsigned b[2];
  } x;
  int i;
  FILE *f;

  f = fopen("book.bin", "r");
  if(f == NULL)
    printf("book.bin not found!\n");
  else {
    for(i=0; i<10; i++) {
      fread(&(x.b[0]), sizeof(unsigned), 1, f);
      fread(&(x.b[1]), sizeof(unsigned), 1, f);
      printf("%llx %x %x\n", x.a, x.b[0], x.b[1]);
    }
    fclose(f);
  }
  return 0;
}


Thank you very much for helping me with this strange problem. I hope I will not have to bother you much more.

Tord

[Jim Ablett]

Different result this time >

Code: Select all

89e91994 e782 89e91994
0 300 0
9fe80007 11f44 9fe80007
0 300 0
de290113 124ef de290113
100000 300 100000
e893a76 1afd3 e893a76
0 300 0
ed3518c0 1b4eb ed3518c0
0 300 0


Jim.

[Tord Romstad]

Different result this time >

Code: Select all

89e91994 e782 89e91994
0 300 0
9fe80007 11f44 9fe80007
0 300 0
de290113 124ef de290113
100000 300 100000
e893a76 1afd3 e893a76
0 300 0
ed3518c0 1b4eb ed3518c0
0 300 0


Aha!

I think we've finally found out what happens. First of all, your compiler does not seem to understand the %llx printf directive (which is not a serious problem, it just made the output in your previous messages a bit confusing). Second, it seems that the upper and lower half of 64-bit ints appear in the opposite order in gcc for Linux compared to in Microsoft's compiler (if that is what you are using) for Windows.

Can I mail you (or some other volunteer) the source code of a slightly modified Glaurung 0.2.0 where I have attempted to solve this problem? I want to be sure the problem is solved before I upload a new version of the source code to my Web site.

Tord

[Jim Ablett]

Fast work my friend!
Sure Tord,
I'd be happy to test it for you.
So what will I be looking for? - Just time losses?


Jim.

[Tord Romstad]

Fast work my friend!
Sure Tord,
I'd be happy to test it for you.

Great, Jim!

I decided to put it on my Web site, but without any link on the main page. You can download the new source code by clicking here.

So what will I be looking for? - Just time losses?

No, you can do something easier than that: For debugging purposes, the version I just made includes the ability to print all book moves in the current position with some statistics. Compile the program, put the executable in the same directory as the book file, and start the program from the command line. Enter the command 'book'. If everything is working, you should see the following output:

Code: Select all

book
e2e4 (+42980, =55832, -30896), 54%, n=129708, p=0.417930
d2d4 (+37420, =50441, -23817), 56%, n=111678, p=0.359836
g1f3 (+10305, =16488, -6813), 55%, n=33606, p=0.108281
c2c4 (+10094, =14052, -6311), 56%, n=30457, p=0.098134
g2g3 (+1101, =1531, -747), 55%, n=3379, p=0.010886
b2b3 (+264, =337, -246), 51%, n=847, p=0.001819
f2f4 (+166, =242, -188), 48%, n=596, p=0.001280
b1c3 (+101, =114, -103), 49%, n=318, p=0.000683
b2b4 (+73, =60, -60), 53%, n=193, p=0.000621
g2g4 (+16, =8, -6), 66%, n=30, p=0.000322
d2d3 (+14, =31, -21), 44%, n=66, p=0.000095
e2e3 (+12, =16, -15), 46%, n=43, p=0.000062
c2c3 (+8, =7, -8), 50%, n=23, p=0.000049


Tord

[milix]

Hi Tord!
BTW, you can make your code endian-free and use the same book binary for both Mac and PC by storing and retrieving bytes and then convert them to integers. For example, to store a 16bit integer you have to:

Code: Select all

fputc(i>>8,fp);
fputc(i-(i>>8),fp);


and to read them back with

Code: Select all

b = fgetc(fp);
i = b<<8;
b = fgetc(fp);
i += b;


.

[Tord Romstad]

Hi Milix!

Hi Tord!
BTW, you can make your code endian-free and use the same book binary for both Mac and PC by storing and retrieving bytes and then convert them to integers.

Yes, I could, and I suppose I should. I was afraid this would slow things down a bit, but perhaps it is not so bad. Thanks for the suggestion!

Tord

[Tord Romstad]

I just realized that there could be a simple solution to this problem without even compiling a new version of my program. Why not just create a special book binary file for Windows with the higher and lower words swapped?

Try to download this book file, and unpack it in the same directory as Glaurung 0.2.0 for Windows. With some luck, this should work. The 'book' command as described in my latest reply to Jim above is not included in Glaurung 0.2.0. The only way to test whether this new book file works in Glaurung is to play games and observe whether or not the program plays book moves.

Tord

[Jim Ablett]

The only way to test whether this new book file works in Glaurung is to play games and observe whether or not the program plays book moves.

No, i'm afraid it isn't Tord.

Jim.

[Jim Ablett]

Actually using the original 'book.bin' it doesn't
play any book moves either.

[Tord Romstad]

OK, here is yet another attempt, using something similar to Milix' suggestion. If everything works as intended, the same book file should now work regardless of endianness and compiler. I have verified that I can now use the same book file in Linux and Mac OS X.

Try to compile Glaurung from this source code, and test it with this book file.

Tord