Looking for linux 64 bit engines & epds

[Daniel Shawul]

Thank you Illari for such a great tool !
I have run about 16,000 games at 40/30 with it on a cluster mostly using only 16 processors.
I only got 2 games missing one from Toga and another from Fruit ?? I don't know if that is the
problem of the engines or if these things happen on clusters.

I am using the pgn of games produced from cutechess-cli to send it to bayeselo. At first, I was
afraid the pgns might be big but it turns out they are really small with -min option. I wrote a small MPI code (just studied it
2 days ago) to manage pairings, match engines etc... If it helps anyone to get started, here is the code

Code: Select all

#include "mpi.h"
#include <math.h>

static const char* const engines[] = {
       "cmd=./scorpio dir=/home/dabdi/Work/scorpio",
       "cmd=./scorpio dir=/home/dabdi/Work/scorpio_241",
   "cmd=./scorpio dir=/home/dabdi/Work/scorpio_24",
        "cmd=./crafty dir=/home/dabdi/Work/crafty-23.1",
   "cmd=./fruit proto=uci dir=/home/dabdi/Work/fruit_21/src",
   "cmd=./glaurung proto=uci option.Threads=1 dir=/home/dabdi/Work/glaurung22/src",
   "cmd=./spike proto=uci dir=/home/dabdi/Work/Spike_12",
       "cmd=./arasanx proto=uci dir=/home/dabdi/Work/arasan-11.6.0/export",
       "cmd=./doch64-134-ja proto=uci dir=/home/dabdi/Work/Doch134/Gcc",
       "cmd=./Toga proto=uci dir=/home/dabdi/Work/Toga",
       "cmd=./Hermann proto=uci dir=/home/dabdi/Work/Hermann25",
   NULL
};

typedef unsigned int JOB;
#define PORTION(x) ((x >> 16) & 0xffff)
#define SECOND(x)  ((x >> 8) & 0xff)
#define FIRST(x)   (x & 0xff)

int main(int argc, char* argv[] ) {
   char str[512];
   int myid,nprocs,namelen,master;
   char processor_name[MPI_MAX_PROCESSOR_NAME];
   MPI_Status status;
   JOB job;

   MPI_Init(&argc,&argv);
   MPI_Comm_size(MPI_COMM_WORLD,&nprocs);
   MPI_Comm_rank(MPI_COMM_WORLD,&myid);
   MPI_Get_processor_name(processor_name, &namelen);

   master = 0;
   nprocs--;

   //master
   if(myid == master) {
      int r,result,njobs,sent,N,NTOTAL;
              const int rounds = 1;
      
              //count number of engines
              N = 0;
          while(engines[N]) N++;
      
       //round robin or gauntlet
      JOB* jobs = new JOB[(N * (N - 1)) / 2];
              njobs = 0;
      for(int i = 0;i < rounds;i++) {
         for(int j = i + 1;j < N;j++) {
            njobs++;
         }
      }
      NTOTAL = njobs;

              //divide pgn
      int divide = int(ceil(nprocs / (float)NTOTAL));
      int dgames = 150 / divide,games = 0;

      FILE* f = fopen("all.pgn","r");
      FILE* f1 = 0;
      char buffer[512];

      strcpy(buffer,"");
      while(fgets(buffer,512,f)) {
         if(strncmp(buffer,"[Event ",7) == 0) {
            if((games % dgames) == 0) {
               if(f1)   fclose(f1);
               sprintf(str,"./divide/all%d.pgn",games / dgames);
               f1 = fopen(str,"w");
            }
            games++;
         }
         fprintf(f1,"%s",buffer);
              }
      if(f1) fclose(f1);
      fclose(f);

              //assign jobs
      njobs = 0;
      for(int k = 0;k < divide;k++) {
                 for(int i = 0;i < rounds;i++) {
            for(int j = i + 1;j < N;j++) {
               jobs[njobs++] = ((k << 16) | (j << 8) | i);
            }
         }
      }
      NTOTAL = njobs;
   
             
      //send slaves to work
      sent = 0;
      njobs = 0;                   
      while(njobs < NTOTAL) {
         sent++;
         MPI_Send(&jobs[njobs++],1,MPI_INT,sent,njobs,MPI_COMM_WORLD);
         if(sent >= nprocs) break;
      }
      while(sent) {
         MPI_Recv(&r,1,MPI_INT,MPI_ANY_SOURCE,MPI_ANY_TAG,MPI_COMM_WORLD,&status);
         if(njobs < NTOTAL) {
            MPI_Send(&jobs[njobs++],1,MPI_INT,r,njobs,MPI_COMM_WORLD);
         } else {
            MPI_Send(MPI_BOTTOM,0,MPI_INT,r,0,MPI_COMM_WORLD);
            sent--;
         }
      }

              //delete
              delete jobs;
   }
   //slave
   else {
      while(true) {
         MPI_Recv(&job,1,MPI_INT,master,MPI_ANY_TAG,MPI_COMM_WORLD,&status);
         if(status.MPI_TAG == 0) {
            break;
         } else {
            sprintf(str,"./cutechess-cli.sh -pgnin ./divide/all%d.pgn -repeat -games 300  -both tc=40/30 -fcp %s -scp %s -pgnout test.pgn min",
                                        PORTION(job),engines[FIRST(job)],engines[SECOND(job)]);
                            //printf("%s\n",str);
                            system(str);
            MPI_Send(&myid,1,MPI_INT,master,status.MPI_TAG,MPI_COMM_WORLD);
         }
      }
   }

   MPI_Finalize();

   return 0;
}


The good news is that i have now a working setup which can allow me to sit tight and press the button
Some random changes, which i thought will improve things, but actually hurt ! I do not have the intention to
nail the elos down to 1 point. +-10 standard error is good enough for me (about 3000 games).

Code: Select all

Num. Name             games   score
   0 Scorpio_2.4.4      299   133.5
   1 Glaurung 2.2       635   465.5
   2 Fruit 2.1          635     306
   3 Scorpio_2.4.1     2740    1249
   4 Toga II 1.3.1      616     413
   5 Arasan 11.6        619   169.5
   6 Hermann 2.5        618   182.5
   7 Scorpio_2.4       2743  1254.5
   8 Spike 1.2 Turin    635     332
   9 Doch64 1.3.4 JA    620   494.5
  10 Crafty-23.1        636     398
Rank Name              Elo    +    - games score oppo. draws
   1 Doch64 1.3.4 JA   208   26   25   620   80%   -27   18%
   2 Glaurung 2.2      148   23   23   635   73%   -27   21%
   3 Toga II 1.3.1      98   23   22   616   67%   -27   22%
   4 Crafty-23.1        65   22   22   636   63%   -27   23%
   5 Spike 1.2 Turin   -10   21   21   635   52%   -27   26%
   6 Scorpio_2.4       -25   11   11  2743   46%     6   24%
   7 Scorpio_2.4.1     -29   11   11  2740   46%     6   24%
   8 Scorpio_2.4.4     -30   31   31   299   45%     7   32%
   9 Fruit 2.1         -41   21   22   635   48%   -27   24%
  10 Hermann 2.5      -184   23   24   618   30%   -27   18%
  11 Arasan 11.6      -199   23   24   619   27%   -27   19%


Thank you
Daniel

[Daniel Shawul]

Hi Dann

Thanks for the code and the epd collection. I forgot that PGN do have FEN capabilities in them
(I am so old for my age ) I will use it as an endless source of start positions
for whatever number of games i want to play.
I want to divide your database into a number of parts (say 300 postions in each sample) so that when i want to do smaller number of games ,
i will use the first, and then the second etc... For that to work, each sample should be representative
of the whole epd suite. Can you sort the epds, in such a way that i have similar probabilities of getting the same result if i select any
of the samples. I hope you are getting my point.

regards,
Daniel

[Ilari Pihlajisto]

Thank you Illari for such a great tool !

I'm glad you find it useful. It's still in an early stage of development, so stay tuned for some impressive features.

I only got 2 games missing one from Toga and another from Fruit ?? I don't know if that is the
problem of the engines or if these things happen on clusters.

What do you mean by missing games? Are they missing just from the PGN output, or did cutechess-cli not play them at all? If an engine crashes or becomes unresponsive, cutechess-cli will stop the whole match, which might explain the missing games. The next version will handle misbehaving engines better: I already have committed many of the fixes to the source code repository, just haven't released the new binaries yet.

I wrote a small MPI code (just studied it
2 days ago) to manage pairings, match engines etc... If it helps anyone to get started, here is the code

Thanks. I've been looking into making cutechess-cli more cluster-friendly (now -concurrency only works on a single machine). Maybe this will help, or at least give some ideas.

BTW, If you want the matches to go faster, you may want to use the -resign and/or -draw options to adjudicate the games when the outcome seems clear.

[Daniel Shawul]

Here is the link to the games http://sites.google.com/site/dshawul/ga ... ects=0&d=1 where two of the games are missing.
Unfortunaely i turned off all log files (of engines, and cutechess-cli). I will atleast turn on the cutechess-cli debug next time.
I read that Bob Hyatt had similar problems now and then, so it may be that it is what it is. The only reason why it made me suspicious is
because it only happend to Fruit and Toga.

Bayeselo output

Code: Select all

Num. Name             games   score
   0 Scorpio_2.4       3000  1384.5
   1 Doch64 1.3.4 JA    900     704
   2 Fruit 2.1          899     432
   3 Toga II 1.3.1      899   604.5
   4 Spike 1.2 Turin    900   461.5
   5 Glaurung 2.2       900   643.5
   6 Crafty-23.1        900   558.5
   7 Hermann 2.5        900   283.5
   8 Arasan 11.6        900   253.5
   9 Scorpio_2.4.1     2999  1376.5
  10 Scorpio_2.4.2     2999    1396
Rank Name              Elo    +    - games score oppo. draws
   1 Doch64 1.3.4 JA   194   21   20   900   78%   -26   18%
   2 Glaurung 2.2      136   20   19   900   72%   -26   20%
   3 Toga II 1.3.1     100   19   19   899   67%   -26   23%
   4 Crafty-23.1        61   19   18   900   62%   -26   24%
   5 Spike 1.2 Turin   -16   18   18   900   51%   -26   27%
   6 Scorpio_2.4.2     -23   11   10  2999   47%     2   26%
   7 Scorpio_2.4       -25   11   10  3000   46%     3   26%
   8 Scorpio_2.4.1     -29   11   10  2999   46%     3   25%
   9 Fruit 2.1         -40   18   18   899   48%   -26   24%
  10 Hermann 2.5      -166   19   19   900   32%   -26   18%
  11 Arasan 11.6      -192   19   20   900   28%   -26   19%


BTW, If you want the matches to go faster, you may want to use the -resign and/or -draw options to adjudicate the games when the outcome seems clear.

Ok i will turn on both options next time. My old versions of scorpio doesn't like it when an opponent engine interrupts them with a "resign" message while
searching during their turn. Inthat case they just hung up. It is because of my messy winboard code. I will check if they can handle cutechess-cli forcing them
to resign.
If you have new versions you want to get tested, I don't mind testing them. You can send them to me at dshawul at yahoo.com

[Ilari Pihlajisto]

Here is the link to the games http://sites.google.com/site/dshawul/ga ... ects=0&d=1 where two of the games are missing.
Unfortunaely i turned off all log files (of engines, and cutechess-cli). I will atleast turn on the cutechess-cli debug next time.
I read that Bob Hyatt had similar problems now and then, so it may be that it is what it is. The only reason why it made me suspicious is
because it only happend to Fruit and Toga.

Thanks. Now I think I know what the problem is: you have multiple cutechess-cli processes running in parallel, and they're all writing the games to the same file. If two or more processes try to write to the file at the same time, you may end up with missing games. There are a couple of possible solutions:
1. You can set a different output file for each cutechess-cli process, and combine the files when you've done
2. I could modify cutechess-cli to use a locked file for PGN output to make sure that write access is always serialized

BTW, If you want the matches to go faster, you may want to use the -resign and/or -draw options to adjudicate the games when the outcome seems clear.

Ok i will turn on both options next time. My old versions of scorpio doesn't like it when an opponent engine interrupts them with a "resign" message while
searching during their turn. Inthat case they just hung up. It is because of my messy winboard code. I will check if they can handle cutechess-cli forcing them
to resign.

The -resign option doesn't mean that Scorpio gets a "resign" message. Cutechess-cli will just stop the game when the resign conditions are met, so it should work with all engines. The output should look like this (here I used "-resign 5 300"):

Code: Select all

70336 <Scorpio_2.4 JA(0): 15 -1064 222 4156140 Kg4-f4 Ba4-d7 Kf4-e5 a5-a4 Ke5-d6 a4-a3 Kd6xd7 Kc3-d4 Bg8-d5 a3-a2 Kd7-c6 a2-a1=Q Kc6xb6 Qa1-b2 Kb6-c6 Qb2-h2 Kc6-d7 Qh2xh4
70447 <Scorpio_2.4 JA(0): 15 -1064 233 4401078 Kg4-f4 Ba4-d7 Kf4-e5 a5-a4 Ke5-d6 a4-a3 Kd6xd7 Kc3-d4 Bg8-d5 a3-a2 Kd7-c6 a2-a1=Q Kc6xb6 Qa1-b2 Kb6-c6 Qb2-h2 Kc6-d7 Qh2xh4
70447 <Scorpio_2.4 JA(0): nodes = 4401078 <68 qnodes> time = 2334ms nps = 1885637
70447 <Scorpio_2.4 JA(0): lazy_eval = 65 splits = 0 badsplits = 0 egbb_probes = 0
70447 <Scorpio_2.4 JA(0): move g4f4
70447 >Glaurung(1): position startpos moves c2c4 e7e6 d2d4 g8f6 b1c3 f8b4 g1f3 e8g8 d1d3 b7b6 c1g5 c8a6 f3e5 d7d6 d3f3 d6e5 d4e5 b8d7 e5f6 d7f6 a1d1 d8e7 e2e3 a8d8 f1d3 h7h6 g5f6 e7f6 f3f6 g7f6 e1e2 b4c3 b2c3 g8g7 h2h4 c7c5 d1d2 d8d7 h1d1 f6f5 a2a4 f8d8 e2e1 a6b7 f2f3 g7f6 d3c2 d7d2 d1d2 d8d2 e1d2 f6e5 d2c1 b7c6 c1d2 a7a6 d2c1 f5f4 c1d2 f7f5 d2d3 f4e3 d3e3 f5f4 e3f2 e5d6 c2d1 e6e5 g2g3 f4g3 f2g2 e5e4 f3e4 c6e4 g2g3 d6e5 g3g4 e4g6 d1f3 g6e8 f3d1 e5e4 d1c2 e4e3 c2f5 a6a5 f5c8 e3d3 c8e6 d3c3 e6g8 e8a4 g4f4
70447 >Scorpio_2.4 JA(0): force
70447 >Scorpio_2.4 JA(0): result 0-1 {Black wins by adjudication}
70447 >Glaurung(1): isready
70447 <Glaurung(1): readyok
Game 1 ended: 0-1 {Black wins by adjudication}
Glaurung wins the game as black
Score of Scorpio_2.4 JA vs Glaurung: 0 - 1 - 0
70646 >Scorpio_2.4 JA(0): quit
70646 >Glaurung(1): quit
70647 <Scorpio_2.4 JA(0): Bye Bye
Finished match


If you have new versions you want to get tested, I don't mind testing them. You can send them to me at dshawul at yahoo.com

If you want bleeding edge, you can compile our sources: http://repo.or.cz/w/sloppygui.git
But I recommend waiting for tagged releases, the next one's coming out soon.

[Dann Corbit]

Hi Dann

Thanks for the code and the epd collection. I forgot that PGN do have FEN capabilities in them
(I am so old for my age ) I will use it as an endless source of start positions
for whatever number of games i want to play.
I want to divide your database into a number of parts (say 300 postions in each sample) so that when i want to do smaller number of games ,
i will use the first, and then the second etc... For that to work, each sample should be representative
of the whole epd suite. Can you sort the epds, in such a way that i have similar probabilities of getting the same result if i select any
of the samples. I hope you are getting my point.

regards,
Daniel

Right now, I am working on a special EPD set. It will have extensive statistics and analysis incorporated.
I have the EPD stored in a SQL*Server database, and so I can perform any sort of operation against it using SQL queries.
One of my tables has 200 million rows in it. That one is a little unweildy for general queries, so I have a second table which summarized the data in the big table.

Let me know exactly what it is that you want and I will provide it.

[Daniel Shawul]

Wow!That is one big collection.
Ok I will tell you what i need roughly.
From your database, you can generate start positions and classify them
based on their ECO codes A,B,C,D,E. Then many samples of same size ,say 150 positions,
are taken with some assumed distribution wrt ECO codes. I would guess normal or something
slightly skewed to the left. Each distribution should have use that same distribution so that
same result is expected which ever one I pick. Later on when i add up the results from the different
samples, I expect the result to quickly converge into a normal distribution according to CLT.
So in short if Sample 1 has 20A's, 30B's 25C's then Sample 2 should have that much.
You can use longer book lines to increase variation but i wouldn't expect that to be a problem
for you with that much games

Thanks
Daniel

[Dann Corbit]

Wow!That is one big collection.
Ok I will tell you what i need roughly.
From your database, you can generate start positions and classify them
based on their ECO codes A,B,C,D,E. Then many samples of same size ,say 150 positions,
are taken with some assumed distribution wrt ECO codes. I would guess normal or something
slightly skewed to the left. Each distribution should have use that same distribution so that
same result is expected which ever one I pick. Later on when i add up the results from the different
samples, I expect the result to quickly converge into a normal distribution according to CLT.
So in short if Sample 1 has 20A's, 30B's 25C's then Sample 2 should have that much.
You can use longer book lines to increase variation but i wouldn't expect that to be a problem
for you with that much games

Thanks
Daniel

Here is the problem:
I have tens of thousands of fully analyzed and statistically classified EPD strings.
However, I do not know the ECO codes for the strings, and I do not have any utility that will classify them for me.
I can input them into a SCID database but it classified every position as A00a and that is not correct.
PGN-EXTRACT by Barnes also does not know how to deal with them.

For instance, here are 1344 carefully analyzed Epd records:
http://cap.connx.com/epd/analyzed.epd.bz2
they are in a format like the following:
rnbqkbnr/pp1p1ppp/4p3/2p5/2P5/2N2N2/PP1PPPPP/R1BQKB1R b KQkq - acd 22; bm Nf6; ce -8; pm Nf6 {649} Nc6 {192} b6 {159} a6 {116} d5 {89}; pv Nf6 e3 Nc6 Be2 d5 cxd5 exd5 d4 cxd4 exd4 Bd6 O-O O-O Qb3 Na5 Qc2 Nc6 Be3 Nb4 Qb3 Bf5 Ne5 Bc2 Qa3 Nd3;

Here is a breakdown of the components of this position:

The base position:
rnbqkbnr/pp1p1ppp/4p3/2p5/2P5/2N2N2/PP1PPPPP/R1BQKB1R b KQkq -

The depth in plies analyzed:
acd 22

The move chosen by computer analysis:
bm Nf6

The computer analysis in centipawns for this position:
ce -8

The statistical counts for actual moves played by strong players for this position, with frequencies in curly braces:
pm Nf6 {649} Nc6 {192} b6 {159} a6 {116} d5 {89}
In this case, Nf6 was played 649 times, Nc6 was played 192 times, b6 was played 159 times, a6 was played 116 times, and d5 was played 89 times. The computer best move coincides with the most frequently played move in this instance.

This the the computer generated pv for this position:
pv Nf6 e3 Nc6 Be2 d5 cxd5 exd5 d4 cxd4 exd4 Bd6 O-O O-O Qb3 Na5 Qc2 Nc6 Be3 Nb4 Qb3 Bf5 Ne5 Bc2 Qa3 Nd3

We could manually enter the positions one at a time into SCID and find out what ECO codes they correspond to by doing a "SET Startup position" followed by examination of the ECO code for the game retrived, but I am far too lazy for that.

[Dann Corbit]

I created a list of 625 "super-neutral" opening positions with this query:

Code: Select all

select Epd + ' bm ' + bm + '; pm ' + rtrim(pm) +'; ce ' + rtrim(convert(char(6), ce)) + '; acd ' + rtrim(convert(char(5), acd)) + '; pv ' + rtrim(pv) + ';'  , acd, ce from Epd
where len(Epd) >= 50 and abs(ce) < 16 and pm like '%{%' and acd > 20 and NOT bm like '%x%'


http://cap.connx.com/epd/neutral.epd.bz2

All positions are +/- 15 centipawns of neutral, and the best move does not consist of a capture. They have also been analyzed to at least 21 plies in depth and contain move frequency statistics.

[Daniel Shawul]

I created a list of 625 "super-neutral" opening positions with this query.

Ok if they are neutral i guess i can simply divide them in to 3 (200 each) and use it. I will run games
with it and see if the result from the three samples is more or less the same.
I thought may be the ECO codes reflect the frequency of each opening in actual games but if all
positions are neutral by a small margin based on your computer analysis, i guess we don't need to classiffy them.
The test positions i am currently using has some gambit openings. I just merged silver,noomen,nunn and gambit pgns.
I don't know this it is right to do this but it game the below distribution which looks ok to me.


I will run the games and come back with results.

[Dann Corbit]

I created a list of 625 "super-neutral" opening positions with this query.

Ok if they are neutral i guess i can simply divide them in to 3 (200 each) and use it. I will run games
with it and see if the result from the three samples is more or less the same.
I thought may be the ECO codes reflect the frequency of each opening in actual games but if all
positions are neutral by a small margin based on your computer analysis, i guess we don't need to classiffy them.
The test positions i am currently using has some gambit openings. I just merged silver,noomen,nunn and gambit pgns.
I don't know this it is right to do this but it game the below distribution which looks ok to me.


I will run the games and come back with results.

All of these positions are very frequently played (you can probably tell by the move frequency data in the pm field list) so that by exercising these positions you will be exercising very frequently played positions in real life among strong competition.

[Aake64]

Hi

How did You compile your MPI c?

I want to test few MPI chess engines,
but my codin skills are -10

Cheers

Aake