Looking for linux 64 bit engines & epds

[Daniel Shawul]

Hi,
I need linux 64 bit engines compiled for amd, which are around scorpio's strength. I have
some opensource engines which I compiled myself. I can only find Spike among the closed source
which distribute exes. Let me know if you know some.
I also need opening positions which can allow me to play upt o 1000 games between engines (500 positions).
Currently i have 150 positions which i got by merging (nunn, noomen, silver,gambit), which i am not sure
is the right thing to do.

@Volker, you may have some engines.

Thanks
Daniel

[Edmund]

hello Daniel,

I am not an expert on linux engines, but I discovered the following thread on CCC recently: http://www.talkchess.com/forum/viewtopic.php?t=31431

There Dann Corbit posted a link to his collection of >5k chess opening positions.

regards,
Edmund

[Volker Pittlik]

Hi,
...
@Volker, you may have some engines.

Thanks
Daniel

This is is the rating list of a calibration I just ran at very fast time controls:

Code: Select all

Rank Name                  Elo    +    - games score oppo. draws
   1 Stockfish 1.6.2       413  197  129    40   96%   -38    8%
   2 Doch 1.3.4            344  156  113    40   94%   -38   13%
   3 Protector 1.3.2       283  133  104    40   89%   -38   13%
   4 spark-0.3a            175  102   89    40   80%   -38   25%
   5 Crafty-23.1           172  106   93    40   78%   -38   15%
   6 Umko 0.4               45   88   84    40   63%   -38   30%
   7 Ruffian 2.1.0          38   88   85    40   61%   -38   28%
   8 Sloppy-0.2.2           18   92   89    40   58%   -38   15%
   9 Yace Paderborn        -27   43   42   200   65%  -154   17%
  10 Hermann 2.5           -36   32   33   360   46%    -2   17%
  11 Spike 1.2 Turin       -40   32   32   360   45%    -1   20%
  12 Xpdnt 091007         -143   47   47   160   53%  -168   13%
  13 Arasan 11.6 (1 cpu)  -156   46   46   160   51%  -167   23%
  14 OliThink 5.2.9       -158   46   46   160   51%  -166   18%
  15 Gaviota 0.74.12      -211   47   47   160   43%  -160   15%
  16 Phalanx XXII Reborn  -336   50   53   160   27%  -144   10%
  17 Beowulf_2.4          -382   52   56   160   22%  -138   11%


There are some 32 bit engines too but that what I use at the moment.

BTW: Scorpio gives me compiler errors here therefore it does not participate yet.

vp

[Daniel Shawul]

Thanks, I missed Doch and Ruffian.

I have uploaded http://sites.google.com/site/dshawul/sc ... ects=0&d=1 . After noticing some warnings on linux 64 bit,
i have discovered that the 64 bit version of 2.4 plays very bad (like 300 elo bad). There is a seemingly easy warning from gcc when
i tried to compile it that is the cause of the problem. Now that and other minor warning are fixed. There are still some warnings but
it works ok now. I suspect that the prolem is there also on windows x64 after getting seeing some inconsistent results.

regards,
Daniel

[Daniel Shawul]

hello Daniel,

I am not an expert on linux engines, but I discovered the following thread on CCC recently: http://www.talkchess.com/forum/viewtopic.php?t=31431

There Dann Corbit posted a link to his collection of >5k chess opening positions.

regards,
Edmund

Hi Edmund, Dann's collection is epd which is an unacceptable format by cutechess-cli. But i will add Franklin's 150 to mine and make 300 pos. Hope there are no duplicate postions. If i don't get more engines i will try to find ways to use Dann's positions.

thanks
Daniel

[nepossiver]

Hi Daniel,

I don't know if you have these, but they aren't in Volker's list: Alfil, DanaSah, Jonny, The Baron & Protector (which may / may not be a derivative from Fruit).

[Ilari Pihlajisto]

Hi Edmund, Dann's collection is epd which is an unacceptable format by cutechess-cli.

EPD support is planned, but in the meantime there are tools that can convert EPD to PGN format. I haven't tried any of them, but it should be really simple stuff.

[Volker Pittlik]

Thanks, I missed Doch and Ruffian....

Ruffian is somewhat private. If you bought the commercial version of it you may try getting it via FQ. He announced it a while ago in a forum which doesn't exist anymore.

Spike was on a CD sold with the German "c't" magazine. There is also a somewhat buggy Shredder Linux version on it. These 3 engines are all 32 bit. However, Ruffian is still a tough opponent for many engines.

vp

[Dann Corbit]

Hi Edmund, Dann's collection is epd which is an unacceptable format by cutechess-cli.

EPD support is planned, but in the meantime there are tools that can convert EPD to PGN format. I haven't tried any of them, but it should be really simple stuff.

Depending upon how the PGN is read, maybe something as simple as this:

Code: Select all

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

char *
getsafe (char *buffer, int count)
{
    char *result = buffer, *np;
    if ((buffer == NULL) || (count < 1))
        result = NULL;
    else if (count == 1)
        *result = '\0';
    else if ((result = fgets (buffer, count, stdin)) != NULL)
        if (np = strchr (buffer, '\n'))
            *np = '\0';
    return result;
}


char string[32767];

int
main (void)
{
    int counter = 0;
    int i;
    while (getsafe (string, sizeof string))
    {

        puts ("\n[Event \"?\"]");
        puts ("[Site \"?\"]");
        puts ("[Date \"????.??.??\"]");
        puts ("[Round \"?\"]");
        puts ("[White \"?\"]");
        puts ("[Black \"?\"]");
        puts ("[Result \"*\"]");
        printf ("[Fen \"%s\"]\n", string);
    }
    return 0;
}



[Miguel A. Ballicora]

Hi Edmund, Dann's collection is epd which is an unacceptable format by cutechess-cli.

EPD support is planned, but in the meantime there are tools that can convert EPD to PGN format. I haven't tried any of them, but it should be really simple stuff.

Depending upon how the PGN is read, maybe something as simple as this:

Code: Select all

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

char *
getsafe (char *buffer, int count)
{
    char *result = buffer, *np;
    if ((buffer == NULL) || (count < 1))
        result = NULL;
    else if (count == 1)
        *result = '\0';
    else if ((result = fgets (buffer, count, stdin)) != NULL)
        if (np = strchr (buffer, '\n'))
            *np = '\0';
    return result;
}


char string[32767];

int
main (void)
{
    int counter = 0;
    int i;
    while (getsafe (string, sizeof string))
    {

        puts ("\n[Event \"?\"]");
        puts ("[Site \"?\"]");
        puts ("[Date \"????.??.??\"]");
        puts ("[Round \"?\"]");
        puts ("[White \"?\"]");
        puts ("[Black \"?\"]");
        puts ("[Result \"*\"]");
        printf ("[Fen \"%s\"]\n", string);
    }
    return 0;
}



I am digressing. Relatively recently I discovered Ruby. It is extremely simple and clear to build small tools like this.

Code: Select all

File.open ARGV[0], "r" do |f|
   lines = f.readlines
   lines.each do |li|
        puts '[Event "?"]'
        puts '[Site "?"]'
        puts '[Date "????.??.??"]'
        puts '[Round "?"]'
        puts '[White "?"]'
        puts '[Black "?"]'
        puts '[Result "*"]'
        puts '[Fen "' +li.strip+ '"]'
      puts
   end
end


Save it as fen2pgn.rb and then run it with
ruby fen2pgn.rb file.fen > file.pgn

Of course, ruby needs to be installed. I think it is worth learning just for this type of things. If the input needs to be parsed, then Ruby really shines. Just to share this.

Miguel

[Ilari Pihlajisto]

Depending upon how the PGN is read, maybe something as simple as this:

Code: Select all

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

char *
getsafe (char *buffer, int count)
{
    char *result = buffer, *np;
    if ((buffer == NULL) || (count < 1))
        result = NULL;
    else if (count == 1)
        *result = '\0';
    else if ((result = fgets (buffer, count, stdin)) != NULL)
        if (np = strchr (buffer, '\n'))
            *np = '\0';
    return result;
}


char string[32767];

int
main (void)
{
    int counter = 0;
    int i;
    while (getsafe (string, sizeof string))
    {

        puts ("\n[Event \"?\"]");
        puts ("[Site \"?\"]");
        puts ("[Date \"????.??.??\"]");
        puts ("[Round \"?\"]");
        puts ("[White \"?\"]");
        puts ("[Black \"?\"]");
        puts ("[Result \"*\"]");
        printf ("[Fen \"%s\"]\n", string);
    }
    return 0;
}



Cutechess-cli doesn't require any other tags, the "FEN" tag (in uppercase letters, as the standard dictates) is enough. A termination marker is needed too, so the final PGN file should look like this:

Code: Select all

[FEN "fen 1"]
*

[FEN "fen 2"]
*

[FEN "fen n"]
*


[Daniel Shawul]

Thank you Illari for such a great tool !
I have run about 16,000 games at 40/30 with it on a cluster mostly using only 16 processors.
I only got 2 games missing one from Toga and another from Fruit ?? I don't know if that is the
problem of the engines or if these things happen on clusters.

I am using the pgn of games produced from cutechess-cli to send it to bayeselo. At first, I was
afraid the pgns might be big but it turns out they are really small with -min option. I wrote a small MPI code (just studied it
2 days ago) to manage pairings, match engines etc... If it helps anyone to get started, here is the code

Code: Select all

#include "mpi.h"
#include <math.h>

static const char* const engines[] = {
       "cmd=./scorpio dir=/home/dabdi/Work/scorpio",
       "cmd=./scorpio dir=/home/dabdi/Work/scorpio_241",
   "cmd=./scorpio dir=/home/dabdi/Work/scorpio_24",
        "cmd=./crafty dir=/home/dabdi/Work/crafty-23.1",
   "cmd=./fruit proto=uci dir=/home/dabdi/Work/fruit_21/src",
   "cmd=./glaurung proto=uci option.Threads=1 dir=/home/dabdi/Work/glaurung22/src",
   "cmd=./spike proto=uci dir=/home/dabdi/Work/Spike_12",
       "cmd=./arasanx proto=uci dir=/home/dabdi/Work/arasan-11.6.0/export",
       "cmd=./doch64-134-ja proto=uci dir=/home/dabdi/Work/Doch134/Gcc",
       "cmd=./Toga proto=uci dir=/home/dabdi/Work/Toga",
       "cmd=./Hermann proto=uci dir=/home/dabdi/Work/Hermann25",
   NULL
};

typedef unsigned int JOB;
#define PORTION(x) ((x >> 16) & 0xffff)
#define SECOND(x)  ((x >> 8) & 0xff)
#define FIRST(x)   (x & 0xff)

int main(int argc, char* argv[] ) {
   char str[512];
   int myid,nprocs,namelen,master;
   char processor_name[MPI_MAX_PROCESSOR_NAME];
   MPI_Status status;
   JOB job;

   MPI_Init(&argc,&argv);
   MPI_Comm_size(MPI_COMM_WORLD,&nprocs);
   MPI_Comm_rank(MPI_COMM_WORLD,&myid);
   MPI_Get_processor_name(processor_name, &namelen);

   master = 0;
   nprocs--;

   //master
   if(myid == master) {
      int r,result,njobs,sent,N,NTOTAL;
              const int rounds = 1;
      
              //count number of engines
              N = 0;
          while(engines[N]) N++;
      
       //round robin or gauntlet
      JOB* jobs = new JOB[(N * (N - 1)) / 2];
              njobs = 0;
      for(int i = 0;i < rounds;i++) {
         for(int j = i + 1;j < N;j++) {
            njobs++;
         }
      }
      NTOTAL = njobs;

              //divide pgn
      int divide = int(ceil(nprocs / (float)NTOTAL));
      int dgames = 150 / divide,games = 0;

      FILE* f = fopen("all.pgn","r");
      FILE* f1 = 0;
      char buffer[512];

      strcpy(buffer,"");
      while(fgets(buffer,512,f)) {
         if(strncmp(buffer,"[Event ",7) == 0) {
            if((games % dgames) == 0) {
               if(f1)   fclose(f1);
               sprintf(str,"./divide/all%d.pgn",games / dgames);
               f1 = fopen(str,"w");
            }
            games++;
         }
         fprintf(f1,"%s",buffer);
              }
      if(f1) fclose(f1);
      fclose(f);

              //assign jobs
      njobs = 0;
      for(int k = 0;k < divide;k++) {
                 for(int i = 0;i < rounds;i++) {
            for(int j = i + 1;j < N;j++) {
               jobs[njobs++] = ((k << 16) | (j << 8) | i);
            }
         }
      }
      NTOTAL = njobs;
   
             
      //send slaves to work
      sent = 0;
      njobs = 0;                   
      while(njobs < NTOTAL) {
         sent++;
         MPI_Send(&jobs[njobs++],1,MPI_INT,sent,njobs,MPI_COMM_WORLD);
         if(sent >= nprocs) break;
      }
      while(sent) {
         MPI_Recv(&r,1,MPI_INT,MPI_ANY_SOURCE,MPI_ANY_TAG,MPI_COMM_WORLD,&status);
         if(njobs < NTOTAL) {
            MPI_Send(&jobs[njobs++],1,MPI_INT,r,njobs,MPI_COMM_WORLD);
         } else {
            MPI_Send(MPI_BOTTOM,0,MPI_INT,r,0,MPI_COMM_WORLD);
            sent--;
         }
      }

              //delete
              delete jobs;
   }
   //slave
   else {
      while(true) {
         MPI_Recv(&job,1,MPI_INT,master,MPI_ANY_TAG,MPI_COMM_WORLD,&status);
         if(status.MPI_TAG == 0) {
            break;
         } else {
            sprintf(str,"./cutechess-cli.sh -pgnin ./divide/all%d.pgn -repeat -games 300  -both tc=40/30 -fcp %s -scp %s -pgnout test.pgn min",
                                        PORTION(job),engines[FIRST(job)],engines[SECOND(job)]);
                            //printf("%s\n",str);
                            system(str);
            MPI_Send(&myid,1,MPI_INT,master,status.MPI_TAG,MPI_COMM_WORLD);
         }
      }
   }

   MPI_Finalize();

   return 0;
}


The good news is that i have now a working setup which can allow me to sit tight and press the button
Some random changes, which i thought will improve things, but actually hurt ! I do not have the intention to
nail the elos down to 1 point. +-10 standard error is good enough for me (about 3000 games).

Code: Select all

Num. Name             games   score
   0 Scorpio_2.4.4      299   133.5
   1 Glaurung 2.2       635   465.5
   2 Fruit 2.1          635     306
   3 Scorpio_2.4.1     2740    1249
   4 Toga II 1.3.1      616     413
   5 Arasan 11.6        619   169.5
   6 Hermann 2.5        618   182.5
   7 Scorpio_2.4       2743  1254.5
   8 Spike 1.2 Turin    635     332
   9 Doch64 1.3.4 JA    620   494.5
  10 Crafty-23.1        636     398
Rank Name              Elo    +    - games score oppo. draws
   1 Doch64 1.3.4 JA   208   26   25   620   80%   -27   18%
   2 Glaurung 2.2      148   23   23   635   73%   -27   21%
   3 Toga II 1.3.1      98   23   22   616   67%   -27   22%
   4 Crafty-23.1        65   22   22   636   63%   -27   23%
   5 Spike 1.2 Turin   -10   21   21   635   52%   -27   26%
   6 Scorpio_2.4       -25   11   11  2743   46%     6   24%
   7 Scorpio_2.4.1     -29   11   11  2740   46%     6   24%
   8 Scorpio_2.4.4     -30   31   31   299   45%     7   32%
   9 Fruit 2.1         -41   21   22   635   48%   -27   24%
  10 Hermann 2.5      -184   23   24   618   30%   -27   18%
  11 Arasan 11.6      -199   23   24   619   27%   -27   19%


Thank you
Daniel

[Daniel Shawul]

Hi Dann

Thanks for the code and the epd collection. I forgot that PGN do have FEN capabilities in them
(I am so old for my age ) I will use it as an endless source of start positions
for whatever number of games i want to play.
I want to divide your database into a number of parts (say 300 postions in each sample) so that when i want to do smaller number of games ,
i will use the first, and then the second etc... For that to work, each sample should be representative
of the whole epd suite. Can you sort the epds, in such a way that i have similar probabilities of getting the same result if i select any
of the samples. I hope you are getting my point.

regards,
Daniel

[Ilari Pihlajisto]

Thank you Illari for such a great tool !

I'm glad you find it useful. It's still in an early stage of development, so stay tuned for some impressive features.

I only got 2 games missing one from Toga and another from Fruit ?? I don't know if that is the
problem of the engines or if these things happen on clusters.

What do you mean by missing games? Are they missing just from the PGN output, or did cutechess-cli not play them at all? If an engine crashes or becomes unresponsive, cutechess-cli will stop the whole match, which might explain the missing games. The next version will handle misbehaving engines better: I already have committed many of the fixes to the source code repository, just haven't released the new binaries yet.

I wrote a small MPI code (just studied it
2 days ago) to manage pairings, match engines etc... If it helps anyone to get started, here is the code

Thanks. I've been looking into making cutechess-cli more cluster-friendly (now -concurrency only works on a single machine). Maybe this will help, or at least give some ideas.

BTW, If you want the matches to go faster, you may want to use the -resign and/or -draw options to adjudicate the games when the outcome seems clear.

[Daniel Shawul]

Here is the link to the games http://sites.google.com/site/dshawul/ga ... ects=0&d=1 where two of the games are missing.
Unfortunaely i turned off all log files (of engines, and cutechess-cli). I will atleast turn on the cutechess-cli debug next time.
I read that Bob Hyatt had similar problems now and then, so it may be that it is what it is. The only reason why it made me suspicious is
because it only happend to Fruit and Toga.

Bayeselo output

Code: Select all

Num. Name             games   score
   0 Scorpio_2.4       3000  1384.5
   1 Doch64 1.3.4 JA    900     704
   2 Fruit 2.1          899     432
   3 Toga II 1.3.1      899   604.5
   4 Spike 1.2 Turin    900   461.5
   5 Glaurung 2.2       900   643.5
   6 Crafty-23.1        900   558.5
   7 Hermann 2.5        900   283.5
   8 Arasan 11.6        900   253.5
   9 Scorpio_2.4.1     2999  1376.5
  10 Scorpio_2.4.2     2999    1396
Rank Name              Elo    +    - games score oppo. draws
   1 Doch64 1.3.4 JA   194   21   20   900   78%   -26   18%
   2 Glaurung 2.2      136   20   19   900   72%   -26   20%
   3 Toga II 1.3.1     100   19   19   899   67%   -26   23%
   4 Crafty-23.1        61   19   18   900   62%   -26   24%
   5 Spike 1.2 Turin   -16   18   18   900   51%   -26   27%
   6 Scorpio_2.4.2     -23   11   10  2999   47%     2   26%
   7 Scorpio_2.4       -25   11   10  3000   46%     3   26%
   8 Scorpio_2.4.1     -29   11   10  2999   46%     3   25%
   9 Fruit 2.1         -40   18   18   899   48%   -26   24%
  10 Hermann 2.5      -166   19   19   900   32%   -26   18%
  11 Arasan 11.6      -192   19   20   900   28%   -26   19%


BTW, If you want the matches to go faster, you may want to use the -resign and/or -draw options to adjudicate the games when the outcome seems clear.

Ok i will turn on both options next time. My old versions of scorpio doesn't like it when an opponent engine interrupts them with a "resign" message while
searching during their turn. Inthat case they just hung up. It is because of my messy winboard code. I will check if they can handle cutechess-cli forcing them
to resign.
If you have new versions you want to get tested, I don't mind testing them. You can send them to me at dshawul at yahoo.com

[Ilari Pihlajisto]

Here is the link to the games http://sites.google.com/site/dshawul/ga ... ects=0&d=1 where two of the games are missing.
Unfortunaely i turned off all log files (of engines, and cutechess-cli). I will atleast turn on the cutechess-cli debug next time.
I read that Bob Hyatt had similar problems now and then, so it may be that it is what it is. The only reason why it made me suspicious is
because it only happend to Fruit and Toga.

Thanks. Now I think I know what the problem is: you have multiple cutechess-cli processes running in parallel, and they're all writing the games to the same file. If two or more processes try to write to the file at the same time, you may end up with missing games. There are a couple of possible solutions:
1. You can set a different output file for each cutechess-cli process, and combine the files when you've done
2. I could modify cutechess-cli to use a locked file for PGN output to make sure that write access is always serialized

BTW, If you want the matches to go faster, you may want to use the -resign and/or -draw options to adjudicate the games when the outcome seems clear.

Ok i will turn on both options next time. My old versions of scorpio doesn't like it when an opponent engine interrupts them with a "resign" message while
searching during their turn. Inthat case they just hung up. It is because of my messy winboard code. I will check if they can handle cutechess-cli forcing them
to resign.

The -resign option doesn't mean that Scorpio gets a "resign" message. Cutechess-cli will just stop the game when the resign conditions are met, so it should work with all engines. The output should look like this (here I used "-resign 5 300"):

Code: Select all

70336 <Scorpio_2.4 JA(0): 15 -1064 222 4156140 Kg4-f4 Ba4-d7 Kf4-e5 a5-a4 Ke5-d6 a4-a3 Kd6xd7 Kc3-d4 Bg8-d5 a3-a2 Kd7-c6 a2-a1=Q Kc6xb6 Qa1-b2 Kb6-c6 Qb2-h2 Kc6-d7 Qh2xh4
70447 <Scorpio_2.4 JA(0): 15 -1064 233 4401078 Kg4-f4 Ba4-d7 Kf4-e5 a5-a4 Ke5-d6 a4-a3 Kd6xd7 Kc3-d4 Bg8-d5 a3-a2 Kd7-c6 a2-a1=Q Kc6xb6 Qa1-b2 Kb6-c6 Qb2-h2 Kc6-d7 Qh2xh4
70447 <Scorpio_2.4 JA(0): nodes = 4401078 <68 qnodes> time = 2334ms nps = 1885637
70447 <Scorpio_2.4 JA(0): lazy_eval = 65 splits = 0 badsplits = 0 egbb_probes = 0
70447 <Scorpio_2.4 JA(0): move g4f4
70447 >Glaurung(1): position startpos moves c2c4 e7e6 d2d4 g8f6 b1c3 f8b4 g1f3 e8g8 d1d3 b7b6 c1g5 c8a6 f3e5 d7d6 d3f3 d6e5 d4e5 b8d7 e5f6 d7f6 a1d1 d8e7 e2e3 a8d8 f1d3 h7h6 g5f6 e7f6 f3f6 g7f6 e1e2 b4c3 b2c3 g8g7 h2h4 c7c5 d1d2 d8d7 h1d1 f6f5 a2a4 f8d8 e2e1 a6b7 f2f3 g7f6 d3c2 d7d2 d1d2 d8d2 e1d2 f6e5 d2c1 b7c6 c1d2 a7a6 d2c1 f5f4 c1d2 f7f5 d2d3 f4e3 d3e3 f5f4 e3f2 e5d6 c2d1 e6e5 g2g3 f4g3 f2g2 e5e4 f3e4 c6e4 g2g3 d6e5 g3g4 e4g6 d1f3 g6e8 f3d1 e5e4 d1c2 e4e3 c2f5 a6a5 f5c8 e3d3 c8e6 d3c3 e6g8 e8a4 g4f4
70447 >Scorpio_2.4 JA(0): force
70447 >Scorpio_2.4 JA(0): result 0-1 {Black wins by adjudication}
70447 >Glaurung(1): isready
70447 <Glaurung(1): readyok
Game 1 ended: 0-1 {Black wins by adjudication}
Glaurung wins the game as black
Score of Scorpio_2.4 JA vs Glaurung: 0 - 1 - 0
70646 >Scorpio_2.4 JA(0): quit
70646 >Glaurung(1): quit
70647 <Scorpio_2.4 JA(0): Bye Bye
Finished match


If you have new versions you want to get tested, I don't mind testing them. You can send them to me at dshawul at yahoo.com

If you want bleeding edge, you can compile our sources: http://repo.or.cz/w/sloppygui.git
But I recommend waiting for tagged releases, the next one's coming out soon.

[Dann Corbit]

Hi Dann

Thanks for the code and the epd collection. I forgot that PGN do have FEN capabilities in them
(I am so old for my age ) I will use it as an endless source of start positions
for whatever number of games i want to play.
I want to divide your database into a number of parts (say 300 postions in each sample) so that when i want to do smaller number of games ,
i will use the first, and then the second etc... For that to work, each sample should be representative
of the whole epd suite. Can you sort the epds, in such a way that i have similar probabilities of getting the same result if i select any
of the samples. I hope you are getting my point.

regards,
Daniel

Right now, I am working on a special EPD set. It will have extensive statistics and analysis incorporated.
I have the EPD stored in a SQL*Server database, and so I can perform any sort of operation against it using SQL queries.
One of my tables has 200 million rows in it. That one is a little unweildy for general queries, so I have a second table which summarized the data in the big table.

Let me know exactly what it is that you want and I will provide it.

[Daniel Shawul]

Wow!That is one big collection.
Ok I will tell you what i need roughly.
From your database, you can generate start positions and classify them
based on their ECO codes A,B,C,D,E. Then many samples of same size ,say 150 positions,
are taken with some assumed distribution wrt ECO codes. I would guess normal or something
slightly skewed to the left. Each distribution should have use that same distribution so that
same result is expected which ever one I pick. Later on when i add up the results from the different
samples, I expect the result to quickly converge into a normal distribution according to CLT.
So in short if Sample 1 has 20A's, 30B's 25C's then Sample 2 should have that much.
You can use longer book lines to increase variation but i wouldn't expect that to be a problem
for you with that much games

Thanks
Daniel

[Dann Corbit]

Wow!That is one big collection.
Ok I will tell you what i need roughly.
From your database, you can generate start positions and classify them
based on their ECO codes A,B,C,D,E. Then many samples of same size ,say 150 positions,
are taken with some assumed distribution wrt ECO codes. I would guess normal or something
slightly skewed to the left. Each distribution should have use that same distribution so that
same result is expected which ever one I pick. Later on when i add up the results from the different
samples, I expect the result to quickly converge into a normal distribution according to CLT.
So in short if Sample 1 has 20A's, 30B's 25C's then Sample 2 should have that much.
You can use longer book lines to increase variation but i wouldn't expect that to be a problem
for you with that much games

Thanks
Daniel

Here is the problem:
I have tens of thousands of fully analyzed and statistically classified EPD strings.
However, I do not know the ECO codes for the strings, and I do not have any utility that will classify them for me.
I can input them into a SCID database but it classified every position as A00a and that is not correct.
PGN-EXTRACT by Barnes also does not know how to deal with them.

For instance, here are 1344 carefully analyzed Epd records:
http://cap.connx.com/epd/analyzed.epd.bz2
they are in a format like the following:
rnbqkbnr/pp1p1ppp/4p3/2p5/2P5/2N2N2/PP1PPPPP/R1BQKB1R b KQkq - acd 22; bm Nf6; ce -8; pm Nf6 {649} Nc6 {192} b6 {159} a6 {116} d5 {89}; pv Nf6 e3 Nc6 Be2 d5 cxd5 exd5 d4 cxd4 exd4 Bd6 O-O O-O Qb3 Na5 Qc2 Nc6 Be3 Nb4 Qb3 Bf5 Ne5 Bc2 Qa3 Nd3;

Here is a breakdown of the components of this position:

The base position:
rnbqkbnr/pp1p1ppp/4p3/2p5/2P5/2N2N2/PP1PPPPP/R1BQKB1R b KQkq -

The depth in plies analyzed:
acd 22

The move chosen by computer analysis:
bm Nf6

The computer analysis in centipawns for this position:
ce -8

The statistical counts for actual moves played by strong players for this position, with frequencies in curly braces:
pm Nf6 {649} Nc6 {192} b6 {159} a6 {116} d5 {89}
In this case, Nf6 was played 649 times, Nc6 was played 192 times, b6 was played 159 times, a6 was played 116 times, and d5 was played 89 times. The computer best move coincides with the most frequently played move in this instance.

This the the computer generated pv for this position:
pv Nf6 e3 Nc6 Be2 d5 cxd5 exd5 d4 cxd4 exd4 Bd6 O-O O-O Qb3 Na5 Qc2 Nc6 Be3 Nb4 Qb3 Bf5 Ne5 Bc2 Qa3 Nd3

We could manually enter the positions one at a time into SCID and find out what ECO codes they correspond to by doing a "SET Startup position" followed by examination of the ECO code for the game retrived, but I am far too lazy for that.

[Dann Corbit]

I created a list of 625 "super-neutral" opening positions with this query:

Code: Select all

select Epd + ' bm ' + bm + '; pm ' + rtrim(pm) +'; ce ' + rtrim(convert(char(6), ce)) + '; acd ' + rtrim(convert(char(5), acd)) + '; pv ' + rtrim(pv) + ';'  , acd, ce from Epd
where len(Epd) >= 50 and abs(ce) < 16 and pm like '%{%' and acd > 20 and NOT bm like '%x%'


http://cap.connx.com/epd/neutral.epd.bz2

All positions are +/- 15 centipawns of neutral, and the best move does not consist of a capture. They have also been analyzed to at least 21 plies in depth and contain move frequency statistics.