Generating Magic Keys for Bitboard Move Generation

Programming Topics (Computer Chess) and technical aspects as test techniques, book building, program tuning etc

Moderator: Andres Valverde

Generating Magic Keys for Bitboard Move Generation

Postby Pradu » 26 Jul 2006, 00:50

Hi all,

I would like to post for some work I'm doing on an algorithmically fast magic key generator for magic bitboard move generation so that we can get better magic keys for smaller move databases. It isn't complete but hopefully with your help it can be finished faster.

For those who haven't heard of "magic bitboard move generation" you might like to read this topic posted by Lasse Hansen:
http://wbforum.volker-pittlik.name/viewtopic.php?t=5015
I'll summarize it here:
Say you had this position and you wanted to generate moves for the white rook.
[diag]8/8/8/2kbR1N1/8/8/4K3/4b3 w - - 0 1[/diag]
8/8/8/2kbR1N1/8/8/4K3/4b3 w - - 0 1

You would first need a couple databases
RookMask[64];
Magic[64];
RookDatabase[64][1<<NumBits]; //Contains rook move bitboards

RookMask[E5] will look like this
Code: Select all
00000000
00001000
00001000
01110110
00001000
00001000
00001000
00000000

For bishops BishopMask[E5] it might look like this
Code: Select all
00000000
00100010
00010100
00000000
00010100
00100010
01000000
00000000

First we need to get the occupancy bitboard which we find like this:
OccupancyBB = AllPieces & RookMask[E5];
Then we multiply it by a magic and shift the "NumBits" top bits down to index to our rook move bitboard database.
Index = ( OccupancyBB * Magic[E5] ) >> ( 64 - NumBits );
RookMoveBB = RookDatabase[E5][Index];

What (OccupancyBB*Magic[E5]) does is "shift" bits up in the occupancy bitboard and you take the top n bits as your index (which will depend on your occupancy bitboard).

But generating this Magic is pretty difficult. It will have to take into account that different occupancy bitboards should hash to an index with the same move. It would be ideal to have all the respective occupancs hash to the same index (as this will reduce our Database size by a lot) but it might not be possible.

===================================

Anyways what I'm trying to do is create magic keys be able to create indecies that only take up 8 bits for rooks and 7 bits for bishops (so that we can reduce the database size):
Code: Select all
Data from the top part of my keygen program to show 8 bits are needed for rooks and 7 bits are needed for bishops.

Possible rook move bitboards for each square
  49  42  70  84  84  70  42  49
  42  36  60  72  72  60  36  42
  70  60 100 120 120 100  60  70
  84  72 120 144 144 120  72  84
  84  72 120 144 144 120  72  84
  70  60 100 120 120 100  60  70
  42  36  60  72  72  60  36  42
  49  42  70  84  84  70  42  49

Possible bishop move bitboards for each square
   7   6  10  12  12  10   6   7
   6   6  10  12  12  10   6   6
  10  10  40  48  48  40  10  10
  12  12  48 108 108  48  12  12
  12  12  48 108 108  48  12  12
  10  10  40  48  48  40  10  10
   6   6  10  12  12  10   6   6
   7   6  10  12  12  10   6   7


In an effort to do this I've noticed that not all bits will affect the top n bits so that we don't have to try the impossible process of iterating through all possible 64 bit magics and instead just iterate through a just select few magics. In order to simplify the analysis I've broken down the multiplication for an 8bit integer:

Code: Select all
00111000 * 01001011 = 00111000<<0 + 00111000<<1 + 00111000<<3 + 00111000<<6
                    = 00111000    + 01110000    + 11000000    + 00000000


We can already see from this simple example that the 6th bit of the multiplier dosen't affect the result at all for how many ever top n bits. This is because the bits of the key gets shifted out of the number. Say we wanted only 2 bits for our index. Then we can see the first expression in the simplification won't even make it up to the top 2 bits -- but it gets a bit more complicated. We have only seen the direct infuence of the shifted bits on the 2 top bits but we haven't seen any the effect of the carry. When the addition process is actually done, we will see that the first expression which we thought wouldn't have an effect, really does because of the second expression which when added to the first will make the bits below the first 2 bits carry up. Likewise there are occations when bits that map directly to the top n bits will carry too far out and really won't have an effect. Currently I'm working on how the carry effects the top n bits, and if anybody can help me out, we can create better magic keys for magic bitboard move generators.
User avatar
Pradu
 
Posts: 343
Joined: 12 Jan 2005, 19:17
Location: Chandler, Arizona, USA

Re: Generating Magic Keys for Bitboard Move Generation

Postby Gerd Isenberg » 26 Jul 2006, 09:22

Hi Pradu,

i am still skeptical to go below 12-rook and 9-bit bishop addresses.
Did you already archived a denser lookup than that due to your ambitious 8,7-bit objectives?
For rook-attacks and the most significant edge-square, how would you squeeze the 12 relevant occupied bits into 8 bits of the address, if you already have 6-occupied bits inside those eight upper bits?

Good luck,
Gerd
Gerd Isenberg
 
Posts: 285
Joined: 31 Jan 2005, 20:31
Location: Hattingen, Germany

Re: Generating Magic Keys for Bitboard Move Generation

Postby Pradu » 26 Jul 2006, 15:53

Gerd Isenberg wrote:Hi Pradu,

i am still skeptical to go below 12-rook and 9-bit bishop addresses.


I'm not sure if it will work, but it is worth a try.

Did you already archived a denser lookup than that due to your ambitious 8,7-bit objectives?


No, I probably won't try to do this either. Instead I'm trying to find patterns in keys that generate minimal bits in the index to work for nxn board (4x4 5x5) in which we can iterate through all magics if we wanted to.

For rook-attacks and the most significant edge-square, how would you squeeze the 12 relevant occupied bits into 8 bits of the address, if you already have 6-occupied bits inside those eight upper bits?


The occupied bits in the eight upper bits will affect carries for those bits and will inturn affect these bits directly so no problem here. Also if such a key for our current occupany isn't possible, then, if necessary, we can introduce redundant bits into our occupancy.

Code: Select all
OccupancyBB = ( AllPieces & RookMask[E5] ) ^ RedundantBits[E5];


So there are still very good chances it can be mapped into 8 bits for rooks and 7 bits for bishops.

Good luck,
Gerd

Thanks.
User avatar
Pradu
 
Posts: 343
Joined: 12 Jan 2005, 19:17
Location: Chandler, Arizona, USA

Re: Generating Magic Keys for Bitboard Move Generation

Postby Gerd Isenberg » 26 Jul 2006, 19:39

So there are still very good chances it can be mapped into 8 bits for rooks and 7 bits for bishops.


You'll receive the computer chess nobel price if you get that - even with an additional xor!
Gerd Isenberg
 
Posts: 285
Joined: 31 Jan 2005, 20:31
Location: Hattingen, Germany

Re: Generating Magic Keys for Bitboard Move Generation

Postby Grant Osborne » 28 Jul 2006, 12:42

Pradu

I have been trying for the last month to find the last 3 rook 11 bit keys (A8, H8 & F1) without success. I can get really close with something like 1900 good collisions but I cannot find the magic number.
As for bishops, I have all 7 bit magic numbers except for D5, E5, D4 and E4 which remain at 9 bits.

Grant
User avatar
Grant Osborne
 
Posts: 69
Joined: 16 Jun 2006, 14:05

Re: Generating Magic Keys for Bitboard Move Generation

Postby Pradu » 28 Jul 2006, 18:04

Grant Osborne wrote:Pradu

I have been trying for the last month to find the last 3 rook 11 bit keys (A8, H8 & F1) without success. I can get really close with something like 1900 good collisions but I cannot find the magic number.
As for bishops, I have all 7 bit magic numbers except for D5, E5, D4 and E4 which remain at 9 bits.

Grant


Hi Grant, is your generator like Lasse's (trying out random numbers) or something else. Can you post here all available 7bit keys for bishops?

Regards,
Pradu
User avatar
Pradu
 
Posts: 343
Joined: 12 Jan 2005, 19:17
Location: Chandler, Arizona, USA

Re: Generating Magic Keys for Bitboard Move Generation

Postby Grant Osborne » 28 Jul 2006, 22:23

Pradu

My magic number generator is basically the same as Lasse's.
Here are the bishop keys you asked for, with the minimum number of bits I have found so far by the side:-

AD6DDDFCDCFF53FF // A8 5
0002040404064001 // B8 5
0010008081000461 // C8 5
1884040080010010 // D8 5
800A0210009101C1 // E8 5
421208020800B000 // F8 5
6804020250440281 // G8 5
02B8823F08434015 // H8 6
000F300418080040 // A7 5
0045043808004088 // B7 5
10D128020400F400 // C7 5
02E3820A02000000 // D7 5
408D2404E0012122 // E7 5
9480AE0202200004 // F7 5
8020060806080409 // G7 5
100AD1840108C200 // H7 5
0940402410442110 // A6 5
6148010210312204 // B6 5
4C08091000801090 // C6 7
02818A9806004000 // D6 7
84C401FB82A00081 // E6 7
0500809808010811 // F6 7
50804014020A1001 // G6 5
C407484201008840 // H6 5
11D0F00A48021010 // A5 5
B0100A0204040410 // B5 5
93A44800A4002408 // C5 7
0008080300820002 // D5 9
003D005001004000 // E5 9
4010002089040100 // F5 7
455402C134023200 // G5 5
5277C2C5E6071400 // H5 5
D8CD1098AA11A000 // A4 5
F0B41A1838027000 // B4 5
0089C03006880040 // C4 7
00A02018001D0050 // D4 9
0440021200022080 // E4 9
1518050310080800 // F4 7
5001010208240200 // G4 5
4722244448160210 // H4 5
041B101190006400 // A3 5
0434040A88100200 // B3 5
824812EA98041000 // C3 7
040115A0B1034800 // D3 7
40488A09A2018400 // E3 7
48405A9090804100 // F3 7
00202F6A14800200 // G3 5
80380E0052000240 // H3 5
00E404020A300000 // A2 5
6E01008210068080 // B2 5
0040020042188680 // C2 5
0080000108D80200 // D2 5
F2048D48B0240820 // E2 5
810040B921030010 // F2 5
0089200440820188 // G2 5
100B30221200C024 // H2 5
A173FEFEFEAE5136 // A1 5
7745BBFBFBFB4CA5 // B1 5
E433BF9FF9BD3C0D // C1 5
8F0BBE9CF98C0405 // D1 5
7E11DFD9DDFBDBF0 // E1 5
9DEBFFFEEFEE7206 // F1 5
FEE2FFFBFBFB155C // G1 5
CB7EDFFD9BFB3F2B // H1 5

Grant
User avatar
Grant Osborne
 
Posts: 69
Joined: 16 Jun 2006, 14:05

Re: Generating Magic Keys for Bitboard Move Generation

Postby Pradu » 03 Aug 2006, 19:55

Hi again all I would like to post a very fast magic-key generator for sets of keys that cause only direct influences to the upper bits (no carry) and where there are no collisions required in the hash. An exmple of this is a magic bitscan.

Click here for my generator

All magic keys were generated in a split second on a Pentium II!

Code: Select all
2 bits  - 0x2
4 bits  - 0x6
8 bits  - 0x3A
16 bits - 0x1EB2
32 bits - 0x0FB5CA62
64 bits - 0x07EDD5E59A4E28C2


The following were also cacluated in a flash:
All possible magics:
8 Bit
16 Bit
32 bit
There are 67108864 possible 64 bit magics (too large to browse through in a simple text file).

For example a 64 bit LSB function might look something like this:

Code: Select all
#define BITSCAN_MAGIC 0x07EDD5E59A4E28C2
#define FirstOne(X) BitScanDatabase[((X&-X)*BITSCAN_MAGIC)>>58]
int BitScanDatabase[64];
void initializeFistOne()
{
   U64 temp=1;
   int i=0;
   do
   {
      BitScanDatabase[i]=(temp*BITSCAN_MAGIC)>>58;
      i++;
      temp<<=1;
   }while(temp);
}


Hopefully it is possible to introduce carries and collisions into an algorithm similar to this for magic move generation.
User avatar
Pradu
 
Posts: 343
Joined: 12 Jan 2005, 19:17
Location: Chandler, Arizona, USA

Re: Generating Magic Keys for Bitboard Move Generation

Postby Gerd Isenberg » 06 Aug 2006, 10:34

Pradu, are you aware that you invented a De Bruijn sequence generator?
How long does it take with your algorithm, to enumerate all 67108864 == 0x4000000 64-bit sequences - without printing all, but only first and last one?

My direct recursive one takes about one minute for that on my amd64 box. Your unique test seems improvable i guess, should be loopless.

I tried De Bruijn sequences as well to hash occupied bits of a masked ray, for a six-bit occupied state of files and diagonals, before Tony came up with random numbers and i understand the fill- and rotate multiplication by 0x0202020202020202 and that like. Tony got much faster results with random numbers, as i with unmodified and modidied (eg. xor by constant) De Bruijn sequences.

Btw. your even 64-bit sequence starts with 1 after mul 2**0, and end with 0 after mul with 2**63. My generator considers the almost same odd sequences (>>1), with phaseshifted wrapped order of all unique six-bit sequences. Thus 0x07EDD5E59A4E28C2 becomes 0x03F6EAF2CD271461 with first index zero but last index 32.

Using de Bruijn Sequences to Index a 1 in a Computer Word
http://supertech.csail.mit.edu/papers/debruijn.pdf

From CCC archives:
http://www.stmintz.com/ccc/index.php?te ... l&search=1

first try:
http://www.stmintz.com/ccc/index.php?id=339184
Build your Unique bitScan routine:
http://www.stmintz.com/ccc/index.php?id=370367

Some thoughts on Dann Corbit's rotated alternative
http://www.stmintz.com/ccc/index.php?id=489664
Bug correction:
http://www.stmintz.com/ccc/index.php?id=490544

Cheers,
Gerd
Gerd Isenberg
 
Posts: 285
Joined: 31 Jan 2005, 20:31
Location: Hattingen, Germany

Re: Generating Magic Keys for Bitboard Move Generation

Postby Pradu » 06 Aug 2006, 19:27

Gerd Isenberg wrote:Pradu, are you aware that you invented a De Bruijn sequence generator?


I had no idea... :?. I've heard of De Burijin from some of your posts before but never really knew exactly what it was. I would also like to note these magics can be used with the division operator (if it helps anyone with their magic generation).

Code: Select all
Index=(magic/key)&(1<<numBits-1)


How long does it take with your algorithm, to enumerate all 67108864 == 0x4000000 64-bit sequences - without printing all, but only first and last one?


Fonzy ran the generator for 64 bits because a PII is a bit slow... It took him about 30 minutes to generate and count all possible 64 bit magics on his 2 ghz 32 bit AMD.

My direct recursive one takes about one minute for that on my amd64 box. Your unique test seems improvable i guess, should be loopless.

I tried De Bruijn sequences as well to hash occupied bits of a masked ray, for a six-bit occupied state of files and diagonals, before Tony came up with random numbers and i understand the fill- and rotate multiplication by 0x0202020202020202 and that like. Tony got much faster results with random numbers, as i with unmodified and modidied (eg. xor by constant) De Bruijn sequences.

Btw. your even 64-bit sequence starts with 1 after mul 2**0, and end with 0 after mul with 2**63. My generator considers the almost same odd sequences (>>1), with phaseshifted wrapped order of all unique six-bit sequences. Thus 0x07EDD5E59A4E28C2 becomes 0x03F6EAF2CD271461 with first index zero but last index 32.

Using de Bruijn Sequences to Index a 1 in a Computer Word
http://supertech.csail.mit.edu/papers/debruijn.pdf

From CCC archives:
http://www.stmintz.com/ccc/index.php?te ... l&search=1

first try:
http://www.stmintz.com/ccc/index.php?id=339184
Build your Unique bitScan routine:
http://www.stmintz.com/ccc/index.php?id=370367

Some thoughts on Dann Corbit's rotated alternative
http://www.stmintz.com/ccc/index.php?id=489664
Bug correction:
http://www.stmintz.com/ccc/index.php?id=490544

Cheers,
Gerd


Thanks for the links Gerd! I'll look through them to see if they can be used for move generation instead. Do you know of any (non-brute force) magic generator that can produce indecies that concider carrys and collisions (I'm currently working on this now)?
Last edited by Pradu on 07 Aug 2006, 15:38, edited 1 time in total.
User avatar
Pradu
 
Posts: 343
Joined: 12 Jan 2005, 19:17
Location: Chandler, Arizona, USA

Re: Generating Magic Keys for Bitboard Move Generation

Postby Gerd Isenberg » 07 Aug 2006, 10:11

While De Bruijn Hashing perfectly maps all 64 single-bit values to a six bit value range from 0..63, it becomes obviously difficult with more bits set. The MIT-guys already need about a 32K-Table to hash bitboards with up to two bits set - and i guess it is already faster to use randoms here to find good magics for that. As we already saw with the trials from Lasse and Grant, it seems, that the population count of the magics is somehow crucial, either a low popcount of approx the number of bits to map, or somehow the one's-complement of that, with a lot of carries around. De Bruijn sequences with exactly 32 one-bits seem not so well suited to map multiple bits.

Gerd
Gerd Isenberg
 
Posts: 285
Joined: 31 Jan 2005, 20:31
Location: Hattingen, Germany

Re: Generating Magic Keys for Bitboard Move Generation

Postby Pradu » 14 Aug 2006, 02:15

I've found an interesting property in multiplication.

Take any set of unique keys, multiply each key by any odd number and you will get another set of unique keys (mabe easier to hash to -- multiply the 2nd set of keys with the magic for the 2nd set of keys and you will have the magic for the 1st set). For example take all 32 bit numbers, multiply it by any odd number and you will get another set of 32 bit numbers except in a different order. You can see the swaps of the numbers going on when you split up the multiply to carry-shift. Mabe this can help with understanding how carries work in multiplication. Also you can take multiplications shifts of any odd number and you will get the respective number of collisions depending on the number you shift the odd number by, so mabe this can help with collisions too.
User avatar
Pradu
 
Posts: 343
Joined: 12 Jan 2005, 19:17
Location: Chandler, Arizona, USA

Re: Generating Magic Keys for Bitboard Move Generation

Postby Gerd Isenberg » 16 Aug 2006, 18:35

Pradu wrote:I've found an interesting property in multiplication.

Take any set of unique keys, multiply each key by any odd number and you will get another set of unique keys (mabe easier to hash to -- multiply the 2nd set of keys with the magic for the 2nd set of keys and you will have the magic for the 1st set). For example take all 32 bit numbers, multiply it by any odd number and you will get another set of 32 bit numbers except in a different order. You can see the swaps of the numbers going on when you split up the multiply to carry-shift. Mabe this can help with understanding how carries work in multiplication. Also you can take multiplications shifts of any odd number and you will get the respective number of collisions depending on the number you shift the odd number by, so mabe this can help with collisions too.


I don't exactly understand what keys you mean - obviously not De Bruijn sequences to map a single bit to an index range. If you have a De Bruijn sequence with 64 unique six-bit sub-sequences, it is a ring of 64+5 bits. So the five leading bits must be equal to the five hidden zeros, "right" of the least significat bit - thus the five leading bits of a 64|6-bit De Bruijn sequence must be zero! If you multiply this sequence by some odd numbers it is likely that those five leading bits become non-zero.

Do you have some sample to prove your thesis?

Thanks,
Gerd
Gerd Isenberg
 
Posts: 285
Joined: 31 Jan 2005, 20:31
Location: Hattingen, Germany

Re: Generating Magic Keys for Bitboard Move Generation

Postby Rolf Sua » 17 Aug 2006, 09:32

Pradu wrote:I've found an interesting property in multiplication.

Take any set of unique keys, multiply each key by any odd number and you will get another set of unique keys (mabe easier to hash to -- multiply the 2nd set of keys with the magic for the 2nd set of keys and you will have the magic for the 1st set). For example take all 32 bit numbers, multiply it by any odd number and you will get another set of 32 bit numbers except in a different order. You can see the swaps of the numbers going on when you split up the multiply to carry-shift. Mabe this can help with understanding how carries work in multiplication. Also you can take multiplications shifts of any odd number and you will get the respective number of collisions depending on the number you shift the odd number by, so mabe this can help with collisions too.


Isn't this just the fact that any odd number is co-prime to 2**n for any n?
Essentially we are doing arithmetic not in the usual integers, but rather in the ring of Z/2**n (i.e. think of the integers from 0 to (2**n)-1 arranged in a circle so that 2**n = 0). In any such ring, all those integers which a co-prime (no common prime factor) with the size of the ring (in this case 2**n) have inverses. Since the only prime factor of 2**n is 2, and no odd number has prime factor 2, all odd numbers are invertible in Z/2**n, which explains your observation.
But I can't see how this is going to help you in finding magic keys...
Rolf Sua
 
Posts: 2
Joined: 03 Jul 2005, 02:52

Re: Generating Magic Keys for Bitboard Move Generation

Postby Pradu » 21 Aug 2006, 03:24

Sorry, been away for awhile...

Gerd Isenberg wrote:
I don't exactly understand what keys you mean - obviously not De Bruijn sequences to map a single bit to an index range. If you have a De Bruijn sequence with 64 unique six-bit sub-sequences, it is a ring of 64+5 bits. So the five leading bits must be equal to the five hidden zeros, "right" of the least significat bit - thus the five leading bits of a 64|6-bit De Bruijn sequence must be zero! If you multiply this sequence by some odd numbers it is likely that those five leading bits become non-zero.

Do you have some sample to prove your thesis?

Thanks,
Gerd


Yes not DeBruijn. By keys I meant the keys to hash (the occupancy keys for example) not the magics.

Rolf Sua wrote:But I can't see how this is going to help you in finding magic keys...
All I'm looking at right now are for any properties I can find that determine the influence of the bits in the keys and the magic in the top n bits. I never thought about overflowing multiplication with a picture of multiplication as numbers arranged in a circle ... I'll think about this and see if it helps any. The original thought was to create another set of unique numbers that will be easier to hash for.
User avatar
Pradu
 
Posts: 343
Joined: 12 Jan 2005, 19:17
Location: Chandler, Arizona, USA

Re: Generating Magic Keys for Bitboard Move Generation

Postby Pradu » 22 Aug 2006, 21:55

I just found out minimal bit magics arn't possible without redundant bits by testing out square D8 for bishops. I guess I'll quit working on finding minimal-bit magics because adding redundant bits will make the math hell complex and brute-force magic generation nearly impossible. Instead I'll concentrate on magics that create the smallest possible databases.

You can find the source to my rather simple magic generator with test code for D8 here.
User avatar
Pradu
 
Posts: 343
Joined: 12 Jan 2005, 19:17
Location: Chandler, Arizona, USA

Next

Return to Programming and Technical Discussions

Who is online

Users browsing this forum: No registered users and 16 guests