Stef Luijten wrote:But a KQkr position with white to move can be made completely equivalent to a KRkq position with black to move. I think there is a color symmetry in this case, so why do we need two separate EGTB files? Isn't one KQKR-file enough, dealing with all cases?
That is the crux: The KQkr EGTB does not contain
any positions with KRkq. To make use of the symmetry you point out, you would first have to include all the KRkq positions, which completely defeats the purpose (which is to reduce the size of the TB).
In fact the use of symmetry in the presence of pawns is of the same questionable value, and more often than not self-defeating: To make use of the left-right mirror symmetry in (say) the KPk end-game with the P in the b file, you would first have to add all positions with pawns in the g file. Thus in the end this saves nothing, it merely adds a needless complication to your EGTB generator, which now has to check all the time if the white King crosses the median.
The key to doing big TBs (like 7 or 8 men) is to
decompose them into fragments that are as independent as possible, so that you can completely work them out independently, without taxing your cache or RAM resources too much and without having to refer (often) to other fragments that are in the mean time stored on a much slower device. Pawns are ideally suited for that: as long as you work on a KPPkpp end-game with c,d vs d,e pawns, for instance, you will
never have to consult the 6-men TB fragments with pawns in other files. Just very small subsets of the 5-men TB's (those with 3 pawns in a limited number of combinations in the c,d and e files), which can be easily cached in memory, while the TB fragment itself contains less than 64x64x6x6x6x6/4 = 1,32M (the division by 4 coming from permuting pawns). And of course to the various 6-men TBs with Queens, which is much more of a problem.
In fact you can make use of the forward nature of the pawns to define even smaller sub-fragments, that do not only confine the pawns to a file, but to a specific square. When you are working on the sub-fragment with pawns on (say) c4,d3-d7,e4 (a sub-fragment containing less than 4096 positions in the KPPkpp end-game!), you only have to refer to 4 other sub-fragments from this 6-men TB, whith each of the pawns advanced by one square. A Commodore 64-could do it! For today's computers it only becomes a (modest) challenge if there are two more pieces on the board, say KRPPkrpp. Then the sub-fragments still only measure 16MB, so that many of them can be cached in DRAM, and you can work your way backwards through the TB at RAM-speed, rater than disk-speed. Of course there will be very many sub-fragments, so it still takes a long time.
The major problem is again conversion to end-games with promoted pawns. KQQRkqqr is quite hope-less. From KRPPkrpp you can directly convert only to KQRPkrpp and KRPPkqrp, though. So the key is to do these extremely skewed TBs. If you are making TBs for DTC rather than DTM you can apply tricks, since for the KRPPkrpp TB you then only have to know
if a position you can convert to in KQRPkrpp is won, and not in how many moves. Presumably you can then solve this latter end-game giving the strong side a handicap aimed at reducing the number of positions or the fragment size. In particular, you could forbid the strong side to have more than one Queen, or forbid him to move his pawns altogheter. I expect the end-game to remain fully won despite such restriction, due to the overwhelming power of the extra Queen.