Creating a positional testset

[Klaus Friedel]

I'm about to compose a positional testset. I think the WM-Test would be a good base. It ships with a subset of 36 positions thats considered to contain mainly positional positions. But some of them focus to much other aspects like king safty or contain tactical aspects. Sure each positional advantage will turn into a tactical one sooner or later. But to have a real positional subset this should happen far beyond the horizon of current engines.

To distinguish between positions with to much tactical impact an pure positional positions I thought about the following test:

1. Solve the testset with your favorite engine at short timecontrols (10sec).
2. Repeat the test at much longer timecontrol (10min)
3. Remove all positions that are not found in 1) but are found in 2)
4. Maybe repeat 1)-3) with other engines.

This might look strange at first glance but if a engine finds a solution in 2) but not in 1) it's very likely that the positional advantage turns into a tactical one to fast (within the horizon of the engine) so your engine might have solved the test by tactics. If the positional evaluation of the engine is good you will find the solution very fast. If the positional evaluation of the engine is wrong it will never find the solution regardless how long it will search (as long as a first tactical bebefit is to deep).

Do you think this is good approach to build a testset ?

Klaus

[Uri Blass]

I disagree.

I believe that it is possible to translate one type of positional advantage that program does not understand to another type of positional advantage that it understand so it is possible that a program is going to find positional move in 10 minutes but not in 10 seconds.

It is also possible that it is possible to find by search that some type of positional advantage get bigger so the program may not choose the move in 10 seconds because it underestimates the positional advantage but find the move in 10 minutes because after 10 minutes it can see that the same type of positional advantage is big enough.

I think that good positional test should include moves when there is no proof that they win material relative to other moves.

I also think that there should be some verification for the claim that the suggested move is correct

I suspect that humans often do not understand chess positions so I do not trust the move to be correct without a verification.

For varification I suggest games that show that choosing the right move lead to better result.

Example:
If the right move is 25.Ne4 and 25.Nd4 is wrong move that engine may choose and if Ne4 is supposed to win the game positionally then we need games that human+computer beat all the top engines with white when they play 25.Ne4 when they draw against the same engines with black after 25.Nd4

Uri

[Klaus Friedel]

I disagree.

I believe that it is possible to translate one type of positional advantage that program does not understand to another type of positional advantage that it understand so it is possible that a program is going to find positional move in 10 minutes but not in 10 seconds.

It is also possible that it is possible to find by search that some type of positional advantage get bigger so the program may not choose the move in 10 seconds because it underestimates the positional advantage but find the move in 10 minutes because after 10 minutes it can see that the same type of positional advantage is big enough.

I think that good positional test should include moves when there is no proof that they win material relative to other moves.

I also think that there should be some verification for the claim that the suggested move is correct

I suspect that humans often do not understand chess positions so I do not trust the move to be correct without a verification.

For varification I suggest games that show that choosing the right move lead to better result.

Example:
If the right move is 25.Ne4 and 25.Nd4 is wrong move that engine may choose and if Ne4 is supposed to win the game positionally then we need games that human+computer beat all the top engines with white when they play 25.Ne4 when they draw against the same engines with black after 25.Nd4

Uri

Good point. Using my method we would remove the tactical ones from the testset, but also the ones with slightly increasing positional advantages.

Playing games as you describe it will surely be the best solution, but you would need a lot of different engines, long timecontrols and a lot of games to get statisticaly significant results.
Im' looking for a faster way to verify a solution has no tactical aspects.

What about solving the position with a good engine and following the PV. Compare the material balance at the first move of the PV with the position after last move. If there is no material gain (there might be a loss, compensated by the positional advantage) it's very likely we have found a "pure positional" position.

Regards,

Klaus

[Kurt Utzinger]

Hi Klaus
To create a good positional testset is the most tricky thing
you can try I think. The problem is to prove that a specific
move is clearly best. For quite some time ago I tried myself
to do something like that but have given up. It happened
too often that the test move (given a !! by top GM's in books)
lead to a small but not decisive advantage (programs could
not win such games) but other moves preferred by the engines
were much more successfull when playing out games with
chess programs.
Best regards
Kurt

[Klaus Friedel]

Hi Klaus
To create a good positional testset is the most tricky thing
you can try I think. The problem is to prove that a specific
move is clearly best. For quite some time ago I tried myself
to do something like that but have given up. It happened
too often that the test move (given a !! by top GM's in books)
lead to a small but not decisive advantage (programs could
not win such games) but other moves preferred by the engines
were much more successfull when playing out games with
chess programs.
Best regards
Kurt

That should be the next step after verifying the position in question has no tactical aspects.
Taking your experiences into account we maybe shouldn't look for the "best move" but for the "best move for a specific engine".
Now we are back to Uri's suggestion of playing shoutouts for the most promising moves of the position in question.

A time consuming approach. And after all this efford your testset would fit one engine only. Or even worse: One version of this particular engine.

Kurt, I slowly realize why you gave up

Klaus

[Kurt Utzinger]

That should be the next step after verifying the position in question has no tactical aspects.
Taking your experiences into account we maybe shouldn't look for the "best move" but for the "best move for a specific engine".
Now we are back to Uri's suggestion of playing shoutouts for the most promising moves of the position in question.

A time consuming approach. And after all this efford your testset would fit one engine only. Or even worse: One version of this particular engine.

Kurt, I slowly realize why you gave up

Klaus

Hi Klaus
Another reason for my unsuccessful try was maybe
that my understanding of chess is not sufficient
Best regards
Kurt

[Klaus Friedel]

Hi Klaus
Another reason for my unsuccessful try was maybe
that my understanding of chess is not sufficient
Best regards
Kurt

I don't think I'd have a better starting position here

Klaus