Feature: signature to authenticate engine

[yoshiharu]

Hello,

here is a quite peculiar feature request: adding to winboard a facility to check the identity of the engine executable.

First the motivations: we (the group of italian chess programmers) are reforming our tournament regulations: we will have two kinds of categories for chess engines, "the originals" and "all the others"; in particular for the former category we request (I'm cutting the details off) that the authors deposit in the hands of a third (trusted) part (basically the referee) the executable(s) of the engines that are going to play; this is only a part of our control mechanism, but by itself it enables the organisation to make the judgements about complaints also after the tournament has finished.

The feature we're requesting would enable referees to check (with reasonable confidence) that the engine playing matches the engine deposited before the tournament, before the beginning of every game, by showing some signature (md5, sha, whatever) of the executable.

Of course we could take care of this ourselves, but we would prefer by far an official release of winboard as this could maybe prompt other GUI developers to introduce this feature in their interfaces.

Can we expect this to be included in a future version of Winboard?

Thanks in advance, Mauro (on behalf of G6)

[H.G.Muller]

Well, I see a couple of problems. For instance, when the engine is UCI, WinBoard would return the md5 of Polyglot, and that would not prove a whole lot. And thinking a bit further along those lines: what would stop a cheater from making his engine really an adapter with some rudimentary chess ability, that when he runs it in the tournament would start up Houdini? Based on some adapter.ini file accessed by the binary he submitted, which WinBoard does not know about?

It seems to me that the only way to verify that people are playing with the engine they submitted is to require a log of the thinking output from their tournament games, select a number of positions from it, and run the submitted binary on them to see if it reproduces that thinking output closely enough.

[yoshiharu]

Well, I see a couple of problems. For instance, when the engine is UCI, WinBoard would return the md5 of Polyglot, and that would not prove a whole lot.

Ok, this is an issue.
We should use a patched Polyglot, maybe.

And thinking a bit further along those lines: what would stop a cheater from making his engine really an adapter with some rudimentary chess ability, that when he runs it in the tournament would start up Houdini? Based on some adapter.ini file accessed by the binary he submitted, which WinBoard does not know about?

It seems to me that the only way to verify that people are playing with the engine they submitted is to require a log of the thinking output from their tournament games, select a number of positions from it, and run the submitted binary on them to see if it reproduces that thinking output closely enough.

Well, I said I was skipping the details
The reason why the engine executable must be deposited, is that we are planning to make this kind of verification offline, after the tournament finishes, iff some author files an official complaint.
So they cannot use the executable as an inbetween, since that would not work on the referee's PC.

Actually we were more focused on matching the moves than the thinking logs, but we have not completely made up our minds on this precise detail, it is likely we'll switch to using either some thinking log or the pgn annotated with engine's final PV and score.

[H.G.Muller]

Well, then I guess it is more important that WinBoard can annotate the PGN with PVs. Currently it only does that in 'Analyze Game' mode.

The binary md5 identification still seems quite pointless to me. If people know what verification is done (and they will, as WinBoard is open source), it should be quite trivial to give any executable they want the correct signature, by just appending a few words to it.

So it seems essential that people would not know how the verification works.

Perhaps you could write a 'verification engine' that gets the name of the executable passed from WinBoard on its command line, and then opens that file for reading, to derive some signature from it. WinBoard could then send the board position (or a hash key derived from it) after every move the program plays to the verification engine, and include the reply (a number derived from the signature and the position) as a comment to the PGN.

You could hand out the verification engines just minutes before the round starts, so people don't have time to reverse engineer them. When UCI engines are run through WinBoard's -fUCI option, WB would be aware of the name of the engine executable. (Provided people did not mess with Polyglot; so you would have to let the verification engine verify Polyglot as well, in such a case.)

[yoshiharu]

Well, then I guess it is more important that WinBoard can annotate the PGN with PVs. Currently it only does that in 'Analyze Game' mode.

I see. Yes, that would be quite useful I think.
I actually for some reason assumed it was already the case, but I didn't check it.

The binary md5 identification still seems quite pointless to me. If people know what verification is done (and they will, as WinBoard is open source), it should be quite trivial to give any executable they want the correct signature, by just appending a few words to it.

Well, one can check for executable file size, too. Anyways, this is meant just as a single layer of verification among others, just to give to the referee the chance to say "something's wrong with the signature" before the first move is played.
The "main" verifications will be carried forward in case of complaint, after the tournament.
To this extent the feature of logging PV in pgn you were mentioning would be very interesting and useful, IMHO: we were going to match the moves played by the engine during the game with those the deposited engine would play, since this is the minimal requirement for an engine (they have got to play moves during the game, don't they? ), but actually engines that don't output any info during the game are very rare, so matching PVs could be done in most cases, I hope.

[H.G.Muller]

Well, you could make it a requirement that the engine outputs a PV. Most on-line tourneys require that the engines kibitz their PVs...

The problem I foresee is that if you let WinBoard print (say) the length of the exe file in a PGN tag, it would be trivial for cheaters to play a test game before the match with WinBoard plus the engine they submitted, see what it prints in the PGN, and then either come with a Houdini padded to the same length, or even more boldly, simply use 'edit tags' to adapt or add that tag with the value they now know it has got to have, or simply make their own WinBoard that they rig to print that fixed verification tag no matter what the executable is they are running with it.

They can't do that if they don't know what has to be printed because it depends on the game (in a way unknown to them in advance) as well as the engine signature.

[mrlissandrello]

Well, I see a couple of problems. For instance, when the engine is UCI, WinBoard would return the md5 of Polyglot, and that would not prove a whole lot.

Hi everyone, is true, this is a problem but...

If WinBoard return the md5 of Polyglot then winboard could read the polyglot.ini and read the EngineName and the EngineDir

in this case will read the md5 of enginename.

To do this winboard should have stored md5 of all versions of polyglot (4).

Sorry my bad English.

Luca Lissandrello (on behalf of G6)