AEGT thoughts

[Klaus Friedel]

Geschrieben von:/Posted by: Klaus Friedel at 26. July 2004 23:19:

Hello,
I don't know how far the draft of AEGT rules is fixed, but I'd like to request some changes.
Running a tournament on very different systems is a diffcult thing, but I think the suggested time adjustments should somehow compensate the different hardware.
But I don't think we should allow different GUIs here. Some engines might behave very different under different GUIs. This is very important if the engine supports both UCI and WB protocol. This is especially true, if you allow learning during the tournament, as some engines only learn using one of the protocols. (Snitch for example only learns when run under WB-protocol because UCI-1 does not send any "End of game" information).
So I propose to use a single GUI on all machines. I would suggest to use Arena here, as it's very flexible and it's free.
Regards,
Klaus Friedel



http://www.friedelprivat.de

[Heinz van Kempen]

Geschrieben von:/Posted by: Heinz van Kempen at 27 July 2004 09:05:24:
Als Antwort auf:/In reply to: AEGT thoughts geschrieben von:/posted by: Klaus Friedel at 26. July 2004 23:19:

Hello,
I don't know how far the draft of AEGT rules is fixed, but I'd like to request some changes.
Running a tournament on very different systems is a diffcult thing, but I think the suggested time adjustments should somehow compensate the different hardware.
But I don't think we should allow different GUIs here. Some engines might behave very different under different GUIs. This is very important if the engine supports both UCI and WB protocol. This is especially true, if you allow learning during the tournament, as some engines only learn using one of the protocols. (Snitch for example only learns when run under WB-protocol because UCI-1 does not send any "End of game" information).
So I propose to use a single GUI on all machines. I would suggest to use Arena here, as it's very flexible and it's free.
Regards,
Klaus Friedel

Hello Klaus,
it is not our goal to deliver a scientifically exact experiment here with all conditions exact, only most.
We want also comparisons. Are there engines doing worse under Fritz (Crafty for example) or under Winboard (Nejmet?). We will keep also different database, one with own books and one with Nunn positions used by some.
The main idea is to do something new with more time where also the testers have fun in and give them the opportunity to get more acquainted, to interchange new ideas. This is nothing productive in economic sense like Thomas wrote before, nothing that makes us to slaves of statistics, it is voluntary and no one has obligations to deliver a certain amount of games.
Please look also at the posting from Roger, where he replies to Uri.
http://f11.parsimony.net/forum16635/messages/69144.htm
Any author of an engine of course can tell us that he do not want to participate in our tournament. This will be respected.
Best Regards
Heinz

[Joachim Rang]

Geschrieben von:/Posted by: Joachim Rang at 27 July 2004 11:23:52:
Als Antwort auf:/In reply to: AEGT thoughts geschrieben von:/posted by: Klaus Friedel at 26. July 2004 23:19:

Hello,
I don't know how far the draft of AEGT rules is fixed, but I'd like to request some changes.
Running a tournament on very different systems is a diffcult thing, but I think the suggested time adjustments should somehow compensate the different hardware.
But I don't think we should allow different GUIs here. Some engines might behave very different under different GUIs. This is very important if the engine supports both UCI and WB protocol. This is especially true, if you allow learning during the tournament, as some engines only learn using one of the protocols. (Snitch for example only learns when run under WB-protocol because UCI-1 does not send any "End of game" information).
So I propose to use a single GUI on all machines. I would suggest to use Arena here, as it's very flexible and it's free.
Regards,
Klaus Friedel

I was about to post the same remark here and asking why play an Amateur event under the Fritz-GUI (especially when crafty is involved). But I think the explanation given by Heinz are valid it will be interesting to see if there are significant differences in different GUIs and between playing with books or from starting positions.
After all one might even get more information out, than "this engines seems to be the best".
regards Joachim

[Luis Smith]

Geschrieben von:/Posted by: Luis Smith at 27 July 2004 11:32:52:
Als Antwort auf:/In reply to: AEGT thoughts geschrieben von:/posted by: Klaus Friedel at 26. July 2004 23:19:

Hello,
I don't know how far the draft of AEGT rules is fixed, but I'd like to request some changes.
Running a tournament on very different systems is a diffcult thing, but I think the suggested time adjustments should somehow compensate the different hardware.
But I don't think we should allow different GUIs here. Some engines might behave very different under different GUIs. This is very important if the engine supports both UCI and WB protocol. This is especially true, if you allow learning during the tournament, as some engines only learn using one of the protocols. (Snitch for example only learns when run under WB-protocol because UCI-1 does not send any "End of game" information).
So I propose to use a single GUI on all machines. I would suggest to use Arena here, as it's very flexible and it's free.
Regards,
Klaus Friedel

I don't think that is a good idea. The fact that all the testers use different GUI's is reflective of how people in this forum also use these different GUI's. If we play games with say Crafty under Winboard, Arena, Fritz GUI, and ChessPartner GUI then all the games will be reflective of how it does on average in all the interfaces and not just one.
Anyway I think we should compile a list of all the interfaces all those participating in AEGT are using. I'll start I guess
Luis Smith - Winboard

[Heinz van Kempen]

Geschrieben von:/Posted by: Heinz van Kempen at 27 July 2004 11:44:47:
Als Antwort auf:/In reply to: Re: AEGT thoughts geschrieben von:/posted by: Joachim Rang at 27 July 2004 11:23:52:

Hello,
I don't know how far the draft of AEGT rules is fixed, but I'd like to request some changes.
Running a tournament on very different systems is a diffcult thing, but I think the suggested time adjustments should somehow compensate the different hardware.
But I don't think we should allow different GUIs here. Some engines might behave very different under different GUIs. This is very important if the engine supports both UCI and WB protocol. This is especially true, if you allow learning during the tournament, as some engines only learn using one of the protocols. (Snitch for example only learns when run under WB-protocol because UCI-1 does not send any "End of game" information).
So I propose to use a single GUI on all machines. I would suggest to use Arena here, as it's very flexible and it's free.
Regards,
Klaus Friedel

I was about to post the same remark here and asking why play an Amateur event under the Fritz-GUI (especially when crafty is involved). But I think the explanation given by Heinz are valid it will be interesting to see if there are significant differences in different GUIs and between playing with books or from starting positions.
After all one might even get more information out, than "this engines seems to be the best".
regards Joachim

Hello Joachim,
as you know fact is that in other fora people use preferably Fritz GUI and are also running there Gothmog, Fruit, Crafty and many more. I promise you when it can be demonstrated that Crafty or many others will perform worse here I will throw my (up to now) favourite GUI away. I suppose you looked a bit at my Nunn tournaments and I do not see any considerable differences from my tournaments and ratings to others with Winboard or Arena (except from normal statistical deviations) and also the new Crafty versions are doing very well. But I am open to be convinced otherwise. On the other hand Olivier asked Odd Gunnar Malin if he can avoid with Wb2UCI what was posted by Mike Byrne in CCC forum. If you do not have it present, write to Olivier plesse. Hope he has time to explain.
Another thing, in past years I did not use Fritz GUI for tournaments because of totally messed up UCI.dll files, at least not for the amateurs. Other people except me noticed that with the new UCI.dll that came with Shredder 8 some things improved or am I dreaming this? I practically do not have the 1 MB bug anymore. Anyone wants to tell me that Arena or Winboard do not have any disadvantages?
I only can answer tomorrow, have a lot of things to prepare still.
Best Regards
Heinz

[Heinz van Kempen]

Geschrieben von:/Posted by: Heinz van Kempen at 27 July 2004 11:47:35:
Als Antwort auf:/In reply to: Re: AEGT thoughts geschrieben von:/posted by: Luis Smith at 27 July 2004 11:32:52:

Hello,
I don't know how far the draft of AEGT rules is fixed, but I'd like to request some changes.
Running a tournament on very different systems is a diffcult thing, but I think the suggested time adjustments should somehow compensate the different hardware.
But I don't think we should allow different GUIs here. Some engines might behave very different under different GUIs. This is very important if the engine supports both UCI and WB protocol. This is especially true, if you allow learning during the tournament, as some engines only learn using one of the protocols. (Snitch for example only learns when run under WB-protocol because UCI-1 does not send any "End of game" information).
So I propose to use a single GUI on all machines. I would suggest to use Arena here, as it's very flexible and it's free.
Regards,
Klaus Friedel

I don't think that is a good idea. The fact that all the testers use different GUI's is reflective of how people in this forum also use these different GUI's. If we play games with say Crafty under Winboard, Arena, Fritz GUI, and ChessPartner GUI then all the games will be reflective of how it does on average in all the interfaces and not just one.
Anyway I think we should compile a list of all the interfaces all those participating in AEGT are using. I'll start I guess
Luis Smith - Winboard

Hello Luis,
thanks for answering, this is an important point. We all have our priorities, favourite GUI, favourite engines, favourite tournament modes etc..
The list is not necessary. In a few days Igor will have all data complete and uploaded in his draft.
Best Regards
Heinz

[Klaus Friedel]

Geschrieben von:/Posted by: Klaus Friedel at 27. July 2004 22:54:
Als Antwort auf:/In reply to: Re: AEGT thoughts geschrieben von:/posted by: Luis Smith at 27 July 2004 11:32:52:

Hello,
I don't know how far the draft of AEGT rules is fixed, but I'd like to request some changes.
Running a tournament on very different systems is a diffcult thing, but I think the suggested time adjustments should somehow compensate the different hardware.
But I don't think we should allow different GUIs here. Some engines might behave very different under different GUIs. This is very important if the engine supports both UCI and WB protocol. This is especially true, if you allow learning during the tournament, as some engines only learn using one of the protocols. (Snitch for example only learns when run under WB-protocol because UCI-1 does not send any "End of game" information).
So I propose to use a single GUI on all machines. I would suggest to use Arena here, as it's very flexible and it's free.
Regards,
Klaus Friedel

I don't think that is a good idea. The fact that all the testers use different GUI's is reflective of how people in this forum also use these different GUI's. If we play games with say Crafty under Winboard, Arena, Fritz GUI, and ChessPartner GUI then all the games will be reflective of how it does on average in all the interfaces and not just one.
Anyway I think we should compile a list of all the interfaces all those participating in AEGT are using. I'll start I guess
Luis Smith - Winboard

You are somehow right. Using diffent GUIs might give a better "over all GUIs an protocols performance" score for each engine. But thats only right, if you test all engines under all GUIs. But as I understand the current rules there might be very different protocol and GUI mixes for each engine.
At least we should store under which GUI a game was played.
Regards,
Klaus Friedel




http://www.friedelprivat.de

[Klaus Friedel]

Geschrieben von:/Posted by: Klaus Friedel at 27. July 2004 23:01:
Als Antwort auf:/In reply to: AEGT thoughts geschrieben von:/posted by: Klaus Friedel at 26. July 2004 23:19:

Hello,
another clarification to the rules. If you enable learning a engine will normaly store its new knowledge to some files. Will you exchange learn-files between testers or will the engine start as dumb as it was before when playing on a other machine ?
Regards,
Klaus Friedel



http://www.friedelprivat.de

[Heinz van Kempen]

Geschrieben von:/Posted by: Heinz van Kempen at 28 July 2004 09:26:30:
Als Antwort auf:/In reply to: Re: AEGT thoughts geschrieben von:/posted by: Klaus Friedel at 27. July 2004 23:01:

Hello,
another clarification to the rules. If you enable learning a engine will normaly store its new knowledge to some files. Will you exchange learn-files between testers or will the engine start as dumb as it was before when playing on a other machine ?
Regards,
Klaus Friedel

Hello Klaus,
the whole learning issue was discussed somewhat heavily here in the forum when you go back a few days in the archive.
We opted to allow book and position learning, but there were a lot that wanted to limit this:
proposals:
---"reset" learning by installing a fresh learnfile, because it might be that for example Crafty 19.15 already played a lot test or tournament games on the machine of one tester
delete (reinstall a fresh file) after a series of games is played
Maybe you understand that the purist testers up to now with big majority always disabled learning. Things will change here in the future I suppose when many engines have it and that we voted in favour of learning is surely because some accepted the points Bryan gave.
It was not discussed, but if I assess correctly the spirit in our group I suppose that exchanging learnfiles would be deemed to exaggerate this stuff. There were opinions from authors that wrote emails to us, telling that this learning might be overestimated especially in case of big books and really only would have effect when playing a series against the same opponent.
Of course other opinions are welcome, but maybe without another long thread debating the learning issue .
What refers to the rules all decisons are fix now at least for this stage of the tournament. The whole project I will explain later, if I see that more testers are willing to collaborate and/or those we have will continue despite of there own projects like Chess War, Infinite Loop, System de Suisse etc..
You will notice that we were forced to leave out a lot of fine engines due to still limited CPU ressources or capacities. So gauntlets are needed anyway for rating calculation.
SOS 4 is one that has to play it, although it is surely rank 3-5 from free engines, but was not voted for because testers were free to apply other criteria. With more testers a Rook B Class or Knight Class would be possible.
So we will first wait for the first results. Most still have to update their homepages with AEGT issues. I will do this by the beginning of next week.
Best Regards
Heinz