Fruit 1.5 parameter test extended

[Robert Allgeuer]

Geschrieben von:/Posted by: Robert Allgeuer at 27 August 2004 19:52:33:

This is an extension of the Fruit 1.5 parameter test that I have run recently (see past post on . It was also posted in CCC, but for some reasons it does not show up in the search engine)

Method:
=======
The test consisted of a round robin tournament of several configurations of Fruit 1.5 and a set of reference engines (please note that this test was run before the El Chinito case has become public).
The Nunn 1 starting positions were used; for each pairing each engine had to play both sides, resulting in 20 games for each pairing and slightly more than 5000 games overall.
The tournament results have been analysed with Elostat and a corresponding rating table has been calculated.

Platform, Tools and Settings:
=============================
Athlon XP 2400+
1.1 GB RAM
Windows XP
Elostat 1.1b
Arena 1.08
Time Control: 5min + 2sec
Ponder off
EGTBs enabled when supported
64MB Hash

Participants:
=============
Ten different configurations of Fruit 1.5, including the default settings, eight settings with exactly one UCI-parameter modified each and one setting with a combination of 3 modified parameters:
Fruit v1.5def: Fruit 1.5 with the default parameter setting
Fruit v1.5nmalways: nullmove search is tried always (instead of in the fail-high case only)
Fruit v1.5noetc: ETC disabled
Fruit v1.5ppushext: pawn push extension (7th rank) enabled
Fruit v1.5nosinglerep: single reply extension disabled
Fruit v1.5noqchecks: quiescence search does not include checking moves
Fruit v1.5nmR2: nullmove reduction set to 2 instead of the default 3
Fruit v1.5qchknm: quiescence search considers checking moves only after a nullmove
Fruit v1.5matvjr: alternative settings for piece values by J. Rang
Fruit v1.5comb1: combination of nullmove always, ETC disabled and pawn push extensions enabled; the three parameter settings that each have resulted in a higher result than the default settings (albeit in some cases only by a very tiny margin)
plus 13 other engines as reference opponents.

Results:
========


    Program                     Elo    +   -   Games   Score   Av.Op.  Draws
 01 Ruffian v1.01             : 2698   24  38   440    70.7 %   2545   21.8 %
 02 List v5.12                : 2664   26  35   440    66.2 %   2547   23.0 %
 03 El Chinito v3.25          : 2648   26  33   440    64.1 %   2548   24.1 %
 04 Gothmog v0.4.8            : 2604   29  31   440    57.7 %   2550   18.6 %
 05 Fruit v1.5nmalways        : 2598   30  29   440    56.8 %   2550   22.3 %
 06 Fruit v1.5noetc           : 2577   31  28   440    53.8 %   2551   21.1 %
 07 Fruit v1.5comb1           : 2576   31  28   440    53.6 %   2551   21.8 %
 08 Fruit v1.5ppushext        : 2573   32  27   440    53.1 %   2551   24.3 %
 09 Fruit v1.5def             : 2568   32  28   440    52.4 %   2551   21.6 %
 10 Fruit v1.5qchknm          : 2560   33  27   440    51.1 %   2552   21.8 %
 11 Fruit v1.5matvjr          : 2557   33  26   440    50.8 %   2552   23.4 %
 12 Fruit v1.5noqchecks       : 2557   33  28   440    50.7 %   2552   19.5 %
 13 Fruit v1.5nosinglerep     : 2554   33  25   440    50.3 %   2552   27.5 %
 14 AnMon v5.21               : 2550   26  33   440    49.7 %   2552   24.3 %
 15 SoS4                      : 2543   26  32   440    48.6 %   2553   24.5 %
 16 Ktulu v5.0                : 2541   25  32   440    48.3 %   2553   29.8 %
 17 Fruit v1.5nmR2            : 2534   27  32   440    47.3 %   2553   25.0 %
 18 Amyan v1.592              : 2533   28  32   440    47.2 %   2553   21.6 %
 19 Yace Paderborn            : 2513   30  30   440    44.1 %   2554   19.1 %
 20 Ufim v5.00                : 2465   33  27   440    37.2 %   2556   22.5 %
 21 Frenzee v1.59             : 2441   36  26   440    33.9 %   2557   19.5 %
 22 Patzer v3.61              : 2427   38  25   440    32.0 %   2558   19.5 %
 23 Sjeng v12.13              : 2415   39  25   440    30.5 %   2558   19.5 %


Essentially all these parameters have pretty much no impact on Fruit´s playing strength with the probable exceptions of:
- Enabling "nullmove always" probably increases playing strength
- Setting nullmove reduction to 2 probably decreases playing strength
Interestingly the combined setting (comb1) scored lower than the pure nmalways setting, but maybe this can be blamed on statistics.

Conclusion:
===========
Generally the impact of the different UCI parameter settings on Fruit´s playing strength is comparatively small, in the end all results still fall within the error margins of ~30.
I personnally am a bit surprised that enabling/disabling the extensions makes pretty much no difference, and would be interested in views as to why this would be the case.
I would have also expected a bigger impact of the modified piece value settings and of disabling/enabling checks in the quiescence search.
Robert

[Fabien Letouzey]

Geschrieben von:/Posted by: Fabien Letouzey at 31 August 2004 15:17:12:
Als Antwort auf:/In reply to: Fruit 1.5 parameter test extended geschrieben von:/posted by: Robert Allgeuer at 27 August 2004 19:52:33:


Hi Robert,
Thanks very much for these tests.
I had seen your first article but long after it was posted.

Fruit v1.5def: Fruit 1.5 with the default parameter setting
Fruit v1.5nmalways: nullmove search is tried always (instead of in the fail-high case only)
Fruit v1.5noetc: ETC disabled
Fruit v1.5ppushext: pawn push extension (7th rank) enabled
Fruit v1.5nosinglerep: single reply extension disabled
Fruit v1.5noqchecks: quiescence search does not include checking moves
Fruit v1.5nmR2: nullmove reduction set to 2 instead of the default 3
Fruit v1.5qchknm: quiescence search considers checking moves only after a nullmove
Fruit v1.5matvjr: alternative settings for piece values by J. Rang
Fruit v1.5comb1: combination of nullmove always, ETC disabled and pawn push extensions enabled; the three parameter settings that each have resulted in a higher result than the default settings (albeit in some cases only by a very tiny margin)
Essentially all these parameters have pretty much no impact on Fruit´s playing strength with the probable exceptions of:
- Enabling "nullmove always" probably increases playing strength
- Setting nullmove reduction to 2 probably decreases playing strength
Interestingly the combined setting (comb1) scored lower than the pure nmalways setting, but maybe this can be blamed on statistics.
Generally the impact of the different UCI parameter settings on Fruit´s playing strength is comparatively small, in the end all results still fall within the error margins of ~30.
I personnally am a bit surprised that enabling/disabling the extensions makes pretty much no difference, and would be interested in views as to why this would be the case.
I would have also expected a bigger impact of the modified piece value settings and of disabling/enabling checks in the quiescence search.
Robert

Joachim also obtained good results for this, but needed more time to confirm. I decided not to postpone the official release and use the more conservative "Fail High". It is probably the most promising change though.
ETC is a "safe bet" choice. I don't expect a better average behaviour but it can help "a lot" in rare cases (late endgames).
The fact that this does not seem to help is especially surprising since v1.5 has no knowledge of dangerous passers (the score only depends on the rank). Joachim had similar results.
In general though I believe in more moves in the quiescence search rather than extending in the whole tree. I have not started working on this however.
Enabling this extension seemed to help Fruit 1.0 a bit (it was disabled by default). Joachim convinced me to keep that extension to partly compensate for lack of king safety.
This should harm Fruit. I remember Joachim aborted the same test due to a significantly bad partial result. Your result is contradictory with Joachim's.
This was the original intent, but trying checks everywhere is more consistent (e.g. allows easier hash table in the quiescence search).
My own tests also show no improvement.
Yes it is suprising, and also noticed by Joachim. Only ETC is a move-ordering heuristic, and as such should be tested using specific methods (e.g. comparing search times on identical positions). One could expect other changes which alter the tree shape should have more impact indeed.
Yes, I suspect this latter test was "random". The only unexpected result in your list.
Thanks again!
Fabien.