Winboard Forum

I was playing around with material values of Fruit 1.5 for fun
and got some interesting results. I was attempting to get the
values closer to the Larry Kaufman material values, but ended
up only seeing it improve by making the Bishop undervalued.

Here are the only values that were changed:

Knight:325 and 280 in the endgame
Bishop:300 and 320 in the endgame
Rook:497 and 526 in the endgame

I was wondering if there was a certain number of games that are
considered enough that a rating is looked at as significant. I
played several games and noticed how it repeats patterns - it
has no opening book by default. I'm not sure how tournaments
have it configured.
--------
Games are 10 0 or 0 15sec./move - on a AMD 1800XP machine
(at least 1 as white, 1 as black)
-----------------------------
3 Fruit_15 [003] : 2497 15 (+ 10,= 1,- 4), 70.0 %

Baron 1.5.0 : 2 (+ 0,= 1,- 1), 25.0 %
Fruit_15 : 2 (+ 1,= 0,- 1), 50.0 %
Chop1099 : 2 (+ 1,= 0,- 1), 50.0 %
Frenzee : 2 (+ 1,= 0,- 1), 50.0 %
Chezzz-1.0.3-windows : 2 (+ 1,= 0,- 1), 50.0 %
Phalanx : 3 (+ 3,= 0,- 0), 100.0 %
Abrok_5_0 : 2 (+ 2,= 0,- 0), 100.0 %
Quark : 2 (+ 2,= 0,- 0), 100.0 %
---

Original (http://www.playwitharena.com/tourney/at ... at-v31.zip):
(41) Fruit 1.5 - 07/05 : 136 (+ 56,= 38,- 42), 55.1 %
(edited....)
Abrok 5.0 : 2 (+ 0,= 0,- 2), 0.0 %
Quark 2.35 : 4 (+ 0,= 1,- 3), 12.5 %
Lamb Chop 10.99 : 2 (+ 0,= 1,- 1), 25.0 %
Phalanx 22 : 6 (+ 2,= 2,- 2), 50.0 %
The Baron 1.30 - 1.40 : 8 (+ 4,= 1,- 3), 56.2 %
Frenzee 1.57 - 1.59 : 4 (+ 3,= 0,- 1), 75.0 %
Chezzz 1.03 : 4 (+ 3,= 0,- 1), 75.0 %

In my view in order to be able to compare two settings should be tested under exactly the same conditions, i.e. against identical opponents and with the same number of games. Given that Fruit has no book I also think in such a case tests should start from the Nunn positions.
As to the number of games: 400 still gives an error margin of around +/-30, which is still not negligable.

Robert

B M Schaeffer wrote:I was playing around with material values of Fruit 1.5 for fun
and got some interesting results. I was attempting to get the
values closer to the Larry Kaufman material values, but ended
up only seeing it improve by making the Bishop undervalued.

Here are the only values that were changed:

Knight:325 and 280 in the endgame
Bishop:300 and 320 in the endgame
Rook:497 and 526 in the endgame

I was wondering if there was a certain number of games that are
considered enough that a rating is looked at as significant. I
played several games and noticed how it repeats patterns - it
has no opening book by default. I'm not sure how tournaments
have it configured.

So you played 15 Games or how many? I agree with Robert to get a reliable result you should play 400 Games. Of course 100 games already give you an indication, but to be (almost) sure...

I too "found" material settings which performed about 25 elopoints stronger in my tests only to see later in other tests, that I was probably wrong and my settings weren't stronger than the old ones.

Robert tried different material values and me too and we both ended up with Kaumann values or even higher material values. Of course we tested development versions where other things were changed as well, so it is not clear whether Fruit 1.5 "likes" other values than Fruit X.

regards Joachim

Thanks for the reply.

I did set up several lightning games v. ruffian and crafty, mainly to see if I could recognize the differences in fruit's play style. It was crushed repeatedly by both. The reverse happened with mscp.

Since Fruit has a rating (http://www.playwitharena.com/tourney/at ... at-v31.zip), I downloaded several engines it was more in league with i.e. my original post.

A few years ago I attempted to modify some engines to see the Larry Kaufman values in action. Borland had problems compiling many of engines, but it had no problems with gerbil and tscp. *Every* version that I modified played slightly worse as compared to the original, and I dropped it. The modified versions of gerbil did show an improvement in performance frequently between moves 25-35, but it tended to drop off and ultimately not be better than the original in terms of win/lose/draw.

One of my conclusions from testing different material values is that engines are relatively unsensitive to changes of them (as long as they stay reasonable). In my view changes in search and eval have a bigger impact.

Robert[/u]

I have run a round-robin tournament to get an idea if there was any real improvement. I chose 3/2 because the machine is used for alot of other things. Overall the new values made Fruit (named Fruit 003 below) performed worse as compared to engines with similar ratings. Most interesting was the performance difference between the original Fruit; the top two finishing engines were dominating Fruit and struggling against the lower rated Fruit 003.

Code: Select all

   Engine               Score        Ba       Ab       Ch       Fr       Fr       Fr       Ph       Ch    S-B
1: Baron 1.5.0          35.0/56 ???????? 101==001 11=0011= 11=1=1=1 10001010 10100=== 01=11111 =111==11  926.00
2: Abrok_5_0            33.5/56 010==110 ???????? 111=01== 10=1110= 11100=1= 01100000 01011101 11111101  889.50
3: Chop1099             32.0/56 00=1100= 000=10== ???????? 1111=1=1 01011100 01=0110= 11111001 11=1==10  846.25
4: Fruit_15             30.0/56 00=0=0=0 01=0001= 0000=0=0 ???????? 1=110111 =1=11111 11011100 11=10=11  761.25
5: Frenzee              26.5/56 01110101 00011=0= 10100011 0=001000 ???????? =1101111 0=101001 =000=011  752.50
6: Fruit_15 [003]       26.0/56 01011=== 10011111 10=1001= =0=00000 =0010000 ???????? =0101001 0=011111  733.75
7: Phalanx              24.0/56 10=00000 10100010 00000110 00100011 1=010110 =1010110 ???????? =0110111  636.75
8: Chezzz-1.0.3-windows 17.0/56 =000==00 00000010 00=0==01 00=01=00 =111=100 1=100000 =1001000 ????????  483.50

224 games played / Tournament finished
Name of the tournament: f3test
Site/ Country: BMS1, United States
Level: Blitz 3/2
Hardware: AuthenticAMD
Operating system: Windows 4.10 Build 67766222

Now I have my answer: slight tweaking of material values really does significantly alter performance, and if Fruit (or any engine) knew what it was up against it probably would do better to load custom settings for each opponent. Food for thought .

or maybe 8 games are not enough to know and even the same engine can get 4.5/8 against the baron in a good day and 1.5/8 against the baron in a bad day so you cannot learn from games against one opponent about better personality against it.

Uri

What are the Kaufman material values?

975 / 500 / 325 / 325 / 100 and 50 for the bishop pair

From an article of Larry Kaufman available e.g. here: http://mywebpages.comcast.net/danheisma ... alance.htm .

I would be interested which exact values were tested above with Fruit 1.5?
Independent of that I agree that with Uri that the number of games for each configuration (56) is still much too small for making conclusions.

Robert

Once again the machine is used for alot of other things.

I would prefer to have 800 games, 400 Fruit 1.5 and 400 Fruit 1.5 with those changes I mentioned (i.e.
Here are the only values that were changed:

Knight:325 and 280 in the endgame
Bishop:300 and 320 in the endgame
Rook:497 and 526 in the endgame

Standard values for these in Fruit 1.5 are:
Pawn: 100 100
Knight: 312 312
Bishop: 315 320
Rook: 500 525
Queen: 1000 980
Bishop Pair:20 40
)

Also have each game be 40 moves in 10 minutes. Or maybe 400 games against each opponent!

Anyway, I just don't see the trend to be overall improvement already, so I'm dropping it. I believe the modified Fruit performed in a way that makes me think there may be a trend of inverted performance against all other engines tested except Chezzz-1.0.3 which finished last.

Code: Select all

(repeated from above)
Engine               Score        Ba       Ab       Ch       Fr     
4: Fruit_15             30.0/56 00=0=0=0 01=0001= 0000=0=0 ????????
6: Fruit_15 [003]       26.0/56 01011=== 10011111 10=1001= =0=00000

Engine               Score        Fr       Fr       Ph       Ch    S-B

4: Fruit_15             30.0/56  1=110111 =1=11111 11011100 11=10=11  761.25
6: Fruit_15 [003]       26.0/56  =0010000 ????????  =0101001 0=011111  733.75

and in case your interested the results:
-----------------Abrok_5_0-----------------
Abrok_5_0 - Baron 1.5.0               : 4.0/8 3-3-2 (010==110)  50%    ?0
Abrok_5_0 - Chezzz-1.0.3-windows      : 7.0/8 7-1-0 (11111101)  88%  +346
Abrok_5_0 - Chop1099                  : 5.5/8 4-1-3 (111=01==)  69%  +139
Abrok_5_0 - Frenzee                   : 5.0/8 4-2-2 (11100=1=)  63%   +92
Abrok_5_0 - Fruit_15                  : 5.0/8 4-2-2 (10=1110=)  63%   +92
Abrok_5_0 - Fruit_15 [003]            : 2.0/8 2-6-0 (01100000)  25%  -191
Abrok_5_0 - Phalanx                   : 5.0/8 5-3-0 (01011101)  63%   +92
-----------------Baron 1.5.0-----------------
Baron 1.5.0 - Abrok_5_0               : 4.0/8 3-3-2 (101==001)  50%    ?0
Baron 1.5.0 - Chezzz-1.0.3-windows    : 6.5/8 5-0-3 (=111==11)  81%  +252
Baron 1.5.0 - Chop1099                : 5.0/8 4-2-2 (11=0011=)  63%   +92
Baron 1.5.0 - Frenzee                 : 3.0/8 3-5-0 (10001010)  38%   -85
Baron 1.5.0 - Fruit_15                : 6.5/8 5-0-3 (11=1=1=1)  81%  +252
Baron 1.5.0 - Fruit_15 [003]          : 3.5/8 2-3-3 (10100===)  44%   -42
Baron 1.5.0 - Phalanx                 : 6.5/8 6-1-1 (01=11111)  81%  +252
-----------------Chezzz-1.0.3-windows-----------------
Chezzz-1.0.3-windows - Abrok_5_0      : 1.0/8 1-7-0 (00000010)  13%  -330
Chezzz-1.0.3-windows - Baron 1.5.0    : 1.5/8 0-5-3 (=000==00)  19%  -252
Chezzz-1.0.3-windows - Chop1099       : 2.5/8 1-4-3 (00=0==01)  31%  -139
Chezzz-1.0.3-windows - Frenzee        : 5.0/8 4-2-2 (=111=100)  63%   +92
Chezzz-1.0.3-windows - Fruit_15       : 2.0/8 1-5-2 (00=01=00)  25%  -191
Chezzz-1.0.3-windows - Fruit_15 [003] : 2.5/8 2-5-1 (1=100000)  31%  -139
Chezzz-1.0.3-windows - Phalanx        : 2.5/8 2-5-1 (=1001000)  31%  -139
-----------------Chop1099-----------------
Chop1099 - Abrok_5_0                  : 2.5/8 1-4-3 (000=10==)  31%  -139
Chop1099 - Baron 1.5.0                : 3.0/8 2-4-2 (00=1100=)  38%   -85
Chop1099 - Chezzz-1.0.3-windows       : 5.5/8 4-1-3 (11=1==10)  69%  +139
Chop1099 - Frenzee                    : 4.0/8 4-4-0 (01011100)  50%    ?0
Chop1099 - Fruit_15                   : 7.0/8 6-0-2 (1111=1=1)  88%  +346
Chop1099 - Fruit_15 [003]             : 4.0/8 3-3-2 (01=0110=)  50%    ?0
Chop1099 - Phalanx                    : 6.0/8 6-2-0 (11111001)  75%  +191
-----------------Frenzee-----------------
Frenzee - Abrok_5_0                   : 3.0/8 2-4-2 (00011=0=)  38%   -85
Frenzee - Baron 1.5.0                 : 5.0/8 5-3-0 (01110101)  63%   +92
Frenzee - Chezzz-1.0.3-windows        : 3.0/8 2-4-2 (=000=011)  38%   -85
Frenzee - Chop1099                    : 4.0/8 4-4-0 (10100011)  50%    ?0
Frenzee - Fruit_15                    : 1.5/8 1-6-1 (0=001000)  19%  -252
Frenzee - Fruit_15 [003]              : 6.5/8 6-1-1 (=1101111)  81%  +252
Frenzee - Phalanx                     : 3.5/8 3-4-1 (0=101001)  44%   -42
-----------------Fruit_15-----------------
Fruit_15 - Abrok_5_0                  : 3.0/8 2-4-2 (01=0001=)  38%   -85
Fruit_15 - Baron 1.5.0                : 1.5/8 0-5-3 (00=0=0=0)  19%  -252
Fruit_15 - Chezzz-1.0.3-windows       : 6.0/8 5-1-2 (11=10=11)  75%  +191
Fruit_15 - Chop1099                   : 1.0/8 0-6-2 (0000=0=0)  13%  -330
Fruit_15 - Frenzee                    : 6.5/8 6-1-1 (1=110111)  81%  +252
Fruit_15 - Fruit_15 [003]             : 7.0/8 6-0-2 (=1=11111)  88%  +346
Fruit_15 - Phalanx                    : 5.0/8 5-3-0 (11011100)  63%   +92
-----------------Fruit_15 [003]-----------------
Fruit_15 [003] - Abrok_5_0            : 6.0/8 6-2-0 (10011111)  75%  +191
Fruit_15 [003] - Baron 1.5.0          : 4.5/8 3-2-3 (01011===)  56%   +42
Fruit_15 [003] - Chezzz-1.0.3-windows : 5.5/8 5-2-1 (0=011111)  69%  +139
Fruit_15 [003] - Chop1099             : 4.0/8 3-3-2 (10=1001=)  50%    ?0
Fruit_15 [003] - Frenzee              : 1.5/8 1-6-1 (=0010000)  19%  -252
Fruit_15 [003] - Fruit_15             : 1.0/8 0-6-2 (=0=00000)  13%  -330
Fruit_15 [003] - Phalanx              : 3.5/8 3-4-1 (=0101001)  44%   -42
-----------------Phalanx-----------------
Phalanx - Abrok_5_0                   : 3.0/8 3-5-0 (10100010)  38%   -85
Phalanx - Baron 1.5.0                 : 1.5/8 1-6-1 (10=00000)  19%  -252
Phalanx - Chezzz-1.0.3-windows        : 5.5/8 5-2-1 (=0110111)  69%  +139
Phalanx - Chop1099                    : 2.0/8 2-6-0 (00000110)  25%  -191
Phalanx - Frenzee                     : 4.5/8 4-3-1 (1=010110)  56%   +42
Phalanx - Fruit_15                    : 3.0/8 3-5-0 (00100011)  38%   -85
Phalanx - Fruit_15 [003]              : 4.5/8 4-3-1 (=1010110)  56%   +42