Latista 1.3 released

[eric_oldre]

I just released Latista version 1.3.

It is available from my website, http://www.oldre.com/chess/latista/

It beats the old version 1.2 about 75% of the time. I'm not sure what
that would translate to in Elo difference.

It features:
-a more selective search.
-An almost completely redone evaluation.
-Many bug fixes.
-Now if comes with it's own opening book (based mostly on normbk03)

I'll welcome any feedback you can provide of course!

Eric

[Alessandro Scotti]

Hi Eric,
according to this table:

http://chess.about.com/library/weekly/aa03a25.htm

a 75% difference translates to +193 elo points. However, this figure is only meaningful "in absolute" if you have played many games against a variety of opponents.

[eric_oldre]

Alessandro,
Thanks for the link, And the advice about broadening the number of different engines I test against.

Recently what I had been doing was running against v1.2 and Janwillem, and running each of the Nunn2 positions. The time control i used was 8+2 fischer time. v 1.3 Scored 59/80 on this test.

I chose Janwillem because it was pretty much exactly the same strength as v1.2.


A couple nights ago I ran a test against kiwi .4d. That build of latista scored 8/20. Much too small to get a good idea. But at least it's something.

Going forward, I'm taking positions 1/9/13/14/19 from the Nunn2 tests. I choose these because they seemed at frist glance to give me a decent variety of opening. And Running a guantlet at time controls of 7+2 on an AMD-64 3200+. It should take about 24 hours to finish. The opponents I'll use are:

Adam_29
Aice 0.92
BigLion 2.20
Kiwi .4d
NagaSkaki (not sure of the version)
NanoSzachy (not sure of the version)
Natwarlal 0.12
Zues (not sure of the version)

This will give me 80 total games, Against a bigger variety of stronger opponents.

I'll let everyone know when the test completes.

[Pallav Nawani]

Hi,

I think it would be good if some of the testers could give us advice on which opponents to use for tests. I recently ran two tests Natwarlal 0.12 + a bugfix against a bunch of engines. The results were strange.

In test 1, Fortress scored high against Natwarlal, which is strange because even Natwarlal 0.09 used to have an upper hand against Fortress. This version absolutely murdered Beowulf, and scored badly against Phalanx, Fruit 1.0, and Fruit 1.5

Next, I added Tord's BFP (Buggy futility pruning) to Natwarlal. Again I ran a test. This time Beowulf scored high, while Phalanx was down. Fortress, Fruit 1.0 and Fruit 1.5 did somewhat worse than they did before. Overall it seemed an improvement, even though BFP version crashed and lost 2 games.

Currently I am now rethinking which opponents to use for my next tests

Pallav

[Alessandro Scotti]

Hi Pallav,
did the two versions play identical openings in both tests? I almost always test using the Noomen positions and after some hundred games against Yace I found that for example Kiwi scored most points in positions 1-22 (King openings), and did *much* worse in the rest (Queen and other openings). Although I don't know why, at least I now have an option of forcing 1. e4 in tournaments!
Also, I like to play against engines that have some good chess knowledge, as opposed to engines that, say, outsearch Kiwi and kill him tactically. This helps me find me what's missing in my evaluation, so I can add it to my growing "todo" list...

[Ralf Schäfer]

Hi Eric, Alessandro and Pallav,

when choosing the opponents for testing, I try to find some engines which are actually a bit stronger than Spike - but not too strong . For example good test candidates are engines which score 52% - 58% against my best version. Another requirement is, that the test candidates should be of course very stable, that means no looses on time or hangs during/after the game at all.

About starting positions: I use the 20 Nunn and the first 30 positions from the Noomen set, 2 games (w/b) for each position per opponent.

The timecontrol here is 2+2 on a fast computer (Centrino 2.0), 2+3 on the slower (P4 2.53).

Ah, and take care that learning is switched off, else your testgames could get very annoying Also the writing of logfiles should be switched off if possible.

best wishes
Ralf

[Pallav Nawani]

Hi Pallav,
did the two versions play identical openings in both tests? I almost always test using the Noomen positions and after some hundred games against Yace I found that for example Kiwi scored most points in positions 1-22.

No, I used Arena + Wizzard book to run those tests. I thought that since Arena would use the same opening book for all engines, things would even out. In retrospect, for such a limited number of games such as mine, I should perhaps use Nunn/Noomen positions. However, I am slighly apprehensive that using Noomen/Nunn positions for testing/benchmarking would tend to make Natwarlal a bit 'tuned' towards those positions, because I would be testing the improvement in a fixed set of positions.

Also, I like to play against engines that have some good chess knowledge, as opposed to engines that, say, outsearch Kiwi and kill him tactically. This helps me find me what's missing in my evaluation, so I can add it to my growing "todo" list...

Yes, this is a good Idea. I will also do this from now on. Ralf's idea about choosing slightly stronger opponents also makes sense.

[Tord Romstad]

Hi all,

Some comments:

I agree, it is definitely necessary to test against more than one opponent (especially if this single opponent is an older version of your own engine). In my opinion it is best to have opponents of a wide variety of strengths and styles. I don't like to only test against stronger engines; it has happened before that changes I have made improve my engine's results against very strong engines, but makes it play much worse against clearly weaker opponents.

My main criterions for choice of sparring partners are stability and the availability of Linux or Mac OS X versions (I don't have any Windows computers). The strength should be within 300-400 Elo points of Glaurung. At the moment, my list of regular sparring partners consist of Diablo, Fruit, Gothmog, Hiarcs, Kiwi, Pepito, Phalanx, Yace, and one top amateur engine which I won't reveal, because the author probably doesn't want everybody to know that I have his source code and has ported his program to Linux and OS X.

I prefer to run test matches without book from a fixed set of opening positions, in order to improve reproducability. I think it is dangerous to use only 20 positions, though. The risk of tuning for these particular positions is very real, in my experience. I therefore prefer the Noomen positions over the Nunn positions.

Recently I have also considered collecting opening positions based on themes, in order to better identify my engine's strengths and weaknesses and to make it easier to test changes which are only relevant in certain classes of positions. For instance, when I wrote the new and improved pawn storm code in Glaurung 0.2.4, I made a collection of 10 different opening positions with castling to opposite sides, and tested Glaurung from these positions with and without the new code against several opponents.

Tord

[Guenther Simon]

[Alessandro:]

I found that for example Kiwi scored most points in positions 1-22 (King openings), and did *much* worse in the rest (Queen and other openings). Although I don't know why, at least I now have an option of forcing 1. e4 in tournaments! Smile
Also, I like to play against engines that have some good chess knowledge, as opposed to engines that, say, outsearch Kiwi and kill him tactically. This helps me find me what's missing in my evaluation, so I can add it to my growing "todo" list... Very Happy

Hi Alessandro,

I noticed that king safety is a very big issue in Kiwi and it might well be
that king safety is much more important in 1.e4 openings (more open,
more tactically...).
May be you have seen that I have added latest Kiwi to my new blitz ranking list? (40 games matches, 40/5 tc)
Here I have seen games, when Kiwi still gave around -0.20/30 for
a few moves, when opponents already climbed up to +2/3 sometimes.


You'll notice also that learning is on and that it has some influence,
how deterministic a program plays, if it does not learn.
Interesting though, most part doubles are played between two
non learners. (Gosu, Petir and Thinker do most of them IIRC)
I would welcome an option for non learning programs to add some
randomness either to evaluation or time management for tournaments
with longer matches.
I have thought a bit about this and came to the conclusion that adding
some small rand value to the first 5-10 plies after book end for
choosing moves or time for those moves would be sufficient to avoid
most of the part doubles.
E.g. choose randomly between the above mentioned plies for a window
of perhaps 0.20, if eval is better than a given value and/or randomly
multiply time per move for above plies by 1.1 to 1.x.
Just a few ideas without changing book choice/randomness, which is
of course an option too.

Best regards,
Guenther

[Tord Romstad]

I would welcome an option for non learning programs to add some randomness either to evaluation or time management for tournaments with longer matches.

Good idea. I will try implement some form of randomness (optional, of course) in the next Glaurung version. An easy way to do this for me would be to add a random number from the set {-1, 0, 1} to each entry of my piece square tables whenever my engine receives the "ucinewgame" command. This should be sufficient to avoid most doubles without hurting playing strength at all.

Tord

[Guenther Simon]

Hi Alessandro,

I noticed that king safety is a very big issue in Kiwi and it might well be
that king safety is much more important in 1.e4 openings (more open,
more tactically...).
May be you have seen that I have added latest Kiwi to my new blitz ranking list? (40 games matches, 40/5 tc)
Here I have seen games, when Kiwi still gave around -0.20/30 for
a few moves, when opponents already climbed up to +2/3 sometimes.

Uh, I completely reversed your results, as you said your results
were much better _with_ 1.e4 openings, thus my theory about
importance of king safety there seems to be nonsense.
Still Kiwi has a problem with king safety

Guenther

[Alessandro Scotti]

Uh, I completely reversed your results, as you said your results
were much better _with_ 1.e4 openings, thus my theory about
importance of king safety there seems to be nonsense.
Still Kiwi has a problem with king safety

LOL... no no, I think you have been right from the beginning! Kiwi does not currently evaluate king safety, except for a rough guess at how good the pawn shelter can be. It's also missing a lot of important stuff such as piece mobility, pawn storms, passed pawns, patterns and so on... it seems that in open positions such missing knowledge hurts somewhat less, isn't it true that beginners should only aim at open games?
I'm now going to rewrite Kiwi's evaluation from scratch and slowly add these features in... it will take some time so for the moment Fabien can sleep tight.
Your suggestion about adding variety is very interesting... I've played 2*100 test games from the Noomen positions against Yace and in the second set Kiwi's score dropped by a good 40% because of learning. I'll try to follow Tord's advice to change the P/S table a little and repeat the above esperiment...
It's great to have Kiwi in the blitz ranking list, thanks! I hope next version improves a little though, so I don't have to scroll the page down all the way to find it!

[eric_oldre]

As promised, here are my initial test results with the released latista 1.3.

Engine Score La
01: Latista 1.3 45.5/100 ??????????
02: Zeus 7.5/10 1111=10101
03: Natwarlal 7.0/10 1==1=1110=
03: Terra33 7.0/10 0111=1110=
05: Adam_29 6.5/10 =1=11101=0
06: Kiwi 6.0/10 1=1101=001
07: Aice 5.5/10 1010011=01
08: NagaSkaki 5.0/10 1===10100=
09: NanoSzachy 4.0/10 000=110=01
09: BigLion 4.0/10 00011=01=0
11: Janwillem 2.0/10 0=000=00==

100 games played / Tournament finished
Name of the tournament: Guantlet8 (latista 1.3)
Site/ Country: ALILAR, United States
Level: Blitz 7/2
Hardware: AMD Athlon(tm) 64 Processor 3200+ 2002 MHz
Operating system: Microsoft Windows XP Build 2600 Service Pack 1
PGN-File: C:\Program Files\Arena\Tournaments\Guantlet8 (latista 1.3).pgn
Website:
E-Mail Address:

[Vadim Bykov]

Hi Eric! Congratiluations with new version!

>Engine Score La
>01: Latista 1.3 45.5/100 ??????????
>02: Zeus 7.5/10 1111=10101


What version of Zeus have been played ? You can simply run .exe file and it typed his version

>Level: Blitz 7/2

7 moves for 2 minutes? Am I right?

[eric_oldre]

Hi Eric! Congratiluations with new version!

>Engine Score La
>01: Latista 1.3 45.5/100 ??????????
>02: Zeus 7.5/10 1111=10101


What version of Zeus have been played ? You can simply run .exe file and it typed his version

>Level: Blitz 7/2

7 moves for 2 minutes? Am I right?

The version of zues is 1.14.
7/2 is fischer time. 7 minutes + 2 seconds per move