Buggy engines and building a bug base

[Guenther Simon]

I am still in the process of adding games to the bottom of my rating list and therefore also
test reasonably stable programs in the lowest range of strength.

Some experiences:

Code: Select all

- Felpo_1001b always crashes here during the game
- Cheoss_0649JA always loses on time here and never manages to reach the cut at move 40,
  except the game finishes before
- PulChess_021rev70 always gets out of sync in move 2 playing as Black
- all ChessAlex versions don't unlock the CPU on my MP system in ponder off

- MicroChess1976 always plays immediately
- ZZZZZZ_349JA still loses on time in dead lost positions when it seems it wants to resign

I will build a bug 'database' (even for older versions!) soon. In case of interest I will add it to my site and give
write permission to other TDs which use WB(and may be Chess GUI too, there is no sense to care about pseudo
bugs which later appear as GUI bugs...). Probably I will upload a first draft of an empty sheet(but with an example)
this weekend.

Guenther

[Olivier Deville]

Hi Guenther !

A very interesting project. I have noticed the same about all engines you mentioned above, except ChessAlex (no such issue here).

And what about Crux ?

Olivier

[Guenther Simon]

Hi Guenther !

A very interesting project. I have noticed the same about all engines you mentioned above, except ChessAlex (no such issue here).

And what about Crux ?

Olivier

Hi Olivier,

You sure remember that we talked already several times in the past about a bug collection base,
but well, we never really started it
Now it is time to do sth about it! I will release a first draft with an example tonight.
I would like to invite known TDs with experience to this project. My current plan is to
make two copies of the file available: an open one for all for reading and a restricted one
for writing and with password protection, which I would mail after request.
A little problem though is that I plan to do it in ods format(which is a free excel equivalent in OpenOffice,
but better and more compact), as I don't use MS Office here.
Of course there would still be the possibility to simply send me mailswith infos and I would enter them myself.

for Crux: that's old news of course, as for Trynyty and dozens of other bugs in dozens of other
programs/versions
You know I always wondered why you did not cancel at least those who have a too high failure rate,
but AFAIK you always told me you wouldn't like to miss _any_ of them.

for ChessAlex: I only noticed this since I have my Quad it doesn't lock a CPU on a single CPU
machine, but note that this is in ponder off and IIRC you do ponder on now?
(It would still be nice if you could perhaps verify this behaviour with PO on one of your multi CPU
machines?)

open draft version available now:
http://rwbc-chess.de/Bugbase/Open/Bugbase.ods

suggestions are welcome

Guenther

[Laurens Winkelhagen]

Nice project Guenter:-)

[Olivier Deville]

Hi Guenther

No problem at all with OpenOffice, I don't use MS Office suite either.

I was not able to reproduce the hanging bug of ChessAlex here with ponder off (Dell Core2Duo).

I'll start testing for the Promo group soon, many bug reports to come

Olivier

[Kirill Kryukov]

Hi Guenther,

Very interesting and needed project. Since recently I also document all the engine bugs and issues I encounter. I post them on engine info pages at my KCEC tourney (see signature for the link). I mark problematic engines with "!" mark in the rating list, in the protocol column (around "wb" and "uci").

Since my tourney is run through UCI interface (and WB2UCI adapter), it's possible that not all of my documented bugs exist under Winboard, but still I believe many of them are relevant under Winboard too.

[Chris Tatham]

Thanks Guenther this is a great idea!

I've played a lot of games with weaker engines in developing and testing my own and gathered some relevant information I can pass on. Is it possible to save the database in an alternative format also (eg .txt) to save installing any additional software as I use MS Excel instead of OpenOffice and can't seem to open it?

Hi Kirill, I hadn't seen KCEC page before but presentation is excellent - nice work!

[Roger Brown]

[SNIP]


A little problem though is that I plan to do it in ods format(which is a free excel equivalent in OpenOffice,
but better and more compact), as I don't use MS Office here.

Guenther

Hello Guenther,

This should not be a problem as OpenOffice was developed to interact with us poor MS Office product users. You can save the sheet as Excel which can be used by those of us who have not yet made the switch.



Of course, maybe it would be simpler to output it as text which was suggested in this thread as well.

Thanks for the willingness to do this free work for the rest of us.

Later.

[Guenther Simon]

[SNIP]


A little problem though is that I plan to do it in ods format(which is a free excel equivalent in OpenOffice,
but better and more compact), as I don't use MS Office here.

Guenther

Hello Guenther,

This should not be a problem as OpenOffice was developed to interact with us poor MS Office product users. You can save the sheet as Excel which can be used by those of us who have not yet made the switch.



Of course, maybe it would be simpler to output it as text which was suggested in this thread as well.

Thanks for the willingness to do this free work for the rest of us.

Later.

Hi Roger,

Yes I know I can export OO formats to excel, but this way I cannot save write protection for sheets.
OTH you and Chris are right that it makes not much sense to use the OO format for the read version at least!
I guess I will change the read version to a cleaned up htm table as in my chronology pages and use csv for
transfers or I other htm and excel export to allow the user sorting himself after he had downloaded
the excel version. Hopefully there will be such a version available tonight and it will also contain
already quite some content then!

Best regards,
Guenther

[Guenther Simon]

I have something new out now for reading only in htm format. Don't expect too much yet, because it is a lot
of work to fill all informations in. So far I just adapted my clean table layout from the chronology pages:

http://rwbc-chess.de/Bugbase/Open/Bugbase.htm

Guenther

[Guenther Simon]

Nice project Guenter:-)

Thanks Laurens, hopefully I find the time for updating my tournament pages too
(I had quite some games played on my quad during the last weeks...)

Guenther

[Guenther Simon]

Hi Guenther

No problem at all with OpenOffice, I don't use MS Office suite either.

I was not able to reproduce the hanging bug of ChessAlex here with ponder off (Dell Core2Duo).

I'll start testing for the Promo group soon, many bug reports to come

Olivier

Hi Olivier,

May be there was a missunderstanding, but I would like to know how exactly you tested it
and which version? Note that it seems it won't happen against Humans. I noticed in 40/15 games,
in the task manager that it always used 25% of my CPUs(= 1CPU) despite being ponder off.
Actually it was not really pondering because it was not thinking or sending any debug output,
but it seems it did some polling stuff and only on my quad not on my single cpu machine.

Guenther

[Guenther Simon]

Thanks Guenther this is a great idea!

I've played a lot of games with weaker engines in developing and testing my own and gathered some relevant information I can pass on. Is it possible to save the database in an alternative format also (eg .txt) to save installing any additional software as I use MS Excel instead of OpenOffice and can't seem to open it?

Hi Kirill, I hadn't seen KCEC page before but presentation is excellent - nice work!

Hi Chris and Kirill,

When I have filled in some of my gathered material I would be happy to look at any other informations
(Note that I prefer WB because it is not only extremely stable, but also because of debugging, which
is important for verification of real bugs. I guess ChessGUI should be good enough too and if another GUI
provides enough serious debugging infos I would probably take those too.)
For the format, you and Olivier are right and I changed it already to the same lightweight htm(and csv later)
as my chronology pages.

Guenther

[Miguel A. Ballicora]

I have something new out now for reading only in htm format. Don't expect too much yet, because it is a lot
of work to fill all informations in. So far I just adapted my clean table layout from the chronology pages:

http://rwbc-chess.de/Bugbase/Open/Bugbase.htm

Guenther

This is a fantastic project. It will help testers and authors to centralize this information. I am glad not to see Gaviota there for now

By the way, I was reported that Gaviota failed to accept certain SAN moves from Arena. The problem was that Arena did not send proper SAN moves (it sent Bb5 rather than Bb5+, for instance). I hope this was fixed in Arena in the last release 2.0. Anybody knows if it was? Otherwise I should make Gaviota to accept this buggy behavior from the GUI, but I would prefer not to do it.

Miguel

[Kirill Kryukov]

Hi Chris,

Hi Kirill, I hadn't seen KCEC page before but presentation is excellent - nice work!

Thanks! Looking forward to adding your engine!

[Kirill Kryukov]

Hi Guenther,

Hi Chris and Kirill,

When I have filled in some of my gathered material I would be happy to look at any other informations
(Note that I prefer WB because it is not only extremely stable, but also because of debugging, which
is important for verification of real bugs. I guess ChessGUI should be good enough too and if another GUI
provides enough serious debugging infos I would probably take those too.)

I run Winboard engines through WB2UCI adapter, which can log all communication (both engine <-> adapter and adapter <-> interface). Also any text-based protocol (like WB or UCI) can be debugged with InBetween. However for the most of the bugs I observe I just record the effect of the bug, when it is apparent, without checking the protocol log.

For the format, you and Olivier are right and I changed it already to the same lightweight htm(and csv later)
as my chronology pages.

Guenther

Looks terrific. Can you also please add sorting by engine name?

Best,
Kirill

[Matthias Gemuh]

This is a fantastic project. It will help testers and authors to centralize this information. I am glad not to see Gaviota there for now

By the way, I was reported that Gaviota failed to accept certain SAN moves from Arena. The problem was that Arena did not send proper SAN moves (it sent Bb5 rather than Bb5+, for instance). I hope this was fixed in Arena in the last release 2.0. Anybody knows if it was? Otherwise I should make Gaviota to accept this buggy behavior from the GUI, but I would prefer not to do it.

Miguel

Hi Miguel,
Gaviota somtimes sends fake SAN moves (I don't mean the "+","#" suffix) !!
I stopped using Gaviota in ChessGUI for that reason. I will test again to get an example for you.
regards,
Matthias.

[Matthias Gemuh]

This is a fantastic project. It will help testers and authors to centralize this information. I am glad not to see Gaviota there for now

By the way, I was reported that Gaviota failed to accept certain SAN moves from Arena. The problem was that Arena did not send proper SAN moves (it sent Bb5 rather than Bb5+, for instance). I hope this was fixed in Arena in the last release 2.0. Anybody knows if it was? Otherwise I should make Gaviota to accept this buggy behavior from the GUI, but I would prefer not to do it.

Miguel

Hi Miguel,
Gaviota somtimes sends fake SAN moves (I don't mean the "+","#" suffix) !!
I stopped using Gaviota in ChessGUI for that reason. I will test again to get an example for you.
regards,
Matthias.

Sorry ! Current ChessGUI does not append "+","#" to any SAN move anymore, so Gaviota will not run under ChessGUI.

Matthias.

[H.G.Muller]

By the way, I was reported that Gaviota failed to accept certain SAN moves from Arena. The problem was that Arena did not send proper SAN moves (it sent Bb5 rather than Bb5+, for instance). I hope this was fixed in Arena in the last release 2.0. Anybody knows if it was? Otherwise I should make Gaviota to accept this buggy behavior from the GUI, but I would prefer not to do it.

It is always a bad idea to make your engine produce pedantic error messages, like refusng input that in principle has all information it needs, based on a technicality (like +# in SAN, or a missing move count in a FEN). It can serve no other purpose than to cause failure.

[Olivier Deville]

Hi Guenther

No problem at all with OpenOffice, I don't use MS Office suite either.

I was not able to reproduce the hanging bug of ChessAlex here with ponder off (Dell Core2Duo).

I'll start testing for the Promo group soon, many bug reports to come

Olivier

Hi Olivier,

May be there was a missunderstanding, but I would like to know how exactly you tested it
and which version? Note that it seems it won't happen against Humans. I noticed in 40/15 games,
in the task manager that it always used 25% of my CPUs(= 1CPU) despite being ponder off.
Actually it was not really pondering because it was not thinking or sending any debug output,
but it seems it did some polling stuff and only on my quad not on my single cpu machine.

Guenther

Hi Guenther

I ran a few games at 40/5 vs other engines with ponder off on my Dell Core2Duo. Version of ChessAlex was 2.0r4. Recent versions of Winboard_F and Fonzy's Polyglot were used. Nothing was hanging in the task manager after the test was over.

Olivier