Will Smirf's Source be published?

[Reinhard Scharnagl]

Hello forum members,

during the last weeks there have been some requests for me to allow some views into Smirf's source code. Well, I understand such wishes but cannot follow them. First Smirf has to be finished and get adult within some tournaments before I can decide to publish the whole program source or some parts or nothing at all.

There have been the discussions on Crafty clons and other cases of code "recycling" which have convinced me, that it would not be good to allow detailed views into foreign source codes. Neither the improvement of chess algorithms would profit from published code, nor the respect of really creative programers. On the other side I am developing Smirf without any "recycling" of borrowed code snippets.

But understanding the interest to know something more detailed on my Smirf project I will publish some information on it now. Smirf is a chess program based on a 14x15 data structure, thus able to support traditional 8x8 boards and the 10x8 capablanca board with its extended piece set (including chancellor and archbishop) within ONE engine, simply switchable by being fed with appropriate FEN strings. Smirf does not use bitboards or something comparable, but it uses a new piece type of bit encoded properties arranged in two double linked and sorted piece lists. Smirf owns a very fast move generator, which produces only legal moves fully informed whether they are captures or check threats (including the direction of the threat, double checking, or mating). Thus the generated moves have very good move presorting characteristics, based even by additionally being overlayed by positional changing weights and MVV/LVA .

Smirf (still) does not see any need for using something like attack tables, and Smirf does not see the need of precise move evaluating to perform selecting decisions. Therefore it might have problems to supply the traditional calculating information viewed by Winboard or UCI GUIs during the engine's thinking process.

I am nevertheless willing to supply interested programers with testing results matching their own data as far as this might be possible facing the very different approaches. For those persons I have created a Smirf beta page: http://www.chessbox.de/beta.html where already some testing results have been published. You will find there moreover an actual version of my still developing Smirf GUI, which is necessary since there are no GUIs yet to support 8x8 and 10x8 chess simultaneously, which my approach does.

To be able to do this, I have invented a new protocol TMCI, which is still changing, because the triple set of Engine, GUI and protocol still are under development.

I hope that people would not be too angry with my closed shop behaviour, but I am willing to answer questions on Smirf, though I am not (yet) willing to make Smirf an open source project.

Regards, Reinhard.

[Dan Honeycutt]

There have been the discussions on Crafty clons and other cases of code "recycling" which have convinced me, that it would not be good to allow detailed views into foreign source codes.

Hi Reinhard

The decision to publish the source is entirely yours. However, I generally don't buy the arguement that open source invites clones. If someone wants to clone an engine there are plenty of strong candidates - Crafty, Phalanx, Arasan, Pepito and Fruit come to mind. Smirf may be a little different though as it supports Capablanca, those I listed do not.

I don't understand the clone mentality. For me, the fun is in screwing with the thing, not how many games it wins. Obviously there are others who feel different.

Dan H.

[Reinhard Scharnagl]

Hi Dan,

The decision to publish the source is entirely yours.

well I hope so.

However, I generally don't buy the arguement that open source invites clones.

In fact OpenSource in my eyes only would make sense, when there would be a customizing process, where software workers could earn some money. But there is nothing to customize in a chess engine, where you would need the help of a programer. Claiming for Open Source chess programs here therefore equals demanding for freeware or cloneware. I (still) cannot see any rational reason for to publish chess sources (may be to show one's huge ego). It might be another idea when some programers would exchange their experiences among a group of active contributors, but for which goal the source should become totally public?

It is a very different thing to write about strategies, heuristics or general ideas. But showing implementations of such creative matter seems to be counter productive.

I don't understand the clone mentality.

Me too. But beside of real cloning, I just have been very frustrated in getting information, why some programers are using e.g. attack tables - it would be of course interesting to know. But I have heard only patchworking arguments like: I have tried it and it was successful. I have not yet heard real arguments, why using attack tables would be better than alternative XY or Z. Similiar would hold for BitBoards I think, where I believe that most people would not use it, if they have not seen into some published code lines. I do use neither. But this does not at all mean that my way would be best. I am of course willing to learn. But I want to get more experienced by analysing arguments of other persons not by simply testing foreign code snippets.

I think that publishing source code would invite for "patchworking" or code "recycling". So I think sources should be shared only with very few selected persons, who would trustably keep the code itself secret.

Reinhard.

[Dan Honeycutt]

Hi Reinhard:

Bruja's source is available in the hopes it will help someone else. If it helps one person and three others misuse it, I'm OK with that.

I hear what you say about recycling and so on. We just disagree. For an amateur engine I see no reason not to make the source available. If someone has designs on producing a commercial engine, that's a whole different ballgame.

Dan H.

[Dan Honeycutt]

Hi again Hi Reinhard:

To your other comment:

I just have been very frustrated in getting information, why some programers are using e.g. attack tables - it would be of course interesting to know.

If you want to know exactly what Bruja does, I'll send you the source. If you want to know "why" that's more difficult. I can tell you that I've tried this, and I've tried that and what I have seems to be the best of what I've tried. But I don't know why.

Dan H.

[Reinhard Scharnagl]

Hi Dan,

Bruja's source is available in the hopes it will help someone else

that sounds very honourable. But indeed it seems that we are differing just here. I am convinced that discussing some detail problems with Bruja's author (you) would for sure be helpful to all involved. That also would hold for writing texts on your ideas and why you would prefer to do things the way you do and not another way. Even giving away source code to a well known person mostly would be ok. But allowing everybody to anonymously copy one's know how?

If someone has designs on producing a commercial engine, that's a whole different ballgame

The difference here is just the pricing. And because zero costs are very cheap, the influence even may be greater.

But it is not the first time, that I have to notice philosophies similar to yours on distributing source code for free. So you are not alone with your point of view, it rather would be me.

Regards, Reinhard.

[Reinhard Scharnagl]

Hi Dan, I see you have written a second time.

If you want to know exactly what Bruja does, I'll send you the source. If you want to know "why" that's more difficult. I can tell you that I've tried this, and I've tried that and what I have seems to be the best of what I've tried. But I don't know why.

Well, my idea on discussing such problems would be: which problem you have detected? how did you try to solve it by yourself? do you look for other solutions and where? why have you stopped finding an own solution? how have you continued in optimizing your own solution?

That would give a discussion on how chess programing could be directed forward. A may be weak on solution for a problem helps me much more to become familiar to chess programing ideas, than to reproduce may be well working solutions of foreign sources.

The questions are not targeting especially you but every chess programer. But it seems as if such discussions are not attractive to others. Nevertheless it could be, that because of English is not my native language, I missunderstand some relevant parts and fail in keeping a discussion interesting.

Reinhard.

[José Carlos]

There're some things you can't reason much about, and testing is the only way. I started off with all original. My search, my evaluation, my book, my hashing... I had a function that scanned the whole board and created a hash key of 128 bits which was unique for every position! Then I read about Zobrist, I asked how to implement it, I tried and... it worked! That's all that matters IMO. I got pretty much the same tree with much less memory and time. There's no "why", it's simply better.
I used char[64] as a board. Then read about 0x88, which has the potential to be faster. I tried, tuned and it worked. I kept it.
I read a lot of ideas for evaluation. Why evaluating squares around the king is a good estimate for dynamic king safety? Well, I'm not sure. I test and see if it works. That's all.
If you read Ed Shroeder web page about REBEL, you'll see how he gets to the top of computer chess: idea->test, test, test->accept/discard
I'm sorry you're frustrated by the answers you got about attack tables. I think you got some interesting answers and many things to think about.

[Dan Honeycutt]

Hi Reinhard:

This time I'll try to get everything said in one post.

First of all, your English is excellent. I've been a member of Spanish language forums and I post here in Spanish at times. I'm well aware of the difficulty in using a language that is not your first. You will never see me criticize the English of a non-native speaker. But I will try to help if it appears there has been a misunderstanding. When you quoted my statement "The decision to publish the source is entirely yours." and answered "well I hope so" I got the impression you may not have understood. In English, stating the obvious is a way of saying "I don't agree with you - but I do respect your opinion".

But indeed it seems that we are differing just here. I am convinced that discussing some detail problems with Bruja's author (you) would for sure be helpful to all involved.

We differ, but that's OK. Sharing code is a help to someone just beginning (at least it was to me) but once a program has reached basic functionality, the code of others is pretty useless. Much better, as you say, is the discussion of ideas, problems and solutions.

Well, my idea on discussing such problems would be: which problem you have detected? how did you try to solve it by yourself? do you look for other solutions and where? why have you stopped finding an own solution? how have you continued in optimizing your own solution?

For me, it starts with an idea. Sometimes I have an original idea, but usually it's triggered by something I read here or in CCC. I may post what I'm considering to get additional input. Then I code. Then I test.

The last step is by far the most difficult. Bruja 1.7 and the current 1.8 are about 200 ELO better than 1.0 to 1.6. Reason is simple. I found a bug in my qsearch where I threw out the good captures and kept the bad ones (hard to believe anyone could be that stupid but it's true). I fixed that and the improvement was immediate and obvious. I wish I could find another 200 ELO bug, but I doubt I will. I have lots of ideas, the hard part is the testing to see which ones are good and which are bad.

But it seems as if such discussions are not attractive to others.

They are to me.

I followed your thread on attack tables. It seemed you were looking for some "absolute truth". I sensed some frustration on your part with the replys but I couldn't add anything to what others posted. There are many ways to "skin the cat" as we say in English and attack tables are just one of them.

Good luck with Smirf. When do you anticipate its first tournament appearance?

Dan H.

[Dan Honeycutt]

Hola Jos?.

Estaba escribiendo cuando pusiste tu contestaci?n. Estamos de acuerdo.

Saludos
Dan H.

[Reinhard Scharnagl]

Hi Jos? and Dan,

I followed your thread on attack tables. It seemed you were looking for some "absolute truth".

as Dan is pointing to - indeed I want to see somehow behind the things. But this is not easy and rarely ending with success.

Like Jos? I worked a lot on data structures, because I realized that there had been big consequences in move generating. I tested some real exotic ideas like e.g. having two boards and pieces distinct only for one color and a unic enemy piece type for the rest. That lead to absolutly symmetric procedures for black and white and gave a double performance. 0x88 structures had been used by me at the times of a Sharp MZ80K using a finally overtuned 2MHz Z80 CPU. As I already have written today I am using bit encoded properties on double linked and sorted piece lists, what for me is working well. I use a self developed five fold associative transposition table with a sophisticated replacement scheme. For that I am xoring a special subtype of random numbers from a self written 32 bit pseudo random number generator building a 32 bit verification key and a (partially used) 32 bit storing address. All this has been done without "recycling" foreign code. When I tried to discuss the theory of using hash codes built from random numbers with equal distributed 16 bits because of a more stable bit switching behaviour, there has been no real interest in newsgroups to hear from that. And this is valid for some of my approaches I developed from my own, not knowing whether there would be parallel inventions published or not.

I have started with a sequential move generator, which gives me a very small memory consumption. But memory is cheap today. So I changed to a generator which creates all valid moves with full information at once. Last year I have been motivated to release a mating engine SmirfMate as a beta version to test my ability to support the UCI protocol. And ignoring the fact that it has not been designed as a special mating engine, it was working fine, but not completely without errors.

Well, after a heavy accident then I decided to write a book on FRC (in German language) to get more of my self confidence back. As you will know neither UCI nor Winboard do support FRC really well, especially in Winboard there are contradicting requests how castlings should be encoded. Finally I understood that there will be no GUI to support FRC within an appropriate protocol. Even Arena, which does a fine job here, is walking besides the rules in neutral territory.

At last I decided to write a program, which is supporting 8x8 and 10x8 capablanca board and piece set, but knowing, that I have to build my own GUI and use my own protocol. That is a very complex job to do, and may be, only a small amount of people will be interested in it.

I hope to put all my parts together and have it play in some days. But the search function first will be it's weakest component. There will be a Gothic Chess event in November I intend to participate. Though I do not regard the idea of patenting variants to be valid or good, I think its a chance to start comparing Smirf to other engines. For to have a more neutral playground I "invented" the Capablanca Random Chess variant, which is free (in my point of view) as my TMCI protocol will also be.

Reinhard.

[José Carlos]

When I tried to discuss the theory of using hash codes built from random numbers with equal distributed 16 bits because of a more stable bit switching behaviour, there has been no real interest in newsgroups to hear from that. And this is valid for some of my approaches I developed from my own, not knowing whether there would be parallel inventions published or not.

I can't speak for others, but I think everyone of us have favorite parts of our chess programs to work on. Personally, I'm not much interested in how to improve performance at very low level, and I enjoy thinking about eval terms or pruning ideas. I know I could get a 5% faster here and there, but I have little time, very little, and I have to choose what to work on.
So my apologies for not being able to talk to you at that level, it's beyond me (like Gerd's assembly in CCC).
But if you want to discuss about high level topics, you can count on me.

[Dan Honeycutt]

Hi Reinhard:

I tested some real exotic ideas

That thins down the participants in a discussion. Exotic enough and it may thin down to you, Gerd and Vincent. But that doesn't mean others aren't interested. We need new techniques. I wish I could say I had something exotic or unique in Bruja.

I am using bit encoded properties on double linked and sorted piece lists, what for me is working well. I use a self developed five fold associative transposition table with a sophisticated replacement scheme.

You lost me here. The linked list is your board representation and then you move on to transition table or do they go together?

So I changed to a generator which creates all valid moves with full information at once.

I started with informed moves. I had if the move gave check by attack (a plus), check by discovery (a big plus), location improvement (by PSTs), hanging on the from square (good idea to move), hanging on the to square (bad idea to move) and the like. I was pretty pleased with it. Early Bruja could actually play a plausible game of chess with no search. But it was too slow. Bruja runs about 60 percent quiescient nodes and ideas I'm considering may push that number higher. I had way more information than I needed for the qsearch.

Now Bruja's moves are on a "need to know" basis. They get informed when I'm ready to sort 'em or search 'em.

Best
Dan H.

[Reinhard Scharnagl]

Hi Jos? and Dan,

if I understood Jos? correctly, he is working on high level improvement technics like make pruning more effective. This theme will become very intersting for me during the next time. I intent to have a search tree formed by something like a Shannon B approach but not with predefined criteria. This is where SMIRF has its name from: "Strategic Game Program with intelligent recursing Facilities". Most of contemporary approaches are Shannon A with alpha/beta. There might be room for interesting discussions.

When I understood Dan right, he might be interested also in my "exotic" internals. The situation is, that I am just about to detect that my approach is somehow exotic, when trying to discuss some questions and finding out, that most of chess programers have solved their problems in very similiar ways - may be with the exception of using BitBoards or not.

To his question: my transposition table and my data structure are completely independent one from the other. They are logically joined by associating a 2x32 bit word (build from a special subtype of random numbers, probably quite uncommon) to each combination of piece type and board koordinate (which seems to be a common approach) to build an address key and verification mask to store information on positions and to verify fetched data to be related to the correct position (which last part obviously seems to be ommitted in other TT approaches).

My data structure is based on a 14x15 matrix of encoded pieces (therefore not at all related to bitboards). The pieces are simultaneously member of a double linked and sorted color specific list, where separated properties of the chesspieces are bit encoded. This facility enables Smirf now to also support piece types as archbishop or chancellor of Capablanca's piece set.

This structure is the result of more then 20 years of thoughts concerning the problem of most efficiently storing and using chess positions. That has lead me to some very exotic approaches, which partially gained huge efficiency and partially not. The most exotic one has been, to use only less piece types PNBRQK and E (Enemy) for the active color und use two parallel boards for each view. The fine effect was, that algorithms were completely identic for black or white, and it has been a big performance step that time. That approach lasted for some years. But my actual structure has been successful during the last time and is not that exotic from my point of view.

Regards, Reinhard.