how much open source code did you read and understand?

[Uri Blass]

I will give my estimate(it is only a fast estimate and I did not spend time in calculating by counting the number of lines that I understand

tscp 90%(I think that I understood all 1.73 but there was a newer version since then that I did not care to read)
Crafty 1%
Fruit 1%
Glaurung 2%
Olithink 2%

I decided to print all fruit's source code and I plan to try to understand it.

Uri

[Reinhard Scharnagl]

Hi Uri,

a long time ago had I taken a short look into the GNU project and then nearly immediately deleted all of it. It has made me very sure, that foreign source code would corrupt my own ideas. Accepting and using data structures not related to my own way of thinking and organizing entities of the chess world hardly could help me to develop my own thoughts and approaches. It is like being infected with a lethal virus which will conquer your whole body and would leave it without any life.

Open Source / GPL chessprogram projects may be a crutch for small-minded people, but they for sure are poisoning the creative.

Reinhard.

[Uri Blass]

Hi Reinhard,

I do not see how understanding source code can prevent me to use a different data structure and ideas that are not in the source.

possible advantages of understanding the source is to know more about programming.



Here is an example(note that knowing more about programming is not only about library functions and it may include tricks to avoid bugs or to do the same thing faster)

Movei is not using the library function
setvbuf and I do not know what it does
Fruit is using it.

Understanding fruit will force me to understand setvbuf
It is a good idea if I understand it and other library functions that are used inside chess programs(I am not interested in understanding all library functions but only functions that are related to chess engines).

At the time of writing this post I still do not understand what does util_init()of fruit doing.

void util_init() {

setvbuf(stdin,NULL,_IONBF,0);
setvbuf(stdout,NULL,_IONBF,0); // _IOLBF breaks on Windows!
}

Uri

[Reinhard Scharnagl]

Hi Uri,

well, if you are that disciplined, you might do this. If you would have a stable base of own ideas and data structures, chances are there.

But think of those many chess programming beginners, who are convinced they could learn about this theme by absorbing GPL knowledge. They are killing their chances to build something new.

Regards, Reinhard.

[Daniel Mehrmann]

Hi Reinhard,



At the time of writing this post I still do not understand what does util_init()of fruit doing.

void util_init() {

setvbuf(stdin,NULL,_IONBF,0);
setvbuf(stdout,NULL,_IONBF,0); // _IOLBF breaks on Windows!
}

Uri

Well, please read the MSDN comment/help:

http://msdn.microsoft.com/library/defau ... etvbuf.asp

MSDN network should be your first step if you don't understand windows code.

[Uri Blass]

Thanks but I still understand nothing.

The relevant parts in this case:

"If mode is _IONBF, the stream is unbuffered and size and buffer are ignored."

"_IONBF
No buffer is used, regardless of buffer or size. "

setvbuf is supposed to control stream buffering and buffer size(not that I understand what it means) but the parameter _IONBF means that size and buffer are ignored so it seems that it even cannot do it and I do not understand what is the effect of that code in fruit .

Uri

[Anonymous]

Standard C input/output functions are buffered by default. This means for example when you have a line

fprintf(somefile, "some data");

somewhere in the implementation of the fprintf() function the Standard library looks how much space is still left in the buffer for somefile. If there are still 9 bytes available, "some data" will only be attached to this buffer. When the buffer is full (or when fclose(somefile), fflush(somefile) and some other functions will be called), it will be written to disk, the buffer will be cleared, and internally the free space in the buffer will be adjusted. This has the advantage, that you will have only few disk accesses with larger chunks of data, instead of many disk accesses with small chunks of data. Typically, one does not care at all as a programmer. Everything works as expected without knowing anything about the stream buffering system used internally in the library.

stdout is typically only line buffered. This means:

printf("some stuff"); /* you see probably nothing on the screen yet */
/* long processing */
printf(" done\n"); /* Only now, things appear on the screen, because of \n */

Other stream are normally fully buffered - which means, the buffering system does not care about line ends at all. This is normally also the most efficient way to work with disk files.

When a chess engine runs under Winboard, stdout is redirected, and the internal system in the library recognizes this. It sets stdout to fully buffered then. For most purposes this would be the most efficient thing, but not for communication with Winboard, where we want, that Winboard sees the engine output immediately.

Things are very similar for input. When you call fopen and fgetc to read one character, the library functions will typically read 1 kB of data or so into a buffer, and return one char to fgetc. The next call to fgetc will need no disk access anymore, and just return a character from the internal buffer.

setvbuf is a Standard C function, to give the progammer the chance to change how the internal buffering system works, in case the defaults are not appropriate.

This all should be explained much better in any reasonable good book about C.

Regards,
Dieter

[Uri Blass]

Dieter,Thanks for the explanation.

I have fflush(stdout); inside of my function to print messgaes.

I know that fflush(stdout) is done in tscp after every printf and I found that it is good enough not to have problems with winboard.

Interesting that fruit has no fflush in the code.
Do you suggest that
setvbuf(stdout,NULL,_IONBF,0) can save the need for
fflush(stdout)?

Even if this is correct I still do not understand the need for
setvbuf(stdin,NULL,_IONBF,0);

[Anonymous]

Do you suggest that
setvbuf(stdout,NULL,_IONBF,0) can save the need for
fflush(stdout)?

Correct. Stetting to _IOLBF should even be sufficient, becaus the WB protocol is line orientated.

Even if this is correct I still do not understand the need for
setvbuf(stdin,NULL,_IONBF,0);

I use this, too. I actually only need it for Unix, not for Windows. As you know, detecting if input is available, is a bit tricky. There is no Standard C way, to do it. So we have to use system specific code. Under Windows for chess engine running under a GUI typically PeekNamedPipe is used. Under Unix we have the select() call which does something similar. But select() does not work with Standard C streams, it rather works with file handles (just integers). Instead of fopen() that returns a FILE *, there is the open() call, that returns a file handle. Things like fprintf() are not available for file handles. Also this lower level input/output system does not use any buffering.

stdin has the file handle 0 always. We can use select, to detect if the file handle of stdin has input. But now the following can happen:

Windows just sent "time 500\notim 500\n". You detect with select(), that input is available. However you use the Standard C function fgets() to read one line of input. What happens will be, when you call fgets() all input currently available (the two lines above) will be read into the internal stream buffer for stdin. You take one line out with fgets(). When you now call select again, it will return "no input available". Because that "otim 500\n" is already in the stream buffer.

When you set stdin to no buffered, there is no buffer for stdin, and "otim 500\n" would still stay available from the file handle of stdin, which select() would detect. And now everything works as expected.

It should actually work the same on Windows when using PeekNamedPipe, but for some reason it didn't for me. Therefore, in my input checking routine, I check stdin->_cnt under Windows (this is of course non portable, other environments will use other variables), which gives the number of bytes available in the stream buffer.

I am actually not sure, how unbuffered can work in general for input. Some Standard functions need to look ahead at least one byte. For example when you use fscanf(file, "%f", ...), it has to read on past the end of the number, to detect when the number ends (Assume input is "0.1enothing"). Only when you read the n of nothing, you detect that the number ended at "e". Also there is the fungetc() function.

Regards,
Dieter

[Pallav Nawani]

I will give my estimate(it is only a fast estimate and I did not spend time in calculating by counting the number of lines that I understand

tscp 90%(I think that I understood all 1.73 but there was a newer version since then that I did not care to read)
Crafty 1%
Fruit 1%
Glaurung 2%
Olithink 2%

I decided to print all fruit's source code and I plan to try to understand it.

Uri

I haven't studied Olithink at all, and I have studied only part of the code of the other engines, but whatever I studied, I had no difficulty in understanding most (Lets say 95%) of it. There were things here and there that I did not understand, but to understand them I would have to understand the data structure of the program. I did not make the effort to do so.

I remember though, that initially I had some trouble understanding crafty code.

Pallav

[Daniel Mehrmann]

I will give my estimate(it is only a fast estimate and I did not spend time in calculating by counting the number of lines that I understand

tscp 90%(I think that I understood all 1.73 but there was a newer version since then that I did not care to read)
Crafty 1%
Fruit 1%
Glaurung 2%
Olithink 2%

I decided to print all fruit's source code and I plan to try to understand it.

Uri

Well, at the beginning i readed TSCP source too and used some structs of it.
TSCP is very good to understand how the basic things in a chessprogram works ) Thanks to Tom !

After that i readed a lot of PDF documnts about different search techniques. That takes a long time.

Basicly i don't like to read oher sources because it may destroy my own ideas.

Of course at the start of my project i looked a little bit into other sources of known engines (Crafty, sjeng).

But i determined fast that this stuff doesn't usefull for me and some source parts are really horrorfull or only bad.

Since 2004 i never looked into other sources again. I did it my way
Exception Fruit, but only search_full.cpp.

Its interessting that fruit used a so agressive PVS, even after the first move, and it looks like that its works. Fruit must have a really good ording to reduce the researches.

At all i never had really problems to understand other sources. Bit operations was a problem, but now any more

[Dann Corbit]

I will give my estimate(it is only a fast estimate and I did not spend time in calculating by counting the number of lines that I understand

tscp 90%(I think that I understood all 1.73 but there was a newer version since then that I did not care to read)
Crafty 1%
Fruit 1%
Glaurung 2%
Olithink 2%

I decided to print all fruit's source code and I plan to try to understand it.

Uri

I usually understand about half when I read it the first time.
Sometimes, I do get it completely wrong what was meant.

But if I want to understand something and I have the source code, for sure I can figure it out if I put some effort into it. On the other hand, a giant machine like crafty is beyond me to grasp its entirety. I have to be happy with understanding little pieces of it.

If after a very long study and finding out what all of the terms mean you still do not understand, try tracing the execution and usually that will help.

It is very good to profile the code and then you will see what is working well and what still needs help.

If you do not understand their idea, it is literally impossible to improve it. Therefore, there is no way to overtake something unless you know how it works. So I think your approach is a good one. It will take a lot of effort to understand even one program, so be smart with your choices of what to study.

It will be even more productive for you to read C books and algorithm books, I think. Also general purpose programming sources. Here are my suggestions:
1. The C Programming Language by K&R
2. Algorithms in C by Sedgewick
3. The Art of Computer Programming by Knuth (3 volumes)

[Peter Fendrich]

There are many levels of understanding. I think that all c-programmers understand most c-code on a c-code level.

Code: Select all

for (square=A1;square <=H8; ++square) Check_if_Pinned(square);

is very easy to understand but how does it fit in the rest of the program? Why here and why done in this way?
If we raise our sight to "function level" it's a bit harder and of course when we come to class or module level it's even more difficult. Finally when we are talking about the whole system (a chess engine can be seen as a system of course) it's almost impossible to have a total understanding of some other programmers thoughts. In the last case I think it's hard to even describe the level of understanding in %-figures. Just a minor misunderstanding can completely mess up the picture. In some other case it's possible to miss the meaning of a whole module but still grasp the overall system.
When it comes to understanding on c-code level we all have 100% (or close to) understanding.
Don't take my principle of division in distinct levels to seriously... I'm only saying that there are different levels.
/Peter

[Ross Boyd]

Hi Uri,

I will give my estimate(it is only a fast estimate and I did not spend time in calculating by counting the number of lines that I understand

tscp 90%(I think that I understood all 1.73 but there was a newer version since then that I did not care to read)
Crafty 1%
Fruit 1%
Glaurung 2%
Olithink 2%

I decided to print all fruit's source code and I plan to try to understand it.

Understanding c-code gets easier the more you use it. That's obvious. C was never my primary language so it has taken ages for me to read it fluently.

TSCP was fairly easy to read because it makes very little use of pointers, and Tom did a good job of keeping it nice and simple.

Fruit is a pleasure to read. Everything is well laid out and elegant. The parts I struggle with now are the pawn eval. BTW, the early versions used a similar pawn structure to TSCP!

Maybe OT, but an extremely important tool to quickly understanding code is a really good programmer's editor....

I have been using Semware's TSEPro for 15 years now. It has the ability to load all the source files into the editor, and then you can search for a variable name across ALL the loaded files and it displays every occurrence in a compressed list. You can easily jump through the code to see how the variable is being used. This saves enormous time when you are trying to understand an application. I spent over a decade maintaining and supporting over 40 BIG software systems, so I'm talking from experience. Without TSEPro I would have been LOST.

A superb editor = a great investment. In the long run, you will save yourself 1000's of hours. That's my best tip for understanding foreign code.

Ross

[Pallav Nawani]

The parts I struggle with now are the pawn eval. BTW, the early versions used a similar pawn structure to TSCP!

Read the pawn eval of older versions more carefully. It is much, much better than Tscp.

Fruit looks simple, but isn't really. I can see that much knowledge and brilliance has gone into it. IIRC, fabien once advised Tord to study AI/coding for other games such as go, othello. Maybe we should all take that advise.

Pallav

[Fabien Letouzey]

IIRC, fabien once advised Tord to study AI/coding for other games such as go, othello. Maybe we should all take that advise.

Hi Pallav,

This was an answer to a question regarding building endgame databases!!! I am sure you will agree the meaning is very different then ...

Fabien.

[Fabien Letouzey]

Fruit is a pleasure to read. Everything is well laid out and elegant. The parts I struggle with now are the pawn eval. BTW, the early versions used a similar pawn structure to TSCP!

Hi Ross,

I plead guilty for the pawn eval.

It's the same as before, but I switched to a branchless design. It leads to table-based programming, which is very hard to read I agree.

The positive side is that I can quickly calculate on-the-fly pawn-only features at evaluation time for instance regarding king safety but also say outposts in the future.

I will decide later whether to revert back to simple board-scan routines.

Fabien.

[Uri Blass]

I am not sure if it is the same as before

You said in the readme

"added (small) king-attack bonus, the last *huge* hole in the eval"

I wanted to understand the king safety evaluation and I could not understand it because I saw that I need to understand what
shelter_file does and shelter_file is based on BitGE that is in pawn.cpp
and I did not understand like other variables

This was the reason that my first question was also about BitGE.
I hoped that understanding it may help me to understand the king safety
code.

My understanding is that the content of BitGE is constant during all the run of Fruit2.1 so I wonder why not to write it as const but to have a function to calculate it.

Cannot fruit be faster in case that BitGE and other arrays in pawn.cpp defined to be const?

Note that I still did not do big effort to understand fruit and I plan to print all the source this week and mark slowly parts that I understand(I may also give comments in the pages that I understand so I do not forget it later).

If I do not understand something I may leave it and try to understand another part with the hope that later it will be easier for me to understand what I do not understand.

Uri

[Pallav Nawani]

IIRC, fabien once advised Tord to study AI/coding for other games such as go, othello. Maybe we should all take that advise.

Hi Pallav,

This was an answer to a question regarding building endgame databases!!! I am sure you will agree the meaning is very different then ...

Fabien.

Ok, my bad. I misquoted you.
Though I still think that it is a good idea.

Pallav

[Ross Boyd]

Hi Pallav,

Read the pawn eval of older versions more carefully. It is much, much better than Tscp.

I said early versions of Fruit employed a similar structure to TSCP... here's what I was referring to...

(edited... TSCP stores the rank of the LEAST advanced pawn on each file, not the MOST advanced pawn)

Early versions of Fruit built an array of the LEAST advanced rank of a pawn on any given file. This is exactly what TSCP does. The array is initialised to 7 and 0 for black and white respectively. This makes it trivial to detect passed pawns.

(edited... TSCP does detect passed pawns correctly, my mistake)

Older versions of Fruit also stored the most advanced pawn rank on each file.

Ross