starting a program from scratch

[Uri Blass]

I failed to make a significant improvement in movei in the last months because the code is not well written.

I wonder what good advice do you have for starting a program again from scratch.

I think to keep my basic data structure but to avoid global variables that are not needed(I have too many global variables instead of struct) and I decided that I do not like trying to replace everything in my code.

I also want to use good ideas in fruit for better debugging so I plan to have function with similiar meaning as fruit in order to check if the board is ok and the same for other structures.

looking at fruit it seems to me that there are 2 files
posix and util that are independent on other files and have nothing to do about chess.

I wonder how do you start from scratch.
Do you start from files that have nothing to do about chess or maybe you start from files to make your move generator and files to check no mistake in the board and not care about things like time function first or maybe the order is not important?

Do you write first on paper all the functions that you plan to write and only later go to the coding process?

I am not sure if it is better if I start from a similiar structure to fruit(I do not plan to change my basic structure of the board so no copy and paste).

I am also not sure if it is better if I delay writing code and start writing code only after trying to figure out a full picture of what I plan to do inspite of the fact that I believe that I am already sure about some parts.

Note that evaluation will probably be the last thing that I plan to do in rewriting the code.

Uri

[Gian-Carlo Pascutto]

I rewrote almost everything in Deep Sjeng in june.

I rewrote step by step, so I could always check if things like perft or nodecounts (when relevant, i.e. when a change should not change them) stayed the same. This helps find new bugs.

Some things like basic datastructures are unchanged. Perhaps something else would be a tiny bit better, but I am more familiar with the old ones.

Another advantage is that you always have a working program.

As for structure, after coding all these years in the old, buggy code, I had a good idea what kind of structure I wanted in most places.

I don't really like the idea of rewriting entirely from scratch.

[Roger Brown]

Hello Gian-Carlo,

This is a non-technical question asked by a non-techie so please forgive it...

Do most of your changes have to do with supporting 64 bit systems and or multiple processors? I know that Deep Sjeng supports more than one processor right now but I was wondering if you were going in the direction of supporting even more processors.

Being a single processor person (oh, the horror) I would be really excited if your changes applied to us single processor, non-64 bit system users as well.

Later.

[Gian-Carlo Pascutto]

The changes don't have anything specifically to do with that. Deep Sjeng has always supported 64 bits and large multiprocessor systems.

[Roger Brown]

Hello Gian-Carlo,

Thanks.

Later.

[Uri Blass]

Note that I also think not to change my basic structures not because they are optimal for speed but because writing a new structure and thinking about all the possible problems is too much time.

The problem is not only not having struct and having global variables but also that my division to files is not smart today and I do not see an easy way to change it.

I have one file that have both most of the evaluation functions and the search function and the qsearch function.
Another file is for making moves and generating moves.


Dividing it to more files can do it slower because of having a lot of global variables.

Every file that I have has #include "data.h" that is a list of almost all my global variables.

I found based on experience that if I divide the program to more files and include that file in more files the program becomes slower and I do not like it.

The correct solution is simply to divide "data.h" to more files when every file only knows about part of "data.h" but dividing it correctly to parts is not easy and this is one of the reasons that I think to start from scratch.

I think to copy and paste parts from my old code when I need it is easier than starting to change the existing code.

Uri

[Daniel Shawul]

I found based on experience that if I divide the program to more files and include that file in more files the program becomes slower and I do not like it.

i don't think the slowdown is significant. You can avoid this problem by including alll of your c files in one file. Then compile only that file to get your exe. "data.h" will be included only once in this case.
daniel

[Alessandro Scotti]

I also want to use good ideas in fruit for better debugging so I plan to have function with similiar meaning as fruit in order to check if the board is ok and the same for other structures.

This saves a lot of time actually. I'm writing a new engine from scratch based on an original (at least for me) data structure and it has been very useful to have functions to: display the data structure; verify the data integrity; test and stress specific functions and... lots of assert (which I have renamed "claim" in my code).

Do you start from files that have nothing to do about chess or maybe you start from files to make your move generator and files to check no mistake in the board and not care about things like time function first or maybe the order is not important?

I don't think the order is particularly important. I started with board representation, then move generation and then SEE, because at that time I wasn't sure on how to proceed and needed a smaller, specific task while designing the rest.

Do you write first on paper all the functions that you plan to write and only later go to the coding process?

Not functions for me, but definitely all the algorithms and the data structures, as well as the overall design.

Note that evaluation will probably be the last thing that I plan to do in rewriting the code.

Same thing here, although I'll add piece/square tables relatively early.

[Reinhard Scharnagl]

To (re-)write a chess program from the sratch is (from time to time) no bad idea to review and reorganize data structures and their representations in segmented files or even better C++ classes.

Using C++ classes does not necessarily imply to slow down performance. In Smirf I encapsulated families of related procedures as static methods into matching once to be created classes.

Now, where Smirf is already playing, the experiences in independently writing an own chess program have grown. Thus it would be a good idea also for me to review and rewrite Smirf and, if there was not that lack of interest in Smirf, make it soon.

Regards, Reinhard.

[Dann Corbit]

Since SMP is clearly going to dominate the future (if you plan to be among the top ten programs and since we have multiple CPUs in core now) I think you should consider SMP when doing your design.

Along those lines, global variables will be an evil thing to be avoided except when they really, really have to be global.

Hash table, for instance, deserves to be global because many threads can benefit from it.

Other things that are needed by many threads should be grouped together into a common object/struct and logically organized.

Here is my opinion on organization of your code:
1. Spend lots of time on designing your system. "Measure twice and cut once." The best place to cure a bug is in design phase. The worst place to cure a bug is in release phase.
2. Keep things simple and clear. It is a big mistake to optimize early. First, make absolutely sure it is correct before you try to make it fast.
3. Focus on improving the algorithms. If you can improve your algorithm (which is going to reduce your branching factor) that will matter far more than little tweaky things.
4. Once you have a simple version of your project operational, profile it to see where the problems are. It will be far easier to fix it early than later on after you have attached 500 pounds of gears to the machine.
5. Look at what Fabian did (more from a programming STYLE perspective). His use of asserts() shows excellent critical thinking. This style of programming is not good just for chess engines but for everything you will ever do. Also the helper functions to diagnose problems are obviously a good idea.

IMO-YMMV.

[smcracraft]

I failed to make a significant improvement in movei in the last months because the code is not well written.

I wonder what good advice do you have for starting a program again from scratch.

I think to keep my basic data structure but to avoid global variables that are not needed(I have too many global variables instead of struct) and I decided that I do not like trying to replace everything in my code.

I also want to use good ideas in fruit for better debugging so I plan to have function with similiar meaning as fruit in order to check if the board is ok and the same for other structures.

looking at fruit it seems to me that there are 2 files
posix and util that are independent on other files and have nothing to do about chess.

I wonder how do you start from scratch.
Do you start from files that have nothing to do about chess or maybe you start from files to make your move generator and files to check no mistake in the board and not care about things like time function first or maybe the order is not important?

Do you write first on paper all the functions that you plan to write and only later go to the coding process?

I am not sure if it is better if I start from a similiar structure to fruit(I do not plan to change my basic structure of the board so no copy and paste).

I am also not sure if it is better if I delay writing code and start writing code only after trying to figure out a full picture of what I plan to do inspite of the fact that I believe that I am already sure about some parts.

Note that evaluation will probably be the last thing that I plan to do in rewriting the code.

Uri

The worst mistake you can make is to begin coding without
multiple weeks of thinking followed by multiple weeks of
writing down outline followed by multiple weeks of writing
down data structures.

Only then begin your most basic routines and work on those
until they are perfect. Exactly as you wish. Don't let the desire
to "play chess" with it hurry your project.

If you do this, you will have a chance to make a bigger improvement.

Stuart

[mathmoi]

I failed to make a significant improvement in movei in the last months because the code is not well written.

I wonder what good advice do you have for starting a program again from scratch.

I think to keep my basic data structure but to avoid global variables that are not needed(I have too many global variables instead of struct) and I decided that I do not like trying to replace everything in my code.

You can _always_ avoid globals variables. Most of the time at no cost. The other times, the benifit (less error prone code) will probably outweight the overhead. learn to use pointers and references to move large structures around at a small cost.

I also want to use good ideas in fruit for better debugging so I plan to have function with similiar meaning as fruit in order to check if the board is ok and the same for other structures.

I'm actually doing this in MatMoi, i put Assertions(cond) everywhere it make sense. Having functions to track down bugs ASAP is a good idea too. My first goal when rewritting my engine (it's what i'm doing now) is to get perft to work.

looking at fruit it seems to me that there are 2 files
posix and util that are independent on other files and have nothing to do about chess.

I wonder how do you start from scratch.
Do you start from files that have nothing to do about chess or maybe you start from files to make your move generator and files to check no mistake in the board and not care about things like time function first or maybe the order is not important?

This time I started with a blank file. However you can keep some fonctions from the previous version of your engine if you want.

Do you write first on paper all the functions that you plan to write and only later go to the coding process?

Definitely. You at least need to write down a plan of wich function will do what. Then look at it and ask yourself, what could be improved. Then modify your plan. Do this until you think your plan is perfect. Then you can open your IDE.

I am not sure if it is better if I start from a similiar structure to fruit(I do not plan to change my basic structure of the board so no copy and paste).

I am also not sure if it is better if I delay writing code and start writing code only after trying to figure out a full picture of what I plan to do inspite of the fact that I believe that I am already sure about some parts.

Note that evaluation will probably be the last thing that I plan to do in rewriting the code.

Same here, but i'll probably implement a piece/square evaluation as soon as my engine is ready to play.

Mathieu Pag?
mathieu.page@gmail.com

[smcracraft]

I failed to make a significant improvement in movei in the last months because the code is not well written.

I wonder what good advice do you have for starting a program again from scratch.

I think to keep my basic data structure but to avoid global variables that are not needed(I have too many global variables instead of struct) and I decided that I do not like trying to replace everything in my code.

You can _always_ avoid globals variables. Most of the time at no cost. The other times, the benifit (less error prone code) will probably outweight the overhead. learn to use pointers and references to move large structures around at a small cost.

I also want to use good ideas in fruit for better debugging so I plan to have function with similiar meaning as fruit in order to check if the board is ok and the same for other structures.

I'm actually doing this in MatMoi, i put Assertions(cond) everywhere it make sense. Having functions to track down bugs ASAP is a good idea too. My first goal when rewritting my engine (it's what i'm doing now) is to get perft to work.

looking at fruit it seems to me that there are 2 files
posix and util that are independent on other files and have nothing to do about chess.

I wonder how do you start from scratch.
Do you start from files that have nothing to do about chess or maybe you start from files to make your move generator and files to check no mistake in the board and not care about things like time function first or maybe the order is not important?

This time I started with a blank file. However you can keep some fonctions from the previous version of your engine if you want.

Do you write first on paper all the functions that you plan to write and only later go to the coding process?

Definitely. You at least need to write down a plan of wich function will do what. Then look at it and ask yourself, what could be improved. Then modify your plan. Do this until you think your plan is perfect. Then you can open your IDE.

I am not sure if it is better if I start from a similiar structure to fruit(I do not plan to change my basic structure of the board so no copy and paste).

I am also not sure if it is better if I delay writing code and start writing code only after trying to figure out a full picture of what I plan to do inspite of the fact that I believe that I am already sure about some parts.

Note that evaluation will probably be the last thing that I plan to do in rewriting the code.

Same here, but i'll probably implement a piece/square evaluation as soon as my engine is ready to play.

Mathieu Pag?
mathieu.page@gmail.com

If I were rewriting my program, I'd use the Fruit example of scaled
evaluation from opening/endgame for pc/sq values, general evaluation
terms, and material. Bob Hyatt said it is critical to avoid hard transitions
between game phases since the search can do weird things if it is not
a smooth function.

I can't retrofit my current program because it would require stripping
out the pc/sq from the makemv/unmakemv, changing all the tables, etc.

Too bad for me.

Stuart