Back-porting WinBoard to xboard

[H.G.Muller]

To make this discussion accessible to everyone, rather than just keep it a private e-mail exchange between Zach and me, I created this thread. Zach is the main person currently working on shaping up the xboard part of the xboard/WinBoard code base so that it will catch up with WinBoard again, and offer all the new features that have been added to WinBoard since version 4.2.7b, (after which development by the GNU-Savannah project effectively ceased).

For clarity: The xboard/WinBoard code base consists of 3 parts:
* A WinBoard-specific part
* An xboard-specific part
* A common part (the 'back-end')
This is necessary because the 'front-end', responsible for communication with the user, the graphics display, management of display windows an the like is very platform-dependent. This makes that any change in the WinBoard-specific code has to be parallelled in the xboard front-end and vice versa, duplicating the amount of work needed for any front-end change. Changes in the back-end, which is responsible for communication with the engines or ICS, and handling all th Chess-related stuff such as legality checking and adjudications, automatically benefit both WinBoard and xboard.

The recent years have seen an enormous development of WinBoard by individuals outside the GNU-Savannah team. As many of these improvements were in the front-end, this development that was not parallelled in xboard at all, and created a situation where the xboard front-end code was no longer compatible with the back-end code used by WinBoard. This meant that xboard users can not even profit anymore from back-end changes that are not dependent on any front-end modifications per se.

We are now going to correct this situation. The upcoming release f WinBoard 4.3.14 will contain also the source code for an xboard front-end that is at least compatible with the back-end used by this most recent WinBoard. Some of the added back-end functions have even be made availabe under xboard, by adding a few new xboard command-line options (mainly for the adjudication of engine-engine games). Existing options for which the new back-end defines additional values (such as the -variant option) of course work automatically. The xboard display handling was quickly patched in order to allow display and handling of the crazyhouse holdings.

But that was really all; no full-scale effort was made yet to make xboard functionally equivalent to WinBoard. In this version (4.3.14), we only tried to restore compatibility with rather minimal effort. So it should be seen as a starting point; the main catching up is still to be done. Especially many of the WinBoard_x options, added by Allessandro Scotti, were front-end options, and require extensiv modification of xboard to work there too. Also in the area of menus xboard severely laggs: none of the new options is available through the menus yet, the back-end options that already do work in xboard can only be controlled through command-line arguments.

In this thread, I want to discuss the plan on how to close the remaining gap.

[H.G.Muller]

In order to minimize the effort future changes in WinBoard / xboard will require, it is important that a good separation between front-end and back-end code will be maintained, and that as much of the code as possible resides in the back-end. Unfortunately, the fact that for som time the development has been entirely on WinBoard has blurred this distinction a bit. Even version 4.2.7 seemed to have a sizable amount of dublication of functions.

Especially in the area of the menus and command-line options, it is ery inconvenient that all the work has to be duplicated. Especially since the menus in WinBoard are made in a very cumbersome way. (I don't know about xboard.) It would be very convenient if the handling of the menus could be split into back-end code, which would be responsible for defining the contents of the menus in a platform-independent way, and a general menu and popup-dialogs-creating routine that would make the necessary system-dependent calls to create a menu/popup according to these specifications. Currently every menu and popup needs its own layout spcefication at a very low level, in different formats for WinBoard and xboard.

For the definition and parsing of command-line options, it is completely unclear to me why this happens in the front-end I see no logical reason why WinBoard and xboard should have different options. Almost all options have their arguments directly copied to a large data structure 'appData' in the back-end. The only difference seems to be the format in which the actual command line is supplied by the operating system. But it is a comaparatively trivial task to convert the Unix-style format (an array of character strings) to the WinBoard format (a single string of characters, separating the arguments by spaces). WinBoard uses the same option parser for handling the command-line, the winboard.ini file and the options hidden in the engine startup commands, an argument determining what to use as source of the data. With only a tiny addition it could be made to accept data from the xboard command line as well.

So I think a good first step would be to take the ParseArgs() routine, and its callbacks FileGet() and StringGet(), plus the array ArgDescriptors[] and its type definition, and almost all of the routine InitAppData (which sets defaults before the arguments are parsed), all out of winboard.c. And then put them into a separate source file args.c, which would belong to the back-end. And then remove all options code from xboard, and replace it by calls to these routines. This would automatically make all options that are implemented entirely in the back-end available to xboard. Parameters in appData for controlling the fron-end would of course only hve an effect if the respective front-end would indeed look at those paramenters (presumably in th function that implements the feature these parameters are supposed to control), but at least the parameters added on behalf of a function in one front-end, would always be there, so the other front-end can also start to use them without th need for any extra work.

[H.G.Muller]

In summary: I recognize the following priority areas in shaping up xboard:

* move all processing of command-line options to the back-end, so they automatically benefit xboard.
* Separate the display sizing code from main() in xboard.c, so that it can be called from any place in the code, so that it becomes possible to re-size and re-format the display while running (rather than just at start-up). This is already essential for switching between crazyhouse and other variants in ICS mode (where the ICS commands such changes), but will become essential in local mode as well, in order for the board-size and variant menu to work.
* Collect all routines that display menus in either front-end, and try to collapse them into a single, universal table-driven menu-handling routine, plus data-tables that have to be passed to that routine. (I know very little about event-driven programming with windows, basically WinBoard is the only program using menus that I have ever seen the source of, but it looks like there are zillions of routines in there that all do almost exactly the same, except for a tiny change of the text they print and the routine they call when the button next to that text is pressed...) If we would convert the WinBoard menu handling this way, most of it could go to the back-end, and all menus would be automatically avalable in xboard as well. This would save us the task of xplicitly designing and adding xboard menus.
* Fix the display of the crazyhouse holdings, which currently does not indicate the number of pieces of a given type that is held in xboard. (In WinBoard there is a display bug in Flip View there, btw.)
* Alter the piece drag animator to handle stacking of pieces in the holdings in a more natural way.
* Make the loading of piece bitmaps (slightly) variant dependent, in order for Shogi to work. Fix potential other problems in Shogi (promotions?).
* Fix xboard such that it no longer pops-up a promotion menu on illegal moves.
* Implement cliking on the clocks in the mouse interrupt handler as a way to adjust the time.
* Move some of the clock-display routines to the back-end, and allow the "flag fell" info to be displayed there in xboard.
* design a better format for the piece bitmaps/pixmaps. The current format is horrendously space wasting. The pixmaps are effectively duplicated now, and us 1 byte per pixel for storage, even where 1 bit would suffice (for the black pieces). Not to talk about the compression factor that could be acieved with run-length coding on such simple bitmaps.
* Implement some equivalent to the font-based rendering in xboard. I would also be satisfied with an external tool that would produce bitmaps from a scalable template (like a true-type font), so that it becomes trivial to create a directory with pixmaps that can be loaded during runtime. Provided WinBoard then acquires the ability to load thes bitmaps at run time too.
* Add the WinBoard_x auxilary windows for engine output and eval graph to xboard. (And move history? Or is that already there?)
* Implement the WinBoard_x backTexture feature into the drawng routines of xboard.
* Implement the -allWhite and -flipBlack options in the piece dawing routine of xboard.
* Convert and add the Japanes Shogi pixmaps to xboard.

[Zach Wegner]

* move all processing of command-line options to the back-end, so they automatically benefit xboard.

As I brought up in the email, this is a bit more complicated than it seems. Xboard has standard X Windows options such as "geometry" that are parsed automatically by the X library calls. I believe that it also looks at the ~/.Xdefaults file for these options. So I'm not sure of the best way to do it. Really, the "geometry" option is useless because it screws up the board, but options like "display" have a useful purpose (I can start Xboard matches remotely via SSH by directing the window output to my monitor at home, whereas otherwise it wouldn't find a display). I'm not sure of the best way to handle this. I think perhaps instead of converting the ParseArgs() routine, we keep it, but we make a portable framework for the command line options that are automatically converted into the native format before they are sent to the library calls, similar to the proposed menu idea. It means that Xboard users still wouldn't get an xboard.ini, but we've been able to do without it for a while.

* Separate the display sizing code from main() in xboard.c, so that it can be called from any place in the code, so that it becomes possible to re-size and re-format the display while running (rather than just at start-up). This is already essential for switching between crazyhouse and other variants in ICS mode (where the ICS commands such changes), but will become essential in local mode as well, in order for the board-size and variant menu to work.
* Collect all routines that display menus in either front-end, and try to collapse them into a single, universal table-driven menu-handling routine, plus data-tables that have to be passed to that routine. (I know very little about event-driven programming with windows, basically WinBoard is the only program using menus that I have ever seen the source of, but it looks like there are zillions of routines in there that all do almost exactly the same, except for a tiny change of the text they print and the routine they call when the button next to that text is pressed...) If we would convert the WinBoard menu handling this way, most of it could go to the back-end, and all menus would be automatically avalable in xboard as well. This would save us the task of xplicitly designing and adding xboard menus.

This one is pretty important, but I don't think it would be too hard. I haven't looked much at the window system, but perhaps the portable data structure could look like this:

Code: Select all

typedef struct{
    char *menu_name;
    char *menu_parent;
    Boolean *argument;
    FuncPointer action;
} MenuEntry;

This looks kind of strange, let me explain. For each menu entry, it can be either a switch (like "Show Thinking"), or have an action (like "Save Game"). So if there's an argument given, it will simply switch it. The argument will also be used to toggle the display of check marks or X's, depending on the port. For some, such as the variant menu, it will be both: there will be a function that turns off every other option. This involves creating an array of BOOLs: variantIsActive[VariantCount];
It's kind of ugly, as only one entry is ever true, but it's only used for menu purposes, and it should simplify the menu structure a lot. Or maybe having different types of arguments, like a Boolean, or an integer, along with a key. So the front end functions, for the integers, will look at the argument, and turn on the display if it matches the key of the menu.

* Fix the display of the crazyhouse holdings, which currently does not indicate the number of pieces of a given type that is held in xboard. (In WinBoard there is a display bug in Flip View there, btw.)
* Alter the piece drag animator to handle stacking of pieces in the holdings in a more natural way.
* Make the loading of piece bitmaps (slightly) variant dependent, in order for Shogi to work. Fix potential other problems in Shogi (promotions?).
* Fix xboard such that it no longer pops-up a promotion menu on illegal moves.
* Implement cliking on the clocks in the mouse interrupt handler as a way to adjust the time.
* Move some of the clock-display routines to the back-end, and allow the "flag fell" info to be displayed there in xboard.
* design a better format for the piece bitmaps/pixmaps. The current format is horrendously space wasting. The pixmaps are effectively duplicated now, and us 1 byte per pixel for storage, even where 1 bit would suffice (for the black pieces). Not to talk about the compression factor that could be acieved with run-length coding on such simple bitmaps.
* Implement some equivalent to the font-based rendering in xboard. I would also be satisfied with an external tool that would produce bitmaps from a scalable template (like a true-type font), so that it becomes trivial to create a directory with pixmaps that can be loaded during runtime. Provided WinBoard then acquires the ability to load thes bitmaps at run time too.
* Add the WinBoard_x auxilary windows for engine output and eval graph to xboard. (And move history? Or is that already there?)
* Implement the WinBoard_x backTexture feature into the drawng routines of xboard.
* Implement the -allWhite and -flipBlack options in the piece dawing routine of xboard.
* Convert and add the Japanes Shogi pixmaps to xboard.

I'd want to convert as much as this as possible into back end functions. Ideally, we would make front end functions such as CreateWindow, AddText, and I suppose some drawing functions. So that way, if done properly, new functions added, wouldn't require _any_ extra code to work in either port. This is a little ambitious, but it can happen slowly...

[H.G.Muller]

* move all processing of command-line options to the back-end, so they automatically benefit xboard.

As I brought up in the email, this is a bit more complicated than it seems. Xboard has standard X Windows options such as "geometry" that are parsed automatically by the X library calls. I believe that it also looks at the ~/.Xdefaults file for these options.

I am not sure I understand the difficulty. Surely there must be a way to request options that were not processed and recognized by the X-library calls to the user application. Or, if this is not possible, pre-process the arguments before invoking the X-library calls, copying them to a string.

How does this .Xdefaults stuff work? Is it something the command shell reads, and prepends to the user-supplied options before passing the lot in the exec call to the new core image? Or is it something the startup code of the new program does, after it has woken up with its memory initialized with the arguments passed through exec? Just reading .Xdefaults, and slipping the lines read there into the argument list, before giving the user control?

In the worst case we can add all X windows argumets to the list of xboard arguments, as dummies, to make sure they are ignored. But I would be surprised if that was necessary.

[Zach Wegner]

I am not sure I understand the difficulty. Surely there must be a way to request options that were not processed and recognized by the X-library calls to the user application. Or, if this is not possible, pre-process the arguments before invoking the X-library calls, copying them to a string.

I'm really not sure, as I don't too much about the X internals. I could research it though.

How does this .Xdefaults stuff work? Is it something the command shell reads, and prepends to the user-supplied options before passing the lot in the exec call to the new core image? Or is it something the startup code of the new program does, after it has woken up with its memory initialized with the arguments passed through exec? Just reading .Xdefaults, and slipping the lines read there into the argument list, before giving the user control?

In the worst case we can add all X windows argumets to the list of xboard arguments, as dummies, to make sure they are ignored. But I would be surprised if that was necessary.

It works differently. Xboard has a big list of command line options, and the actions to take, and passes it to XtAppInitialize, which parses all the command line both against standard X options and the supplied ones. I looked a little closer though, and it shouldn't be too hard: XtAppInitialize modifies argc and argv so that any unrecognized options can be parsed separately. So either we pass a NULL pointer for the custom options, or a pointer to an empty set. The man page for XtAppInitialize is very terse and doesn't go into much detail about this. But I guess it should be easier than I thought. I think the ParseArgs from Winboard should be broken up just to pull the strings apart, and then they should be passed into the back end, so that xboard doesn't have to merge all the strings together, only to merge them back. In fact, that probably wouldn't work anyways: if an argument is given on the command line, like this: xboard -fcp gnuchess -fd "/home/chess and stuff/engines and other stuff/gnuchess", then the shell automatically takes care of the quotes and gives the path as just one string, without the quotes included. The back end wouldn't know which arguments were which once they were merged together.

[Zach Wegner]

OK, I was looking at the Winboard source, and I saw that all the menus aren't defined in the source code, but rather in the winboard.rc file. Yuck!! Do any of the Windows gurus know how to dynamically generate them?

[H.G.Muller]

It works differently. Xboard has a big list of command line options, and the actions to take, and passes it to XtAppInitialize, which parses all the command line both against standard X options and the supplied ones. I looked a little closer though, and it shouldn't be too hard: XtAppInitialize modifies argc and argv so that any unrecognized options can be parsed separately. So either we pass a NULL pointer for the custom options, or a pointer to an empty set. The man page for XtAppInitialize is very terse and doesn't go into much detail about this. But I guess it should be easier than I thought. I think the ParseArgs from Winboard should be broken up just to pull the strings apart, and then they should be passed into the back end, so that xboard doesn't have to merge all the strings together, only to merge them back.

I had something much simpler in mind. Move ParseArgs and associated stuff to the back-end. Just provide a third type of GetFunc for it (next to FileGet() to read from a file and StringGet() to read the next string character): ArgGet(). This would be defined in the front-end. In winboard.c it would be an identical copy of StringGet (which was moved to the back-end).

But in xboard, it would be a function that returns the next character from the current arguument, or, if that runs out, it returns a space, and sets the 'pointer' with which it is called to the first character of the next argument. The pointer would be a kind of two-dimensional pointer, the high-order byte would code for the argument number, and the low-orderr byte for the character. So basically:

Code: Select all

ArgGet(void **p)
{
    int *ip = (int **) p; char c;

    if((*ip >> 8) >= argc) return EOF; // no more arguments
    c = argv[(*ip)>>8][(*ip)&255]; // next char of current arg
    if(c != 0) { (*ip)++; retiurn(c); }
    *ip = (*ip | 255) + 1;       // skip to first char of next arg
    return(' ');
}


Only hack is that argc and argv have to be globally available.

[Zach Wegner]

I had something much simpler in mind. Move ParseArgs and associated stuff to the back-end. Just provide a third type of GetFunc for it (next to FileGet() to read from a file and StringGet() to read the next string character): ArgGet(). This would be defined in the front-end. In winboard.c it would be an identical copy of StringGet (which was moved to the back-end).

But in xboard, it would be a function that returns the next character from the current arguument, or, if that runs out, it returns a space, and sets the 'pointer' with which it is called to the first character of the next argument. The pointer would be a kind of two-dimensional pointer, the high-order byte would code for the argument number, and the low-orderr byte for the character. So basically:

Code: Select all

ArgGet(void **p)
{
    int *ip = (int **) p; char c;

    if((*ip >> 8) >= argc) return EOF; // no more arguments
    c = argv[(*ip)>>8][(*ip)&255]; // next char of current arg
    if(c != 0) { (*ip)++; retiurn(c); }
    *ip = (*ip | 255) + 1;       // skip to first char of next arg
    return(' ');
}


Only hack is that argc and argv have to be globally available.

I think we should try to avoid that kind of code as much as possible.

Seriously though, much of the parsing code in ParseArgs is completely unnecessary for xboard. I had in mind separating that code out of ParseArgs and having a function that grabs the next string. It could be passed a void *arg, which in winboard would simply be a char*, and in xboard it would be a struct { int argc; char **argv; }. So that way we don't have a weird fudge to parse arguments in xboard, and I think it will make the winboard code cleaner too. Here's what it could look like:

Code: Select all

char *c;
    c = getstr(cl);
    if (c == NULL) break;
    if (*c == ';') {
      /* Comment to end of line */
      continue;
    } else if (*c == '/' || *c == '-') {
      /* Switch */
      for (ad = argDescriptors; ad->argName != NULL; ad++)
        if (strcmp(ad->argName, c + 1) == 0) break;

      if (ad->argName == NULL)
        ExitArgError("Unrecognized argument", c);

    } else if (*c == '@') {
      /* Indirection file */
      ad = argDescriptorIndirection;
    } else {
      /* Positional argument */
/* What the hell does this do?? It just grabs straight from the array of valid command line options, which means that the user must supply every command line option in the order in winboard.c... very strange... */
      ad = &argDescriptors[posarg++];
      strcpy(argName, ad->argName);
    }
....

The parsing function is pretty simple too. There just has to be an argument whether the next string is a command line option or an argument of one.

Code: Select all

char buffer[123], *c, ch;
    do
        ch = get(cl);
    while (ch == ' ' || ch == '\n' || ch == '\t');
    if (ch == NULLCHAR) return NULL;
    if (ch == ';') {
      /* Comment to end of line */
      c = buffer;
      do {
          *c = ch;
          ch = get(cl);
      } while (ch != '\n' && ch != NULLCHAR);
      *c = NULLCHAR;
      return buffer;
    } else if (ch == '/' || ch == '-') {
      /* Switch */
      c = buffer;
      while (ch != ' ' && ch != '=' && ch != ':' && ch != NULLCHAR &&
             ch != '\n' && ch != '\t') {
        *c++ = ch;
        ch = get(cl);
      }
      *q = NULLCHAR;
    } else if (ch == '@') {
      /* Indirection file */
      buffer[0] = '@';
      buffer[1] = NULLCHAR;
      return buffer;
    } else {
      /* Positional argument */
      buffer[0] = NULLCHAR;
      return buffer;
    }
...

[H.G.Muller]

Although I am not sure what you do mean by the distinction "command-line option" and "argument of one", I agree that the parsing is very simple. The reason that it is a pain not to have it in the back-end is the enormous data-structure with options, which has different formats in xboard and WinBoard, so that you have to add each new option twice.

What I like about the xboard format is that the default value is in the list. What I like about the WinBoard format is that it contains a Boolean to specify if the option should be written to the winboard.ini file, or not. I think we should combine these features, by adding an extra element to the ArgType structure (which could be a union, so that we do not need the kludgey casts that the xboard default values use).

This way we can get rid of most of InitAppData() in winboard.c as well.

SaveSettings() in winboard.c must be split into a routine interrogating the window placements, and a part doing the actual saving. That latter part should then go to the back-end (calling the former part, for which an xboard equivalent will have to be written if we want to provide that functionality in xboard).

[Zach Wegner]

I didn't think that would be clear. I'll just give an example.

Command line option:

Code: Select all

-tc

Argument:

Code: Select all

1

It actually might not be necessary, but the way ParseArgs works now parses the first one differently than the second. For instance, you can only use quotes in the "arguments". The question is if this distinction is necessary, or if it's only there because it's easier in the big ParseArgs function.

I think your idea is good though, merging the data structures is essential.

[Zach Wegner]

Huh, one thing I just realized is that Winboard has the string coming either from the command line or from a file, while xboard's options are already preparsed by the shell. So I'm not sure the best way to do this. Maybe we could use the same parsing function to read command lines in winboard, and files in both xboard and winboard...

[H.G.Muller]

That pre-parsing only involves removal of quotes of string arguments, not? I guess that makes the ArgGet a little bit more complicated: when it wants to return the first character of the next argument, it should scan the argument for spaces, and slip in a quote.

I guess there is a subtlety here concerning the treatment of the backslash: with single or double quoting WinBoard accepts this as an escape character, but not if the argument type is filename.

Is there any harm in forbidding escapes like \n, \r and \t in file names used by xboard? Seems to me you have to be pretty malicious to use those in file names. Of course we could let the back-end ask to the front-end if it should treat the \ as an escape character in filenames, or not. That might be the simplest solution:

Code: Select all

int Escapes() { return TRUE; } // in xboard.c
int Escapes() { return FALSE; } // in  winboard.c


[Zach Wegner]

I put a quick draft of this idea up on SF. It might not even compile, I can't test it. There is also a flaw (though this was already there) that the strings aren't checked for a buffer overrun.

That pre-parsing only involves removal of quotes of string arguments, not? I guess that makes the ArgGet a little bit more complicated: when it wants to return the first character of the next argument, it should scan the argument for spaces, and slip in a quote.

No, it takes a full string, with quotes or whatever, and breaks it up into chunks. I'm not sure exactly what you are referring to, but the code I put up doesn't require any fudging around with the arguments.

I guess there is a subtlety here concerning the treatment of the backslash: with single or double quoting WinBoard accepts this as an escape character, but not if the argument type is filename.

Is there any harm in forbidding escapes like \n, \r and \t in file names used by xboard? Seems to me you have to be pretty malicious to use those in file names. Of course we could let the back-end ask to the front-end if it should treat the \ as an escape character in filenames, or not. That might be the simplest solution:

Code: Select all

int Escapes() { return TRUE; } // in xboard.c
int Escapes() { return FALSE; } // in  winboard.c


I think xboard users can live without those escapes in their filenames. But the treatment of backslashes works the same way. I put in an argument to an argDescriptor which specifies what type of argument it is, and filenames don't use escapes.

[Eric Mullins]

I'm interested in this project, and can assist some. Is there any interest in migrating the frontend to Qt or similar framework?

[Zach Wegner]

I'm interested in this project, and can assist some. Is there any interest in migrating the frontend to Qt or similar framework?

Certainly a very interesting idea. There hasn't really been much work on the front/back end separation yet, so now would certainly be a good time to decide.

I don't really know anything about Qt. GTK is also a reasonable alternative (makes pretty nice looking applications, including Firefox). GTK has the advantage of being in C, as well as being GPL. On first glance it looks easier to integrate into the x/winboard code.

Whatever the case, I think a portable GUI framework would be a great idea.

[Eric Mullins]

I don't really know anything about Qt. GTK is also a reasonable alternative (makes pretty nice looking applications, including Firefox). GTK has the advantage of being in C, as well as being GPL. On first glance it looks easier to integrate into the x/winboard code.

Whatever the case, I think a portable GUI framework would be a great idea.

That's why I mentioned it. If ever there was a good time to merge xboard and Winboard, this seems to be it.

I've been a windows developer for a long time. I don't know anything about Qt, GTK, or even Xaw for that matter. I figured if I must learn one to help out, a modern one is preferable.