Input redirection problem

[mridul]

Hi all,

I just interfaced my smp engine with winboard and I am facing an unexpected problem.
When I run my engine with two processes , winboard reads only from the first child and not from the second.
A console window opened up and the output from the second child process (of same engine) goes into that window.

Is there some way for a spawned child to inherit the console descriptors of the parent ?
Essentially , I want outputs of both processes to go to winboard.

Thanks,
Mridul

PS : I am not sure if I have made my problem clear above ...

[H.G.Muller]

Hi all,

I just interfaced my smp engine with winboard and I am facing an unexpected problem.
When I run my engine with two processes , winboard reads only from the first child and not from the second.
A console window opened up and the output from the second child process (of same engine) goes into that window.

Is there some way for a spawned child to inherit the console descriptors of the parent ?
Essentially , I want outputs of both processes to go to winboard.

Thanks,
Mridul

PS : I am not sure if I have made my problem clear above ...

You want the engine process, that has a pipe towards the winboard process, to fork, so that both engine processes share the same pipe and can each shove messages into it? If I remember correctly, under UNIX this is absolutely standard: after a fork the child inherits all open file descriptors from the parent. This is how pipes are set up in the first place: the parent creates a pipe and gets handed an input and output file descriptor for it from the OS. Then it forks and one of the new processes then uses only the input part of the pipe and the other the output (usually immediately closing the end that they don't need).

My knowledge could be outdated by several decades, though...

[Tony van Roon-Werten]

Hi all,

I just interfaced my smp engine with winboard and I am facing an unexpected problem.
When I run my engine with two processes , winboard reads only from the first child and not from the second.
A console window opened up and the output from the second child process (of same engine) goes into that window.

Is there some way for a spawned child to inherit the console descriptors of the parent ?
Essentially , I want outputs of both processes to go to winboard.

Thanks,
Mridul

PS : I am not sure if I have made my problem clear above ...

I don't think it works that way. How can winboard decide wich engine to follow ? Which one gives the best move ?

The best way might be to let the first process get the output from all the other processes, and let him send it to winboard.

Tony

[H.G.Muller]

This is true, if two processes write the same pipe, the reader at the other end can not know who did the writing (unless that info is included in the message). If two processes read from the same pipe, the first to try will get the data, and the other gets nothing. There is no duplication of data, and no way to put the data back if you thiscover it was not meant for you (i.e. things like ungetchar() will just put the data back into your own input buffer, not make it available to any other pipe readers).

By the way, is there an easy guide about how to interface an engine to Winboard? As far as I am concerned only the tiniest subset of commands needs to be implemented, not much more than just the moves going back and forth between Winboard and the engine. My engine is ready to completely ignore any other command that Winboard would send it, as long as that is acceptable to Winboard and does not create a dead-lock.

Does Winboard cooperate with programs that run under Cygwin?

[Sven Schüle]

Hi all,

I just interfaced my smp engine with winboard and I am facing an unexpected problem.
When I run my engine with two processes , winboard reads only from the first child and not from the second.
A console window opened up and the output from the second child process (of same engine) goes into that window.

Is there some way for a spawned child to inherit the console descriptors of the parent ?
Essentially , I want outputs of both processes to go to winboard.

Thanks,
Mridul

PS : I am not sure if I have made my problem clear above ...

Hi mridul,

WinBoard communicates with two players. Each player can be an engine (/fcp or /scp option), and if it is, WinBoard forks one process for each engine and opens two pipes to that process, one for reading (engine moves) and one for writing. Only that process communicates via the pipe. If that process has one or more child processes, these can't communicate with WinBoard (it is simply impossible because WinBoard has opened the pipes).

One question: why do you use child processes for your SMP engine instead of threads?

Sven

[mridul]

Hi all,

I just interfaced my smp engine with winboard and I am facing an unexpected problem.
When I run my engine with two processes , winboard reads only from the first child and not from the second.
A console window opened up and the output from the second child process (of same engine) goes into that window.

Is there some way for a spawned child to inherit the console descriptors of the parent ?
Essentially , I want outputs of both processes to go to winboard.

Thanks,
Mridul

PS : I am not sure if I have made my problem clear above ...

Hi all,

I guess I was correct in assuming that I was not communicating the problem correctly
Just got back from work - so will try to explain my problem better : all/any suggestions welcome !

The idea is this - depending on who owns the root , one of the two processes (when I run with 2 processes) will spit out the pv line.

When in a , if I redirect the output to a file - I do get everything in order and as expected.
In winboard though , I observe the following -

* If I start with a single process (and so the winboard spawned process is the only engine searching) , it works fine : as expected. There is no visible console window that pops up , etc.

* When I launch with two processes , winboard continues to interface properly with the first child (which it spawned) , but not with the second.
Instead I get a command window opened up , and all the stdout messages from the second process goes to that.

The bahaviour is as expected when I manually launch from cli - and yet different when I launch from winboard : hence my question.

Note : the loss of functionality is that
a) I have an extra console window
b) I miss some pv line updates (when second process becomes the root owner)

Ofcourse , when every iteration completes , the initial parent does send the pv to winboard - so I am not having severe issues.
But I find this behaviour inconsistent and plain annonying.

I suspect this is because of how winboard launches the engine to avoid the console window , etc : I have not yet looked at its code though and am quite ignorent about win32 api.

Thanks in advance for your suggestions,
Regards,
Mridul

[H.G.Muller]

Hmm, this new description does not seem to be different from what I understood from the first one. What you don't state explicitly, is which of the first two process spawns the third one. I suppose it is the engine process, so that it is actually a grand child of the Winboard process. So Winboard knows nothing about it, and hence it is no surprise that there is no explicit attempt to interface with it.

As I understand pipes, the grand child should inherit the open input end of the pipe from the first engine, and anything written to it by either process should end up at Winboard, which does not know from which engine it comes or even thatthere were multiple engines.

Perhaps pipes work differently under Windows. But seems to me that your call for spawning a new process does more that it should: it seems to close open file descriptors for input and output and resets them to a (new) console. So I would suspect the spawn call, rather than any peculiar property of pipes.

[mathmoi]

Hi all,

I just interfaced my smp engine with winboard and I am facing an unexpected problem.
When I run my engine with two processes , winboard reads only from the first child and not from the second.
A console window opened up and the output from the second child process (of same engine) goes into that window.

Is there some way for a spawned child to inherit the console descriptors of the parent ?
Essentially , I want outputs of both processes to go to winboard.

Thanks,
Mridul

PS : I am not sure if I have made my problem clear above ...

Hi mridul,

The question is : Why don't you use threads instead of process to create an SMP engine ?

MP

[mridul]

Hi all,

I just interfaced my smp engine with winboard and I am facing an unexpected problem.
When I run my engine with two processes , winboard reads only from the first child and not from the second.
A console window opened up and the output from the second child process (of same engine) goes into that window.

Is there some way for a spawned child to inherit the console descriptors of the parent ?
Essentially , I want outputs of both processes to go to winboard.

Thanks,
Mridul

PS : I am not sure if I have made my problem clear above ...

Hi mridul,

The question is : Why don't you use threads instead of process to create an SMP engine ?

MP

I have been boring way too many people with those for quite a while now - so wont repeat them
Let me see if I can find a link to what I said latest at CCC ... everything is based on my expierence , so please take it with a pinch of salt.

Or is your question specifically targeted at this problem I am seeing ? Would be an overkill to change design to get around a stupid bug

Mridul

[Sven Schüle]

Hi mridul,

Would be an overkill to change design to get around a stupid bug

which bug do you mean here? Some of us have tried to explain that the existing WinBoard architecture does not allow what you intend to do. The engine process being forked by WinBoard has to communicate with the GUI, and the GUI does not allow other partners than those it created itself.

You also can't split a pipe such that either one process or another may write into it. (Neither on Windows nor on UNIX, and this was also never possible in the past.)

If you really need separate processes instead of threads then you might perhaps define the initial engine process as a "master" and maintain a communication protocol to forward messages from a secondary process via the master to the GUI. That's exactly what Tony had already suggested.

Still I think threads would even be the better choice. But you seem to have good reasons against it.

I hope that you will find a good design for your SMP engine, at least I believe it's not an easy task (I never tried it for a chess engine myself).

Sven

[Tony van Roon-Werten]

Hi mridul,

Would be an overkill to change design to get around a stupid bug

which bug do you mean here? Some of us have tried to explain that the existing WinBoard architecture does not allow what you intend to do. The engine process being forked by WinBoard has to communicate with the GUI, and the GUI does not allow other partners than those it created itself.

You also can't split a pipe such that either one process or another may write into it. (Neither on Windows nor on UNIX, and this was also never possible in the past.)

If you really need separate processes instead of threads then you might perhaps define the initial engine process as a "master" and maintain a communication protocol to forward messages from a secondary process via the master to the GUI. That's exactly what Tony had already suggested.

Still I think threads would even be the better choice. But you seem to have good reasons against it.

I hope that you will find a good design for your SMP engine, at least I believe it's not an easy task (I never tried it for a chess engine myself).

Sven

Actually, for duals, multiprocessing takes away a lot of the difficulties. You only need to share the hashtable and some struct wich has all the relevant (root) data for the other process to search.

You can keep all your globals so you a) don't have to adjust your code and b) don't need to pass around pointers to structs so you save at least one register.

Another advantage is that you'll be using exactly the same move selection function etc.

If you want "easy" smp, multiprocessing is best choice IMO.

Tony

[H.G.Muller]

You also can't split a pipe such that either one process or another may write into it. (Neither on Windows nor on UNIX, and this was also never possible in the past.)

I most emphatically disagree. I have the UNIX programmer's manuals in front of me, (alas, a paper version...) and they clearly state that on a fork() the child inherits all open file descriptors. No exception is made for file descriptors that refer to a pipe.

In the section on pipes says that reading an (empty) pipe produces an error if all write file descriptors on that pipe are closed, a remark that would make no sense if their could be only one process writing the pipe.

And finally, because the proof of the pipe is in the smoking, I can run the following program under Cygwin (the Unix emulator under Windows) compiled with gcc:

Code: Select all

#include <stdio.h>

main()
{
        int i, fd[2];
        char buf[100];

        pipe(fd);
        if(fork())
        {
                printf("I am the parent\n");
                sleep(10); /* wait until offspring is done */
                read(fd[0], buf, 31);
                buf[31] = 0;
                printf("message from my grandchildren:\n%s\n", buf);
        } else
        {
                printf("I am the child\n");
                if(fork())
                {
                        printf("I am Alice\n");
                        write(fd[1],"Alice says Hi!\n", 15);
                        printf("Alice exits\n");
                } else
                {
                        printf("I am Bob\n");
                        write(fd[1],"Bob says Hello!\n", 16);
                        printf("Bob exits\n");
                }
        }
}


Running it shows clearly that two processes (Alice and Bob) can write to the same pipe (fd[1]), and that their messages are then merged and read by the parent (from fd[0]). Despite the fact that the parent is totally unaware that its original child has forked!

Of course this does not proof that under Windows pipes work like this, but when these MicroSoft guys steal an idea the usually steal it entirely...

[mridul]

You also can't split a pipe such that either one process or another may write into it. (Neither on Windows nor on UNIX, and this was also never possible in the past.)

I most emphatically disagree. I have the UNIX programmer's manuals in front of me, (alas, a paper version...) and they clearly state that on a fork() the child inherits all open file descriptors. No exception is made for file descriptors that refer to a pipe.

In the section on pipes says that reading an (empty) pipe produces an error if all write file descriptors on that pipe are closed, a remark that would make no sense if their could be only one process writing the pipe.

And finally, because the proof of the pipe is in the smoking, I can run the following program under Cygwin (the Unix emulator under Windows) compiled with gcc:

Code: Select all

#include <stdio.h>

main()
{
        int i, fd[2];
        char buf[100];

        pipe(fd);
        if(fork())
        {
                printf("I am the parent\n");
                sleep(10); /* wait until offspring is done */
                read(fd[0], buf, 31);
                buf[31] = 0;
                printf("message from my grandchildren:\n%s\n", buf);
        } else
        {
                printf("I am the child\n");
                if(fork())
                {
                        printf("I am Alice\n");
                        write(fd[1],"Alice says Hi!\n", 15);
                        printf("Alice exits\n");
                } else
                {
                        printf("I am Bob\n");
                        write(fd[1],"Bob says Hello!\n", 16);
                        printf("Bob exits\n");
                }
        }
}


Running it shows clearly that two processes (Alice and Bob) can write to the same pipe (fd[1]), and that their messages are then merged and read by the parent (from fd[0]). Despite the fact that the parent is totally unaware that its original child has forked!

Of course this does not proof that under Windows pipes work like this, but when these MicroSoft guys steal an idea the usually steal it entirely...

Hi,

In *nix world , what I am asking for is the natural way to code - you inherit the fd's , so your child processes will write to the same fd's.

The order in which it get sent to the other end depends on the actual order in which it was written , etc - I handle those : so my best move will not come from multiple processes , etc... common everyday parallel programming issues these , we will neglect these since I handle them.

My guess is that console applications in windows are not really non-gui applications.
My guess is , they are by default attached to a console (window ?) or something like that and when you spawn a process , this gets poped up when the parent did not have a console .... ofcourse , wild guesses these , so please do correct me , but dont flame me

Thanks,
Mridul

[mridul]

Hi,

Just found this method : AllocConsole.
According to it :
"This function is primarily used by graphics applications to create a console window. Graphics applications are initialized without a console. Console applications are normally initialized with a console, unless they are created as detached processes (by calling the CreateProcess function with the DETACHED_PROCESS flag). "

Tried with 'DETACHED_PROCESS' , and voila !

Thanks for all the suggestions , comments , help - got me thinking in the right direction finally

Regards.
Mridul

[H.G.Muller]

Apparently the Windows CreateProcess call does a lot more than a simple UNIX fork().

So you know how to suppress the console now, but do both engine processes now indeed merge their output into the same pipe, with Winboard listening at the other end?

[mathmoi]

Apparently the Windows CreateProcess call does a lot more than a simple UNIX fork().

So you know how to suppress the console now, but do both engine processes now indeed merge their output into the same pipe, with Winboard listening at the other end?

CreateProcess does not fork() the actual process, but actually create a new process. This new process can be from the same program as the one launching it or it can be a completly different one (ie. an other .exe file).

I think that Windows does not offer any API comparable to fork().

MP

[mridul]

Apparently the Windows CreateProcess call does a lot more than a simple UNIX fork().

So you know how to suppress the console now, but do both engine processes now indeed merge their output into the same pipe, with Winboard listening at the other end?

In my specific case the children will start spitting out pvlines only when the root subtree ownership changes from the initial parent to the children and there is a best score change at the root.
Ofcourse , at the end of the iteration , the original parent outputs the pvline again , so there is no loss of functionality if the output is not coming from the children.

That being said , it is working as expected.
The output from the various children are also being read by winboard - the same behaviour as in *nix

Thanks !
Mridul