[Idea] Analyze Mode: Hinting multiple lines and more

[Jonas Thiem]

Is there a way a GUI can request specific moves of an engine in the current position in the analyze mode to be rated? (Instead of just obtaining a hint for a single best one without getting information about any others)

I am thinking about 1.) multiple good moves (not only a single best one) which could be used for some diversity in hinting the user what he could possibly do now 2.) rating a very specific move e.g. for examining the move the user does and warning him if that move is e.g. considerably bad or something.

First idea: extended hint query with blocking and inverse ratings
I guess a possible issue with both is that engines like to cutoff the non-best moves and it might not be possible that easily to obtain ratings for worse moves. But it would be already helpful if the GUI could query the best move while blocking a specific move or multiple ones off (e.g. the previous best move) to get the second-to-best aswell (or the third with blocking the best and second-to-best). Also querying the worst move with inverse ratings for the enemy and then the second worst move with the same blocking technique to get just a bit more information then simply a single "this is what I'd do now" move would be great.

Second idea: extended hint reply to the currently existing simple "hint" request with multiple, ordered/rated answers and also anti-hints
Of course an engine could use this technique just internally to provide multiple hinted moves in order of rating (with probably also a more specific number to indicate the rating falloff) and also multiple anti-hints for warning about bad moves. But then again, the protocol would need to allow such an extended hint reply (multiple answers with ordering/rating and anti-hints).


As I take it, CECP does currently not support this [both the multiple hints response and the extended query with blocking as outlined above]? Or does it in some form (e.g. through the stat01 lines) which I missed? Just for the record, can UCI do something like this (information about more moves than simply the considered best one)?

What do you think about an extension to the simple hint command to allow for inverse ratings (worst move) and blocking one or multiple moves from being examined to also obtain second-to-best ratings and so on? Or an extension that allows the engine to give a more diverse answer to the hint command with multiple entries and ratings and also anti-hints?

--
I'm also thinking about a more diverse rating of a position similar to Fritz does it, e.g. explosiveness/possible potencial of a position or how dangerous it is at the moment (possibly split up in black, white), but I guess that's too specific now, is it? Has anyone ever thought about a system to make up such additional rating values an engine could provide in case the GUI can deal with it?

[H.G.Muller]

This is more a matter of what engines can do, rather than the GUI or the protocol. Some remarks:

The WB hint command is a bit obsolete, and I don't think people nowadays use that anymore to analyze games. The modern way is to use analyze mode, where the computer will be thinking and reporting moves and scores for whoever has the move, without ever playing a move, but in stead allow the user to play and take-back moves on the board.

Usually engines only report a single best move in analyze mode, but many engines can be put in a 'multi-PV' mode, where they report mutiple moves and scores. This is purely a matter of the engine, that does not require any protocol extension, and hardly any GUI support. The GUI just displays what the engine decides to send it. Switching on muti-PV, and requesting the number of PVs or the score threshold are usually engine-specific options, that can be set through the GUI engine-settings dialogs. Fruit and Glaurung are examples of UCI engines supporting multi-PV, Fairy-Max and Gaviota are examples of native WinBoard engines that do.

UCI has a command to forbid certain moves, so that a search (single or multi-PV) would report on the best of the remaining moves. However, if you want to know the score of a move that is very late in the ranking (e.g. because you don't see why it is so bad, and would like to know), it is much more convenient to just play that move in analyze mode, and get the score reported for the opponent (which you then would have to flip in sign to know your own score in the previous position), rather than somehow having to enter all moves that you want not played.

Asking for the worst move does not seem generally useful, as it will almost always come up with some trivial way to blunder away your Queen. In the rare case you want it, you can simply use multi-PV mode asking for 256 PVs or a threshold of 100,000 centiPawn, then you will get sores for all moves,and can easily conclude which is the best. Engines woud likely implement a 'worst-move' command like this anyway, I cannot imagne they would go to the trouble to implement an entirely different search algorith for finding worst moves, just to implement such a command. (More likely they would not implement it at all...)

[Giorgio Medeot]

UCI has a command to restrict the search to specific moves too. You can send "searchmoves <a list of moves>" along with the "go" command, IIRC.

Giorgio

[Jonas Thiem]

Usually engines only report a single best move in analyze mode, but many engines can be put in a 'multi-PV' mode, where they report mutiple moves and scores.

Then is there a documentation of that multi-PV output somewhere?
Also, I think you are wrong on one thing, which is:

This is more a matter of what engines can do, rather than the GUI or the protocol.

I think this is wrong because if the GUI can actually activate that analysis itself (e.g. through an xboard protocol command rather than a GUI option that is only understood by the user) and interpret it itself, it can create very useful graphical representations (e.g. multiple recommended moves and things) on the board that are much more helpful to the beginner than just that cryptic multi-PV output in a text box.

Also, standardizing it would help to make that semi-standard more streamlined and help both new engine developers and GUI developers to adapt this feature better.

Asking for the worst move does not seem generally useful

This is probably true, now as I think about it.

I really think xboard should standardise this in a way similar to UCI does (which specifies an exact syntax for analysis lines if I remember that right?) for engines that support it to ease automatic use through the GUI and also working this into graphical graphs.

..., I cannot imagne they would go to the trouble to implement...

Just a note on that: Since I'm developing a custom engine to be shipped with the GUI, I won't have that problem. I would strongly prefer some standardised way of using this multi-PV analysis through CECP though (with a proper feature to indicate the capability of the engine) instead of some hacking that doesn't work with all engines (because the GUI option is named differently, not present, or the output differs, etc) and would only properly work with e.g. my own and a limited set of other engines which I manually added special cases for.

Since CECPv2 allows extensions so easily, have you ever considered writing an extension for this? (some "multipv=1" feature that will then allow the GUI to query multi-PV through "multipv-centipawns <centipawns>" or "multipv-count <amount of PV lines>" or something like that from the engine with a standardised response line) One could also easily look a bit at UCI here.

[H.G.Muller]

I have considered that feature, but when I tried starting to implement it, I realized that there was really nothing that would change the behavior of the GUI depending on multipv=0 or multipv=1. The main reason to standardize options is to be able to bulk-enable or adjust them for a tourney. E.g. setting the hash size: I would want that to apply to all engines, rather than having to change the setting for each and every engine separately, when I decide I want to run with another hash size.

Multi-PV is not of that nature. It is not used in tournaments. I don't see much benifit in a centralized knob to control it. You are likely to use it only on a few engines that you installed especially for analysis. And the most natural dialog to have the control for it would be in the engine-specific option settings. Which is just where it would appear if it was an engine-defined option. If there is a need to standardize, we could of course standardize the name of this engine-defined option, so engines supporting it could announce it as featur option="multiPV -spin 0 100 1"

The thinking output of engines is already precisely prescribed in WB protocol.

I am not sure what you mean by "the GUI could activate this by itself". How can a GUI possibly know how many PVs the user wants to see? The user would somehow have to announce that it wants to see more than one. The fact that the engine supports multi-PV does not mean that we want it to be switched on. In fact we hardly do.

[Jonas Thiem]

I am not sure what you mean by "the GUI could activate this by itself". How can a GUI possibly know how many PVs the user wants to see?

The entire purpose of this would be graphical indications on the chessboard displayed by the GUI for different moves or being able to suggest more than just one good move to the user. Just displaying the engine output as it floats in is the basic "go read it yourself"-approach, but to make the nice board highlights for extended playing help and an easy overview over a position I simply need more than just a single hint.

so engines supporting it could announce it as featur option="multiPV -spin 0 100 1"

That would already be helpful. Maybe there should be two ways for either setting it per centipawn and the amount of moves to be shown or something like that.

The fact that the engine supports multi-PV does not mean that we want it to be switched on

Probably not you, but the GUI wants to colour some fields on the board based on the output, no matter if you are atm willing to look at engine output or not (this is rather for internal processing by the GUI than for being dumped into an engine analysis box without any filtering). This can be still treated and filtered differently. I simply want to go further than the usual dropping engine output into a text box approach, and for that more standardisation might be really helpful to avoid breakage with other engines.

The GUI could still filter out just the best-move line if you don't want extended analysis output in your engine window, if there is any.

To me, beginner-friendly board colouring and having that analysis data in the GUI to do something with it (e.g. create nice graphs etc) is more important than the engine output I could possibly dump into a text box as I personally never found that engine output that easy to read and want to create a different experience by processing the data myself. I can still apply GUI switches afterwards to control which of the processed output the user actually wants to see.

The main feature I think of is if you open some game to analyse it with the GUI, it will automatically use this multi-PV approach to obtain useful data to e.g. display many nice moves starting from a position, or during an actual game, if you activate the beginners help feature that would suggest *multiple* good moves to the user (e.g. by showing unobtrusive transparent arrows on the board during your turn). Requiring to search the engine options and then actually understanding what "multi-PV" even means to turn it on involves quite some user knowledge beginners might not have. Also that universal engine options dialog wants to be implemented yet. During normal tournaments without being in the GUI's analysis screen or activating extensive help for beginners, that multi-PV wouldn't be constantly activated obviously.

I realized that there was really nothing that would change the behavior of the GUI depending on multipv=0 or multipv=1.

In my case the GUI would, since it is actually processing that output in case it needs it. There should be an explicit command to activate/deactivate it DURING a game though to react to a user switching on/off e.g. the extensive beginners' help described above. Also some standardized syntax for the multi-PV lines as being sent by the engine would be nice (I guess Winboard/CECP engines like to differ there?).

[Jonas Thiem]

To make my intention even more clear, as soon as I read up more on that multi PV output and checked out how it looks for other engines I will add this to my own little engine and then process this with the GUI, no matter if this turns out being implemented specifically in the CECP protocol,

but if it does I could allow the user to use practically any supporting engine for analysis and not just the few for which I tested that automatic multi-PV output control which the GUI would base the features described above on. That would simply seem better and more universal than an approach that works e.g. with my engine or just crafty but not with most other engines out there because they differ in ways to activate it and in their specific output.

[H.G.Muller]

Somehow I get the impression you think this multi-PV information is free. But it is far from that. Asking 5 PVs from a multi-PV engine efectively makes it 4-5 times slower, becausee it now has to distribute its search time over 5 moves, rather than concentrate it on a single move. So it is not a matter of always letting the engine provide the information, and filtereing in the GUI what you want to display. The engine must really know is it has to spend all this effort on the extra moves, or not.

Multi-PV really is a very specialized feature of an engine, that happens to generate information that can already be perfectly transmitted through existing protocol. If you would like to have multi-PV output from an engine that is not programmed to generate it, in principle the GUI could evoke the information from the engine by just forcing all possible moves into it, and obtain scores and PVs ffor them by analyzing the daughter posititions. This would be tantamount to moving part of the search process to the GUI, putting a much larger burdon on the latter. But this can be perfectly done with the existing protocol.

I am not sure how you imagine a GUI should display multiple 'move hints' other than as text. Currently the only way WinBoard displays moves is when it actually makes them on the board. Having it make several moves at the same time does not seem an option. Indicating several moves with arrows would be rather messy, and raise the problem how you would know which is best, which second, etc. And even if you would, say, color-code that, it would still be of limited help if you have no idea about the score difference between first and second.

[Jonas Thiem]

Short version:
I think that criticism of how this is actually useful or if I understand it quite right is a bit unrelated to the actual question whether it should be added to CECP or not. As I said in my post above:

I will add this to my own little engine and then process this with the GUI, no matter if this turns out being implemented specifically in the CECP protocol,

but if it does I could allow the user to use practically any supporting engine for analysis and not just the few for which I tested that automatic multi-PV output control which the GUI would base the features described above on.

I just hope that there will be standardization for this so I don't need to go with own, specific hacks that won't work for the majority and another GUI dev goes for his hack, another one for his etc. given other people also think about doing this.

In the end I think that analysis information is nice as an optional feature and shows how diverse a game is (or how clear a situation if there's just one good move) and nobody says WinBoard/XBoard needs to add this (and displaying it with multiple arrows which are thicker or opaque for good, thinner/transparent for worse moves is possible and IMHO very clear and obvious, much more than the cryptic engine multi-pv output).

I just need to obtain some information from the engine and a standard in the CECP specs (not necessarily implemented in WinBoard/XBoard!) would be helpful (and keep others from coming up with parallel hacks that do the same and work differently, leading to incompatibilities).
-----
Long version:

Asking 5 PVs from a multi-PV engine efectively makes it 4-5 times slower, becausee it now has to distribute its search time over 5 moves

Well... that's technical details now. For today's engines, it won't hurt them too much if they get a bit slower giving more diverse advise to a beginner (optional! for people that want to use it!).

For the usefulness: It also is some sort of indication of how dangerous/explosive a position is since if you got many good moves, it's rather calm/open and otherwise rather a closed, cut thing that needs to be dealt with care by the player for not missing out on an advantage and have the game turn. Some people (including me) might consider that sort of analysis interesting - nobody says everyone needs to be using this now, I just want to add this as an optional feature to my GUI.

For your criticism of ways of displaying (although I don't see how this is related to this protocol request?): well, arrows can have thickness, differing transparency etc to indicate various move ratings on the board.

Again, this is intended to be an optional GUI feature I want to add to my own GUI where people can activate it if they want to, or not do it. I am really not asking you to include this feature into XBoard/WinBoard! I am just seeking for a standardized protocol addition to the CECP specs for automatic multi-PV querying so it will possibly work with more engines than some hacking on my own that will work with just a few. At least to me, this doesn't mean that XBoard/Winboard actually needs to implement this (since it'd just be an extension that engines offer which support it).

in principle the GUI could evoke the information from the engine by just forcing all possible moves into it, and obtain scores and PVs ffor them by analyzing the daughter posititions. This would be tantamount to moving part of the search process to the GUI, putting a much larger burdon on the latter.

This is a good idea. Still, I think this is probably even slower than if the engine knows how many moves to search for with appropriate cutoff/pruning in one go?


Since your last post didn't seem to include much apart from criticism of how this could be just of any use to anyone (apart from that useful proposal of how to enforce it for non-multi-PV engines), maybe we should just stop that discussion:

As I indicated I'll add it to my GUI somehow with possibly making up such a "multipv=1" extension that is used by my engine and it finally is up to you whether you want to standardize it in CECP.

I simply made this thread because I'd really greatly appreciate a standardization and consider that a much better idea than a lonewolf solution just for my GUI that works with some, doesn't work with others and is just watering the CECP protocol instead of using a cleanly standardized way that everyone may use. Custom protocol variations to do the same thing just aren't that funny and make you end up with lots of incompatibilities (which I'm already enough used to in regards of the Internet Relay Chat protocol). Therefore I simply thought it would be a good idea to ask here for an "official" way of doing this which could possibly written as a CECP specs addition.

If you ever come around to standardize, be sure I will use that standard to get this as smoothly working for as many engines as possible.

If the reason for non-inclusion is the additional work needed for this I can also make up a more complete proposal for this if it helps.

If you don't want to add this to XBoard/WinBoard, that's just fine. That really wasn't what I asked for (I am just voting for a standardized way of doing this in the CECP protocl specs for those people that want to implement it, e.g. me).

[H.G.Muller]

Well, I am still at the stage of understanding what exactly you want to implement, to see if the current protocol already supports this, or if extensions are called for, and if so, which.

It seems that for the engine->GUI data flow the current protocol can easily handle everything that is required. An engine is free to send as many PVs at any depth as it wants, and in fact many engines do exactly that, even when in normal (single-PV) mode, when they happen to find non-best PVs that are later overturned. This is all the GUI needs to make fancy displays of alternatives, and how exactly the user can control what is displayed is a matter between GUI and user, not concerning the protocol.

So the only matter is how the engine behavior should be controlled from the GUI. One problem here is that not all engines implement multi-PV in the same way. In some you can request a fixed number of PVs, in some you can set a threshold, in others you can do both. I don't think it is a good idea to restrict what engines are allowed to do by forcing to adopt some standardization of limited capability, as there could be valid reasons for doing it the way they do it. Not everyone uses engines for the same purposes, and the protocol must cater to all possible needs. For this reason I think it would be a bad idea to have a single feature multipv=1 that would then force the engine to obey both the setting of a margin and the requesting of a fixed (or maximum, or minimum) number of PVs. Note that in general a GUI would be capable of concluding the engine is in multi-PV mode by looking at this thinking output: the most common implementations of multi-PV would first find the best move at a new depth, and then follow it with a number of lines with lower score, while in single-PV mode engines that send multible lines will only show increasing scores.

It still seems to me that the best way to standardize this is to standardize the name of an engine-defined option. That allows maximal (future) flexibility for the engine developer, and if an engine developer thinks of a clever new mode to select and produce alternative lines (e.g. show "avoid moves"), he could add its own engine-defined option for it that would to the user apper on par with the other multiPV settings. If the names for a few such options tht commonly appear in engines would be standardized, a specialized multi-PV GUI could check for their presense, and use them to its advantage. So I would propose to use spin options with the name multiPV or multi-PV for indicating a fixed number of PVs, and multi-PV margin for setting a score range (relative to the best move) in which all PVs have to be calculated.

To solve the efficiency issue when a GUI would try advanced manipulations to emulate multi-PV on an engine that does not have it, it would be useful to add a new command to the protocol that could set an aspiration window. E.g.

window ALPHA BETA

could set a window to be used for the next search that is to be started (so it would typically be sent just before forcing in a move or giving an undo command in analyze mode), instructing the engine not to resolve any fail low or fail high w.r.t. this window.

[Jonas Thiem]

For some reason the forum probably ate my post although I was pretty sure I have already seen it in the thread view meh. Maybe I forgot to press submit or something >.>

*snip* gah, forget what I wrote previously here - standardizing some basic multi-pv GUI options would already be sufficient and pretty neat! I say go for it

To solve the efficiency issue when a GUI would try advanced manipulations to emulate multi-PV on an engine that does not have it, it would be useful to add a new command to the protocol that could set an aspiration window. E.g.

window ALPHA BETA

I wonder if any engine dev would actually implement this instead of just doing a full multi-pv output right away, but would certainly help to solve issues with those that don't go for multi-pv (given they still care about this command then if they don't)