Hi,
I would like to make a few suggestions for the next draft of the GTP version 2 spec. Just my $0.02:
a) For consistency reasons, is it possible to rename showboard to show_board? There is already clear_board.
b) In section 6.2 Protocol Subsets, could we add subsection 6.2.3 Debug? That would be: --------------------------------------------- 6.2.3 Debug The debug subset adds the commands: show_board ---------------------------------------------
c) In subsection 6.2.2 Regression, I would suggest the addition of: get_random_seed, set_random_seed Both commands are already implemented in GNU Go and I assume most, if not all engines would introduce a degree of randomness based on a seed. An engine that does not use pseudo-randomness (?) should output 1 for the get_random_seed command and save the parameter of the set_random_seed command in an otherwise unused variable, returning it if requested by the get_random_seed command.
d) For the showboard (or show_board) command, I would like to suggest the following description: * show_board arguments none effects none output the board in human-readable format. See Gnu Go output for example. fails board not cleared comments This command should fail if sent before the board is properly initialized (i.e. using the clear_board command).
e) I had some doubts about whether Wallyplus should output the string "pass" or "resign" when it wants to pass. In the end, I implemented "pass", even though I was under the impression that the present spec is "resign". I would be grateful if this issue could be clarified (in the genmove command description).
f) Is there a command that tells GNU Go to **explain** why it generated a specific move? Could such a command be included in the protocol? That would be in the Debug command subset, I believe. It could be something like: * comment_last_move arguments none effects none output reason, explanation or comments about the last move generated by the engine. fails no move generated comments This command should fail if sent before the engine is asked to generate any move. Returns an empty string if the engine cannot or does not want to comment its move.
I guess that last suggestion could be the matter of some debate, so I would like to explain why I am making it. Wallyplus can output a few lines of comments for every move it generates. I could save these comments in a string variable and just output them as a reply to the above command. It could prove quite useful to understand why an engine makes a particular move, without actually using a debugger while the engine is playing.
Again, these are just my $0.02. Please take with a ton of salt, flames to /dev/null, etc.
Greetings,
Andrew wrote:
a) For consistency reasons, is it possible to rename showboard to show_board? There is already clear_board.
I guess it's possible but I think I prefer to stay with the GTP version 1 name here.
b) In section 6.2 Protocol Subsets, could we add subsection 6.2.3 Debug? That would be:
6.2.3 Debug The debug subset adds the commands: show_board
It doesn't seem very useful to formalize this as a protocol subset. The idea with these subsets is that an application should be able to say "requires the GTP tournament subset" to quickly declare what commands it needs. An application definitely shouldn't need a debug command in the first place.
c) In subsection 6.2.2 Regression, I would suggest the addition of: get_random_seed, set_random_seed Both commands are already implemented in GNU Go and I assume most, if not all engines would introduce a degree of randomness based on a seed. An engine that does not use pseudo-randomness (?) should output 1 for the get_random_seed command and save the parameter of the set_random_seed command in an otherwise unused variable, returning it if requested by the get_random_seed command.
I'm sceptical about this. First because I doubt that all engines use a scalar random seed like GNU Go does. Second because programs with a more sophisticated time handling than GNU Go probably has other, hard to control, sources of indeterminism.
Furthermore I'm not sure what purpose you see for these commands. Apparently you intend them for regression use but for reg_genmove it's advisable to turn off all random variations anyway, in order to avoid meaningless regression fluctuations.
d) For the showboard (or show_board) command, I would like to suggest the following description:
- show_board
arguments none effects none output the board in human-readable format. See Gnu Go output for example. fails board not cleared comments This command should fail if sent before the board is properly initialized (i.e. using the clear_board command).
I see no reason for this command to fail before clear_board has been invoked. Although the board configuration is unknown per the specification there is no reason for the engine to initialize it to something that's internally invalid. For your own sanity I'd recommend that you start up with an empty board. This is what GNU Go does, unless command line options tell it to load an sgf file before starting. My draft (not yet published) for the command looks like this:
* showboard arguments none effects none output board string*& board - A diagram of the board position. fails never comments The engine may draw the board as it likes. This command is only intended to help humans with debugging and the output should never need to be parsed by another program.
e) I had some doubts about whether Wallyplus should output the string "pass" or "resign" when it wants to pass. In the end, I implemented "pass", even though I was under the impression that the present spec is "resign". I would be grateful if this issue could be clarified (in the genmove command description).
Pass is correct. In section 3.2.1 a vertex entity is defined as a board coordinate or "pass", so I thought this would be clear. Of course I can add a comment in the command description if that will help. The purpose of "resign" is to give up the game.
f) Is there a command that tells GNU Go to **explain** why it generated a specific move?
No such command has been implemented. There is a command top_moves to list the top ten moves and their move values, but that's not the same thing.
Could such a command be included in the protocol?
Seems more appropriate for private extensions, I think.
I guess that last suggestion could be the matter of some debate, so I would like to explain why I am making it. Wallyplus can output a few lines of comments for every move it generates. I could save these comments in a string variable and just output them as a reply to the above command. It could prove quite useful to understand why an engine makes a particular move, without actually using a debugger while the engine is playing.
In GNU Go this kind of information is available as trace and debug messages (controlled by a couple of command line options), written to stderr.
/Gunnar
I definitely see the point of having control over the randomness, but I personally think it should be external to the protocol.
In my programs, I have a configuration parameter that sets whether the program is deterministic or not. I'm nitpicking, but a random seed implies a particular implementation, but setting "determinism on" is more general.
Don
c) In subsection 6.2.2 Regression, I would suggest the addition of: get_random_seed, set_random_seed Both commands are already implemented in GNU Go and I assume most, if not all engines would introduce a degree of randomness based on a seed. An engine that does not use pseudo-randomness (?) should output 1 for the get_random_seed command and save the parameter of the set_random_seed command in an otherwise unused variable, returning it if requested by the get_random_seed command.
I'm sceptical about this. First because I doubt that all engines use a scalar random seed like GNU Go does. Second because programs with a more sophisticated time handling than GNU Go probably has other, hard to control, sources of indeterminism.
Furthermore I'm not sure what purpose you see for these commands. Apparently you intend them for regression use but for reg_genmove it's advisable to turn off all random variations anyway, in order to avoid meaningless regression fluctuations.
Dear Don,
Thanks a lot for your comments.
I still have to reflect on this issue and yours and Gunnar's comments. Some reflections:
For example, a Go engine that would "learn" to never repeat its mistakes should, in a very deterministic way, never play exactly the same moves if they lead to a loss. So in this case even if I set "determinism on", I'll still get regression fluctuations - and that's exactly what would be expected of the engine in this case. Those would be "desirable" regression fluctuations.
And of course I cannot expect all programs to have a scalar random seed, or even to include randomness at all, in fact, as pointed out by both you and Gunnar. Immediate example: AmiGo does not have randomness.
So I guess that Gunnar is 100% correct: the two suggested commands are not "standardizable".
On Saturday 05 October 2002 01:58, Don Dailey wrote:
I definitely see the point of having control over the randomness, but I personally think it should be external to the protocol.
In my programs, I have a configuration parameter that sets whether the program is deterministic or not. I'm nitpicking, but a random seed implies a particular implementation, but setting "determinism on" is more general.
Don
c) In subsection 6.2.2 Regression, I would suggest the addition of: get_random_seed, set_random_seed Both commands are already implemented in GNU Go and I assume most, if not all engines would introduce a degree of randomness based on a seed. An engine that does not use pseudo-randomness (?) should output 1 for the get_random_seed command and save the parameter of the set_random_seed command in an otherwise unused variable, returning it if requested by the get_random_seed command.
I'm sceptical about this. First because I doubt that all engines use a scalar random seed like GNU Go does. Second because programs with a more sophisticated time handling than GNU Go probably has other, hard to control, sources of indeterminism.
Furthermore I'm not sure what purpose you see for these commands. Apparently you intend them for regression use but for reg_genmove it's advisable to turn off all random variations anyway, in order to avoid meaningless regression fluctuations.
gtp mailing list gtp@lists.lysator.liu.se http://lists.lysator.liu.se/mailman/listinfo/gtp
Hello Gunnar,
Thanks for taking the time to answer my beginner's questions and sharing your experience. Here are some comments to your detailed reply:
On Friday 04 October 2002 23:46, Gunnar Farnebäck wrote:
Andrew wrote:
...
b) In section 6.2 Protocol Subsets, could we add subsection 6.2.3 Debug? That would be:
6.2.3 Debug The debug subset adds the commands: show_board
It doesn't seem very useful to formalize this as a protocol subset. The idea with these subsets is that an application should be able to say "requires the GTP tournament subset" to quickly declare what commands it needs. An application definitely shouldn't need a debug command in the first place.
I intend to write a nice KDE GTP 2 client, which everybody will be able to use to debug their own engines. In such a situation, I can clearly see that my application will want to say: "requires the standard GTP debug subset".
And IMHO an experimental Go engine definitely does need debug commands. Perhaps I am not making myself clear: this is not to debug programming errors, this is actually to debug/optimize/fine-tune the engine's algorithm.
So why not include such commands in the standard?
c) In subsection 6.2.2 Regression, I would suggest the addition of: get_random_seed, set_random_seed Both commands are already implemented in GNU Go and I assume most, if not all engines would introduce a degree of randomness based on a seed. An engine that does not use pseudo-randomness (?) should output 1 for the get_random_seed command and save the parameter of the set_random_seed command in an otherwise unused variable, returning it if requested by the get_random_seed command.
I'm sceptical about this. First because I doubt that all engines use a scalar random seed like GNU Go does. Second because programs with a more sophisticated time handling than GNU Go probably has other, hard to control, sources of indeterminism.
Furthermore I'm not sure what purpose you see for these commands. Apparently you intend them for regression use but for reg_genmove it's advisable to turn off all random variations anyway, in order to avoid meaningless regression fluctuations.
I think I need more time to think about the (in)determinism issues. But yes, I agree that these commands do not make sense in the standard, due to the variety of ways a program can deal with randomness/determinism. Thanks.
d) For the showboard (or show_board) command, I would like to suggest the following description:
- show_board
arguments none effects none output the board in human-readable format. See Gnu Go output for example. fails board not cleared comments This command should fail if sent before the board is properly initialized (i.e. using the clear_board command).
I see no reason for this command to fail before clear_board has been invoked. Although the board configuration is unknown per the specification there is no reason for the engine to initialize it to something that's internally invalid. For your own sanity I'd recommend that you start up with an empty board. This is what GNU Go does, unless command line options tell it to load an sgf file before starting. My draft (not yet published) for the command looks like this:
- showboard
arguments none effects none output board string*& board - A diagram of the board position. fails never comments The engine may draw the board as it likes. This command is only intended to help humans with debugging and the output should never need to be parsed by another program.
OK, but now I have another question: I have noticed that in response to the showboard command, GNU Go sends its output to stderr, not to stdout!!!
Wallyplus outputs the board to stdout.
I had earlier understood that the standard specified that the showboard command would cause the engine to output the board to stdout, in the normal response stream.
What is the correct behaviour?? Output to stderr or stdout?
e) I had some doubts about whether Wallyplus should output the string "pass" or "resign" when it wants to pass. In the end, I implemented "pass", even though I was under the impression that the present spec is "resign". I would be grateful if this issue could be clarified (in the genmove command description).
Pass is correct. In section 3.2.1 a vertex entity is defined as a board coordinate or "pass", so I thought this would be clear. Of course I can add a comment in the command description if that will help. The purpose of "resign" is to give up the game.
OK, thanks for the clarification.
f) Is there a command that tells GNU Go to **explain** why it generated a specific move?
No such command has been implemented. There is a command top_moves to list the top ten moves and their move values, but that's not the same thing.
Could such a command be included in the protocol?
Seems more appropriate for private extensions, I think.
I guess that last suggestion could be the matter of some debate, so I would like to explain why I am making it. Wallyplus can output a few lines of comments for every move it generates. I could save these comments in a string variable and just output them as a reply to the above command. It could prove quite useful to understand why an engine makes a particular move, without actually using a debugger while the engine is playing.
In GNU Go this kind of information is available as trace and debug messages (controlled by a couple of command line options), written to stderr.
I guess again I refer to the case where I want a GUI client to communicate with the engine. It seems practical to have a standard command that will cause the engine to "explain" its last move.
When dealing with a new engine, I just would turn on this option in my GUI client and would get those explanations in a window; that would be very helpful to understand the logic of this new engine.
I have just finished rewiting AmiGo to play using GTP V2. Similarly to Wallyplus, I have called the new program Amigoplus. I have reused most of the front-end written for Wallyplus, and in fact I still have little understanding of the engine itself. So the debug commands would be useful in such a situation.
Again, I am very thankful to you and all that have worked on the GTP. It is a nice protocol to work with.
Regards,
Andrew wrote:
I intend to write a nice KDE GTP 2 client, which everybody will be able to use to debug their own engines. In such a situation, I can clearly see that my application will want to say: "requires the standard GTP debug subset".
And IMHO an experimental Go engine definitely does need debug commands. Perhaps I am not making myself clear: this is not to debug programming errors, this is actually to debug/optimize/fine-tune the engine's algorithm.
Actually, the primary thought behind including showboard is that it should come in handy when debugging gtp communication problems.
The kind of commands you want may be better to call diagnostic, or something like that.
So why not include such commands in the standard?
That's not unreasonable, but I'm somewhat doubtful that different engines have sufficiently similar internals for it to be meaningful. Also, as Don pointed out, this is not the right time to add such commands.
Still I would encourage you to do experiments with a GUI and multiple engines to find out how well diagnostic commands work in practice. This may well be stuff for a future extension of the protocol.
OK, but now I have another question: I have noticed that in response to the showboard command, GNU Go sends its output to stderr, not to stdout!!!
Wallyplus outputs the board to stdout.
I had earlier understood that the standard specified that the showboard command would cause the engine to output the board to stdout, in the normal response stream.
What is the correct behaviour?? Output to stderr or stdout?
The simple answer is that in GTP version 2 stdout is correct. The original GNU Go implementation was to output to stderr, mainly because that was easiest to implement, but also because it sufficed for our purposes. If you are observant you may notice that if stderr and stdout are merged, the output is not syntactically correct, but on stdout the output is a correct empty response. Current GNU Go development versions still use GTP version 1, but this will change before 3.4.
The somewhat more complex answer is that the output should be over the GTP communication channel, but it's up to the program whether it communicates through stdout, stderr, a socket, or whatever. For a typical unix program, though, using stdin/stdout would usually make most sense.
/Gunnar
Dear Gunnar,
Thanks a lot, your answers really makes sense. Comments follow:
On Wednesday 09 October 2002 01:07, Gunnar FarnebXck wrote: ...
The kind of commands you want may be better to call diagnostic, or something like that.
Yes, that is correct. ...
Still I would encourage you to do experiments with a GUI and multiple engines to find out how well diagnostic commands work in practice. This may well be stuff for a future extension of the protocol.
OK, will do! My point is that for heuristic engines like GNU Go, Wallyplus, Amigoplus or Baduki, it is interesting to know the exact heuristic used by the engine to generate its move. And it would be nice to have a GTP command for this purpose, so that this information can be made available to the opponent (when the engine is playing against a human) or observer (when two engines are playing against each other) or programmer of the engine (in a regression testing situation).
For a non-heuristic engine (e.g. Neurogo), I still believe that it would be interesting to know in some way how the engine generated its move, although the answer will not be of the form: "Using rule XXX to generate move". ...
What is the correct behaviour?? Output to stderr or stdout?
The simple answer is that in GTP version 2 stdout is correct.
Aaah! OK!
The original GNU Go implementation was to output to stderr, mainly because that was easiest to implement, but also because it sufficed for our purposes. If you are observant you may notice that if stderr and stdout are merged, the output is not syntactically correct, but on stdout the output is a correct empty response.
Yes, I did observe that somehow and I was curious about it. That prompted my question.
Current GNU Go development versions still use GTP version 1, but this will change before 3.4.
OK, that is good news, in the sense that it would be useful to have a GTP V2 reference implementation.
The somewhat more complex answer is that the output should be over the GTP communication channel, but it's up to the program whether it communicates through stdout, stderr, a socket, or whatever. For a typical unix program, though, using stdin/stdout would usually make most sense.
Yes, I have understood also that GTP does not specify the streams through which the programs should communicate. IMHO, that is a Good Thing. Different environments will need different modes of communication, and specifying e.g. stdin/stdout would restrict the protocol without any advantage.
OTOH, I saw no reason to make one command (showboard) different from all the others, by responding on a different output stream.
Regards,
-
Andrew Derrick Balsa -
Don Dailey -
Gunnar Farneb�ck