This thread started in the GNU Go list but I'm cc'ing the GTP list
because the second part of this post is pertinent to the protocol.
(The first part is more GNU Go specific.)
Trevor wrote:
> >The problem with this is that you must be very selective with what you
> >generate sgf traces for. If you tried to do something like that for an
> >entire genmove (for technical reasons you can't, but let's ignore
> >that) of average complexity you would be completely drowned in
…
[View More]> >variations and get a huge sgf file which would probably crash your sgf
> >viewer or at least render it useless.
>
> I would not want the tree generated for a genmove command, but
> rather, say, for an owl_attack or owl_defend. Perhaps an optional
> parameter with those commands requesting the SGF output?
This could be implemented fairly easily.
But do we really want it?
The sgf files generated by attack and defend --decide-worm can
be enormous. For --decide-dragon we have an a priori bound of
about 1000 nodes, so the file can still be big. Each node gets
a comment, remember.
Once one has identified a reading or owl mistake one starts
to tune, and the methodology is:
(1) Generate sgf file with variations;
(2) Examine file, try to fix something. Still broke? Go back to (1).
(3) Now that you've fixed it, run the regressions and see what else
you broke.
If you add an owl defense pattern, you may have to add some
attack patterns too.
For this process it is probably easier to generate the sgf file
from the command line than from the gtp.
>>> 2) Relatedly, how about enhancing loadsgf to accept the SGF file
>>> directly on the GTP command line, like:
>>> loadsgf (;FF[4]GM[1]SZ[19]AP[Jago]AB[pd][dp][nq]AW[pp][dd])
>>
>>This has been discussed before and I'm strongly against it. One reason
>>is that we want to have external sgf files for the regressions because
>>it allows them to be examined by other tools (sgf viewers) and other
>>GNU Go modes. But most importantly I want to keep the GTP protocol
>>simple and easy to inspect. Inline sgf would not be a development in
>>the right direction.
>>
>>> This would avoid necessity for temp files, and would make a network
>>> based GTP easier to use.
>>
>>The answer is to use native GTP commands to set up the position.
>Right, I agree w/ you now. For example, use boardsize, white, and
>black GTP commands to set up a position.
I think there needs to be a way of transferring the board position
over the GTP without resorting to (potentially) several hundred
calls to black and white.
Currently in GNU Go the command worm_stones is a convenient
way of transferring the board position from the engine to the
controller. What about the other direction?
One possibility is to generalize black and white so that instead
of a single stone they take a sequence of stones.
Dan
[View Less]
I've put up a web page for the GTP at
http://www.lysator.liu.se/~gunnar/gtp
This contains working documents, sample implementations and all other
relevant information. There are probably still some missing items, so
please alert me to errors and omissions on the page. Suggestions for
general improvements are of course also welcome.
/Gunnar
Phil,
I am currently preparing a formal specification of
the grammar (based on your draft) which can be fed
into flex/bison to generate a GTP command parser.
I will integrate this in a very simple application
so that one can get a feeling for the behaviour of
the protocol by manually feeding it commands and
observe the answers. Actually this is already half
way done (error recovery is missing yet); I will
make it available ASAP.
The formal spec perhaps can help to cleanup syntax
issues in the …
[View More]GTP docs as well.
Hellwig
[View Less]
Hi folks,
I joined this mailing list only very recently, so please
be patient with me if I missed one point or another in
the preceding discussions.
I am interested in implementing the Go Text Protocol (GTP)
Version 2, for which I downloaded the draft spec revision 9.
I assume this to be the most recent published one, right?
First of all, thanks for the hard work which already went
into this document. But during my implementation work I ran
into some questions which I want to discuss here.
…
[View More]1. What kind of return (success or failure) does the engine
have to return if it does not implement the command which was
requested? The spec says "..should return 'not_implemented'",
but does not specify the return type (by the way, there is a
typo in the section on failure: it must read "Here the '?'
character..." instead of "Here the '=' character ..."). Only
much later, in the section on error messages, one can guess
that this return must be a failure return.
I propose to phrase this sentence something like that:
"Commands that are not recognized by the engine should return
'not_implemented' in a failure return."
2. The grammar as specified cannot be used to parse commands
because it is ambiguous in several places. Let me give an
example: 'command-arguments' can be a 'command-argument' which
in turn can be 'text'. 'text' consists of 'printable' characters.
The hash "#" is a 'SYM' and therefore a 'printable' character.
This is in conflict with its use as the character which starts
a comment.
By the way, the 'text' "a1" may as well be a 'coordinate' in a
command argument and thus has two different derivations also.
I see two possible solutions:
a) Cleanup the specification so that a parser for commands and
responses can be generated by tools (e.g. flex & bison). I could
do that, in case this is really wanted. This requires some extra
work and I am not sure if it is worth the effort.
b) Drop the attempt to formally specify the protocol. Use
explanations in plain English instead. This is not as bad as
it sounds, and it is used already in some places in the current
spec (e.g., response-message = { depends on the ... }).
3. A few other grammar curiosities:
a) Where is 'WS' (white space) used?
b) "date-time = date | date time" is missing an 'SP' between date
and time. By the way: does 'SP' really mean a single space character?
So one cannot, for example, separate two arguments with two spaces.
Frequently, horizontal tabs ('\t') are also considered "white space".
c) A comment following a command argument must not be preceded by
an 'SP'. Is this really intended?
d) A command which solely consists of a command name may not be
followed by a comment. Really?
4. Another typo:
"The core command set consists of the following eleven (14) commands:"
5. The error messages specified for various commands sometimes
include 'not_implemented' and sometimes they don't. What is the
reason? This has apparently nothing to do with the different
command sets: gen_move is in the core set and nevertheless may
be answered with 'not_implemented'.
By the way: is it spelled "gen_move" or "genmove"?
6.What is the exact format of the answer to 'help'? Specifically,
is the first command returned on the same line as the '=' or on
the next one?
7. It is explicitly stated that coordinates and colors are case
insensitive. This does not hold for "yes"/"no" arguments in
a suicide command, for example, nor for the argument of the
'scoring_system' command. What is the reason?
(One can define this at will, of course, but IMHO consistency
is a desirable goal.)
8. To write a competitive engine, the concept of time has to be
incorporated. Then the human opponent (or an arbiter in case of
a computer tournament) has to fix the timing system. How should
this be communicated to the engine?
(Perhaps this goes under the "Tournament Command Set", or you have
discussed it already; please enlighten me on this subject. The
communication through command-line arguments during engine start-up
is IMHO a non-solution because you open up a second way to transmit
information to the program: the very next question is 'why not send
<insert, e.g., boardsize here> in this way?'.)
Please don't get me wrong: I do not want to exercise fault-finding
on your specification draft. I only tried to implement things and
I think that my experience could help to improve an already well-
written document.
Regards,
Hellwig
[View Less]
Daniel Bump <bump(a)match.Stanford.EDU> writes:
>
> So it is worth reviewing how GNU Go 2.7.253 handles the newlines
> in the GTP.
>
> Newlines are written in gtp.c and play_gtp.c with the standard
> C function printf(" ... \n"). On Unix this makes a LF=\012.
> On DOS it makes CRLF. David stated that on Macintosh it makes
> just a Carriage return but we believe this is not true with
> OS X. If it were, things would be worse broke than they are now.
>
…
[View More]> (Does anyone know?)
FWIW... just discovered that the perlport man page has some discussion on this :
ISSUES
Newlines
In most operating systems, lines in files are terminated by newlines. Just what is used as a newline may
vary from OS to OS. Unix traditionally uses \012, one kind of Windows I/O uses \015\012, and Mac OS uses
\015.
Perl uses \n to represent the "logical" newline, where what is logical may depend on the platform in use. In
MacPerl, \n always means \015. In DOSish perls, \n usually means \012, but when accessing a file in "text"
mode, STDIO translates it to (or from) \015\012.
Due to the "text" mode translation, DOSish perls have limitations of using seek and tell when a file is being
accessed in "text" mode. Specifically, if you stick to seek-ing to locations you got from tell (and no
others), you are usually free to use seek and tell even in "text" mode. In general, using seek or tell or
other file operations that count bytes instead of characters, without considering the length of \n, may be
non-portable. If you use binmode on a file, however, you can usually use seek and tell with arbitrary values
quite safely.
A common misconception in socket programming is that \n eq \012 everywhere. When using protocols such as
common Internet protocols, \012 and \015 are called for specifically, and the values of the logical \n and \r
(carriage return) are not reliable.
print SOCKET "Hi there, client!\r\n"; # WRONG
print SOCKET "Hi there, client!\015\012"; # RIGHT
[NOTE: this does not necessarily apply to communications that are filtered by another program or module
before sending to the socket; the the most popular EBCDIC webserver, for instance, accepts \r\n, which
translates those characters, along with all other characters in text streams, from EBCDIC to ASCII.]
However, using \015\012 (or \cM\cJ, or \x0D\x0A) can be tedious and unsightly, as well as confusing to those
maintaining the code. As such, the Socket module supplies the Right Thing for those who want it.
use Socket qw(:DEFAULT :crlf);
print SOCKET "Hi there, client!$CRLF" # RIGHT
When reading from a socket, remember that the default input record separator ($/) is \n, but code like this
should recognize $/ as \012 or \015\012:
while (<SOCKET>) {
# ...
}
Better:
use Socket qw(:DEFAULT :crlf);
local($/) = LF; # not needed if $/ is already \012
while (<SOCKET>) {
s/$CR?$LF/\n/; # not sure if socket uses LF or CRLF, OK
# s/\015?\012/\n/; # same thing
}
And this example is actually better than the previous one even for Unix platforms, because now any \015's
(\cM's) are stripped out (and there was much rejoicing).
An important thing to remember is that functions that return data should translate newlines when appropriate.
Often one line of code will suffice:
$data =~ s/\015?\012/\n/g;
return $data;
dd
--
dave.denholm(a)insignia.com http://www.insignia.com
[View Less]
Currently, at least with GNU Go there doesn't seem to
be any way to get the engine's opinion about boardsize,
komi, etc.
Normally the gtp controller will know these before the
engine does but there is an exception after the command
loadsgf is executed.
Because of this exception, there should be a way of getting
the boardsize from the engine.
Actually there should be a way of getting the whole position
from the engine but for my purposes just getting the boardsize
will be enough.
There …
[View More]seem to be two obvious ways of doing this:
(1) allow the command 'boardsize' without arguments. The
gtp response is to return the boardsize without setting it.
(2) a new command query_boardsize with obvious functionality:
loadsgf games/nicklas/nicklas9.sgf
= black
query_boardsize
= 9
I've patched GNU Go 3.1.2 to add this command (because I
need it) but I wonder if this is something that is likely to
become part of the protocol or not.
Dan
[View Less]
Well things have been quiet lately, so I think I'll put out a set of
commands for server play to think about. I've been going from Phil's
straw man version of the spec; Phil, you can plug these into the next
revision if you want.
It seems that only 5 more commands are needed. The trickiest thing is
setting up games; and the related problem of specifying the server game
exactly when somebody challenges the program. To do things well, we need
to be able to tell the engine exactly what the …
[View More]time system is. Engines
can choose to ignore this of course.
Having written a client for IGS/NNGS/etc. and for KGS, I'm pretty sure
that these five commands will give us the functionality we need for all
these servers. If anybody has information on dashn, cyberkiwon, etc.,
and thinks this will have to be changed for them, that's no problem.
**********************************************************************
GTP commands for go servers
----------------------------------------------------------------------
Command: play_mode
This is the bridge's attempt to find out whether the go engine is
looking for games, or if it intends to play a particular game vs. a
particular opponent.
Engine Responses:
If the engine is in general looking for games:
any
If the engine is looking to play a particular opponent, with particular
settings:
<color> <user> <handicap> <handicap fixed?> <ko setting> <suicide>
<scoring system> <time system>
Time system format:
<style>[,<main_time>[,<byo-yomi time>,(<stones>|<# of periods>)]]
<style> is one of "none", "absolute", "canadian", "byoyomi"
----------------------------------------------------------------------
Command: game_ok <color> <user> <handicap> <handicap fixed?> <ko setting>
<suicide> <scoring system> <time>
This is the bridge's query to the engine, as to whether a particular game
is acceptable. Note that the arguments are exactly the same as the
response to "play_mode".
Engine Responses:
"y" or "n"
"y" will probably be followed soon by a serios of "komi", "board_size", etc.
commands then finally a "new_game".
----------------------------------------------------------------------
Command: undo_ok
Here the bridge is saying that the opponent wants an undo. The engine can
allow or disallow.
Responses:
"y" or "n"
----------------------------------------------------------------------
Command: query_dead
The game is over. The brige is asking the engine what stones it thinks are
dead. If the engine does not know, then it answering with a not implemented
error is OK...in this case, it will be up to the opponent to decide which
stones are dead.
responses: List all locations on the board that contain dead stones.
----------------------------------------------------------------------
Command: time_left <color> <seconds left in period> [<stones left>]
The server is telling the engine how much time is left on the clocks for
the color specified. The time reported should be the server's idea of how
much time is left, so if there is netlag the engine's clock will not
drift from the server's.
----------------------------------------------------------------------
Command: comment
The server is telling you that somebody is trying to talk to the engine.
Any text in the response will be sent back to the user who was talking.
**********************************************************************
--
Bill Shubert (wms(a)igoweb.org) <mailto:wms@igoweb.org>
http://www.igoweb.org/~wms/ <http://igoweb.org/%7Ewms/>
[View Less]
Thanks, David, now it's working.
After I saw your message I added fflush(stdout) after the printf
to gnugo.
That isn't the first thing I tried but I don't really want to
tell everyone how stupid I was. But now that I've rebooted my
computer ...
This may have fixed the other problem too which was apparently
caused by the EOF that comes down stdin when you type a ^D (one
way to terminate metamachine) not getting flushed. Unclear,
but I don't have a lingering gnugo process this time.
The …
[View More]current (working) version looks like this.
Dan
#include <stdio.h>
#include <unistd.h>
void error(const char *msg);
/* This program uses two pipes to communicate with the
* GNU Go client. To an external client it appears to
* be a gtp engine. It accomplishes this by intercepting
* gtp commands and passing them on to GNU Go.
*
* This program in and of itself is not useful but it
* can be the basis of useful programs. Example: hack
* in gmp.c and get a gtp / gmp translator.
*
* Pipe a: client --> gnugo
* Pipe b: gnugo --> client
*/
int
main()
{
int pfd_a[2];
int pfd_b[2];
char gnugo_line[128], client_line[128];
int length = 0;
FILE *to_gnugo_stream, *from_gnugo_stream;
if (pipe(pfd_a) == -1)
error("can't open pipe a");
if (pipe(pfd_b) == -1)
error("can't open pipe b");
switch(fork()) {
case -1:
error("fork failed (try chopsticks)");
case 0:
/* Attach pipe a to stdin */
if (close(0) == -1)
error("close(0) failed");
if (dup(pfd_a[0]) != 0)
error("dup pfd_a[0] failed");
/* attach pipe b to stdout" */
if (close(1) == -1)
error("close(1) failed");
if (dup(pfd_b[1]) != 1)
error("dup pfd_b[1] failed");
execlp("gnugo", "gnugo", "--mode", "gtp", "--quiet", NULL);
error("execlp failed");
}
/* We use stderr to communicate with the client since stdout is needed. */
/* Attach pipe a to to_gnugo_stream */
to_gnugo_stream = fdopen(pfd_a[1], "w");
/* Attach pipe b to from_gnugo_stream */
from_gnugo_stream = fdopen(pfd_b[0], "r");
while (1) {
int length = 0;
if (!fgets(client_line, 128, stdin))
error("can't get command\n");
if (!strncmp(client_line, "quit", 4)) {
fprintf(to_gnugo_stream, "%s", "quit\n");
fflush(to_gnugo_stream);
return 1;
}
if (fprintf(to_gnugo_stream, "%s", client_line) < 0)
error("can't write command to to_gnugo_stream");
fflush(to_gnugo_stream);
while (length != 1) {
if (!fgets(gnugo_line, 128, from_gnugo_stream))
error("can't get response");
length = strlen(gnugo_line);
printf(gnugo_line);
fflush(stdout);
}
}
}
void
error(const char *msg)
{
fprintf(stderr, "metamachine: %s\n", msg);
abort();
}
[View Less]
Daniel Bump <bump(a)match.stanford.edu> writes:
> Back to Problem 2 from my previous message.
>
> Here's what is happening. Suppose I run:
>
> perl twogtp-a --black "metamachine" --white "gnugo --mode gtp --quiet"
>
> (There are two versions of twogtp in interface/gtp_examples, and
> twogtp-a is a little easier for me to parse.)
>
> I can do this from perldb and as soon as pidb is open I can
> attach gdb to metamachine and/or gnugo. Metamachine stops
…
[View More]> at line 61:
>
> if (!fgets(client_line, 128, stdin))
>
> waiting for a command in stdin. This it should get when
> twogtp-a gets up to line 115 and prints "boardsize 19"
> but at this point deadlock occurs. Note that there is
> no deadlock if metamachine is run from the shell.
One difference is that stdin tends to be fully buffered
unless it is a shell. I'd guess that the stdio library
is being initialised before you drop in your replacements
for fileid 0 and 1 (which is possibly a bad idea..?)
So when you run from the shell, stdin is set to be
line-buffered, and then you drop a pipe into fd 0.
But when it is spawned from twogtp, stdin is set
fully buffered ? Ditto for stdout.
For this sort of application I would tend to use raw
reads and writes, omitting the stdio library entirely.
Forcing (no more than) line buffering in stdio is probably
essential.
>
> I have an email from David referring to man IPC::Open2
> on the dangers of open2. But I can't say I really
> understand why this deadlock occurs or how it can be
> avoided.
The problem is that when you do fprintf(), there is no
guarantee that the data has actually been written out of
the process buffers. So process A thinks it has sent the
command and is blocked waiting for the reply. But the
message is still in A's stdio buffers, and so process
is still blocked waiting to read it. => deadlock.
dd
--
dave.denholm(a)insignia.com http://www.insignia.com
[View Less]