hi,
pike documentation now at your fingertips!
please take a look at the new doc command in hilfe. it is far from perfect, but i hope it is usable.
one flaw that needs fixing is the ability to look at docs from class members without instantiating the class. currently an actual function reference is needed.
if you find other issues, please let me know.
i am not sure how much overhead the added cache_source() will create. but i hope it is acceptable.
actual parsing of docs will only happen when the show_doc() function is used, and then only one class at a time, which should be fast enough for interactive use.
the big thing that is missing is of course the docs from the c code.
i am not yet sure how to handle those, but i think the best option would be to create a compact version of plaintext docs or possibly encode the AutoDoc objects into files, while pike is being compiled. these files could then be distributed with pike as "runtime docs"
formatting of the output might also be improved. right now the docs are displayed as is. for the most part this is readable enough, but so far i have only looked at a few examples.
greetings, martin.
actually, you should be able to do an end run around the whole thing by using the extracted autodoc blob. if there were an xpath implementation, getting the documentation for any method/constant/etc would be a trivial operation. even without xpath, it's still probably easier than reinventing the wheel.
Bill
i am not yet sure how to handle those, but i think the best option would be to create a compact version of plaintext docs or possibly encode the AutoDoc objects into files, while pike is being compiled. these files could then be distributed with pike as "runtime docs"
for the c-docs sure, that's something to look into.
for pike docs the current way has the advantage of working on any class, even stuff that does not exist in files. (consider things like classes in sTeam which are stored in a database, now it makes sense to document them and show the documentation within sTeam.
greetings, martin.
I'd gladly contribute to a bounty for a proper C XPath implementation; I often miss one in pike. It's somewhat unfortunate that libxml2 isn't unicode sane, but the penalty for munging data back and forth through an utf8 codec might not be catastrophical.
Adding another dependent library to Pike core might not be kosher (I'd presume several or most others to be -1 rather than +0), but I would not personally mind libxml2's XPath implementation if there isn't a better one around.
Well, I think you'd probably need to properly define "unicode sane"... libxml2's internal encoding is utf8, which is pretty standard for unix libraries these days.
The public.parser.xml2 glue does auto-conversion of data passed to functions other than parse_(x|ht)ml(), as raw markup strings are supposed to be utf8 unless otherwise specified.
Now, there are compatibility and design differences between my glue and the existing xml code.
I'd gladly contribute to a bounty for a proper C XPath implementation; I often miss one in pike. It's somewhat unfortunate that libxml2 isn't unicode sane, but the penalty for munging data back and forth through an utf8 codec might not be catastrophical.
Adding another dependent library to Pike core might not be kosher (I'd presume several or most others to be -1 rather than +0), but I would not personally mind libxml2's XPath implementation if there isn't a better one around.
because then i'd need to parse the xml and turn it back into text, whereas this way the text for the most part can be displayed as is.
greetings, martin.
but you still don't have any of the c language docs, so why reinvent the wheel? it's not as though you need to parse the xml by hand, right? :)
presumably, you're also parsing the autodoc syntax, which seems like a lot of work compared to using the extracted xml, but that's just my opinion :)
bill
On Mon, 11 Jun 2007, Martin Baehr wrote:
because then i'd need to parse the xml and turn it back into text, whereas this way the text for the most part can be displayed as is.
greetings, martin.
i am not reinventing the wheel, i am doing: docs = Tools.AutoDoc.PikeExtractor.extractClass(source);
that is all. the rest of the code deals with figuring out which docs to actually display. the whole code is actually rather simple.
most of the work was figuring out how to do it to get the desired results.
greetinfs, martin.
but you still don't have any of the c language docs, so why reinvent the wheel? it's not as though you need to parse the xml by hand, right? :)
presumably, you're also parsing the autodoc syntax, which seems like a lot of work compared to using the extracted xml, but that's just my opinion :)
The blob also has the bonus that any references are already resolved.
bill
On Mon, 11 Jun 2007, Martin Baehr wrote:
because then i'd need to parse the xml and turn it back into text, whereas this way the text for the most part can be displayed as is.
? As far as I can see, reading the autodoc syntax has the same problem.
greetings, martin.
On Mon, Jun 11, 2007 at 03:55:01PM +0000, Henrik Grubbstr�m (Lysator) @ Pike (-) developers forum wrote:
because then i'd need to parse the xml and turn it back into text, whereas this way the text for the most part can be displayed as is.
? As far as I can see, reading the autodoc syntax has the same problem.
no, because the AutoDoc module already does it for me:
object docs = master()->show_doc(module); write("%{%s\n%}\n", docs->objects->print()); write(docs->documentation->text);
most of the docs can be displayed just like that.
xml would need some xml->text conversion.
keep in mind that one of the points is to display docs for classes that are not part of the pike distribution. the autodoc blob generated during build won't get to see those. to make this generic, parsing at runtime is the easiest option.
greetings, martin.
It's probably reasonable to first try to use the autodoc.xml, and if that fails continue with code inspection. I'm not sure I want this code in Hilfe though. Overall I think some caution is adviced. The start up time for Hilfe has gone up 30% with these two additions (tab completion and documentation).
It should be possible to delay initialization of this until it's actually used.
Btw, a small bug: The text shown by "help doc" doesn't end with a newline, so the next prompt gets wrong:
help doc
Show documentation for pike modules and classes.>
It'd also be good to get a "No doc found" or something when no doc is found, instead of just a couple of empty lines.
The "doc" command could use a bit better error handling. I got these two errors just by simple typing mistakes. Both causes hilfe to bail out completely.
doc write;
Bad argument 1 to Gmp.mpz(). Expected int|float|Gmp.mpz|Gmp.mpf|Gmp.mpq. Unknown program: Gmp.mpz(Fd(1)->write) src/modules/Gmp/mpz_glue.c:2234: 0->create(@0=Fd(1)->write) devel/lib/modules/Tools.pmod/Hilfe.pmod:2377: Tools.Hilfe->resolv(Tools.Hilfe.StdinHilfe(),({";","\n" "\n"}),@0,"object") devel/lib/modules/Tools.pmod/Hilfe.pmod:2368: Tools.Hilfe->resolv(@1=Tools.Hilfe.StdinHilfe(),({"write",";","\n" "\n"}),@0,UNDEFINED) devel/lib/modules/Tools.pmod/Hilfe.pmod:316: Tools.Hilfe.CommandDoc()->exec(@1,"doc write;",({"doc","write;"}),({"doc"," ","write ",";","\n" "\n"})) devel/lib/modules/Tools.pmod/Hilfe.pmod:1581: Tools.Hilfe.StdinHilfe()->add_buffer("doc write;") devel/lib/modules/Tools.pmod/Hilfe.pmod:1563: Tools.Hilfe.StdinHilfe()->add_input_line("doc write;") devel/lib/modules/Tools.pmod/Hilfe.pmod:2463: Tools.Hilfe.StdinHilfe()->create(UNDEFINED) devel/lib/modules/Tools.pmod/Hilfe.pmod:0: Tools.Hilfe->StdinHilfe()
doc
Bad argument 1 to Program.defined(). Expected program. Unknown program: Program.defined(0) src/module.c:360: _static_modules.Builtin()->program_defined(0) devel/build/linux-i686/master.pike:829: master()->show_doc(UNDEFINED) devel/lib/modules/Tools.pmod/Hilfe.pmod:319: Tools.Hilfe.CommandDoc()->exec(Tools.Hilfe.StdinHilfe(),"doc",({"doc"}),({"doc","\n" "\n"})) devel/lib/modules/Tools.pmod/Hilfe.pmod:1581: Tools.Hilfe.StdinHilfe()->add_buffer("doc") devel/lib/modules/Tools.pmod/Hilfe.pmod:1563: Tools.Hilfe.StdinHilfe()->add_input_line("doc") devel/lib/modules/Tools.pmod/Hilfe.pmod:2463: Tools.Hilfe.StdinHilfe()->create(UNDEFINED) devel/lib/modules/Tools.pmod/Hilfe.pmod:0: Tools.Hilfe->StdinHilfe()
On Sun, Jun 17, 2007 at 01:15:00PM +0000, Martin Stjernholm, Roxen IS @ Pike developers forum wrote:
The "doc" command could use a bit better error handling.
true. i have been focusing on making sure the functionality works, and neglected testing the nonfunctionality.
doc write;
Bad argument 1 to Gmp.mpz(). Expected int|float|Gmp.mpz|Gmp.mpf|Gmp.mpq. Unknown program: Gmp.mpz(Fd(1)->write)
that is a puzzling one. i don't understand how the error relates to the code in that place.
doc
Bad argument 1 to Program.defined(). Expected program.
fixed. thank you!
greetings, martin.
Speaking of start up time and overhead, I saw now that the master keeps the sources of all compiled files that doesn't come from dumps, just to have them handy for extracting the docs, I presume.
That does not work. Really. Granted, pike isn't very small footprint to begin with, but this is in a different league.
I don't think there should be any space or time overhead in the master until docs actually are requested, and even then I think it should be up to the caller that requests the doc to keep caches and stuff.
I said it was best to add stuff to the master that needs to meddle with the dirnode/joinnode internals. That means functions that maps objects/functions/programs to the paths of their source files (if known), but nothing more than that. Anything else is beyond the scope of the master.
well, i agree, but i could not find a way to do it otherwise. when pike compiles the classes, the source gets thrown away. so unless it is cought at that point it is lost, and no docs can be extracted.
the code only keeps the source it it is not coming from a file. (otherwise it just stores the filename)
the actual extraction of docs does not hapen until it is requested through show_doc()
i did not find a way to control whether docs should be available or not, early enough in the process to not lose some docs. even a comandline swich does not get processed early enough it seems, let alone anything defined in a users main class.
i am aware that this code is not the final solution, but i wanted to put it in because it works and also to get a discussion base to improve on it.
there is definetly much room for improvement.
greetings, martin.
Currently the code caches all source that is read from .pike files (not .pmod), but I guess that's a bug then.
Anyway, it's not anywhere near worth the cost to store the source for pure compiled strings for this. Afterall, at least 99.995% of all pike processes won't use any doc extraction at all, and of those that do I'd say at least 80% is perfectly fine with that it only works for normal files.
If you want to cope with code that comes from odd places I think you should provide an API to let other programmers explicitly hook in their class loaders into your doc extraction system. (And such an API should be that the doc extractor can ask the loader for the source when needed, not the other way around.)
In specific projects one can also extend the master to do this all the time, if it seems like a good idea.
i did not find a way to control whether docs should be available or not, early enough in the process to not lose some docs.
I don't understand this. All pike sources that are used during the bootstrap comes from normal .pike and .pmod files in the lib directory. It's all there, even the master itself.
And btw, I do think it's a very neat feature to get online docs in hilfe. (It's not all gripes. One shouldn't forget the kudos.)
On Tue, Jun 12, 2007 at 12:25:05PM +0000, Martin Stjernholm, Roxen IS @ Pike developers forum wrote:
Currently the code caches all source that is read from .pike files (not .pmod), but I guess that's a bug then.
yup, that was not intended. fixed now. thanks.
Anyway, it's not anywhere near worth the cost to store the source for pure compiled strings for this. Afterall, at least 99.995% of all pike processes won't use any doc extraction at all, and of those that do I'd say at least 80% is perfectly fine with that it only works for normal files.
true. actually, now that the above problem is fixed, a simple switch that can be turned in any script will be enough to start collecting docs. i was under the impression that somehow not all classes created in the beginning came from files. that is of course nonsense. where should they come from?
If you want to cope with code that comes from odd places I think you should provide an API to let other programmers explicitly hook in their class loaders into your doc extraction system. (And such an API should be that the doc extractor can ask the loader for the source when needed, not the other way around.)
can you elaborate on that please?
I don't understand this. All pike sources that are used during the bootstrap comes from normal .pike and .pmod files in the lib directory. It's all there, even the master itself.
right. see above. my mistake.
And btw, I do think it's a very neat feature to get online docs in hilfe. (It's not all gripes. One shouldn't forget the kudos.)
thanks!
greetings, martin.
/.../ (And such an API should be that the doc extractor can ask the loader for the source when needed, not the other way around.)
can you elaborate on that please?
The API would allow custom class loaders to hook in callbacks that the doc extractor can call when it needs to get the source for a program or function. The doc extractor would first try Program.defined/ Function.defined, then see if the master can find the location, and then go through all the registered callbacks.
That way it's possible for a custom class loader to hook itself in and provide the sources for its programs on behalf of the doc extractor.
could you give an example of how code to use that api might look like?
the thing that i am concerned about is, that people would have to make sure they use the right way, if they want to provide the docs and a simple compile_file() or compile_string() or even cast to program would not cut it anymore.
i'd like to be able to go into any existing pike application, add a setting or line of code in the beginning and then be sure to not miss any docs from there on.
btw, i am also considering to extend the doc parser to create structures for classes and functions that have no docs, because even without a description, being able to look up a function definition can be helpful. (and i remember that someone has asked for such a feature in the past)
greetings, martin.
could you give an example of how code to use that api might look like?
Sorry, but my high-level description will have to do. It's not rocket surgery in any way, but it's also not any simpler or more convenient to use than it looks like.
the thing that i am concerned about is, that people would have to make sure they use the right way, if they want to provide the docs and a simple compile_file() or compile_string() or even cast to program would not cut it anymore.
Indeed it wouldn't, and it can't in any real-world case, where doc extraction, after all, isn't the main thing a program does, or even a remotely common thing. The cost of tapping the source all the time is absolutely nowhere near in proportion to the gain, not even in an application where you would find it nice to provide an integrated API doc browsing feature (and that isn't just a playground like hilfe).
I.e. even if you were to provide a function like OnlinePikeDoc.compile_string_and_tap_source which would be a source-tapping alternative to compile_string, I couldn't use it on, say, the modules in Roxen to make an API doc browser in the admin interface. The sources have to be reloaded on demand only.
i'd like to be able to go into any existing pike application, add a setting or line of code in the beginning and then be sure to not miss any docs from there on.
This would be much like turning on extra debug modes, I take it: I.e. being able to investigate apps without changing them. Sure, I don't have a problem with that, as long as it defaults to off. A define would be best since it means zero runtime overhead when not used.
On Thu, Jun 14, 2007 at 10:15:01PM +0000, Martin Stjernholm, Roxen IS @ Pike developers forum wrote:
i'd like to be able to go into any existing pike application, add a setting or line of code in the beginning and then be sure to not miss any docs from there on.
This would be much like turning on extra debug modes, I take it: I.e. being able to investigate apps without changing them.
exactly. half of my current job is doing just that.
Sure, I don't have a problem with that, as long as it defaults to off.
well, the current code does default to off. with a function to turn it on.
A define would be best since it means zero runtime overhead when not used.
right now it is one simple if in compile_string. could probably wrap that into a define too.
the remaining functions are never called until docs are actually requested anywhere, so they should have no impact on performance.
greetings, martin.
can you elaborate on how you measured that?
i am especially curious as to howthe tab_completion can impact the startup time, as it is just a bit more code to be compiled, and a few function calls to add the keybinding. there is no code running other than that, until someone actually hits tab.
greetings, martin.
Nothing really scientific. Just ran
void main() { object t = System.Timer(); object h = master()->resolv("Tools.Hilfe.StdinHilfe"); werror("%O ms\n", t->peek()*1000); }
a few times.
ok, i removed all the unnecesary cache_source calls, now that i understand how to get the actual program of a module.
i can't see any speed difference with the code now.
greetings, martin.
On Wed, Jun 13, 2007 at 03:30:31PM +0200, Martin Baehr wrote:
i can't see any speed difference with the code now.
actually, there is a speed difference. the new code is slightly faster than the previous incarnation.
i didn't test with a version of the code from before starting to mess with it. so i can't comment on how much slower the current incarnation is compared to that.
greetings, martin.
Can you give an example on how to use it? I cannot get it to work for anything that I've tried so far. And most importantly, please give me ^D back as exit/forward-delete...
oh, sorry, i never use ^D for editing. but i just see now that backspace is mapped to ^D too. ooops. can you suggest a better key?
ctrl-? maybe?
greetings, martin.
argh, that's nonsense, it's ctrl-? that backspace is mapped to. so that one is out too.
if anyone has any ideas for a better key, please just change it. i don't use emacs editing keys much, so i don't know what most of them are.
greetings, martin.
how about F1? that is often used for help. would that conflict with anything?
(F1 is "^[OP" if someone wants to make the change)
greetings, martin.
I suggest C-h (as in Emacs ;). In the old times it was difficult to tell apart from DEL (aka Backspace) but nowadays that ought not be a problem.
C-H works in my terminal. although i like F1 better because it is easiert to hit one key instead of two, especially if you are in the middle of typing a command. but i can't figure out how to catch that one (apart from hardcoding a particular sequence)
greetings, martin.
Maybe both then? I for instance have mapped the F* keys directly to my window manager, so they are essentially inaccessible to apps. That works well since almost all unix apps prefers Ctrl-foo keys; I only run into trouble when I access a windoze box through remote desktop.
although i like F1 better because it is easiert to hit one key instead of two, especially if you are in the middle of typing a command. but i can't figure out how to catch that one (apart from hardcoding a particular sequence)
Huh? Didn't we just decide that "\!k1" is used to catch F1?
___Hilfe->readline->get_input_controller()->bind("\!k1", lambda(){write("Hello!");});
works fine here. (In an xterm. In gnome-terminal, F1 is eaten by Gnome and not delivered to the command-line app.)
i missread your previous comment yesterday and thought that "\!k1" was for YOUR terminal.
sorry about that.
greetings, martin.
Nope, ! is the escape to look something up in the terminfo database. See all the default bindings for cursor keys etc in Readline.pike. (Note that terminfo based bindings should be made after hardcoded ones, so that they can override. If for example cursor left is "^H" on some terminal, the binding order
({ "^H", backward_delete_char }), ({ "\!kl", backward_char }),
makes sure that cursor left generates backward_char on that terminal.)
no worries, i already suspected that. thanks for the confirmation :-)
greetings, martin.
I'd suggest using ^D as forward-delete/menu-complete/exit instead, and opt for the latter when used on an empty line.
No thanks. It's annoying enough as a default in some shells. Please don't infect my Hilfe with it.
which modules are you trying?
type "doc" at the prompt and use tab completion to get to the function/module that you want to see the docs for:
doc Stdio.write_file<enter>
write_file(string filename, string str, int | void access)
Write the string @[str] onto the file @[filename]. Any existing data in the file is overwritten. ...
or use it inside some code:
Image.ANY.decode(Stdio.read_file<hit ^D, to see the docs for read_file()>
you can even do things like assigning functions or objects to variables:
object foo = Stdio.Readline(); doc foo->message
message(string msg)
Print a message to the output device
greetings, martin.
Ok, didn't understand I had to type "doc" first. However, I mostly get empty lines or garbled output so it seems to interact badly with terminfo/readline and my SSH connection (TERM set to vt102). And now I got this for combine_path^D:
combine_pathDivision on different types.
Unknown program: `/(0,"->") build/darwin-8.9.1-i386/master.pike:782: master()->show_doc(function) lib/modules/Tools.pmod/Hilfe.pmod:317: Tools.Hilfe.CommandDoc()->exec(Tools.Hilfe.StdinHilfe(),"doc combine_path" ,({"doc","combine_path"}),({"doc"," ","combine_path","\n" "\n"})) lib/modules/Tools.pmod/Hilfe.pmod:1574: Tools.Hilfe.StdinHilfe()->add_buffer("doc combine_path") lib/modules/Tools.pmod/Hilfe.pmod:1556: Tools.Hilfe.StdinHilfe()->add_input_line("doc combine_path") lib/modules/Tools.pmod/Hilfe.pmod:2495: Tools.Hilfe.StdinHilfe()->handle_doc("\4") lib/modules/Stdio.pmod/Readline.pike:486: Stdio.Readline()->InputController()->process_input("\4") lib/modules/Stdio.pmod/Readline.pike:565: Stdio.Readline()->InputController()->run_blocking() lib/modules/Stdio.pmod/Readline.pike:1643: Stdio.Readline()->edit("",UNDEFINED,UNDEFINED) lib/modules/Tools.pmod/Hilfe.pmod:2445: Tools.Hilfe.StdinHilfe()->create(UNDEFINED) lib/modules/Tools.pmod/Hilfe.pmod:0: Tools.Hilfe->StdinHilfe() lipton:7.7 $ lipton:7.7 $
On Mon, Jun 11, 2007 at 06:10:02PM +0000, Jonas Walld�n @ Pike developers forum wrote:
Ok, didn't understand I had to type "doc" first.
it's
doc function|program|object<enter>
or
code ... function|program|object<hotkey when cursor is here>more code...
However, I mostly get empty lines or garbled output so it seems to interact badly with terminfo/readline and my SSH connection (TERM set to vt102). And now I got this for combine_path^D:
no, that is a general problem of not catching warnings. i fixed that for tab-completion but aparently it needs fixing here too.
combine_pathDivision on different types.
Unknown program: `/(0,"->")
made the code slightly more paranoid to catch that error. thanks.
greetings, martin.
pike-devel@lists.lysator.liu.se
-
H. William Welliver III -
Henrik Grubbstr�m (Lysator) ＠ Pike (-) developers forum -
Johan Sundstr�m (Achtung Liebe!) ＠ Pike (-) developers forum
-
Jonas Walld�n ＠ Pike developers forum
-
Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) ＠ Pike (-) developers forum
-
Martin Baehr -
Martin Bähr
-
Martin Nilsson (Opera Mini - AFK!) ＠ Pike (-) developers forum
-
Martin Stjernholm, Roxen IS ＠ Pike developers forum
-
Peter Bortas ＠ Pike developers forum