I think having both LSP https://microsoft.github.io/language-server-protocol/ and DAP https://microsoft.github.io/debug-adapter-protocol/ implementations for Pike would be awsome, but I feel that this discussion is getting further away from it's main point. We've been talking about LSP for a little during the Conference. There is a huge interest in it and I think we should move the LSP discussion to another mailing thread to avoid mixing contexts.
Having a Pike debug adapter would also be great, but since DAP is used for estabilishing communication between IDE and the debugger, we can't move on with it until we actually have a debugger.
From what I understand the greatest challenge now would be to reach the
local variables. I've heard that long time ago, the language got some optimizations that makes retrieving local variable names very challenging. Is it true? If yes, how we can jump this through?
I would be more than happy to help crossing-out the remaining TO-DOs, if I only could get some itroduction and clues.
Hi.
I think having both LSP https://microsoft.github.io/language-server-protocol/ and DAP https://microsoft.github.io/debug-adapter-protocol/ implementations for Pike would be awsome, but I feel that this discussion is getting further away from it's main point. We've been talking about LSP for a little during the Conference. There is a huge interest in it and I think we should move the LSP discussion to another mailing thread to avoid mixing contexts.
Having a Pike debug adapter would also be great, but since DAP is used for estabilishing communication between IDE and the debugger, we can't move on with it until we actually have a debugger.
From what I understand the greatest challenge now would be to reach the local variables. I've heard that long time ago, the language got some optimizations that makes retrieving local variable names very challenging. Is it true? If yes, how we can jump this through?
As far as I know, information about the names and types of local variables have never been available to the runtime. Prior to 2006-02-25, the allocator for local variables was broken, causing local variables to never be reused (which could be considered to have it be less challenging then).
I would be more than happy to help crossing-out the remaining TO-DOs, if I only could get some itroduction and clues.
Sure.
Most of the code for handling local variables and their types and names is located at the end of language.yacc. Of interest are probably the functions {low_,}add_local_name() and {low_,lexical_,}islocal() in language.yacc, as well as {low_,}pop_local_variables() in pike_compiler.cmod.
The main problem in the case of debugging, is that this information is thrown away by the compiler when no longer needed.
One possible place to store the information would be in the linenumber table, but doing that would probably require adding suitable new nodes to the parse-tree (las.c), corresponding new virtual opcodes (docode.c, et al) to let the code that generates the linenumber table (peep.c) reach it. Note also that adding these extra nodes and opcodes may affect some of the optimizers.
/grubba
On Dec 21, 2018, at 6:35 AM, Henrik Grubbström (Lysator) @ Pike (-) developers forum 10353@lyskom.lysator.liu.se wrote:
Most of the code for handling local variables and their types and names is located at the end of language.yacc. Of interest are probably the functions {low_,}add_local_name() and {low_,lexical_,}islocal() in language.yacc, as well as {low_,}pop_local_variables() in pike_compiler.cmod.
The main problem in the case of debugging, is that this information is thrown away by the compiler when no longer needed.
One possible place to store the information would be in the linenumber table, but doing that would probably require adding suitable new nodes to the parse-tree (las.c), corresponding new virtual opcodes (docode.c, et al) to let the code that generates the linenumber table (peep.c) reach it. Note also that adding these extra nodes and opcodes may affect some of the optimizers.
I looked at this code briefly last week but was wondering: would it be possible to store the variable definitions in a separate structure in the program and then use the INIT_FRAME opcode to also connect the variables up in the pike frame while allocating the locals?
/grubba
I agree that LSP may be a bit off-topic in this discussion, but I think DAP is very much relevant: implementing it, or the relevant portion of it, as the means of communicating with the in-process debug agent gets us a large bit of the way to having support in a range of debugger clients (VS Code, Eclipse, emacs). Put another way, if hilfe-over-socket isn’t going to be the debug interface (and I never really had it in mind for that to be the ultimate solution), then we’d need to implement something else and I’m not sure there’s a better option.
There’s plenty of functionality present in the proof of concept to be able to start working against the actual debug agent, or a stub at the very least. Yes, there’s still a lot of low-level work required but it doesn’t make sense to wait for that to be completed before getting started with higher level bits… especially if someone wants to help out but isn’t comfortable working in the depths of the interpreter.
I’ve already used the POC to explore some programs and have learned a few interesting things.
A few examples of things that could be worked in parallel:
- The DAP message specification is contained within a JSON schema. There are a lot of message types to be implemented. It would be really handy if there were a way to spit out marshaling code from the schema. There’s a tool called QuickType[1] that can do this, and can be extended to support new languages. Perhaps someone can come up with an approach to using Standards.JSON to marshal/unmarshal JSON messages and implement a Pike class generator for QuickType? I think this would be valuable and given the amount of work maintaining those classes manually, could save a lot of time.
- There probably needs to be some thought given to calculating code blocks so that step-in and step-out can work. I’m not sure what the best approach here would be, but if speed were not a concern, it could be a layer of pike code that sits on top of single stepping and monitoring the code position against a rudimentary tree generated by parsing the pike code under debug.
- There will probably need to be an agent process responsible for starting and monitoring the process under debug, possibly passing portions of the DAP message stream on to the in-process debug agent.
- Does optimization hurt the ability to debug? Breakpoints on constant expressions or on a for loop, as examples, won’t get hit how you might expect them to. Is that okay, or should we explore ways to skip optimization at runtime?
Bill
[1] https://github.com/quicktype/quicktype
On Dec 21, 2018, at 5:27 AM, Mateusz Krawczuk krawczukmat@gmail.com wrote:
I think having both LSP https://microsoft.github.io/language-server-protocol/ and DAP https://microsoft.github.io/debug-adapter-protocol/ implementations for Pike would be awsome, but I feel that this discussion is getting further away from it's main point. We've been talking about LSP for a little during the Conference. There is a huge interest in it and I think we should move the LSP discussion to another mailing thread to avoid mixing contexts.
Having a Pike debug adapter would also be great, but since DAP is used for estabilishing communication between IDE and the debugger, we can't move on with it until we actually have a debugger.
From what I understand the greatest challenge now would be to reach the local variables. I've heard that long time ago, the language got some optimizations that makes retrieving local variable names very challenging. Is it true? If yes, how we can jump this through?
I would be more than happy to help crossing-out the remaining TO-DOs, if I only could get some itroduction and clues.
H. William Welliver III wrote:
- Does optimization hurt the ability to debug?
Breakpoints on constant expressions or on a for loop, as examples, won???t get hit how you might expect them to. Is that okay, or should we explore ways to skip optimization at runtime?
I personally prefer that optimisation levels are decided at compile time only. Debugging an optimised version would imply that: - Yes, you cannot put breakpoints everywhere (some code has been optimised out), the debugger should pick the next nearest spot it *can* insert a breakpoint. - Yes, some variables are inaccessible because they have been optimised out. - Yes, single stepping can seem somewhat chaotic, because of code reordering.
But that is what I want, because otherwise I cannot debug optimised programs. Sometimes bugs only show up in optimised code. Being able to debug that without altering the runtime behaviour (timingwise or registerwise), would be essential.
By "at runtime" I meant when pike code is compiled, not when the interpreter is compiled. Presumably this would be an option configurable at runtime.
I understand your sentiment in terms of aim, but it's important to remember that just about any means of debugging (from printf to a debugger) is going to alter execution timing in some largely unquantifiable way, so tracking down races and timing related problems will still be subject to variation even if the exact same bytecode is used.
Bill
December 28, 2018 12:13 PM, "Stephen R. van den Berg" srb@cuci.nl wrote:
H. William Welliver III wrote:
- Does optimization hurt the ability to debug?
Breakpoints on constant expressions or on a for loop, as examples, won???t get hit how you might expect them to. Is that okay, or should we explore ways to skip optimization at runtime?
I personally prefer that optimisation levels are decided at compile time only. Debugging an optimised version would imply that:
- Yes, you cannot put breakpoints everywhere (some code has been optimised out),
the debugger should pick the next nearest spot it *can* insert a breakpoint.
- Yes, some variables are inaccessible because they have been optimised out.
- Yes, single stepping can seem somewhat chaotic, because of code reordering.
But that is what I want, because otherwise I cannot debug optimised programs. Sometimes bugs only show up in optimised code. Being able to debug that without altering the runtime behaviour (timingwise or registerwise), would be essential. -- Stephen.
H. William Welliver III wrote:
By "at runtime" I meant when pike code is compiled, not when the interpreter is compiled. Presumably this would be an option configurable at runtime.
That is what I meant too.
I understand your sentiment in terms of aim, but it's important to remember that just about any means of debugging (from printf to a debugger) is going to alter execution timing in some largely unquantifiable way, so tracking down races and timing related problems will still be subject to variation even if the exact same bytecode is used.
True, but nonetheless, being able to debug in "optimised" mode is still a very desirable feature (even if that means that there will be a lot of shortcomings then: missing variables, chaotic single step, breakpoints not possible everywhere).
That's convincing. Having a debug adapter would allow to select from multiple debugger clients already available. I'll try and write one according to your suggestions.
After your recent updates to the debugger-concept branch I'm no longer able to set a breakpoint. Can you please review the wiki and ensure the example scenario you described there is relevant to the current state? This would let me move on with the adapter work.
pt., 28 gru 2018, 15:46: H. William Welliver III william@welliver.org napisał(a):
I agree that LSP may be a bit off-topic in this discussion, but I think DAP is very much relevant: implementing it, or the relevant portion of it, as the means of communicating with the in-process debug agent gets us a large bit of the way to having support in a range of debugger clients (VS Code, Eclipse, emacs). Put another way, if hilfe-over-socket isn’t going to be the debug interface (and I never really had it in mind for that to be the ultimate solution), then we’d need to implement something else and I’m not sure there’s a better option.
There’s plenty of functionality present in the proof of concept to be able to start working against the actual debug agent, or a stub at the very least. Yes, there’s still a lot of low-level work required but it doesn’t make sense to wait for that to be completed before getting started with higher level bits… especially if someone wants to help out but isn’t comfortable working in the depths of the interpreter.
I’ve already used the POC to explore some programs and have learned a few interesting things.
A few examples of things that could be worked in parallel:
- The DAP message specification is contained within a JSON schema. There
are a lot of message types to be implemented. It would be really handy if there were a way to spit out marshaling code from the schema. There’s a tool called QuickType[1] that can do this, and can be extended to support new languages. Perhaps someone can come up with an approach to using Standards.JSON to marshal/unmarshal JSON messages and implement a Pike class generator for QuickType? I think this would be valuable and given the amount of work maintaining those classes manually, could save a lot of time.
- There probably needs to be some thought given to calculating code blocks
so that step-in and step-out can work. I’m not sure what the best approach here would be, but if speed were not a concern, it could be a layer of pike code that sits on top of single stepping and monitoring the code position against a rudimentary tree generated by parsing the pike code under debug.
- There will probably need to be an agent process responsible for starting
and monitoring the process under debug, possibly passing portions of the DAP message stream on to the in-process debug agent.
- Does optimization hurt the ability to debug? Breakpoints on constant
expressions or on a for loop, as examples, won’t get hit how you might expect them to. Is that okay, or should we explore ways to skip optimization at runtime?
Bill
[1] https://github.com/quicktype/quicktype
On Dec 21, 2018, at 5:27 AM, Mateusz Krawczuk krawczukmat@gmail.com wrote:
I think having both LSP https://microsoft.github.io/language-server-protocol/ and DAP https://microsoft.github.io/debug-adapter-protocol/ implementations for Pike would be awsome, but I feel that this discussion is getting further away from it's main point. We've been talking about LSP for a little during the Conference. There is a huge interest in it and I think we should move the LSP discussion to another mailing thread to avoid mixing contexts.
Having a Pike debug adapter would also be great, but since DAP is used for estabilishing communication between IDE and the debugger, we can't move on with it until we actually have a debugger.
From what I understand the greatest challenge now would be to reach the local variables. I've heard that long time ago, the language got some optimizations that makes retrieving local variable names very challenging. Is it true? If yes, how we can jump this through?
I would be more than happy to help crossing-out the remaining TO-DOs, if I only could get some itroduction and clues.
Yes, I think that DAP allows us to make the most of limited developer cycles.
Also, I started writing domain objects and serialization code for the DAP protocol messages, but after a few hours, it seemed like generating code from the schema specification might be the more useful approach, especially if LSP uses schemas to describe its messages (I haven't checked). I think that adding support for generating Pike classes to something like QuickTypes might be the best option at the moment.
What sort of problem are you having setting breakpoints? I don't think I've pushed any changes that broke anything... does the process under debug print out any useful information when you tell it to go?
Bill
December 28, 2018 2:56 PM, "Mateusz Krawczuk" <krawczukmat@gmail.com (mailto:%22Mateusz%20Krawczuk%22%20krawczukmat@gmail.com)> wrote: That's convincing. Having a debug adapter would allow to select from multiple debugger clients already available. I'll try and write one according to your suggestions. After your recent updates to the debugger-concept branch I'm no longer able to set a breakpoint. Can you please review the wiki and ensure the example scenario you described there is relevant to the current state? This would let me move on with the adapter work.
I've started a separate thread for QuickType support in Pike. But now I'm thinking perhaps it would be even better to have some kind of TypeScript -> Pike class transformer, since both DAP and LSP JSON-RPC message specifications are in TypeScript. Does anyone know any tools that might be able to do it?
Bill, can you please share the Pike language parser plugin for Eclipse, along with the JSON-RPC implementation you mentioned? Might turn out handy.
pt., 28 gru 2018 o 21:20 H. William Welliver III william@welliver.org napisał(a):
Yes, I think that DAP allows us to make the most of limited developer cycles.
Also, I started writing domain objects and serialization code for the DAP protocol messages, but after a few hours, it seemed like generating code from the schema specification might be the more useful approach, especially if LSP uses schemas to describe its messages (I haven't checked). I think that adding support for generating Pike classes to something like QuickTypes might be the best option at the moment.
What sort of problem are you having setting breakpoints? I don't think I've pushed any changes that broke anything... does the process under debug print out any useful information when you tell it to go?
Bill
December 28, 2018 2:56 PM, "Mateusz Krawczuk" <krawczukmat@gmail.com <%22Mateusz%20Krawczuk%22%20%3Ckrawczukmat@gmail.com%3E>> wrote:
That's convincing. Having a debug adapter would allow to select from multiple debugger clients already available. I'll try and write one according to your suggestions. After your recent updates to the debugger-concept branch I'm no longer able to set a breakpoint. Can you please review the wiki and ensure the example scenario you described there is relevant to the current state? This would let me move on with the adapter work.
I started looking at the JSON-RPC code before the holidays, I will put what I have in a repository somewhere... it's not a lot of code, as the spec is pretty simple. Note that this won't be helpful for DAP, as it doesn't use JSON-RPC (though LSP does).
I think (but am not 100% sure), that the definitive specifications for DAP is expressed via a JSON schema... my guess is that a TypeScript converter would probably be a good bit of work, unless the changes required would be very minimal, though I haven't really thought too much about it. TypeScript is certainly what Microsoft uses for a lot of their tooling, but I'm not sure if that code is considered the original specification or just a reference implementation.
I will try to dig out the Java parser code. I did spend a day or two looking at it, but there were two problems: 1) there is some dependency on Eclipse editor classes and b) an older version of the parser and lexer libraries is fairly old. I tried to tackle both of the problems at once and it ended up being a bit of a mess.
There's a LR parser module in Pike, but I haven't looked at whether it could be used with a version of the pike language syntax to generate the kinds of data we need.
January 2, 2019 12:04 PM, "Mateusz Krawczuk" <krawczukmat@gmail.com (mailto:%22Mateusz%20Krawczuk%22%20krawczukmat@gmail.com)> wrote: I've started a separate thread for QuickType support in Pike. But now I'm thinking perhaps it would be even better to have some kind of TypeScript -> Pike class transformer, since both DAP and LSP JSON-RPC message specifications are in TypeScript. Does anyone know any tools that might be able to do it?
Bill, can you please share the Pike language parser plugin for Eclipse, along with the JSON-RPC implementation you mentioned? Might turn out handy.
Ah! Right, sorry, my bad, actually the specification is a JSON schema and can be found here: https://github.com/Microsoft/debug-adapter-protocol/blob/gh-pages/debugAdapt...
śr., 2 sty 2019, 20:23: H. William Welliver III william@welliver.org napisał(a):
I started looking at the JSON-RPC code before the holidays, I will put what I have in a repository somewhere... it's not a lot of code, as the spec is pretty simple. Note that this won't be helpful for DAP, as it doesn't use JSON-RPC (though LSP does).
I think (but am not 100% sure), that the definitive specifications for DAP is expressed via a JSON schema... my guess is that a TypeScript converter would probably be a good bit of work, unless the changes required would be very minimal, though I haven't really thought too much about it. TypeScript is certainly what Microsoft uses for a lot of their tooling, but I'm not sure if that code is considered the original specification or just a reference implementation.
I will try to dig out the Java parser code. I did spend a day or two looking at it, but there were two problems: 1) there is some dependency on Eclipse editor classes and b) an older version of the parser and lexer libraries is fairly old. I tried to tackle both of the problems at once and it ended up being a bit of a mess.
There's a LR parser module in Pike, but I haven't looked at whether it could be used with a version of the pike language syntax to generate the kinds of data we need.
January 2, 2019 12:04 PM, "Mateusz Krawczuk" <krawczukmat@gmail.com <%22Mateusz%20Krawczuk%22%20%3Ckrawczukmat@gmail.com%3E>> wrote:
I've started a separate thread for QuickType support in Pike. But now I'm thinking perhaps it would be even better to have some kind of TypeScript -> Pike class transformer, since both DAP and LSP JSON-RPC message specifications are in TypeScript. Does anyone know any tools that might be able to do it?
Bill, can you please share the Pike language parser plugin for Eclipse, along with the JSON-RPC implementation you mentioned? Might turn out handy.
I've issued a pull request [1] with quicktype-generated Debug Adapter Protocol pmod containing domain objects, along with serializing/deserializing subroutines. It leaves a lot to be desired, but it's something we can move the work on for now. One of the problems with this specification is the fact that it's delivered as a JSON Schema. AFAIK Schema's purpose is to validate JSONs - not to represent a JSON structure. For example, this specification is attempting to represent inheritance relation with an `allOf` keyword, while Schema reference explicitly states that `allOf` is not meant to represent such relation. I've also encountered some problems with an absurd amount of generated enums, so I had to hack QuickType's Pike renderer a little. More details in the commit's doc/worklog.md.
Notice that the problem is within how QuickType handles this schema, and it's not related specifically to Pike's renderer.
I'm really concerned about poor serialization and deserialization performance of the generated implementation. We'll see in practice how big a problem it is. One way would be to gradually refactor the generated implementation manually, and taking into account that it's almost 7k lines long, it might take some time.
[1] https://github.com/pikelang/Pike/pull/22
śr., 2 sty 2019 o 23:25 Mateusz Krawczuk krawczukmat@gmail.com napisał(a):
Ah! Right, sorry, my bad, actually the specification is a JSON schema and can be found here: https://github.com/Microsoft/debug-adapter-protocol/blob/gh-pages/debugAdapt...
śr., 2 sty 2019, 20:23: H. William Welliver III william@welliver.org napisał(a):
I started looking at the JSON-RPC code before the holidays, I will put what I have in a repository somewhere... it's not a lot of code, as the spec is pretty simple. Note that this won't be helpful for DAP, as it doesn't use JSON-RPC (though LSP does).
I think (but am not 100% sure), that the definitive specifications for DAP is expressed via a JSON schema... my guess is that a TypeScript converter would probably be a good bit of work, unless the changes required would be very minimal, though I haven't really thought too much about it. TypeScript is certainly what Microsoft uses for a lot of their tooling, but I'm not sure if that code is considered the original specification or just a reference implementation.
I will try to dig out the Java parser code. I did spend a day or two looking at it, but there were two problems: 1) there is some dependency on Eclipse editor classes and b) an older version of the parser and lexer libraries is fairly old. I tried to tackle both of the problems at once and it ended up being a bit of a mess.
There's a LR parser module in Pike, but I haven't looked at whether it could be used with a version of the pike language syntax to generate the kinds of data we need.
January 2, 2019 12:04 PM, "Mateusz Krawczuk" <krawczukmat@gmail.com <%22Mateusz%20Krawczuk%22%20%3Ckrawczukmat@gmail.com%3E>> wrote:
I've started a separate thread for QuickType support in Pike. But now I'm thinking perhaps it would be even better to have some kind of TypeScript -> Pike class transformer, since both DAP and LSP JSON-RPC message specifications are in TypeScript. Does anyone know any tools that might be able to do it?
Bill, can you please share the Pike language parser plugin for Eclipse, along with the JSON-RPC implementation you mentioned? Might turn out handy.
Forget that pull request, I've got something way better!
I gave up on hacking quicktype to make it spit out the protocol objects they way I wanted and started to rewriting it manually. I also came up with a proof-of-concept Debug Adapter written in Pike.
If you would like to take a test ride, just clone my repository[1]. More detailed description and instructions are in the readme.
Originally, the adapter is meant to be a mediator between a debugging editor and the debugger client, but since we don't have a debugger client we just might go with plugging the adapter inside the debugger and use it as a client (which will actually make it a debug server).
At this point we have to think how the debug mode should be launched. I don't believe the `--debugger` CLI arugment is the best approach. Maybe something GDB- or Python-style?
Now I'll be working to integrate what I already have here with Bill's concept debugger so it will be possible to set/stop at breakpoints using an editor of choice.
[1] https://github.com/mkrawczuk/pike-debug-adapter-playground
czw., 17 sty 2019 o 18:27 Mateusz Krawczuk krawczukmat@gmail.com napisał(a):
I've issued a pull request [1] with quicktype-generated Debug Adapter Protocol pmod containing domain objects, along with serializing/deserializing subroutines. It leaves a lot to be desired, but it's something we can move the work on for now. One of the problems with this specification is the fact that it's delivered as a JSON Schema. AFAIK Schema's purpose is to validate JSONs - not to represent a JSON structure. For example, this specification is attempting to represent inheritance relation with an `allOf` keyword, while Schema reference explicitly states that `allOf` is not meant to represent such relation. I've also encountered some problems with an absurd amount of generated enums, so I had to hack QuickType's Pike renderer a little. More details in the commit's doc/worklog.md.
Notice that the problem is within how QuickType handles this schema, and it's not related specifically to Pike's renderer.
I'm really concerned about poor serialization and deserialization performance of the generated implementation. We'll see in practice how big a problem it is. One way would be to gradually refactor the generated implementation manually, and taking into account that it's almost 7k lines long, it might take some time.
[1] https://github.com/pikelang/Pike/pull/22
śr., 2 sty 2019 o 23:25 Mateusz Krawczuk krawczukmat@gmail.com napisał(a):
Ah! Right, sorry, my bad, actually the specification is a JSON schema and can be found here: https://github.com/Microsoft/debug-adapter-protocol/blob/gh-pages/debugAdapt...
śr., 2 sty 2019, 20:23: H. William Welliver III william@welliver.org napisał(a):
I started looking at the JSON-RPC code before the holidays, I will put what I have in a repository somewhere... it's not a lot of code, as the spec is pretty simple. Note that this won't be helpful for DAP, as it doesn't use JSON-RPC (though LSP does).
I think (but am not 100% sure), that the definitive specifications for DAP is expressed via a JSON schema... my guess is that a TypeScript converter would probably be a good bit of work, unless the changes required would be very minimal, though I haven't really thought too much about it. TypeScript is certainly what Microsoft uses for a lot of their tooling, but I'm not sure if that code is considered the original specification or just a reference implementation.
I will try to dig out the Java parser code. I did spend a day or two looking at it, but there were two problems: 1) there is some dependency on Eclipse editor classes and b) an older version of the parser and lexer libraries is fairly old. I tried to tackle both of the problems at once and it ended up being a bit of a mess.
There's a LR parser module in Pike, but I haven't looked at whether it could be used with a version of the pike language syntax to generate the kinds of data we need.
January 2, 2019 12:04 PM, "Mateusz Krawczuk" <krawczukmat@gmail.com <%22Mateusz%20Krawczuk%22%20%3Ckrawczukmat@gmail.com%3E>> wrote:
I've started a separate thread for QuickType support in Pike. But now I'm thinking perhaps it would be even better to have some kind of TypeScript -> Pike class transformer, since both DAP and LSP JSON-RPC message specifications are in TypeScript. Does anyone know any tools that might be able to do it?
Bill, can you please share the Pike language parser plugin for Eclipse, along with the JSON-RPC implementation you mentioned? Might turn out handy.
I think there’s a sense that the parts of the debug infrastructure that run within the process under debug should be as lightweight as possible, and not be active unless the process is being debugged. In keeping with this line of thought, I think the in-process portion of the agent shouldn’t be fully featured, as that would imply that it would be parsing code for metadata like block locations and the like (for stepping over, etc). The —debugger option is intended to be the mechanism which activates the in-process debugging agent.
If your concern is that connecting to a hilfe session is not an ideal debugging tool, that is true, but it was never my intention for that to be the final state. Rather I had some code that served a similar purpose and it didn’t make sense to go off building a lot of infrastructure if the premise wasn’t workable.
Something along these lines is what I had in mind:
Debugging Tool <— DAP —> Debug Agent <— DAP-lite —> Pike process under debug
Now, the debugging tool and Debug Agent could theoretically be a single process, such as a command line debug utility along the lines of gdb, and it could possibly know how to start a pike process for debugging or could also just allow connecting to a running pike process which has the debugger enabled. This is basically how the Java debug agent works, and it allows local and remote debugging, which is pretty handy.
Bill
On Jan 30, 2019, at 1:46 PM, Mateusz Krawczuk krawczukmat@gmail.com wrote:
Forget that pull request, I've got something way better!
I gave up on hacking quicktype to make it spit out the protocol objects they way I wanted and started to rewriting it manually. I also came up with a proof-of-concept Debug Adapter written in Pike.
If you would like to take a test ride, just clone my repository[1]. More detailed description and instructions are in the readme.
Originally, the adapter is meant to be a mediator between a debugging editor and the debugger client, but since we don't have a debugger client we just might go with plugging the adapter inside the debugger and use it as a client (which will actually make it a debug server).
At this point we have to think how the debug mode should be launched. I don't believe the `--debugger` CLI arugment is the best approach. Maybe something GDB- or Python-style?
Now I'll be working to integrate what I already have here with Bill's concept debugger so it will be possible to set/stop at breakpoints using an editor of choice.
[1] https://github.com/mkrawczuk/pike-debug-adapter-playground https://github.com/mkrawczuk/pike-debug-adapter-playground
Update:
I took a little time this morning and merged grubba's local variable names branch with the debugger branch, and it looks pretty good.
Seems to me that the next piece of infrastructure that's needed probably involves some sort of Pike parser. I've looked at the java parser a bit and I'm not really happy with it. Perhaps it would make sense to extend Parser.LR to work with Parser.C.Tokens and generate a pike grammar for use with it?
I looked a little bit at Parser.LR and it looks like some features we'd want might not be complete (like %token) or that would make maintaining similarity with the existing grammar (like |). Does anyone have much experience using Parser.LR or might have some real examples?
Bill
February 6, 2019 8:20 PM, "H. William Welliver III" <william@welliver.org (mailto:%22H.%20William%20Welliver%20III%22%20william@welliver.org)> wrote: I think there’s a sense that the parts of the debug infrastructure that run within the process under debug should be as lightweight as possible, and not be active unless the process is being debugged. In keeping with this line of thought, I think the in-process portion of the agent shouldn’t be fully featured, as that would imply that it would be parsing code for metadata like block locations and the like (for stepping over, etc). The —debugger option is intended to be the mechanism which activates the in-process debugging agent. If your concern is that connecting to a hilfe session is not an ideal debugging tool, that is true, but it was never my intention for that to be the final state. Rather I had some code that served a similar purpose and it didn’t make sense to go off building a lot of infrastructure if the premise wasn’t workable. Something along these lines is what I had in mind: Debugging Tool <— DAP —> Debug Agent <— DAP-lite —> Pike process under debug Now, the debugging tool and Debug Agent could theoretically be a single process, such as a command line debug utility along the lines of gdb, and it could possibly know how to start a pike process for debugging or could also just allow connecting to a running pike process which has the debugger enabled. This is basically how the Java debug agent works, and it allows local and remote debugging, which is pretty handy. Bill
What exactly is this hypothetical parser needed for? Can you give more detail on what is needed?
I've been working on the debug adapter for a while now and made it possible to work with DAP-compliant IDEs, like VScode. You can check it out at the 'debugger' branch of my Pike fork [1]. Breakpoints, stack trace and variable browsing seems to work pretty well. I've also made it possible to attach to a running Pike process and set breakpoints at runtime. To make it possible I've introduced a program cache, which is string-program mapping. It exists in the master object and is updated every time a program is loaded. It is needed because the way we handle breakpoint initialization deferring is insufficient for this scenario. I wonder how does it affect memory optimization.
One major thing that is missing is setting breakpoints on class methods. Any clues how to implement it?
[1] https://github.com/mkrawczuk/Pike/tree/debugger
czw., 7 lut 2019 o 19:14 H. William Welliver III william@welliver.org napisał(a):
Update:
I took a little time this morning and merged grubba's local variable names branch with the debugger branch, and it looks pretty good.
Seems to me that the next piece of infrastructure that's needed probably involves some sort of Pike parser. I've looked at the java parser a bit and I'm not really happy with it. Perhaps it would make sense to extend Parser.LR to work with Parser.C.Tokens and generate a pike grammar for use with it?
I looked a little bit at Parser.LR and it looks like some features we'd want might not be complete (like %token) or that would make maintaining similarity with the existing grammar (like |). Does anyone have much experience using Parser.LR or might have some real examples?
Bill
February 6, 2019 8:20 PM, "H. William Welliver III" <william@welliver.org <%22H.%20William%20Welliver%20III%22%20%3Cwilliam@welliver.org%3E>> wrote:
I think there’s a sense that the parts of the debug infrastructure that run within the process under debug should be as lightweight as possible, and not be active unless the process is being debugged. In keeping with this line of thought, I think the in-process portion of the agent shouldn’t be fully featured, as that would imply that it would be parsing code for metadata like block locations and the like (for stepping over, etc). The —debugger option is intended to be the mechanism which activates the in-process debugging agent. If your concern is that connecting to a hilfe session is not an ideal debugging tool, that is true, but it was never my intention for that to be the final state. Rather I had some code that served a similar purpose and it didn’t make sense to go off building a lot of infrastructure if the premise wasn’t workable. Something along these lines is what I had in mind: Debugging Tool <— DAP —> Debug Agent <— DAP-lite —> Pike process under debug Now, the debugging tool and Debug Agent could theoretically be a single process, such as a command line debug utility along the lines of gdb, and it could possibly know how to start a pike process for debugging or could also just allow connecting to a running pike process which has the debugger enabled. This is basically how the Java debug agent works, and it allows local and remote debugging, which is pretty handy. Bill
CCing dozzie (aka Jarowit). He is proficient with parsers and is eager to put some effort to carry this step on.
pon., 11 mar 2019 o 17:43 Mateusz Krawczuk krawczukmat@gmail.com napisał(a):
What exactly is this hypothetical parser needed for? Can you give more detail on what is needed?
I've been working on the debug adapter for a while now and made it possible to work with DAP-compliant IDEs, like VScode. You can check it out at the 'debugger' branch of my Pike fork [1]. Breakpoints, stack trace and variable browsing seems to work pretty well. I've also made it possible to attach to a running Pike process and set breakpoints at runtime. To make it possible I've introduced a program cache, which is string-program mapping. It exists in the master object and is updated every time a program is loaded. It is needed because the way we handle breakpoint initialization deferring is insufficient for this scenario. I wonder how does it affect memory optimization.
One major thing that is missing is setting breakpoints on class methods. Any clues how to implement it?
[1] https://github.com/mkrawczuk/Pike/tree/debugger
czw., 7 lut 2019 o 19:14 H. William Welliver III william@welliver.org napisał(a):
Update:
I took a little time this morning and merged grubba's local variable names branch with the debugger branch, and it looks pretty good.
Seems to me that the next piece of infrastructure that's needed probably involves some sort of Pike parser. I've looked at the java parser a bit and I'm not really happy with it. Perhaps it would make sense to extend Parser.LR to work with Parser.C.Tokens and generate a pike grammar for use with it?
I looked a little bit at Parser.LR and it looks like some features we'd want might not be complete (like %token) or that would make maintaining similarity with the existing grammar (like |). Does anyone have much experience using Parser.LR or might have some real examples?
Bill
February 6, 2019 8:20 PM, "H. William Welliver III" <william@welliver.org <%22H.%20William%20Welliver%20III%22%20%3Cwilliam@welliver.org%3E>> wrote:
I think there’s a sense that the parts of the debug infrastructure that run within the process under debug should be as lightweight as possible, and not be active unless the process is being debugged. In keeping with this line of thought, I think the in-process portion of the agent shouldn’t be fully featured, as that would imply that it would be parsing code for metadata like block locations and the like (for stepping over, etc). The —debugger option is intended to be the mechanism which activates the in-process debugging agent. If your concern is that connecting to a hilfe session is not an ideal debugging tool, that is true, but it was never my intention for that to be the final state. Rather I had some code that served a similar purpose and it didn’t make sense to go off building a lot of infrastructure if the premise wasn’t workable. Something along these lines is what I had in mind: Debugging Tool <— DAP —> Debug Agent <— DAP-lite —> Pike process under debug Now, the debugging tool and Debug Agent could theoretically be a single process, such as a command line debug utility along the lines of gdb, and it could possibly know how to start a pike process for debugging or could also just allow connecting to a running pike process which has the debugger enabled. This is basically how the Java debug agent works, and it allows local and remote debugging, which is pretty handy. Bill
That’s cool. I suspect that Parser.LR may require a bit of work to make it nicer to build a parser of this complexity, but I think it would be worth the effort, as it would allow a lot of interesting information to be extracted from pike classes… dependency graphs, data to allow refactoring, various types of analysis, etc.
I’ll also take a look at your branch; I sort of gave up trying to keep my branch in sync with some of the changes Grubba has been making to support live backtrace frames and such. I also have some changes to the thread object that permits setting of names on threads (very useful when debugging) and being able to break execution on a thread regardless of whatever it’s running, etc.
Bill
On Mar 13, 2019, at 8:48 AM, Mateusz Krawczuk krawczukmat@gmail.com wrote:
CCing dozzie (aka Jarowit). He is proficient with parsers and is eager to put some effort to carry this step on.
pon., 11 mar 2019 o 17:43 Mateusz Krawczuk <krawczukmat@gmail.com mailto:krawczukmat@gmail.com> napisał(a): What exactly is this hypothetical parser needed for? Can you give more detail on what is needed?
I've been working on the debug adapter for a while now and made it possible to work with DAP-compliant IDEs, like VScode. You can check it out at the 'debugger' branch of my Pike fork [1]. Breakpoints, stack trace and variable browsing seems to work pretty well. I've also made it possible to attach to a running Pike process and set breakpoints at runtime. To make it possible I've introduced a program cache, which is string-program mapping. It exists in the master object and is updated every time a program is loaded. It is needed because the way we handle breakpoint initialization deferring is insufficient for this scenario. I wonder how does it affect memory optimization.
One major thing that is missing is setting breakpoints on class methods. Any clues how to implement it?
[1] https://github.com/mkrawczuk/Pike/tree/debugger https://github.com/mkrawczuk/Pike/tree/debugger
czw., 7 lut 2019 o 19:14 H. William Welliver III <william@welliver.org mailto:william@welliver.org> napisał(a): Update:
I took a little time this morning and merged grubba's local variable names branch with the debugger branch, and it looks pretty good.
Seems to me that the next piece of infrastructure that's needed probably involves some sort of Pike parser. I've looked at the java parser a bit and I'm not really happy with it. Perhaps it would make sense to extend Parser.LR to work with Parser.C.Tokens and generate a pike grammar for use with it?
I looked a little bit at Parser.LR and it looks like some features we'd want might not be complete (like %token) or that would make maintaining similarity with the existing grammar (like |). Does anyone have much experience using Parser.LR or might have some real examples?
Bill
February 6, 2019 8:20 PM, "H. William Welliver III" <william@welliver.org mailto:%22H.%20William%20Welliver%20III%22%20%3Cwilliam@welliver.org%3E> wrote: I think there’s a sense that the parts of the debug infrastructure that run within the process under debug should be as lightweight as possible, and not be active unless the process is being debugged. In keeping with this line of thought, I think the in-process portion of the agent shouldn’t be fully featured, as that would imply that it would be parsing code for metadata like block locations and the like (for stepping over, etc). The —debugger option is intended to be the mechanism which activates the in-process debugging agent. If your concern is that connecting to a hilfe session is not an ideal debugging tool, that is true, but it was never my intention for that to be the final state. Rather I had some code that served a similar purpose and it didn’t make sense to go off building a lot of infrastructure if the premise wasn’t workable. Something along these lines is what I had in mind: Debugging Tool <— DAP —> Debug Agent <— DAP-lite —> Pike process under debug Now, the debugging tool and Debug Agent could theoretically be a single process, such as a command line debug utility along the lines of gdb, and it could possibly know how to start a pike process for debugging or could also just allow connecting to a running pike process which has the debugger enabled. This is basically how the Java debug agent works, and it allows local and remote debugging, which is pretty handy. Bill
That's good news, looking forward to see the update!
My branch is a little specific - I've completely disabled the Hilfe debugger, replacing it by the debug adapter. For this reason you need a DAP-compliant IDE to test on my branch. The examples I provided earlier should suffice, but if you have any problems running it - do not hesitate to ask.
Previously I expressed myself incorrectly. I did not mean class methods (usually called static methods). What was on my mind were the methods, i.e. functions defined inside a class (nested program, in Pike terminology), that can be called on objects instantiating the given class. You currently can't break on them. For example, given
class C { int a; void create() { a = 1; werror("%d\n", a); }
void f() { ++a; werror("%d\n", a); } }
int main(int arc, array(string) argv) { C c = C(); c->f(); werror("c->a: %d\n", c->a); }
It will not stop on breakpoints set inside C's create() and f(). Setting breakpoints inside main() should go fine though.
I did a little research and I might have a potential fix for it. Looks like nested programs are getting their own separate entry. So all the data related to nested programs, like line offsets, are stored in a different object for each defined class. What is done now in debug_breakpoint class's low_enable_breakpoint, is calling low_get_offset_for_line only for one program object that is directly related to the file. What should be done instead, is checking all the programs originating from a given file, so the nested programs are also checked for the needed line offset.
So what I need is to get a list of program objects from a provided file string so I can loop through it. It needs to be available in builtin.cmod. Nothing comes to my mind on how to do it, any help is strongly appreciated.
śr., 13 mar 2019 o 15:01 H. William Welliver III william@welliver.org napisał(a):
That’s cool. I suspect that Parser.LR may require a bit of work to make it nicer to build a parser of this complexity, but I think it would be worth the effort, as it would allow a lot of interesting information to be extracted from pike classes… dependency graphs, data to allow refactoring, various types of analysis, etc.
I’ll also take a look at your branch; I sort of gave up trying to keep my branch in sync with some of the changes Grubba has been making to support live backtrace frames and such. I also have some changes to the thread object that permits setting of names on threads (very useful when debugging) and being able to break execution on a thread regardless of whatever it’s running, etc.
Bill
On Mar 13, 2019, at 8:48 AM, Mateusz Krawczuk krawczukmat@gmail.com wrote:
CCing dozzie (aka Jarowit). He is proficient with parsers and is eager to put some effort to carry this step on.
pon., 11 mar 2019 o 17:43 Mateusz Krawczuk krawczukmat@gmail.com napisał(a):
What exactly is this hypothetical parser needed for? Can you give more detail on what is needed?
I've been working on the debug adapter for a while now and made it possible to work with DAP-compliant IDEs, like VScode. You can check it out at the 'debugger' branch of my Pike fork [1]. Breakpoints, stack trace and variable browsing seems to work pretty well. I've also made it possible to attach to a running Pike process and set breakpoints at runtime. To make it possible I've introduced a program cache, which is string-program mapping. It exists in the master object and is updated every time a program is loaded. It is needed because the way we handle breakpoint initialization deferring is insufficient for this scenario. I wonder how does it affect memory optimization.
One major thing that is missing is setting breakpoints on class methods. Any clues how to implement it?
[1] https://github.com/mkrawczuk/Pike/tree/debugger
czw., 7 lut 2019 o 19:14 H. William Welliver III william@welliver.org napisał(a):
Update:
I took a little time this morning and merged grubba's local variable names branch with the debugger branch, and it looks pretty good.
Seems to me that the next piece of infrastructure that's needed probably involves some sort of Pike parser. I've looked at the java parser a bit and I'm not really happy with it. Perhaps it would make sense to extend Parser.LR to work with Parser.C.Tokens and generate a pike grammar for use with it?
I looked a little bit at Parser.LR and it looks like some features we'd want might not be complete (like %token) or that would make maintaining similarity with the existing grammar (like |). Does anyone have much experience using Parser.LR or might have some real examples?
Bill
February 6, 2019 8:20 PM, "H. William Welliver III" < william@welliver.org <%22H.%20William%20Welliver%20III%22%20%3Cwilliam@welliver.org%3E>> wrote:
I think there’s a sense that the parts of the debug infrastructure that run within the process under debug should be as lightweight as possible, and not be active unless the process is being debugged. In keeping with this line of thought, I think the in-process portion of the agent shouldn’t be fully featured, as that would imply that it would be parsing code for metadata like block locations and the like (for stepping over, etc). The —debugger option is intended to be the mechanism which activates the in-process debugging agent. If your concern is that connecting to a hilfe session is not an ideal debugging tool, that is true, but it was never my intention for that to be the final state. Rather I had some code that served a similar purpose and it didn’t make sense to go off building a lot of infrastructure if the premise wasn’t workable. Something along these lines is what I had in mind: Debugging Tool <— DAP —> Debug Agent <— DAP-lite —> Pike process under debug Now, the debugging tool and Debug Agent could theoretically be a single process, such as a command line debug utility along the lines of gdb, and it could possibly know how to start a pike process for debugging or could also just allow connecting to a running pike process which has the debugger enabled. This is basically how the Java debug agent works, and it allows local and remote debugging, which is pretty handy. Bill
I've verified my hypothesis and made breakpoints in classes work with commit [1]. The way I did it solves the problem, but I believe it should have been done on a lower level.
I've also looked up the program cache and verified that there in fact already is such a structure. I'll work on avoiding the necessity to double it.
[1] https://github.com/mkrawczuk/Pike/commit/bbdf2c23fb536ef42f0d83d14af84a06849...
wt., 2 kwi 2019 o 14:55 Mateusz Krawczuk krawczukmat@gmail.com napisał(a):
That's good news, looking forward to see the update!
My branch is a little specific - I've completely disabled the Hilfe debugger, replacing it by the debug adapter. For this reason you need a DAP-compliant IDE to test on my branch. The examples I provided earlier should suffice, but if you have any problems running it - do not hesitate to ask.
Previously I expressed myself incorrectly. I did not mean class methods (usually called static methods). What was on my mind were the methods, i.e. functions defined inside a class (nested program, in Pike terminology), that can be called on objects instantiating the given class. You currently can't break on them. For example, given
class C { int a; void create() { a = 1; werror("%d\n", a); }
void f() { ++a; werror("%d\n", a); } }
int main(int arc, array(string) argv) { C c = C(); c->f(); werror("c->a: %d\n", c->a); }
It will not stop on breakpoints set inside C's create() and f(). Setting breakpoints inside main() should go fine though.
I did a little research and I might have a potential fix for it. Looks like nested programs are getting their own separate entry. So all the data related to nested programs, like line offsets, are stored in a different object for each defined class. What is done now in debug_breakpoint class's low_enable_breakpoint, is calling low_get_offset_for_line only for one program object that is directly related to the file. What should be done instead, is checking all the programs originating from a given file, so the nested programs are also checked for the needed line offset.
So what I need is to get a list of program objects from a provided file string so I can loop through it. It needs to be available in builtin.cmod. Nothing comes to my mind on how to do it, any help is strongly appreciated.
śr., 13 mar 2019 o 15:01 H. William Welliver III william@welliver.org napisał(a):
That’s cool. I suspect that Parser.LR may require a bit of work to make it nicer to build a parser of this complexity, but I think it would be worth the effort, as it would allow a lot of interesting information to be extracted from pike classes… dependency graphs, data to allow refactoring, various types of analysis, etc.
I’ll also take a look at your branch; I sort of gave up trying to keep my branch in sync with some of the changes Grubba has been making to support live backtrace frames and such. I also have some changes to the thread object that permits setting of names on threads (very useful when debugging) and being able to break execution on a thread regardless of whatever it’s running, etc.
Bill
On Mar 13, 2019, at 8:48 AM, Mateusz Krawczuk krawczukmat@gmail.com wrote:
CCing dozzie (aka Jarowit). He is proficient with parsers and is eager to put some effort to carry this step on.
pon., 11 mar 2019 o 17:43 Mateusz Krawczuk krawczukmat@gmail.com napisał(a):
What exactly is this hypothetical parser needed for? Can you give more detail on what is needed?
I've been working on the debug adapter for a while now and made it possible to work with DAP-compliant IDEs, like VScode. You can check it out at the 'debugger' branch of my Pike fork [1]. Breakpoints, stack trace and variable browsing seems to work pretty well. I've also made it possible to attach to a running Pike process and set breakpoints at runtime. To make it possible I've introduced a program cache, which is string-program mapping. It exists in the master object and is updated every time a program is loaded. It is needed because the way we handle breakpoint initialization deferring is insufficient for this scenario. I wonder how does it affect memory optimization.
One major thing that is missing is setting breakpoints on class methods. Any clues how to implement it?
[1] https://github.com/mkrawczuk/Pike/tree/debugger
czw., 7 lut 2019 o 19:14 H. William Welliver III william@welliver.org napisał(a):
Update:
I took a little time this morning and merged grubba's local variable names branch with the debugger branch, and it looks pretty good.
Seems to me that the next piece of infrastructure that's needed probably involves some sort of Pike parser. I've looked at the java parser a bit and I'm not really happy with it. Perhaps it would make sense to extend Parser.LR to work with Parser.C.Tokens and generate a pike grammar for use with it?
I looked a little bit at Parser.LR and it looks like some features we'd want might not be complete (like %token) or that would make maintaining similarity with the existing grammar (like |). Does anyone have much experience using Parser.LR or might have some real examples?
Bill
February 6, 2019 8:20 PM, "H. William Welliver III" < william@welliver.org <%22H.%20William%20Welliver%20III%22%20%3Cwilliam@welliver.org%3E>> wrote:
I think there’s a sense that the parts of the debug infrastructure that run within the process under debug should be as lightweight as possible, and not be active unless the process is being debugged. In keeping with this line of thought, I think the in-process portion of the agent shouldn’t be fully featured, as that would imply that it would be parsing code for metadata like block locations and the like (for stepping over, etc). The —debugger option is intended to be the mechanism which activates the in-process debugging agent. If your concern is that connecting to a hilfe session is not an ideal debugging tool, that is true, but it was never my intention for that to be the final state. Rather I had some code that served a similar purpose and it didn’t make sense to go off building a lot of infrastructure if the premise wasn’t workable. Something along these lines is what I had in mind: Debugging Tool <— DAP —> Debug Agent <— DAP-lite —> Pike process under debug Now, the debugging tool and Debug Agent could theoretically be a single process, such as a command line debug utility along the lines of gdb, and it could possibly know how to start a pike process for debugging or could also just allow connecting to a running pike process which has the debugger enabled. This is basically how the Java debug agent works, and it allows local and remote debugging, which is pretty handy. Bill
Update:
with regard to multi-process debugging, some time ago I wrote a fun little feature that lets you open a debug server by sending a SIGUSR1 to a selected process/thread. A temporary substitute to multi-threaded debugging, also useful for debugging forked processes. I haven't pushed it to any remote repository yet, but I'm planning to.
I'm really keen on seeing your your changes related to thread naming and debugging, Bill, that would be a huge step forward. Maybe you could push the changes to the repo?
I'm also thinking about a huge clean-up of the debugger branch, even branching out of the freshest master and merging/rewriting previous changes in a more elegant way.
Any chance for me to get access to the official Pike repository? It would me much more comfortable to have my own branch/branches instead of working with a forked github repo mirror.
niedz., 7 kwi 2019 o 12:44 Mateusz Krawczuk krawczukmat@gmail.com napisał(a):
I've verified my hypothesis and made breakpoints in classes work with commit [1]. The way I did it solves the problem, but I believe it should have been done on a lower level.
I've also looked up the program cache and verified that there in fact already is such a structure. I'll work on avoiding the necessity to double it.
[1] https://github.com/mkrawczuk/Pike/commit/bbdf2c23fb536ef42f0d83d14af84a06849...
wt., 2 kwi 2019 o 14:55 Mateusz Krawczuk krawczukmat@gmail.com napisał(a):
That's good news, looking forward to see the update!
My branch is a little specific - I've completely disabled the Hilfe debugger, replacing it by the debug adapter. For this reason you need a DAP-compliant IDE to test on my branch. The examples I provided earlier should suffice, but if you have any problems running it - do not hesitate to ask.
Previously I expressed myself incorrectly. I did not mean class methods (usually called static methods). What was on my mind were the methods, i.e. functions defined inside a class (nested program, in Pike terminology), that can be called on objects instantiating the given class. You currently can't break on them. For example, given
class C { int a; void create() { a = 1; werror("%d\n", a); }
void f() { ++a; werror("%d\n", a); } }
int main(int arc, array(string) argv) { C c = C(); c->f(); werror("c->a: %d\n", c->a); }
It will not stop on breakpoints set inside C's create() and f(). Setting breakpoints inside main() should go fine though.
I did a little research and I might have a potential fix for it. Looks like nested programs are getting their own separate entry. So all the data related to nested programs, like line offsets, are stored in a different object for each defined class. What is done now in debug_breakpoint class's low_enable_breakpoint, is calling low_get_offset_for_line only for one program object that is directly related to the file. What should be done instead, is checking all the programs originating from a given file, so the nested programs are also checked for the needed line offset.
So what I need is to get a list of program objects from a provided file string so I can loop through it. It needs to be available in builtin.cmod. Nothing comes to my mind on how to do it, any help is strongly appreciated.
śr., 13 mar 2019 o 15:01 H. William Welliver III william@welliver.org napisał(a):
That’s cool. I suspect that Parser.LR may require a bit of work to make it nicer to build a parser of this complexity, but I think it would be worth the effort, as it would allow a lot of interesting information to be extracted from pike classes… dependency graphs, data to allow refactoring, various types of analysis, etc.
I’ll also take a look at your branch; I sort of gave up trying to keep my branch in sync with some of the changes Grubba has been making to support live backtrace frames and such. I also have some changes to the thread object that permits setting of names on threads (very useful when debugging) and being able to break execution on a thread regardless of whatever it’s running, etc.
Bill
On Mar 13, 2019, at 8:48 AM, Mateusz Krawczuk krawczukmat@gmail.com wrote:
CCing dozzie (aka Jarowit). He is proficient with parsers and is eager to put some effort to carry this step on.
pon., 11 mar 2019 o 17:43 Mateusz Krawczuk krawczukmat@gmail.com napisał(a):
What exactly is this hypothetical parser needed for? Can you give more detail on what is needed?
I've been working on the debug adapter for a while now and made it possible to work with DAP-compliant IDEs, like VScode. You can check it out at the 'debugger' branch of my Pike fork [1]. Breakpoints, stack trace and variable browsing seems to work pretty well. I've also made it possible to attach to a running Pike process and set breakpoints at runtime. To make it possible I've introduced a program cache, which is string-program mapping. It exists in the master object and is updated every time a program is loaded. It is needed because the way we handle breakpoint initialization deferring is insufficient for this scenario. I wonder how does it affect memory optimization.
One major thing that is missing is setting breakpoints on class methods. Any clues how to implement it?
[1] https://github.com/mkrawczuk/Pike/tree/debugger
czw., 7 lut 2019 o 19:14 H. William Welliver III william@welliver.org napisał(a):
Update:
I took a little time this morning and merged grubba's local variable names branch with the debugger branch, and it looks pretty good.
Seems to me that the next piece of infrastructure that's needed probably involves some sort of Pike parser. I've looked at the java parser a bit and I'm not really happy with it. Perhaps it would make sense to extend Parser.LR to work with Parser.C.Tokens and generate a pike grammar for use with it?
I looked a little bit at Parser.LR and it looks like some features we'd want might not be complete (like %token) or that would make maintaining similarity with the existing grammar (like |). Does anyone have much experience using Parser.LR or might have some real examples?
Bill
February 6, 2019 8:20 PM, "H. William Welliver III" < william@welliver.org <%22H.%20William%20Welliver%20III%22%20%3Cwilliam@welliver.org%3E>> wrote:
I think there’s a sense that the parts of the debug infrastructure that run within the process under debug should be as lightweight as possible, and not be active unless the process is being debugged. In keeping with this line of thought, I think the in-process portion of the agent shouldn’t be fully featured, as that would imply that it would be parsing code for metadata like block locations and the like (for stepping over, etc). The —debugger option is intended to be the mechanism which activates the in-process debugging agent. If your concern is that connecting to a hilfe session is not an ideal debugging tool, that is true, but it was never my intention for that to be the final state. Rather I had some code that served a similar purpose and it didn’t make sense to go off building a lot of infrastructure if the premise wasn’t workable. Something along these lines is what I had in mind: Debugging Tool <— DAP —> Debug Agent <— DAP-lite —> Pike process under debug Now, the debugging tool and Debug Agent could theoretically be a single process, such as a command line debug utility along the lines of gdb, and it could possibly know how to start a pike process for debugging or could also just allow connecting to a running pike process which has the debugger enabled. This is basically how the Java debug agent works, and it allows local and remote debugging, which is pretty handy. Bill
Previously, on Pike Debugger:
we've ended up with a working proof-of-concept debugger with a set of cool features, including breakpoints, stepping, variable inspection, and an ability to debug a selected thread by firing SIGUSR1 at it (as a way ti debug multi-threaded programs). Unfortunately, the debugger branch got so far away from the master branch, that it became practically impossible to merge it.
I took some (a lot, actually) of my time and rewrote the code manually on top of a fresh master. It's now available at 'mkrawczuk/debugger' branch in the main Pke repo (not the github mirror, as previously).
Could you guys please take some time on the meetup and review my last commit on this branch (6386f2de4b1) and consider merging it into master? The core part for breakpoints is (except some minor TODOs) done, and there's much more work in the Debugger module. But having those changes inside master I won't have to worry about future merge conflicts similar to those we've experienced the last time.
As previously, the debugger is available only if Pike is compiled --with-debug. To try it out, you need an IDE that supports Microsoft's Debug Adapter Protocol (Visual Studio Code is one of them, although there are plugins for other popular editors like emacs or vim). run $ pike --debugger --debugger-wait=(n) (your-program.pike) where n is a number of seconds the debugger will wait for a connection from the debug client, and your-program.pike is a program you'd like to debug. If everything goes ok, the debug server will listen on port 3333. Connect your debug adapter to it.
Voila! Although a little buggy, breakpoints and stack inspection should work fine.
As I understand, commits e13e50ad and 710fa97fa are work towards proper local variable inspection and setting in debugger. Previously we've relied on a different mechanism introduced in branch grubba/wop-local-variables-debug-info. I'll re-enable this feature basing on this mechanism in my free time.
I'll provide a more detailed documentation for this feature but what is the most important for me now is merging the current code into master, and the upcoming meetup (which I'll miss, unfortunately) seems to be a perfect occasion for it. Let me know is there anything standing n the way.
niedz., 7 kwi 2019 o 12:44 Mateusz Krawczuk krawczukmat@gmail.com napisał(a):
I've verified my hypothesis and made breakpoints in classes work with commit [1]. The way I did it solves the problem, but I believe it should have been done on a lower level.
I've also looked up the program cache and verified that there in fact already is such a structure. I'll work on avoiding the necessity to double it.
[1] https://github.com/mkrawczuk/Pike/commit/bbdf2c23fb536ef42f0d83d14af84a06849...
wt., 2 kwi 2019 o 14:55 Mateusz Krawczuk krawczukmat@gmail.com napisał(a):
That's good news, looking forward to see the update!
My branch is a little specific - I've completely disabled the Hilfe debugger, replacing it by the debug adapter. For this reason you need a DAP-compliant IDE to test on my branch. The examples I provided earlier should suffice, but if you have any problems running it - do not hesitate to ask.
Previously I expressed myself incorrectly. I did not mean class methods (usually called static methods). What was on my mind were the methods, i.e. functions defined inside a class (nested program, in Pike terminology), that can be called on objects instantiating the given class. You currently can't break on them. For example, given
class C { int a; void create() { a = 1; werror("%d\n", a); }
void f() { ++a; werror("%d\n", a); } }
int main(int arc, array(string) argv) { C c = C(); c->f(); werror("c->a: %d\n", c->a); }
It will not stop on breakpoints set inside C's create() and f(). Setting breakpoints inside main() should go fine though.
I did a little research and I might have a potential fix for it. Looks like nested programs are getting their own separate entry. So all the data related to nested programs, like line offsets, are stored in a different object for each defined class. What is done now in debug_breakpoint class's low_enable_breakpoint, is calling low_get_offset_for_line only for one program object that is directly related to the file. What should be done instead, is checking all the programs originating from a given file, so the nested programs are also checked for the needed line offset.
So what I need is to get a list of program objects from a provided file string so I can loop through it. It needs to be available in builtin.cmod. Nothing comes to my mind on how to do it, any help is strongly appreciated.
śr., 13 mar 2019 o 15:01 H. William Welliver III william@welliver.org napisał(a):
That’s cool. I suspect that Parser.LR may require a bit of work to make it nicer to build a parser of this complexity, but I think it would be worth the effort, as it would allow a lot of interesting information to be extracted from pike classes… dependency graphs, data to allow refactoring, various types of analysis, etc.
I’ll also take a look at your branch; I sort of gave up trying to keep my branch in sync with some of the changes Grubba has been making to support live backtrace frames and such. I also have some changes to the thread object that permits setting of names on threads (very useful when debugging) and being able to break execution on a thread regardless of whatever it’s running, etc.
Bill
On Mar 13, 2019, at 8:48 AM, Mateusz Krawczuk krawczukmat@gmail.com wrote:
CCing dozzie (aka Jarowit). He is proficient with parsers and is eager to put some effort to carry this step on.
pon., 11 mar 2019 o 17:43 Mateusz Krawczuk krawczukmat@gmail.com napisał(a):
What exactly is this hypothetical parser needed for? Can you give more detail on what is needed?
I've been working on the debug adapter for a while now and made it possible to work with DAP-compliant IDEs, like VScode. You can check it out at the 'debugger' branch of my Pike fork [1]. Breakpoints, stack trace and variable browsing seems to work pretty well. I've also made it possible to attach to a running Pike process and set breakpoints at runtime. To make it possible I've introduced a program cache, which is string-program mapping. It exists in the master object and is updated every time a program is loaded. It is needed because the way we handle breakpoint initialization deferring is insufficient for this scenario. I wonder how does it affect memory optimization.
One major thing that is missing is setting breakpoints on class methods. Any clues how to implement it?
[1] https://github.com/mkrawczuk/Pike/tree/debugger
czw., 7 lut 2019 o 19:14 H. William Welliver III william@welliver.org napisał(a):
Update:
I took a little time this morning and merged grubba's local variable names branch with the debugger branch, and it looks pretty good.
Seems to me that the next piece of infrastructure that's needed probably involves some sort of Pike parser. I've looked at the java parser a bit and I'm not really happy with it. Perhaps it would make sense to extend Parser.LR to work with Parser.C.Tokens and generate a pike grammar for use with it?
I looked a little bit at Parser.LR and it looks like some features we'd want might not be complete (like %token) or that would make maintaining similarity with the existing grammar (like |). Does anyone have much experience using Parser.LR or might have some real examples?
Bill
February 6, 2019 8:20 PM, "H. William Welliver III" < william@welliver.org <%22H.%20William%20Welliver%20III%22%20%3Cwilliam@welliver.org%3E>> wrote:
I think there’s a sense that the parts of the debug infrastructure that run within the process under debug should be as lightweight as possible, and not be active unless the process is being debugged. In keeping with this line of thought, I think the in-process portion of the agent shouldn’t be fully featured, as that would imply that it would be parsing code for metadata like block locations and the like (for stepping over, etc). The —debugger option is intended to be the mechanism which activates the in-process debugging agent. If your concern is that connecting to a hilfe session is not an ideal debugging tool, that is true, but it was never my intention for that to be the final state. Rather I had some code that served a similar purpose and it didn’t make sense to go off building a lot of infrastructure if the premise wasn’t workable. Something along these lines is what I had in mind: Debugging Tool <— DAP —> Debug Agent <— DAP-lite —> Pike process under debug Now, the debugging tool and Debug Agent could theoretically be a single process, such as a command line debug utility along the lines of gdb, and it could possibly know how to start a pike process for debugging or could also just allow connecting to a running pike process which has the debugger enabled. This is basically how the Java debug agent works, and it allows local and remote debugging, which is pretty handy. Bill
Hi Mateusz,
We had a long discussion about this, and the main reservation was if having one global debug object in master is the correct final solution. But no one had plans on an alternative solution right now, and we absolutely do not want to stand in the way of something that finally provides a new debugger.
So the compromise is this: Your changes should be merged, but if someone comes up with a better design this might be ripped out and replaced, don't consider that part a stable API.
Before it's merged, this should be done in your branch:
* Divide the changes into semantic commits, something like: - One commit with the master changes - At least one with the src/* changes' - One with the Debug module * set_debugger() should probably not be an efun. * Most to all of the stuff in builtin.cmod could probably live in modules/_Debug. * printf() within PIKE_DEBUG (or otherwise) should be fprintf() to stderr. * In a PIKECLASS, CVARS are automagically initialised to zero, there is no need to manually do that in INIT. * The introduction of low_low_backtrace seems completely redundant, there is already a flags argument to low_backtrace, and low_bracktrace already keeps a ref to the bp. * f_cq___debug_backtrace seems unnecessary, backtrace(1) already exists. * The preference would be to have a function in the master that returns the debugger handler instead of calling resolv in the master and then calling the result of resolve(). * The malloc on sizeof(svalue) line in interpret.c is broken and needs to just be removed. The code should be good if that line is removed. Then feel free to remove the TODO above. * The change in low_mega_apply; s -> obj_name is a bugfix and is now commited separately on master. * There is no need for an extra "stepping_mode" in struct thread_state, a new debug_flags can be used instead. * There is no need for the program pointer in the struct debug_breakpoints, it's already in the program. * Move low_enable_breakpoint and low_disable_breakpoint to program.c
TODO, but not required for merge:
* Change the comments in DebugAdapterProtocol into autodoc compatible syntax. More autodoc in general would be nice. * Update the pike.1 man page with the new options. * The low_get_offset_for_line API should return a set of offsets, since a 1:1 mapping is not guaranteed. * low_get_offset_for_line is probably broken in regards to local variable information. * The breakpoint handling is will be very slow, but we'll handle this in a better manner together. * There is no need for the debug_breakpoints to be a double linked list since it's only traversed in one direction.
Regards,
Thanks for the feedback! Trying to limit time spent at the computer during summertime, but I'll certainly fix the issues before the next meetup.
niedz., 4 sie 2019, 18:37 użytkownik Peter Bortas bortas@gmail.com napisał:
Hi Mateusz,
We had a long discussion about this, and the main reservation was if having one global debug object in master is the correct final solution. But no one had plans on an alternative solution right now, and we absolutely do not want to stand in the way of something that finally provides a new debugger.
So the compromise is this: Your changes should be merged, but if someone comes up with a better design this might be ripped out and replaced, don't consider that part a stable API.
Before it's merged, this should be done in your branch:
- Divide the changes into semantic commits, something like:
- One commit with the master changes
- At least one with the src/* changes'
- One with the Debug module
- set_debugger() should probably not be an efun.
- Most to all of the stuff in builtin.cmod could probably live in modules/_Debug.
- printf() within PIKE_DEBUG (or otherwise) should be fprintf() to stderr.
- In a PIKECLASS, CVARS are automagically initialised to zero, there is no need to manually do that in INIT.
- The introduction of low_low_backtrace seems completely redundant, there is already a flags argument to low_backtrace, and low_bracktrace already keeps a ref to the bp.
- f_cq___debug_backtrace seems unnecessary, backtrace(1) already exists.
- The preference would be to have a function in the master that returns the debugger handler instead of calling resolv in the master and then calling the result of resolve().
- The malloc on sizeof(svalue) line in interpret.c is broken and needs to just be removed. The code should be good if that line is removed. Then feel free to remove the TODO above.
- The change in low_mega_apply; s -> obj_name is a bugfix and is now commited separately on master.
- There is no need for an extra "stepping_mode" in struct thread_state, a new debug_flags can be used instead.
- There is no need for the program pointer in the struct debug_breakpoints, it's already in the program.
- Move low_enable_breakpoint and low_disable_breakpoint to program.c
TODO, but not required for merge:
- Change the comments in DebugAdapterProtocol into autodoc compatible syntax. More autodoc in general would be nice.
- Update the pike.1 man page with the new options.
- The low_get_offset_for_line API should return a set of offsets, since a 1:1 mapping is not guaranteed.
- low_get_offset_for_line is probably broken in regards to local variable information.
- The breakpoint handling is will be very slow, but we'll handle this in a better manner together.
- There is no need for the debug_breakpoints to be a double linked list since it's only traversed in one direction.
Regards,
Peter Bortas
On Sat, Aug 3, 2019 at 3:37 PM Mateusz Krawczuk krawczukmat@gmail.com wrote:
Previously, on Pike Debugger:
we've ended up with a working proof-of-concept debugger with a set of
cool features, including breakpoints, stepping, variable inspection, and an ability to debug a selected thread by firing SIGUSR1 at it (as a way ti debug multi-threaded programs). Unfortunately, the debugger branch got so far away from the master branch, that it became practically impossible to merge it.
I took some (a lot, actually) of my time and rewrote the code manually
on top of a fresh master. It's now available at 'mkrawczuk/debugger' branch in the main Pke repo (not the github mirror, as previously).
Could you guys please take some time on the meetup and review my last
commit on this branch (6386f2de4b1) and consider merging it into master? The core part for breakpoints is (except some minor TODOs) done, and there's much more work in the Debugger module. But having those changes inside master I won't have to worry about future merge conflicts similar to those we've experienced the last time.
As previously, the debugger is available only if Pike is compiled
--with-debug.
To try it out, you need an IDE that supports Microsoft's Debug Adapter
Protocol (Visual Studio Code is one of them, although there are plugins for other popular editors like emacs or vim).
run $ pike --debugger --debugger-wait=(n) (your-program.pike) where n is
a number of seconds the debugger will wait for a connection from the debug client, and your-program.pike is a program you'd like to debug.
If everything goes ok, the debug server will listen on port 3333.
Connect your debug adapter to it.
Voila! Although a little buggy, breakpoints and stack inspection should
work fine.
As I understand, commits e13e50ad and 710fa97fa are work towards proper
local variable inspection and setting in debugger. Previously we've relied on a different mechanism introduced in branch grubba/wop-local-variables-debug-info. I'll re-enable this feature basing on this mechanism in my free time.
I'll provide a more detailed documentation for this feature but what is
the most important for me now is merging the current code into master, and the upcoming meetup (which I'll miss, unfortunately) seems to be a perfect occasion for it. Let me know is there anything standing n the way.
niedz., 7 kwi 2019 o 12:44 Mateusz Krawczuk krawczukmat@gmail.com
napisał(a):
I've verified my hypothesis and made breakpoints in classes work with
commit [1].
The way I did it solves the problem, but I believe it should have been
done on a lower level.
I've also looked up the program cache and verified that there in fact
already is such a structure. I'll work on avoiding the necessity to double it.
[1]
https://github.com/mkrawczuk/Pike/commit/bbdf2c23fb536ef42f0d83d14af84a06849...
wt., 2 kwi 2019 o 14:55 Mateusz Krawczuk krawczukmat@gmail.com
napisał(a):
That's good news, looking forward to see the update!
My branch is a little specific - I've completely disabled the Hilfe
debugger, replacing it by the debug adapter. For this reason you need a DAP-compliant IDE to test on my branch. The examples I provided earlier should suffice, but if you have any problems running it - do not hesitate to ask.
Previously I expressed myself incorrectly. I did not mean class
methods (usually called static methods). What was on my mind were the methods, i.e. functions defined inside a class (nested program, in Pike terminology), that can be called on objects instantiating the given class.
You currently can't break on them. For example, given
class C { int a; void create() { a = 1; werror("%d\n", a); }
void f() { ++a; werror("%d\n", a); } }
int main(int arc, array(string) argv) { C c = C(); c->f(); werror("c->a: %d\n", c->a); }
It will not stop on breakpoints set inside C's create() and f().
Setting breakpoints inside main() should go fine though.
I did a little research and I might have a potential fix for it. Looks like nested programs are getting their own separate entry. So
all the data related to nested programs, like line offsets, are stored in a different object for each defined class.
What is done now in debug_breakpoint class's low_enable_breakpoint, is
calling low_get_offset_for_line only for one program object that is directly related to the file. What should be done instead, is checking all the programs originating from a given file, so the nested programs are also checked for the needed line offset.
So what I need is to get a list of program objects from a provided
file string so I can loop through it. It needs to be available in builtin.cmod.
Nothing comes to my mind on how to do it, any help is strongly
appreciated.
śr., 13 mar 2019 o 15:01 H. William Welliver III william@welliver.org
napisał(a):
That’s cool. I suspect that Parser.LR may require a bit of work to
make it nicer to build a parser of this complexity, but I think it would be worth the effort, as it would allow a lot of interesting information to be extracted from pike classes… dependency graphs, data to allow refactoring, various types of analysis, etc.
I’ll also take a look at your branch; I sort of gave up trying to
keep my branch in sync with some of the changes Grubba has been making to support live backtrace frames and such. I also have some changes to the thread object that permits setting of names on threads (very useful when debugging) and being able to break execution on a thread regardless of whatever it’s running, etc.
Bill
On Mar 13, 2019, at 8:48 AM, Mateusz Krawczuk krawczukmat@gmail.com
wrote:
CCing dozzie (aka Jarowit). He is proficient with parsers and is
eager to put some effort to carry this step on.
pon., 11 mar 2019 o 17:43 Mateusz Krawczuk krawczukmat@gmail.com
napisał(a):
What exactly is this hypothetical parser needed for? Can you give
more detail on what is needed?
I've been working on the debug adapter for a while now and made it
possible to work with DAP-compliant IDEs, like VScode. You can check it out at the 'debugger' branch of my Pike fork [1].
Breakpoints, stack trace and variable browsing seems to work pretty
well. I've also made it possible to attach to a running Pike process and set breakpoints at runtime. To make it possible I've introduced a program cache, which is string-program mapping. It exists in the master object and is updated every time a program is loaded. It is needed because the way we handle breakpoint initialization deferring is insufficient for this scenario. I wonder how does it affect memory optimization.
One major thing that is missing is setting breakpoints on class
methods. Any clues how to implement it?
[1] https://github.com/mkrawczuk/Pike/tree/debugger
czw., 7 lut 2019 o 19:14 H. William Welliver III <
william@welliver.org> napisał(a):
> > Update: > > I took a little time this morning and merged grubba's local
variable names branch with the debugger branch, and it looks pretty good.
> > Seems to me that the next piece of infrastructure that's needed
probably involves some sort of Pike parser. I've looked at the java parser a bit and I'm not really happy with it. Perhaps it would make sense to extend Parser.LR to work with Parser.C.Tokens and generate a pike grammar for use with it?
> > I looked a little bit at Parser.LR and it looks like some features
we'd want might not be complete (like %token) or that would make maintaining similarity with the existing grammar (like |). Does anyone have much experience using Parser.LR or might have some real examples?
> > Bill > > February 6, 2019 8:20 PM, "H. William Welliver III" <
william@welliver.org> wrote:
> > I think there’s a sense that the parts of the debug infrastructure
that run within the process under debug should be as lightweight as possible, and not be active unless the process is being debugged. In keeping with this line of thought, I think the in-process portion of the agent shouldn’t be fully featured, as that would imply that it would be parsing code for metadata like block locations and the like (for stepping over, etc). The —debugger option is intended to be the mechanism which activates the in-process debugging agent.
> If your concern is that connecting to a hilfe session is not an
ideal debugging tool, that is true, but it was never my intention for that to be the final state. Rather I had some code that served a similar purpose and it didn’t make sense to go off building a lot of infrastructure if the premise wasn’t workable.
> Something along these lines is what I had in mind: > Debugging Tool <— DAP —> Debug Agent <— DAP-lite —> Pike process
under debug
> Now, the debugging tool and Debug Agent could theoretically be a
single process, such as a command line debug utility along the lines of gdb, and it could possibly know how to start a pike process for debugging or could also just allow connecting to a running pike process which has the debugger enabled. This is basically how the Java debug agent works, and it allows local and remote debugging, which is pretty handy.
> Bill
As discussed, I did the pre-merge cleanup, except:
- set_debugger() should probably not be an efun.
Apparently it has to be, else it appears as an unknown identifier in master.pike.
- Move low_enable_breakpoint and low_disable_breakpoint to program.c
I didn't really see any point in this, since these are helper methods of the debug_breakpoint pikeclass that change it's state.
I will go further with the TODOs, but I would feel much more comfortable if the changes in src and master were on master branch, since it would spare me regular sync nightmares like merge conflicts, etc. So if there is anything else to correct, please let me know. Otherwise - feel free to merge the changes.
They can be found on branch mkrawczuk_debugger_refactor. By the way, I was really unhappy about breaking the branch naming convention. The pre-receive hook kept telling me that "Common topic branch names are not allowed to contain /.", and I wanted to name my branch "mkrawczuk/debugger_refactor".
Again, thanks for a great conference and hoping to see you soon.
pike-devel@lists.lysator.liu.se
-
H. William Welliver III -
Henrik Grubbstr�m (Lysator) ＠ Pike (-) developers forum -
Mateusz Krawczuk -
Peter Bortas -
Stephen R. van den Berg