Hello,
We have a memory leak inside Camas (and it exists most propably on IMHO as well). I've tried to debug in order to find where the leak is (I used Pike 7.2). We know the leak is inside the session object, it contains all sort of variables including other objects, mapping, string and so on (people interested in the source can get it here http://cvsweb.caudiumforge.net/cgi-bin/cvsweb.cgi/camas/server/modules/camas/camas.pike?rev=1.337&content-type=text/x-cvsweb-markup&cvsroot=caudium).
I've tried to force gc() every x minutes and when the user logout but it doesn't change the problem. I also added this in the destroy function of the session:
void destroy_object(object o) { if(o && objectp(o)) destruct(o); } void destroy() { foreach(indices(this_object()), string var) { if(this_object()[var] && !intp(this_object()[var])) { int refs = _refs(this_object()[var]); if(refs > 2) write("variable %s is referenced %d times\n", var, refs); } } destroy_object(lock); destroy_object(imapclient); destroy_object(cmail); destroy_object(prefsmailobj); destroy_object(extendedabook); destroy_object(ebookmailobj); destruct(this_object()); }
Which shows me lot of variables are referenced several times and for some of then, more than 7000 times. I also noted that the variables are usually referenced by similar number of other variables, for example I have this output:
variable draftmail is referenced 1107 times variable misspelled is referenced 7067 times variable spelling is referenced 7067 times variable deletemails is referenced 7067 times variable deletemboxes is referenced 7067 times variable prefuids is referenced 7067 times variable searchstring is referenced 1107 times variable session is referenced 1107 times
I also saw the destroy or dump_me functions are referenced 8 times while they are not used by other things in the program.
So my first questions are:
. How can I have so much objects referenced ? Is it because of some sort of reference loop ? Is _refs() working good on Pike 7.2 ? . What does a reference for a function mean ? . Is it possible to know which things reference which variables ? . What is the effect of adding a destroy() inside an object ?
I also enable debug in gc.c to find the problem but the output looks quite cryptic to me. At the beginning of the destroy I have this:
Garbage collecting ... | check: 33151 references checked, counted 489 weak refs | mark: 15799 markers referenced, 0 weak references freed, | 0 things really freed, got 0 tricky weak refs
Then lot of enqueue 0x89223ec [(nil)]: back=(nil), prev=0x81d1990, next=(nil), data=0x850cd2 4, cycle=0, flags=0x01 gc_cycle_push, recurse 0x850cd24 [(nil)] marker at 0x89495ec: flags =0x6446, refs=1, weak=-1, xrefs=0, saved=0, frame=0x89223ec [back=(nil), prev=0x 81d1990, next=(nil), data=0x850cd24, cycle=0, flags=0x01] ... dequeue 0x89222b4 [0x89222cc]: back=0x89222cc, prev=0x89223ec, next=(nil), data= 0x88ade1c, cycle=0, flags=0x01 gc_cycle_pop, pop off 0x88ade1c [0x8606e0c] marker at 0x898bb2c: flags=0x040e, refs=1, weak=0, xrefs=0, saved=1, frame=0x89222b4 [back=0xfffffff f, prev=0x89223ec, next=(nil), data=0x88ade1c, cycle=0, flags=0x09] .... gc_cycle_push, no recurse 0x88ad8ac [(nil)] marker at 0x89986b4: flags =0x040e, refs=1, weak=0, xrefs=0, saved=1, frame=(nil) and finally: | cycle: 554 internal things visited, 14 cycle ids used, | 0 weak references freed, 0 things really freed, | space for 48 gc frames used | free: 26 really freed, 528 left with live references | Killing 0x850e4c4 with 25 refs .... | Killing 0x8606e0c with 25 refs | kill: 23 objects killed, 528 things really freed | destruct: 0 things really freed done (freed 554 of 14900 objects)
. Can someone explained a little the output if it worth it ? Especially I would like to know what is a weak reference ? Why some references are weaker than others ? what does object killed mean ?
I also want to know if _next() and _prev() can be of any help to me and if yes how can I use it ?
Thank you for your time.
variable draftmail is referenced 1107 times variable misspelled is referenced 7067 times variable spelling is referenced 7067 times variable deletemails is referenced 7067 times variable deletemboxes is referenced 7067 times variable prefuids is referenced 7067 times variable searchstring is referenced 1107 times variable session is referenced 1107 times
Are all these really variables in your object? If so, do they point to functions in some global module or something like that?
. How can I have so much objects referenced ?
I've no idea. The only things I can think of that get that many references are global classes like Stdio.File and efuns like aggregate_mapping and `+.
Is it because of some sort of reference loop ?
You mean like objects that refer to each other? No, that doesn't accumulate refs transitively or something like that.
Is _refs() working good on Pike 7.2 ?
Yes, fairly well. One must just be aware that there'll be an extra ref on the stack during the call.
. What does a reference for a function mean ?
It's the same thing as a reference to the object. A function reference is really a reference to the object containing the function, with an additional offset for the function identifier.
. Is it possible to know which things reference which variables ?
There's one tool: If you compile with rtl-debug, there's a function _locate_references that walks through all things and prints the ones that reference the argument (in debug style). It's really useful, especially in 7.4 and later where it has been improved to report on the more odd sorts of references.
. What is the effect of adding a destroy() inside an object ?
Not much. It's handled a bit differently inside the gc to ensure a somewhat well defined destruct order.
I also enable debug in gc.c to find the problem but the output looks quite cryptic to me. /.../
I'm not surprised. ;) Really, that output isn't useful to hunt down leaks. _locate_references is, and if you suspect C-level leaks, the dmalloc debug mode.
If you suspect a bug inside the gc itself you can compile with rtl-debug and run with -d. That enables a whole suite of self tests inside it.
Especially I would like to know what is a weak reference ?
It's a reference that "doesn't count". It behaves like a normal reference except that the gc will ignore it when determining the structures that still are referenced.
Why some references are weaker than others ?
Not sure how to respond to this. Have you looked at the comment blurb near the top of gc.c?
what does object killed mean ?
It means that an object with a destroy function was destructed because the gc determined that it wasn't referenced anymore.
I also want to know if _next() and _prev() can be of any help to me and if yes how can I use it ?
Perhaps. You can use them to make a memory walker which allows you to see which objects have most clones etc. I suggest you take a look at server/config_interface/action/debug_info.pike in a recent Roxen release if you want to do that.
/ Martin Stjernholm, Roxen IS
Previous text:
2003-06-03 01:01: Subject: Debugging memory problems
Hello,
We have a memory leak inside Camas (and it exists most propably on IMHO as well). I've tried to debug in order to find where the leak is (I used Pike 7.2). We know the leak is inside the session object, it contains all sort of variables including other objects, mapping, string and so on (people interested in the source can get it here http://cvsweb.caudiumforge.net/cgi-bin/cvsweb.cgi/camas/server/modules/camas/camas.pike?rev=1.337&content-type=text/x-cvsweb-markup&cvsroot=caudium).
I've tried to force gc() every x minutes and when the user logout but it doesn't change the problem. I also added this in the destroy function of the session:
void destroy_object(object o) { if(o && objectp(o)) destruct(o); } void destroy() { foreach(indices(this_object()), string var) { if(this_object()[var] && !intp(this_object()[var])) { int refs = _refs(this_object()[var]); if(refs > 2) write("variable %s is referenced %d times\n", var, refs); } } destroy_object(lock); destroy_object(imapclient); destroy_object(cmail); destroy_object(prefsmailobj); destroy_object(extendedabook); destroy_object(ebookmailobj); destruct(this_object()); }
Which shows me lot of variables are referenced several times and for some of then, more than 7000 times. I also noted that the variables are usually referenced by similar number of other variables, for example I have this output:
variable draftmail is referenced 1107 times variable misspelled is referenced 7067 times variable spelling is referenced 7067 times variable deletemails is referenced 7067 times variable deletemboxes is referenced 7067 times variable prefuids is referenced 7067 times variable searchstring is referenced 1107 times variable session is referenced 1107 times
I also saw the destroy or dump_me functions are referenced 8 times while they are not used by other things in the program.
So my first questions are:
. How can I have so much objects referenced ? Is it because of some sort of reference loop ? Is _refs() working good on Pike 7.2 ? . What does a reference for a function mean ? . Is it possible to know which things reference which variables ? . What is the effect of adding a destroy() inside an object ?
I also enable debug in gc.c to find the problem but the output looks quite cryptic to me. At the beginning of the destroy I have this:
Garbage collecting ... | check: 33151 references checked, counted 489 weak refs | mark: 15799 markers referenced, 0 weak references freed, | 0 things really freed, got 0 tricky weak refs
Then lot of enqueue 0x89223ec [(nil)]: back=(nil), prev=0x81d1990, next=(nil), data=0x850cd2 4, cycle=0, flags=0x01 gc_cycle_push, recurse 0x850cd24 [(nil)] marker at 0x89495ec: flags =0x6446, refs=1, weak=-1, xrefs=0, saved=0, frame=0x89223ec [back=(nil), prev=0x 81d1990, next=(nil), data=0x850cd24, cycle=0, flags=0x01] ... dequeue 0x89222b4 [0x89222cc]: back=0x89222cc, prev=0x89223ec, next=(nil), data= 0x88ade1c, cycle=0, flags=0x01 gc_cycle_pop, pop off 0x88ade1c [0x8606e0c] marker at 0x898bb2c: flags=0x040e, refs=1, weak=0, xrefs=0, saved=1, frame=0x89222b4 [back=0xfffffff f, prev=0x89223ec, next=(nil), data=0x88ade1c, cycle=0, flags=0x09] .... gc_cycle_push, no recurse 0x88ad8ac [(nil)] marker at 0x89986b4: flags =0x040e, refs=1, weak=0, xrefs=0, saved=1, frame=(nil) and finally: | cycle: 554 internal things visited, 14 cycle ids used, | 0 weak references freed, 0 things really freed, | space for 48 gc frames used | free: 26 really freed, 528 left with live references | Killing 0x850e4c4 with 25 refs .... | Killing 0x8606e0c with 25 refs | kill: 23 objects killed, 528 things really freed | destruct: 0 things really freed done (freed 554 of 14900 objects)
. Can someone explained a little the output if it worth it ? Especially I would like to know what is a weak reference ? Why some references are weaker than others ? what does object killed mean ?
I also want to know if _next() and _prev() can be of any help to me and if yes how can I use it ?
Thank you for your time.
-- David Gourdelier
/ Brevbäraren
On Tue, 3 Jun 2003, Martin Stjernholm, Roxen IS @ Pike developers forum wrote:
Hi,
variable draftmail is referenced 1107 times variable misspelled is referenced 7067 times variable spelling is referenced 7067 times variable deletemails is referenced 7067 times variable deletemboxes is referenced 7067 times variable prefuids is referenced 7067 times variable searchstring is referenced 1107 times variable session is referenced 1107 times
Are all these really variables in your object? If so, do they point to functions in some global module or something like that?
Yes there are variables inside my object. There are not referenced by other global modules.
. Is it possible to know which things reference which variables ?
There's one tool: If you compile with rtl-debug, there's a function _locate_references that walks through all things and prints the ones that reference the argument (in debug style). It's really useful, especially in 7.4 and later where it has been improved to report on the more odd sorts of references.
Ok I will try that.
I also enable debug in gc.c to find the problem but the output looks quite cryptic to me. /.../
I'm not surprised. ;) Really, that output isn't useful to hunt down
Hehe :)
leaks. _locate_references is, and if you suspect C-level leaks, the dmalloc debug mode.
I don't think it's a problem at the C level but more propably at Camas or 123sessions level.
Why some references are weaker than others ?
Not sure how to respond to this. Have you looked at the comment blurb near the top of gc.c?
Yes.
On Tue, 3 Jun 2003, David Gourdelier wrote:
Are all these really variables in your object? If so, do they point to functions in some global module or something like that?
Yes there are variables inside my object. There are not referenced by other global modules.
. Is it possible to know which things reference which variables ?
There's one tool: If you compile with rtl-debug, there's a function _locate_references that walks through all things and prints the ones that reference the argument (in debug style). It's really useful, especially in 7.4 and later where it has been improved to report on the more odd sorts of references.
Ok I will try that.
Ok I've got some news. I've install Camas on a Caudium 1.3/Pike 7.4 and added this code:
foreach(indices(this_object()), string var) { if(this_object()[var] && !intp(this_object()[var])) { int refs = _refs(this_object()[var]); if(refs > 2) { write("\n------------------------------------\n" "variable %s is referenced %d times\n", var, refs); #if constant(_locate_references) write("by %O\n", _locate_references(var)); #endif } } }
And the output is ... well... strange :)
------------------------------------ variable destroy is referenced 8 times **Looking for references to 0x8377314: ***One ref found on current interpreter stack. -> at location 0x28337460 ----------end------------ **Done looking for references to 0x8377314. by "destroy"
------------------------------------ variable files is referenced 701 times **Looking for references to 0x83c4d24: ***One ref found on current interpreter stack. -> at location 0x28337460 ----------end------------ **Done looking for references to 0x83c4d24. by "files"
------------------------------------ variable name is referenced 87 times **Looking for references to 0x844085c: ***One ref found on current interpreter stack. -> at location 0x28337460 ----------end------------ **Done looking for references to 0x844085c. by "name"
And so on, every variables are referenced by themselves and the location address is the same. The location address is the same if the user login and logout, maybe it points to a module ? So is it possible to know what is at this address ?
Thanks,
You probably want to call _locate_references(this_object()[var]), otherwise you'll find all references to the string which contains the variable name...
/ Fredrik (Naranek) Hubinette (Real Build Master)
Previous text:
2003-06-03 19:43: Subject: Re: Debugging memory problems
On Tue, 3 Jun 2003, David Gourdelier wrote:
Are all these really variables in your object? If so, do they point to functions in some global module or something like that?
Yes there are variables inside my object. There are not referenced by other global modules.
. Is it possible to know which things reference which variables ?
There's one tool: If you compile with rtl-debug, there's a function _locate_references that walks through all things and prints the ones that reference the argument (in debug style). It's really useful, especially in 7.4 and later where it has been improved to report on the more odd sorts of references.
Ok I will try that.
Ok I've got some news. I've install Camas on a Caudium 1.3/Pike 7.4 and added this code:
foreach(indices(this_object()), string var) { if(this_object()[var] && !intp(this_object()[var])) { int refs = _refs(this_object()[var]); if(refs > 2) { write("\n------------------------------------\n" "variable %s is referenced %d times\n", var, refs); #if constant(_locate_references) write("by %O\n", _locate_references(var)); #endif } } }
And the output is ... well... strange :)
variable destroy is referenced 8 times **Looking for references to 0x8377314: ***One ref found on current interpreter stack. -> at location 0x28337460 ----------end------------ **Done looking for references to 0x8377314. by "destroy"
variable files is referenced 701 times **Looking for references to 0x83c4d24: ***One ref found on current interpreter stack. -> at location 0x28337460 ----------end------------ **Done looking for references to 0x83c4d24. by "files"
variable name is referenced 87 times **Looking for references to 0x844085c: ***One ref found on current interpreter stack. -> at location 0x28337460 ----------end------------ **Done looking for references to 0x844085c. by "name"
And so on, every variables are referenced by themselves and the location address is the same. The location address is the same if the user login and logout, maybe it points to a module ? So is it possible to know what is at this address ?
Thanks,
-- David Gourdelier
/ Brevbäraren
On Tue, 3 Jun 2003, Fredrik (Naranek) Hubinette (Real Build Master) @ Pike (-) developers forum wrote:
You probably want to call _locate_references(this_object()[var]), otherwise you'll find all references to the string which contains the variable name...
Oh, yes thanks :)
I get lot more output this way :)
/ Fredrik (Naranek) Hubinette (Real Build Master)
Previous text:
2003-06-03 19:43: Subject: Re: Debugging memory problems
On Tue, 3 Jun 2003, David Gourdelier wrote:
Are all these really variables in your object? If so, do they point to functions in some global module or something like that?
Yes there are variables inside my object. There are not referenced by other global modules.
. Is it possible to know which things reference which variables ?
There's one tool: If you compile with rtl-debug, there's a function _locate_references that walks through all things and prints the ones that reference the argument (in debug style). It's really useful, especially in 7.4 and later where it has been improved to report on the more odd sorts of references.
Ok I will try that.
Ok I've got some news. I've install Camas on a Caudium 1.3/Pike 7.4 and added this code:
foreach(indices(this_object()), string var) { if(this_object()[var] && !intp(this_object()[var])) { int refs = _refs(this_object()[var]); if(refs > 2) { write("\n------------------------------------\n" "variable %s is referenced %d times\n", var, refs); #if constant(_locate_references) write("by %O\n", _locate_references(var)); #endif } } }
And the output is ... well... strange :)
variable destroy is referenced 8 times **Looking for references to 0x8377314: ***One ref found on current interpreter stack. -> at location 0x28337460 ----------end------------ **Done looking for references to 0x8377314. by "destroy"
variable files is referenced 701 times **Looking for references to 0x83c4d24: ***One ref found on current interpreter stack. -> at location 0x28337460 ----------end------------ **Done looking for references to 0x83c4d24. by "files"
variable name is referenced 87 times **Looking for references to 0x844085c: ***One ref found on current interpreter stack. -> at location 0x28337460 ----------end------------ **Done looking for references to 0x844085c. by "name"
And so on, every variables are referenced by themselves and the location address is the same. The location address is the same if the user login and logout, maybe it points to a module ? So is it possible to know what is at this address ?
Thanks,
-- David Gourdelier
/ Brevbäraren
On Tue, 3 Jun 2003, David Gourdelier wrote:
. Is it possible to know which things reference which variables ?
There's one tool: If you compile with rtl-debug, there's a function _locate_references that walks through all things and prints the ones that reference the argument (in debug style). It's really useful, especially in 7.4 and later where it has been improved to report on the more odd sorts of references.
Ok I will try that.
After some debugging it looks like there is a problem with call_out. For example taking this code:
class C { mixed call;
void timeout() { }
void create() { call = call_out(timeout, 1); }
void close() { write("call=%O\n", call); remove_call_out(timeout); write("call=%O\n", call); } };
int main() { object o = C(); o->close(); write("call_out_info=%O\n", call_out_info()); write("refs=%d\n", _refs(o)); _locate_references(o); return -1; }
I get the following results:
call=({ /* 1 element */ memory()->C()->timeout }) call=({ /* 1 element */ memory()->C()->timeout }) call_out_info=({ }) refs=3 **Looking for references to 0x83a1ea0: ***One ref found in an array. -> from array 0x83a98c0 offset 28 **Location: 0x83a98c0 Type: array Refs: 1 **Describing array: Location=0x83a98c0 Refs=1, next=0x83f4100, prev=0x822d1e0, flags=0x0, size=1, malloced_size=1 Type field = array mapping multiset object function program string type int float <10> <11> <12> <13> <14> ({ timeout }) ******************* ----------end------------ ***One ref found on current interpreter stack. -> at location 0x283370f8 ----------end------------ **Done looking for references to 0x83a1ea0.
The problem is that timeout should not reference the object anymore.
However it works if I set call = 0 in close() : call=({ /* 1 element */ memory()->C()->timeout }) call=0 call_out_info=({ }) refs=2 **Looking for references to 0x83a1ea0: ***One ref found on current interpreter stack. -> at location 0x283370f8 ----------end------------ **Done looking for references to 0x83a1ea0.
Any ideas ?
The problem is that timeout should not reference the object anymore.
You mean the call variable? Yes, you have to zero it yourself; remove_call_out doesn't do that for you.
As you've noticed, the result from call_out is an array containing the function. When you put it in a variable in the same object, you'll get a classic reference cycle from the object via the array to itself. Thus, when the object runs out of external references it will still be around for a while, but it's not really a leak since the gc will get it eventually.
If you can't rely on the gc then you have to make sure that you break up all reference cycles, just like when you assign zero to the call variable. A robust way to do that is to destruct the object explicitly (but that can bring its own problems if there are multiple threads that access it etc). Roxen destructs the RequestID objects explicitly for this reason.
/ Martin Stjernholm, Roxen IS
Previous text:
2003-06-04 13:08: Subject: Re: Debugging memory problems
On Tue, 3 Jun 2003, David Gourdelier wrote:
. Is it possible to know which things reference which variables ?
There's one tool: If you compile with rtl-debug, there's a function _locate_references that walks through all things and prints the ones that reference the argument (in debug style). It's really useful, especially in 7.4 and later where it has been improved to report on the more odd sorts of references.
Ok I will try that.
After some debugging it looks like there is a problem with call_out. For example taking this code:
class C { mixed call;
void timeout() { }
void create() { call = call_out(timeout, 1); }
void close() { write("call=%O\n", call); remove_call_out(timeout); write("call=%O\n", call); } };
int main() { object o = C(); o->close(); write("call_out_info=%O\n", call_out_info()); write("refs=%d\n", _refs(o)); _locate_references(o); return -1; }
I get the following results:
call=({ /* 1 element */ memory()->C()->timeout }) call=({ /* 1 element */ memory()->C()->timeout }) call_out_info=({ }) refs=3 **Looking for references to 0x83a1ea0: ***One ref found in an array. -> from array 0x83a98c0 offset 28 **Location: 0x83a98c0 Type: array Refs: 1 **Describing array: Location=0x83a98c0 Refs=1, next=0x83f4100, prev=0x822d1e0, flags=0x0, size=1, malloced_size=1 Type field = array mapping multiset object function program string type int float <10> <11> <12> <13> <14> ({ timeout })
----------end------------ ***One ref found on current interpreter stack. -> at location 0x283370f8 ----------end------------ **Done looking for references to 0x83a1ea0.
The problem is that timeout should not reference the object anymore.
However it works if I set call = 0 in close() : call=({ /* 1 element */ memory()->C()->timeout }) call=0 call_out_info=({ }) refs=2 **Looking for references to 0x83a1ea0: ***One ref found on current interpreter stack. -> at location 0x283370f8 ----------end------------ **Done looking for references to 0x83a1ea0.
Any ideas ?
-- David Gourdelier
/ Brevbäraren
Hello,
The problem is that timeout should not reference the object anymore.
You mean the call variable? Yes, you have to zero it yourself; remove_call_out doesn't do that for you.
Ok thank you.
If you can't rely on the gc then you have to make sure that you break up all reference cycles, just like when you assign zero to the call variable. A robust way to do that is to destruct the object explicitly (but that can bring its own problems if there are multiple threads that access it etc). Roxen destructs the RequestID objects explicitly for this reason.
Ok. I have another problem with cycle reference. Given this code:
class session { object imapclient; string a = "toto"; }
class imapclient { object session; void create() { } void start(object _session) { session = _session; } void stop() { session = 0; } };
int main() { object sess = session(); sess->imapclient = imapclient(); sess->imapclient->start(sess); sess->imapclient->stop(); _locate_references(sess->imapclient); write("a=%s\n", sess->a); return -1; }
I get:
**Looking for references to 0x83a1e40: ***One ref found in an object. -> from object 0x83a1ea0 offset -445904 **In variable imapclient **In storage for inherit 0 **Location: 0x83a1ea0 Type: object Refs: 1 **Program id: 65621 **Describing program of object: **Program id: 65621, flags: f **Location: /home/david/scripts/memory.pike:52 **There is no parent (any longer?) ******************* ----------end------------ **Done looking for references to 0x83a1e40.
The problem is that I can't destruct(sess) because I need it after in main().
How is that a cycle? Or a problem, for that matter?
sess references the session object, which references the imapclient object. It's a single acyclic reference chain.
/ Marcus Comstedt (ACROSS) (Hail Ilpalazzo!)
Previous text:
2003-06-05 16:49: Subject: Re: Debugging memory problems
Hello,
The problem is that timeout should not reference the object anymore.
You mean the call variable? Yes, you have to zero it yourself; remove_call_out doesn't do that for you.
Ok thank you.
If you can't rely on the gc then you have to make sure that you break up all reference cycles, just like when you assign zero to the call variable. A robust way to do that is to destruct the object explicitly (but that can bring its own problems if there are multiple threads that access it etc). Roxen destructs the RequestID objects explicitly for this reason.
Ok. I have another problem with cycle reference. Given this code:
class session { object imapclient; string a = "toto"; }
class imapclient { object session; void create() { } void start(object _session) { session = _session; } void stop() { session = 0; } };
int main() { object sess = session(); sess->imapclient = imapclient(); sess->imapclient->start(sess); sess->imapclient->stop(); _locate_references(sess->imapclient); write("a=%s\n", sess->a); return -1; }
I get:
**Looking for references to 0x83a1e40: ***One ref found in an object. -> from object 0x83a1ea0 offset -445904 **In variable imapclient **In storage for inherit 0 **Location: 0x83a1ea0 Type: object Refs: 1 **Program id: 65621 **Describing program of object: **Program id: 65621, flags: f **Location: /home/david/scripts/memory.pike:52 **There is no parent (any longer?)
----------end------------ **Done looking for references to 0x83a1e40.
The problem is that I can't destruct(sess) because I need it after in main().
-- David Gourdelier
/ Brevbäraren
There's a reference back through sess->imapclient->session.
/ Martin Stjernholm, Roxen IS
Previous text:
2003-06-05 17:05: Subject: Re: Debugging memory problems
How is that a cycle? Or a problem, for that matter?
sess references the session object, which references the imapclient object. It's a single acyclic reference chain.
/ Marcus Comstedt (ACROSS) (Hail Ilpalazzo!)
On Thu, 5 Jun 2003, Martin Stjernholm, Roxen IS @ Pike developers forum wrote:
Hi,
Here are some more notes I have regarding this problem:
. It seems like empty array or string with the same content share the same references. That's why I got lot of references for the empty array. So I put something like ({ "ThisiSaRandDomString232323239084 "}) in every arrays of the objects and they are not referenced anymore. Is it enough to set those variables to 0 ? . _refs always report lot of references but _locate_references doesn't find them. For example I have lot of things like this for strings:
variable searchstring ("") is referenced 1044 times **Looking for references to 0x83773ec: **Done looking for references to 0x83773ec. variable defaultsentfolder ("") is referenced 1044 times **Looking for references to 0x83773ec: **Done looking for references to 0x83773ec.
Is it normal ?
. When I call destruct(), destroy() is sometimes not called after some time or after the user login again or if I reload the module.
/ David Gourdelier
Please ignore that; I missed the stop function.
/ Martin Stjernholm, Roxen IS
Previous text:
2003-06-05 17:20: Subject: Re: Debugging memory problems
There's a reference back through sess->imapclient->session.
/ Martin Stjernholm, Roxen IS
. It seems like empty array or string with the same content share the same references. That's why I got lot of references for the empty array. So I put something like ({ "ThisiSaRandDomString232323239084 "}) in every arrays of the objects and they are not referenced anymore.
Ok, that explains it. (But there's no use avoiding the empty string and array just becaused they're referenced from a lot of places.)
Is it enough to set those variables to 0 ?
Depends on what you want to accomplish. If you want to avoid garbage that's left around until the next gc then you only need to zero variables if their values directly or indirectly refers to the object they're in. Normal acyclic structures are freed as soon as possible through reference counting.
. _refs always report lot of references but _locate_references doesn't find them. For example I have lot of things like this for strings:
_locate_references doesn't work well for strings. It uses the gc to go through the memory and the gc doesn't care about strings since they can't reference anything else and therefore can't be part of cycles.
. When I call destruct(), destroy() is sometimes not called after some time or after the user login again or if I reload the module.
I assume there's a "not" too much in that sentence. Seems like destroy is called by the gc or as a result of that the object runs out of references. Are you absolutely sure that you call destruct on the right object? I often put in some debug code like this to make sure I can tell objects apart:
static int obj_counter; class MyObject { static int id = ++obj_counter; string _sprintf (int flag) {return flag == 'O' && "MyObject(" + id + ")";} }
/ Martin Stjernholm, Roxen IS
Previous text:
2003-06-10 12:05: Subject: Re: Debugging memory problems
On Thu, 5 Jun 2003, Martin Stjernholm, Roxen IS @ Pike developers forum wrote:
Hi,
Here are some more notes I have regarding this problem:
. It seems like empty array or string with the same content share the same references. That's why I got lot of references for the empty array. So I put something like ({ "ThisiSaRandDomString232323239084 "}) in every arrays of the objects and they are not referenced anymore. Is it enough to set those variables to 0 ? . _refs always report lot of references but _locate_references doesn't find them. For example I have lot of things like this for strings:
variable searchstring ("") is referenced 1044 times **Looking for references to 0x83773ec: **Done looking for references to 0x83773ec. variable defaultsentfolder ("") is referenced 1044 times **Looking for references to 0x83773ec: **Done looking for references to 0x83773ec.
Is it normal ?
. When I call destruct(), destroy() is sometimes not called after some time or after the user login again or if I reload the module.
/ David Gourdelier
Please ignore that; I missed the stop function.
/ Martin Stjernholm, Roxen IS
Previous text:
2003-06-05 17:20: Subject: Re: Debugging memory problems
There's a reference back through sess->imapclient->session.
/ Martin Stjernholm, Roxen IS
-- David Gourdelier
/ Brevbäraren
Hello,
Is it enough to set those variables to 0 ?
Depends on what you want to accomplish. If you want to avoid garbage that's left around until the next gc then you only need to zero variables if their values directly or indirectly refers to the object they're in. Normal acyclic structures are freed as soon as possible through reference counting.
Ok. Thank you for the help :)
. When I call destruct(), destroy() is sometimes not called after some time or after the user login again or if I reload the module.
I assume there's a "not" too much in that sentence. Seems like destroy is called by the gc or as a result of that the object runs out of references. Are you absolutely sure that you call destruct on the right object? I often put in some debug code like this to make sure I can tell objects apart:
This one was my fault, I did a destruct on the NULL value and didn't see it. It works now.
I also noted that arrays in Pike can share their content. For example if I fill 30MB in this arrays:
$ ps auxw|grep pike david 446 13.5 1.1 8784 4360 pts/4 S 23:25 0:00 pike
array memtest = ({ Stdio.File("/dev/zero", "r")->read(31457280) });
$ ps axuw|grep pike david 446 1.9 9.1 39632 35320 pts/4 S 23:25 0:01 pike
array memtest2 = ({ Stdio.File("/dev/zero", "r")->read(31457280) });
$ ps axuw|grep pike david 446 2.5 9.1 39632 35320 pts/4 S 23:25 0:01 pike
array memtest3 = ({ Stdio.File("/dev/zero", "r")->read(31457280) +
"kjshjkzerhzjkerhzkjer" }); $ ps axuw|grep pike david 446 1.6 17.0 70356 66044 pts/4 S 23:25 0:02 pike
That explains why I don't see the leak sometime. For example if I login as the same user again on the same computer (so I get the same session).
I also have another question. Why not taking an already existing garbage collector (for example http://www.hpl.hp.com/personal/Hans_Boehm/gc/) and adapt it to Pike ?
/ David Gourdelier
On Thu, 5 Jun 2003, Martin Stjernholm, Roxen IS @ Pike developers forum wrote:
Hi,
Here are some more notes I have regarding this problem:
. It seems like empty array or string with the same content share the same references. That's why I got lot of references for the empty array. So I put something like ({ "ThisiSaRandDomString232323239084 "}) in every arrays of the objects and they are not referenced anymore. Is it enough to set those variables to 0 ? . _refs always report lot of references but _locate_references doesn't find them. For example I have lot of things like this for strings:
variable searchstring ("") is referenced 1044 times **Looking for references to 0x83773ec: **Done looking for references to 0x83773ec. variable defaultsentfolder ("") is referenced 1044 times **Looking for references to 0x83773ec: **Done looking for references to 0x83773ec.
Is it normal ?
. When I call destruct(), destroy() is sometimes not called after some time or after the user login again or if I reload the module.
/ David Gourdelier
Please ignore that; I missed the stop function.
/ Martin Stjernholm, Roxen IS
Previous text:
2003-06-05 17:20: Subject: Re: Debugging memory problems
There's a reference back through sess->imapclient->session.
/ Martin Stjernholm, Roxen IS
-- David Gourdelier
/ Brevbäraren
I also have another question. Why not taking an already existing garbage collector (for example http://www.hpl.hp.com/personal/Hans_Boehm/gc/) and adapt it to Pike ?
The Pike garbage collector is also already existing, and what's more, it is already adapted to Pike. Hence, it seems the logical choice. ;-)
/ Marcus Comstedt (ACROSS) (Hail Ilpalazzo!)
Previous text:
2003-06-11 23:40: Subject: Re: Debugging memory problems
Hello,
Is it enough to set those variables to 0 ?
Depends on what you want to accomplish. If you want to avoid garbage that's left around until the next gc then you only need to zero variables if their values directly or indirectly refers to the object they're in. Normal acyclic structures are freed as soon as possible through reference counting.
Ok. Thank you for the help :)
. When I call destruct(), destroy() is sometimes not called after some time or after the user login again or if I reload the module.
I assume there's a "not" too much in that sentence. Seems like destroy is called by the gc or as a result of that the object runs out of references. Are you absolutely sure that you call destruct on the right object? I often put in some debug code like this to make sure I can tell objects apart:
This one was my fault, I did a destruct on the NULL value and didn't see it. It works now.
I also noted that arrays in Pike can share their content. For example if I fill 30MB in this arrays:
$ ps auxw|grep pike david 446 13.5 1.1 8784 4360 pts/4 S 23:25 0:00 pike
array memtest = ({ Stdio.File("/dev/zero", "r")->read(31457280) });
$ ps axuw|grep pike david 446 1.9 9.1 39632 35320 pts/4 S 23:25 0:01 pike
array memtest2 = ({ Stdio.File("/dev/zero", "r")->read(31457280) });
$ ps axuw|grep pike david 446 2.5 9.1 39632 35320 pts/4 S 23:25 0:01 pike
array memtest3 = ({ Stdio.File("/dev/zero", "r")->read(31457280) +
"kjshjkzerhzjkerhzkjer" }); $ ps axuw|grep pike david 446 1.6 17.0 70356 66044 pts/4 S 23:25 0:02 pike
That explains why I don't see the leak sometime. For example if I login as the same user again on the same computer (so I get the same session).
I also have another question. Why not taking an already existing garbage collector (for example http://www.hpl.hp.com/personal/Hans_Boehm/gc/) and adapt it to Pike ?
/ David Gourdelier
On Thu, 5 Jun 2003, Martin Stjernholm, Roxen IS @ Pike developers forum wrote:
Hi,
Here are some more notes I have regarding this problem:
. It seems like empty array or string with the same content share the same references. That's why I got lot of references for the empty array. So I put something like ({ "ThisiSaRandDomString232323239084 "}) in every arrays of the objects and they are not referenced anymore. Is it enough to set those variables to 0 ? . _refs always report lot of references but _locate_references doesn't find them. For example I have lot of things like this for strings:
variable searchstring ("") is referenced 1044 times **Looking for references to 0x83773ec: **Done looking for references to 0x83773ec. variable defaultsentfolder ("") is referenced 1044 times **Looking for references to 0x83773ec: **Done looking for references to 0x83773ec.
Is it normal ?
. When I call destruct(), destroy() is sometimes not called after some time or after the user login again or if I reload the module.
/ David Gourdelier
Please ignore that; I missed the stop function.
/ Martin Stjernholm, Roxen IS
Previous text:
2003-06-05 17:20: Subject: Re: Debugging memory problems
There's a reference back through sess->imapclient->session.
/ Martin Stjernholm, Roxen IS
-- David Gourdelier
/ Brevbäraren
-- David Gourdelier
/ Brevbäraren
I also noted that arrays in Pike can share their content. /.../
What you're seeing is that strings are shared; there's only one instance of every unique string in a pike process. That's why e.g. indexing mappings with strings is so fast.
I also have another question. Why not taking an already existing garbage collector (for example http://www.hpl.hp.com/personal/Hans_Boehm/gc/) and adapt it to Pike ?
There's a tight integration between the gc and the data types in Pike, so I suspect adapting an existing one would essentially mean rewriting it.
The one you pointed out scans through the memory looking for patterns that can be references to or into allocated blocks. Very convenient, but it means it can keep blocks around too long just because there are values that happen to look like pointers, and it also means quite a bit more work scanning the blocks during gc. The one in Pike can do better since it knows where the pointers are.
/ Martin Stjernholm, Roxen IS
Previous text:
2003-06-11 23:40: Subject: Re: Debugging memory problems
Hello,
Is it enough to set those variables to 0 ?
Depends on what you want to accomplish. If you want to avoid garbage that's left around until the next gc then you only need to zero variables if their values directly or indirectly refers to the object they're in. Normal acyclic structures are freed as soon as possible through reference counting.
Ok. Thank you for the help :)
. When I call destruct(), destroy() is sometimes not called after some time or after the user login again or if I reload the module.
I assume there's a "not" too much in that sentence. Seems like destroy is called by the gc or as a result of that the object runs out of references. Are you absolutely sure that you call destruct on the right object? I often put in some debug code like this to make sure I can tell objects apart:
This one was my fault, I did a destruct on the NULL value and didn't see it. It works now.
I also noted that arrays in Pike can share their content. For example if I fill 30MB in this arrays:
$ ps auxw|grep pike david 446 13.5 1.1 8784 4360 pts/4 S 23:25 0:00 pike
array memtest = ({ Stdio.File("/dev/zero", "r")->read(31457280) });
$ ps axuw|grep pike david 446 1.9 9.1 39632 35320 pts/4 S 23:25 0:01 pike
array memtest2 = ({ Stdio.File("/dev/zero", "r")->read(31457280) });
$ ps axuw|grep pike david 446 2.5 9.1 39632 35320 pts/4 S 23:25 0:01 pike
array memtest3 = ({ Stdio.File("/dev/zero", "r")->read(31457280) +
"kjshjkzerhzjkerhzkjer" }); $ ps axuw|grep pike david 446 1.6 17.0 70356 66044 pts/4 S 23:25 0:02 pike
That explains why I don't see the leak sometime. For example if I login as the same user again on the same computer (so I get the same session).
I also have another question. Why not taking an already existing garbage collector (for example http://www.hpl.hp.com/personal/Hans_Boehm/gc/) and adapt it to Pike ?
/ David Gourdelier
On Thu, 5 Jun 2003, Martin Stjernholm, Roxen IS @ Pike developers forum wrote:
Hi,
Here are some more notes I have regarding this problem:
. It seems like empty array or string with the same content share the same references. That's why I got lot of references for the empty array. So I put something like ({ "ThisiSaRandDomString232323239084 "}) in every arrays of the objects and they are not referenced anymore. Is it enough to set those variables to 0 ? . _refs always report lot of references but _locate_references doesn't find them. For example I have lot of things like this for strings:
variable searchstring ("") is referenced 1044 times **Looking for references to 0x83773ec: **Done looking for references to 0x83773ec. variable defaultsentfolder ("") is referenced 1044 times **Looking for references to 0x83773ec: **Done looking for references to 0x83773ec.
Is it normal ?
. When I call destruct(), destroy() is sometimes not called after some time or after the user login again or if I reload the module.
/ David Gourdelier
Please ignore that; I missed the stop function.
/ Martin Stjernholm, Roxen IS
Previous text:
2003-06-05 17:20: Subject: Re: Debugging memory problems
There's a reference back through sess->imapclient->session.
/ Martin Stjernholm, Roxen IS
-- David Gourdelier
/ Brevbäraren
-- David Gourdelier
/ Brevbäraren
Then you can clear either sess->imapclient or imapclient->session, or destruct the imapclient object.
/ Martin Stjernholm, Roxen IS
Previous text:
2003-06-05 16:49: Subject: Re: Debugging memory problems
Hello,
The problem is that timeout should not reference the object anymore.
You mean the call variable? Yes, you have to zero it yourself; remove_call_out doesn't do that for you.
Ok thank you.
If you can't rely on the gc then you have to make sure that you break up all reference cycles, just like when you assign zero to the call variable. A robust way to do that is to destruct the object explicitly (but that can bring its own problems if there are multiple threads that access it etc). Roxen destructs the RequestID objects explicitly for this reason.
Ok. I have another problem with cycle reference. Given this code:
class session { object imapclient; string a = "toto"; }
class imapclient { object session; void create() { } void start(object _session) { session = _session; } void stop() { session = 0; } };
int main() { object sess = session(); sess->imapclient = imapclient(); sess->imapclient->start(sess); sess->imapclient->stop(); _locate_references(sess->imapclient); write("a=%s\n", sess->a); return -1; }
I get:
**Looking for references to 0x83a1e40: ***One ref found in an object. -> from object 0x83a1ea0 offset -445904 **In variable imapclient **In storage for inherit 0 **Location: 0x83a1ea0 Type: object Refs: 1 **Program id: 65621 **Describing program of object: **Program id: 65621, flags: f **Location: /home/david/scripts/memory.pike:52 **There is no parent (any longer?)
----------end------------ **Done looking for references to 0x83a1e40.
The problem is that I can't destruct(sess) because I need it after in main().
-- David Gourdelier
/ Brevbäraren
Are all these really variables in your object? If so, do they point to functions in some global module or something like that?
Yes there are variables inside my object. There are not referenced by other global modules.
You misunderstood me. I asked what they pointed to, not what was pointing to them (or rather, their object). With your use of _refs you're really looking at the refs of the things the variables contain.
/ Martin Stjernholm, Roxen IS
Previous text:
2003-06-03 17:01: Subject: Re: Debugging memory problems
On Tue, 3 Jun 2003, Martin Stjernholm, Roxen IS @ Pike developers forum wrote:
Hi,
variable draftmail is referenced 1107 times variable misspelled is referenced 7067 times variable spelling is referenced 7067 times variable deletemails is referenced 7067 times variable deletemboxes is referenced 7067 times variable prefuids is referenced 7067 times variable searchstring is referenced 1107 times variable session is referenced 1107 times
Are all these really variables in your object? If so, do they point to functions in some global module or something like that?
Yes there are variables inside my object. There are not referenced by other global modules.
. Is it possible to know which things reference which variables ?
There's one tool: If you compile with rtl-debug, there's a function _locate_references that walks through all things and prints the ones that reference the argument (in debug style). It's really useful, especially in 7.4 and later where it has been improved to report on the more odd sorts of references.
Ok I will try that.
I also enable debug in gc.c to find the problem but the output looks quite cryptic to me. /.../
I'm not surprised. ;) Really, that output isn't useful to hunt down
Hehe :)
leaks. _locate_references is, and if you suspect C-level leaks, the dmalloc debug mode.
I don't think it's a problem at the C level but more propably at Camas or 123sessions level.
Why some references are weaker than others ?
Not sure how to respond to this. Have you looked at the comment blurb near the top of gc.c?
Yes.
-- David Gourdelier
/ Brevbäraren
pike-devel@lists.lysator.liu.se
-
David Gourdelier -
Fredrik (Naranek) Hubinette (Real Build Master) ＠ Pike (-) developers forum -
Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) ＠ Pike (-) developers forum
-
Martin Stjernholm, Roxen IS ＠ Pike developers forum