Ok, so I have a Caudium server running either Pike 7.0 or 7.2. It's a search engine using an in-memory mapping based data structure for the search. Problem is, it's leaking memory. The 7.0 is leaking a lot faster than 7.2 it seems (have only tried running 7.2 a short period of time).
It never stabilizes which indicates that's it's not just ordinary increase due to fragmentation, caching etc. Using the only available tool, Debug info for developers, I do clearly see especially mappings but also strings increase in size. This MIGHT be a mapping-based cache that's caching more than it should but it's close to impossible to find out.
A snapshot of the debug info is available at http://david.hedbor.org/tmp/debug.gif One interesting thing to note is that there are -12862 - i.e there's obviously a bug in the object counting.
Since the number of mappings doesn't change as much, and sometimes decreases even though memory increases, it seems like it IS something that is being added to rather than a pure leak.. As a matter of fact, since the GIF snapshot, there are only 8 more mappings but almost 30 MB more memory used by them.
Does anyone have any good suggestions on how to figure this out? An optimal solution would be to be able to get a list of all mappings and their memory usage (or at least the ones using significant amount of memory, since there are 6.3 million mappings...).
Ok, this is interesting - it seems like do_gc (Pike 7.2) is called about once a every 70-90 seconds on this box. That's VERY often and very bad since it takes quite a long time (1.5 GB worth of data to iterate through).
I had previously disabled gc manually and will do so again, but does anyone have any idea why it's called so often (I added breakpoint to both do_gc and f_gc and only do_gc was called so it's not done manually).
/ David Hedbor
Previous text:
2002-12-09 20:49: Subject: Memory leak, how to find it?
Ok, so I have a Caudium server running either Pike 7.0 or 7.2. It's a search engine using an in-memory mapping based data structure for the search. Problem is, it's leaking memory. The 7.0 is leaking a lot faster than 7.2 it seems (have only tried running 7.2 a short period of time).
It never stabilizes which indicates that's it's not just ordinary increase due to fragmentation, caching etc. Using the only available tool, Debug info for developers, I do clearly see especially mappings but also strings increase in size. This MIGHT be a mapping-based cache that's caching more than it should but it's close to impossible to find out.
A snapshot of the debug info is available at http://david.hedbor.org/tmp/debug.gif One interesting thing to note is that there are -12862 - i.e there's obviously a bug in the object counting.
Since the number of mappings doesn't change as much, and sometimes decreases even though memory increases, it seems like it IS something that is being added to rather than a pure leak.. As a matter of fact, since the GIF snapshot, there are only 8 more mappings but almost 30 MB more memory used by them.
Does anyone have any good suggestions on how to figure this out? An optimal solution would be to be able to get a list of all mappings and their memory usage (or at least the ones using significant amount of memory, since there are 6.3 million mappings...).
/ David Hedbor
Obviously there's something wrong with the gc threshold algorithm. It'd be helpful if you periodically (once every ten minutes or something) could log the gc status returned by _gc_status.
/ Martin Stjernholm, Roxen IS
Previous text:
2002-12-09 22:14: Subject: Memory leak, how to find it?
Ok, this is interesting - it seems like do_gc (Pike 7.2) is called about once a every 70-90 seconds on this box. That's VERY often and very bad since it takes quite a long time (1.5 GB worth of data to iterate through).
I had previously disabled gc manually and will do so again, but does anyone have any idea why it's called so often (I added breakpoint to both do_gc and f_gc and only do_gc was called so it's not done manually).
/ David Hedbor
Right now I disabled gc() manually (defining callback add to be nothing).
/ David Hedbor
Previous text:
2002-12-09 23:33: Subject: Memory leak, how to find it?
Obviously there's something wrong with the gc threshold algorithm. It'd be helpful if you periodically (once every ten minutes or something) could log the gc status returned by _gc_status.
/ Martin Stjernholm, Roxen IS
It's required for us however. We can't afford 20-30 second lockups unless we control the gc() manually. These are production boxes serving live audiences with rather high frequency.
/ David Hedbor
Previous text:
2002-12-10 02:30: Subject: Memory leak, how to find it?
That's not helpful.
/ Martin Stjernholm, Roxen IS
Run Pike 7.4?
/ Martin Nilsson (hehe Torgny)
Previous text:
2002-12-10 02:43: Subject: Memory leak, how to find it?
It's required for us however. We can't afford 20-30 second lockups unless we control the gc() manually. These are production boxes serving live audiences with rather high frequency.
/ David Hedbor
Why something wrong? Perhaps it actually does a collect 20% garbage on each run on average?
/ Fredrik (Naranek) Hubinette (Real Build Master)
Previous text:
2002-12-09 23:33: Subject: Memory leak, how to find it?
Obviously there's something wrong with the gc threshold algorithm. It'd be helpful if you periodically (once every ten minutes or something) could log the gc status returned by _gc_status.
/ Martin Stjernholm, Roxen IS
Seems very unlikely because with the garbage collector not disabled, it doesn't grow really (not counting the leak, which I believe is fixed).
/ David Hedbor
Previous text:
2002-12-25 08:23: Subject: Memory leak, how to find it?
Why something wrong? Perhaps it actually does a collect 20% garbage on each run on average?
/ Fredrik (Naranek) Hubinette (Real Build Master)
pike-devel@lists.lysator.liu.se
-
David Hedbor ＠ Pike developers forum -
Fredrik (Naranek) Hubinette (Real Build Master) ＠ Pike (-) developers forum
-
Martin Nilsson (hehe Torgny) ＠ Pike (-) developers forum
-
Martin Stjernholm, Roxen IS ＠ Pike developers forum