...

...

I have a production machine running Roxen which crashes on an error like this roughly once a day. I'm willing to make custom changes to the

And four more backtraces since Thursday.

Anyone (mast?) have an idea where to dig for the program_id mentioned? Or do I need to add more logging to the gc.c file? It would probably be interesting to find out when this object was created. I think it should be possible to add another field to the object which logs the time of creation. Then the backtrace from that moment would be interesting; anyone have a better idea than storing a backtrace for every object as soon as it is created, yet discarding the backtrace (to conserve space) as soon as the object gets a reference?

/data/src/gpike/src/gc.c:2519: GC fatal: Marked a thing without refs. **Block: 0xb3ef74e8 Type: object Refs: 0 **Got gc marker at 0xae2c1b68: flags=0x00000 refs=0 weak=0 xrefs=0 saved=-1 frame=(nil) **Program id: 66233 **The object is not on the object link list. **The object is destructed but program found from id. ******************* **Looking for references to 0xb3ef74e8: **Done looking for references to 0xb3ef74e8, found 0 refs. Pike was in GC stage 200 when this fatal occured. Backtrace at time of fatal: /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:469: RXML.PXml(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set))->remove_runtime_tag(UserTag(selectcommunity),UNDEFINED) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:423: RXML.PXml(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set))->add_runtime_tag(@0=UserTag(selectcommunity)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:6139: RXML.t_any_text(text/*, xml)->get_parser(Context(),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set),RXML.RenewablePCode(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set)),0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4684: Frame(if)->_eval(@1=Context(),@2=RXML.RenewablePCode(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set)),RXML.t_any_text(text/*, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.RenewablePCode(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set))->_eval(@1,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:9206: RXML.RenewablePCode(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set))->_eval(@1,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:3865: UserTag.Frame(celement)->_exec_array(@1,RXML.PXml(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(emit+rxml_tag_set)),({@2}),"do_return") /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4825: UserTag.Frame(celement)->_eval(@1,@3=RXML.PXml(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(emit+rxml_tag_set)),@4=RXML.t_any_text(text/*, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:459: UserTag(celement)->_p_xml_handle_tag(@3,mapping[5],"") /usr/local/pike/7.8.409/lib/modules/Parser.pmod/_parser.so:1: RXML.PXml(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(emit+rxml_tag_set))->finish("\r\n<celement\r\n title="&var.d_&_.p7;;"\r\n label="p7"\r\n elmcheck="&_.p7;>=10000"\r\n elmlabel="propval7" elmval="&form.propval7:none;"\r\n />\r\n") /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:389: RXML.PXml(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(emit+rxml_tag_set))->finish("\r\n<celement\r\n title="&var.d_&_.p7;;"\r\n label="p7"\r\n elmcheck="&_.p7;>=10000"\r\n elmlabel="propval7" elmval="&form.propval7:none;"\r\n />\r\n") /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4705: Frame(if)->_eval(@1,RXML.PXml(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(emit+rxml_tag_set)),@4) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:459: TagIf(if)->_p_xml_handle_tag(@5=RXML.PXml(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(emit+rxml_tag_set)),mapping[1],,,0) /usr/local/pike/7.8.409/lib/modules/Parser.pmod/_parser.so:1: RXML.PXml(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(emit+rxml_tag_set))->finish("\r\n\r\n<if variable="var.d_&_.p0;">\r\n<celement\r\n title="&var.d_&_.p0;;"\r\n label="p0"\r\n elmcheck="&_.p0;>=10000"\r\n elmlabel="propval0" elmval="&form.propval0:none;"\r\n />\r\n</if>\r\n\r\n<if variable="var.d_"+[3665]) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:389: RXML.PXml(RXML.t_any_text(text/*, xml),RXML.CompositeTagSet(emit+rxml_tag_set))->finish("\r\n\r\n<if variable="var.d_&_.p0;">\r\n<celement\r\n title="&var.d_&_.p0;;"\r\n label="p0"\r\n elmcheck="&_.p0;>=10000"\r\n elmlabel="propval0" elmval="&form.propval0:none;"\r\n />\r\n</if>\r\n\r\n<if variable="var.d_"+[3665]) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4705: Frame(eval)->_eval(@1,RXML.PXml(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set)),RXML.t_html(text/html, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:459: TagEval(eval)->_p_xml_handle_tag(@6,([]),"\r\n\r\n<if variable="var.d_&_.p0;">\r\n<celement\r\n title="&var.d_&_.p0;;"\r\n label="p0"\r\n elmcheck="&_.p0;>=10000"\r\n "+[3750]) /usr/local/pike/7.8.409/lib/modules/Parser.pmod/_parser.so:1: RXML.PXml(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set))->finish("\r\n\r\n<emit query="SELECT * FROM pdlprop WHERE 0\r\n\r\n OR id_desc='&_.p0;'\r\n\r\n OR id_desc='&_.p1;'\r\n\r\n OR id_desc='&_.p2;'\r\n\r\n OR id_desc='&_.p3;'\r\n\r\n OR id_desc='&_.p4;'\r\n\r\n OR id_desc='&_.p5;'"+[4404]) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:389: RXML.PXml(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set))->finish("\r\n\r\n<emit query="SELECT * FROM pdlprop WHERE 0\r\n\r\n OR id_desc='&_.p0;'\r\n\r\n OR id_desc='&_.p1;'\r\n\r\n OR id_desc='&_.p2;'\r\n\r\n OR id_desc='&_.p3;'\r\n\r\n OR id_desc='&_.p4;'\r\n\r\n OR id_desc='&_.p5;'"+[4404]) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4705: Frame(emit)->_eval(@1,RXML.PXml(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),@7=RXML.t_html(text/html, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:459: TagEmit(emit)->_p_xml_handle_tag(@8,mapping[2],"\r\n\r\n<emit query="SELECT * FROM pdlprop WHERE 0\r\n\r\n OR id_desc='&_.p0;'\r\n\r\n OR id_desc='&_.p1;'\r\n\r\n OR "+[4495]) /usr/local/pike/7.8.409/lib/modules/Parser.pmod/_parser.so:1: RXML.PXml(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->finish("\r\n\r\n<emit source="sql"\r\n query="SELECT * FROM pdlnode\r\n WHERE id_pdlnode='&form._id_pdlnode;';">\r\n\r\n<emit query="SELECT * FROM pdlprop WHERE 0\r\n\r\n OR id_desc='&_.p0;'\r\n\r\n OR id_desc='&_.p1"+[4517]) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:389: RXML.PXml(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->finish("\r\n\r\n<emit source="sql"\r\n query="SELECT * FROM pdlnode\r\n WHERE id_pdlnode='&form._id_pdlnode;';">\r\n\r\n<emit query="SELECT * FROM pdlprop WHERE 0\r\n\r\n OR id_desc='&_.p0;'\r\n\r\n OR id_desc='&_.p1"+[4517]) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:3819: Frame(eval)->_exec_array(@1,RXML.PXml(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),({"\r\n\r\n<emit source="sql"\r\n query="SELECT * FROM pdlnode\r\n WHERE id_pdlnode='&form._id_pdlnode;';">\r"+[4610]}),"do_return") /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4825: Frame(eval)->_eval(@1,@9=RXML.PXml(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),@7) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:459: TagEval(eval)->_p_xml_handle_tag(@9,([]),"\r\n<noparse>\r\n<emit source="sql"\r\n query="SELECT * FROM pdlnode\r\n WHERE id_pdlnode='&form._id_pdlnode;';">\r\n</noparse>\r\n<maketag name=emit type=container>\r\n <attrib name=source>sql</att"+[674]) /usr/local/pike/7.8.409/lib/modules/Parser.pmod/_parser.so:1: RXML.PXml(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->finish("\n<if\n match="hidden == hidden"\n ><if not variable="form._rdonly"\n ><input type=hidden name=_id_pdlnode value="&form.id_pdlnode:html;"/></if></if><else\n ><tr><td></td><td colspan="&var.columns;"><"+[16928]) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:389: RXML.PXml(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->finish("\n<if\n match="hidden == hidden"\n ><if not variable="form._rdonly"\n ><input type=hidden name=_id_pdlnode value="&form.id_pdlnode:html;"/></if></if><else\n ><tr><td></td><td colspan="&var.columns;"><"+[16928]) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4705: Frame(else)->_eval(@1,RXML.PXml(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),@7) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:459: TagElse(else)->_p_xml_handle_tag(@10,([]),"\n<if\n match="hidden == hidden"\n ><if not variable="form._rdonly"\n ><input type=hidden name=_id_pdlnode value="+[17014]) /usr/local/pike/7.8.409/lib/modules/Parser.pmod/_parser.so:1: RXML.PXml(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->finish("\n<form method=post action="&page.url:none;">\n<table border=0>\n\n<define tag=buttons>\n<if not variable="form._rdonly">\n<table width="100%"><tr><td width="50%" align=left nowrap>\n<if variable="var._g"+[30512]) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:389: RXML.PXml(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->finish("\n<form method=post action="&page.url:none;">\n<table border=0>\n\n<define tag=buttons>\n<if not variable="form._rdonly">\n<table width="100%"><tr><td width="50%" align=left nowrap>\n<if variable="var._g"+[30512]) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4705: Frame(else)->_eval(@1,RXML.PXml(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),@7) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:459: TagElse(else)->_p_xml_handle_tag(@11,([]),"\n<form method=post action="&page.url:none;">\n<table border=0>\n\n<define tag=buttons>\n<if not variable="form._rd"+[30598]) /usr/local/pike/7.8.409/lib/modules/Parser.pmod/_parser.so:1: RXML.PXml(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->finish("<if\n variable="form._duplicate">\n <set variable="form._paste" value="_duplicate"/>\n</if\n><if variable="form._paste">\n<set\n variable="form.id_pdloffer" value="&cookie.clipboard;"\n/>\n</if>\n\n\n\n\n\n\n\n<i"+[47475]) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:389: RXML.PXml(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->finish("<if\n variable="form._duplicate">\n <set variable="form._paste" value="_duplicate"/>\n</if\n><if variable="form._paste">\n<set\n variable="form.id_pdloffer" value="&cookie.clipboard;"\n/>\n</if>\n\n\n\n\n\n\n\n<i"+[47475]) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:3819: Frame(template)->_exec_array(@1,RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),({"<if\n variable="form._duplicate">\n <set variable="form._paste" value="_duplicate"/>\n</if\n><if variable="+[47569]}),"do_return") /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4825: Frame(template)->_eval(@1,@12=RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),@7) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->_eval(@1,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4775: Frame(if)->_eval(@1,RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),@7) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->_eval(@1,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8243: RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->eval(@1,UNDEFINED) modules/tags/rxmlparse.pike:171: RoxenModule(entreeding.com/rxmlparse#0)->handle_file_extension(Stdio.File("/home/entreeding/info/businesscard-invitem.rxml", "r", 666 /* fd=88 */),"rxml",RequestID(/info/businesscard-invitem.rxml?parent=397601&sort=&invno=12)) base_server/configuration.pike:2357: Configuration(entreeding.com)->low_get_file(@13=RequestID(/info/businesscard-invitem.rxml?parent=397601&sort=&invno=12),UNDEFINED) base_server/configuration.pike:2491: Configuration(entreeding.com)->get_file(@13,UNDEFINED,UNDEFINED) base_server/configuration.pike:2466: Configuration(entreeding.com)->handle_request(@13,UNDEFINED) protocols/http.pike:2823: RequestID(/info/businesscard-invitem.rxml?parent=397601&sort=&invno=12)->handle_request() base_server/roxen.pike:849: roxen->handler_thread(20)

/data/src/gpike/src/gc.c:2519: GC fatal: Marked a thing without refs. **Block: 0xb1e9fef0 Type: object Refs: 0 **Got gc marker at 0xa9bd248: flags=0x00000 refs=0 weak=0 xrefs=0 saved=-1 frame=(nil) **Program id: 66124 **The object is not on the object link list. **The object is destructed but program found from id. ******************* **Looking for references to 0xb1e9fef0: **Done looking for references to 0xb1e9fef0, found 0 refs. Pike was in GC stage 200 when this fatal occured. Backtrace at time of fatal: /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:5233: RXML->rxml_index(mapping[9],"minpropval","var",Context(),RXML.t_any_text(text/*, none)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:1766: Context()->get_var(({"minpropval"}),"var",@0=RXML.t_any_text(text/*, none)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7455: RXML.VarRef(var.minpropval)->get(@1=Context()) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8885: RXML.PCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+rxml_tag_set))->_eval(@1,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4775: Frame(if)->_eval(@1,RXML.RenewablePCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+rxml_tag_set)),RXML.t_xml(text/xml, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.RenewablePCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+rxml_tag_set))->_eval(@1,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:9206: RXML.RenewablePCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+rxml_tag_set))->_eval(@1,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:3865: UserTag.Frame(checkproperty)->_exec_array(@1,RXML.PCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+rxml_tag_set)),({@2=RXML.RenewablePCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+rxml_tag_set))}),"do_return") /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4825: UserTag.Frame(checkproperty)->_eval(@1,@3=RXML.PCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+rxml_tag_set)),@4=RXML.t_xml(text/xml, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+rxml_tag_set))->_eval(@1,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4775: Frame(define)->_eval(@1,RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),RXML.t_html(text/html, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->_eval(@1,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4775: Frame(if)->_eval(@1,RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),@5=RXML.t_html(text/html, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->_eval(@1,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8243: RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->eval(@1,UNDEFINED) modules/tags/rxmlparse.pike:171: RoxenModule(entreeding.com/rxmlparse#0)->handle_file_extension(Stdio.File("/home/entreeding/info/zoekresultaat.rxml", "r", 666 /* fd=69 */),"rxml",RequestID(/info/zoekresultaat.rxml?_id_pdlnode=1686)) base_server/configuration.pike:2357: Configuration(entreeding.com)->low_get_file(@6=RequestID(/info/zoekresultaat.rxml?_id_pdlnode=1686),UNDEFINED) base_server/configuration.pike:2491: Configuration(entreeding.com)->get_file(@6,UNDEFINED,UNDEFINED) base_server/configuration.pike:2466: Configuration(entreeding.com)->handle_request(@6,UNDEFINED) protocols/http.pike:2823: RequestID(/info/zoekresultaat.rxml?_id_pdlnode=1686)->handle_request() base_server/roxen.pike:849: roxen->handler_thread(24)

/data/src/gpike/src/gc.c:2519: GC fatal: Marked a thing without refs. **Block: 0xa9d8980 Type: object Refs: 0 **Got gc marker at 0xb1d65b80: flags=0x00000 refs=0 weak=0 xrefs=0 saved=-1 frame=(nil) **Program id: 66463 **The object is not on the object link list. **The object is destructed but program found from id. ******************* **Looking for references to 0xa9d8980: **Done looking for references to 0xa9d8980, found 0 refs. Pike was in GC stage 200 when this fatal occured. Backtrace at time of fatal: -:1: _static_modules.Builtin()->single_string_replace()->`()("/images/user/254231/art.100351200.jpeg") base_server/configuration.pike:3003: Configuration(entreeding.com)->real_file("/images/user/254231/art.100351200.jpeg",RequestID(/info/businesscard-inventory.rxml?parent=254231)) modules/graphics/cimg.pike:328: RoxenModule(entreeding.com/cimg#0)->get_my_args(mapping[10],@0=RequestID(/info/businesscard-inventory.rxml?parent=254231)) modules/graphics/cimg.pike:430: Frame(cimg)->do_return(@0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4821: Frame(cimg)->_eval(Context(),RXML.PCode(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set)),RXML.t_html(text/html, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set))->_eval(@1=Context(),0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4775: Frame(if)->_eval(@1,RXML.PCode(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set)),@2=RXML.t_html(text/html, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set))->_eval(@1,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4775: Frame(emit)->_eval(@1,RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),@2) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->_eval(@1,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4775: Frame(if)->_eval(@1,RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),@2) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->_eval(@1,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8243: RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->eval(@1,UNDEFINED) modules/tags/rxmlparse.pike:171: RoxenModule(entreeding.com/rxmlparse#0)->handle_file_extension(Stdio.File("/home/entreeding/info/businesscard-inventory.rxml", "r", 666 /* fd=76 */),"rxml",@0) base_server/configuration.pike:2357: Configuration(entreeding.com)->low_get_file(@0,UNDEFINED) base_server/configuration.pike:2491: Configuration(entreeding.com)->get_file(@0,UNDEFINED,UNDEFINED) base_server/configuration.pike:2466: Configuration(entreeding.com)->handle_request(@0,UNDEFINED) protocols/http.pike:2823: RequestID(/info/businesscard-inventory.rxml?parent=254231)->handle_request() base_server/roxen.pike:849: roxen->handler_thread(6)

/data/src/gpike/src/gc.c:2519: GC fatal: Marked a thing without refs. **Block: 0xaf32900 Type: object Refs: 0 **Got gc marker at 0xb921bb8: flags=0x00000 refs=0 weak=0 xrefs=0 saved=-1 frame=(nil) **Program id: 65540 **The object is not on the object link list. **The object is destructed but program found from id. ******************* **Looking for references to 0xaf32900: **Done looking for references to 0xaf32900, found 0 refs. Pike was in GC stage 200 when this fatal occured. Backtrace at time of fatal: /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8910: RXML.PCode(RXML.t_string(string, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set))->_eval(Context(),0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4775: Frame(replace)->_eval(@0=Context(),RXML.PCode(RXML.t_any(any, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set)),RXML.t_any(any, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_any(any, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set))->_eval(@0,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4775: Frame(set)->_eval(@0,RXML.RenewablePCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set)),RXML.t_xml(text/xml, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.RenewablePCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set))->_eval(@0,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:9206: RXML.RenewablePCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set))->_eval(@0,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:3865: UserTag.Frame(converttexttourl)->_exec_array(@0,RXML.PCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set)),({@1=RXML.RenewablePCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set))}),,,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4825: UserTag.Frame(converttexttourl)->_eval(@0,@2=RXML.PCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set)),@3=RXML.t_xml(text/xml, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_xml(text/xml, xml),RXML.CompositeTagSet(_user_tag+emit+rxml_tag_set))->_eval(@0,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4775: Frame(define)->_eval(@0,RXML.PCode(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set)),RXML.t_html(text/html, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set))->_eval(@0,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4775: Frame(emit)->_eval(@0,RXML.PCode(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set)),@4=RXML.t_html(text/html, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set))->_eval(@0,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4775: Frame(emit)->_eval(@0,RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),@4) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8835: RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->_eval(@0,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8243: RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->eval(@0,UNDEFINED) modules/tags/rxmlparse.pike:171: RoxenModule(entreeding.com/rxmlparse#0)->handle_file_extension(Stdio.File("/home/entreeding/extern/nieuws.rxml", "r", 666 /* fd=42 */),"rxml",RequestID(/extern/nieuws.rxml?id=1557)) base_server/configuration.pike:2357: Configuration(entreeding.com)->low_get_file(@5=RequestID(/extern/nieuws.rxml?id=1557),UNDEFINED) base_server/configuration.pike:2491: Configuration(entreeding.com)->get_file(@5,UNDEFINED,UNDEFINED) base_server/configuration.pike:2466: Configuration(entreeding.com)->handle_request(@5,UNDEFINED) protocols/http.pike:2823: RequestID(/extern/nieuws.rxml?id=1557)->handle_request() base_server/roxen.pike:849: roxen->handler_thread(13)

Martin Stjernholm, Roxen IS @ Pike developers forum wrote:

Looks a lot like https://bugzilla.roxen.com/bugzilla/show_bug.cgi?id=5072. Here's a bit of what I wrote in that ticket:

It sure does.

This is from a fresh 7.8? More specifically, one from cvs after Nov 28th, when I added some debug for [bug 5072]? Can you run with a version compiled using --with-rtldebug? It's not that much slower.

The traces I gave you are *with* rtldebug already (you wouldn't be seeing the problem otherwise), and yes, it's from a recent version (like the end of January 2010).

In relation to the suspicions on binary modules, the server where this happens in is compiled without Java support, and it doesn't use any other databases than mysql.

...

Anyone (mast?) have an idea where to dig for the program_id mentioned?

The id is bigger than PROG_DYNAMIC_ID_START, so it's a program without a fixed id. It's also the fourth program that got registered by low_allocate_program, so it's probably a C program. You could set a breakpoint there and check current_program_id to find out which it is.

Running the production server in gdb is going to be tricky, so this is not something I'm likely to try first (unless you say it helps tremendously in locating the problem).

...

Or do I need to add more logging to the gc.c file?

What would be useful is debug when gc_mark_enqueue is run - log the pointer (data) and the number of refs (*(INT32 *) data). But it's probably not feasible if your server has a big memory footprint; not even GC_VERBOSE logs that.

I'll see what I can do. I have over 100GB of spare room on a logpartition, so if this doesn't slow down the server too much, it might be doable.

...

It would probably be interesting to find out when this object was created.

Yes, that could give a clue, but the really valuable info is what kind of things were referencing the object, e.g. some weak mapping as suspected above.

The first clue to that is finding out what this object contained (or was supposed to contain); looking at the current trace, it seems like this intel is already lost at this point?

...

I think it should be possible to add another field to the object which logs the time of creation. Then the backtrace from that moment would be interesting; anyone have a better idea than storing a backtrace for every object as soon as it is created, yet discarding the backtrace (to conserve space) as soon as the object gets a reference?

--with-dmalloc-c-stack-trace does that, but then you're running with dmalloc which is probably too slow. It should be possible to rip out the location tracking from dmalloc so that it only logs this. Then I it could be fast enough. But afaik there's no ready-made define for that.

I'll look into this.

Martin Stjernholm, Roxen IS @ Pike developers forum wrote:

Hmm, considering the debug check at svalue.c:2316 it must happen while the object is in the queue, and that should be easy to check. Here's a patch that strengthens those debug checks. Could you please try it? Be prepared to back it out quickly though, because I'm not entirely sure it's correct (that o->next != o condition is suspicious).

Running with it since last night. So far, so good.

...

...

Hmm, considering the debug check at svalue.c:2316 it must happen while the object is in the queue, and that should be easy to check. Here's a patch that strengthens those debug checks. Could you please try it? Be prepared to back it out quickly though, because I'm not entirely sure it's correct (that o->next != o condition is suspicious).

I'm running with these and with GC_VERBOSE turned on now. So far the slowdown is insignificant, and the amount of logdata is not a real problem.

Martin Stjernholm, Roxen IS @ Pike developers forum wrote:

...

I'm running with these and with GC_VERBOSE turned on now. So far the slowdown is insignificant, and the amount of logdata is not a real problem.

Oh? Very well then. Ought to give something to go on, I hope.

Well, here are the results of the jury, and the results are inconclusive as far as I am concerned, but maybe you see more in it:

Garbage collecting... | check: 8767 references in 472 things, counted 3 weak refs | mark: 2936 markers referenced, 0 weak references freed, | 3 things to free, got 0 tricky weak refs | cycle: 23 internal things visited, | 0 weak references freed, 0 more things to free, | 0 mark live visits, 0 frame rotations, | 22 links searched, used max 31 link frames, | 9 rec frames and 0 free extra frames | free: 23 unreferenced, 23 really freed, 0 left with live references | kill: 0 objects killed, 0 things really freed | destruct: 0 things really freed done (23 of 472 were unreferenced), 1 ms Garbage collecting... | check: 13557 references in 2109 things, counted 14 weak refs | mark: 5437 markers referenced, 0 weak references freed, | 13 things to free, got 0 tricky weak refs | cycle: 13 internal things visited, | 0 weak references freed, 0 more things to free, | 0 mark live visits, 0 frame rotations, | 0 links searched, used max 2 link frames, | 1 rec frames and 0 free extra frames | free: 13 unreferenced, 13 really freed, 0 left with live references | kill: 0 objects killed, 0 things really freed | destruct: 0 things really freed done (13 of 2109 were unreferenced), 2 ms Garbage collecting... | check: 45721 references in 7083 things, counted 184 weak refs | mark: 15850 markers referenced, 0 weak references freed, | 83 things to free, got 0 tricky weak refs | cycle: 83 internal things visited, | 0 weak references freed, 0 more things to free, | 0 mark live visits, 0 frame rotations, | 0 links searched, used max 2 link frames, | 1 rec frames and 0 free extra frames | free: 83 unreferenced, 83 really freed, 0 left with live references | kill: 0 objects killed, 0 things really freed | destruct: 0 things really freed done (83 of 7083 were unreferenced), 8 ms Done [1496.2ms] Garbage collecting... | check: 100566 references in 18550 things, counted 863 weak refs | mark: 29590 markers referenced, 0 weak references freed, | 230 things to free, got 1 tricky weak refs | cycle: 1058 internal things visited, | 0 weak references freed, 1 more things to free, | 35 mark live visits, 0 frame rotations, | 4 links searched, used max 224 link frames, | 206 rec frames and 0 free extra frames | zap weak: freed 0 external weak refs, 1 internal still around, | 0 more things to free | free: 1058 unreferenced, 82 really freed, 976 left with live references | Killing 0xa0dfc48 with 1 refs, prog src/program.c:9997 | Killing 0xa0dee98 with 1 refs, prog src/program.c:9997 | Killing 0xa0e6950 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e6938 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa0e6968 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa0e62a8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa0e6290 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa0e62f0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa0e62c0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5f48 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5f30 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5f90 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:936 | Killing 0xa0e5f60 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e6068 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1064 | Killing 0xa0e6020 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1095 | Killing 0xa0e6050 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5fd8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5fc0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1806 | Killing 0xa0e5ee8 with 1 refs, prog src/modules/Gmp/mpz_glue.c:2467 | Killing 0xa0e5ed0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1250 | Killing 0xa0e5f00 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa0e5eb8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5ea0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5e88 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5e70 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1250 | Killing 0xa0e5e58 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5e40 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5e28 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa0e5e10 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa0e5df8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5de0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5dc8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5db0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa0e5d98 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa0e5d80 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1258 | Killing 0xa0e5d68 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa0e5d50 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa0e5d38 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1258 | Killing 0xa0e5d20 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa0e5d08 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1258 | Killing 0xa0e5cf0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa0e57c8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e55e8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:936 | Killing 0xa0e55d0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa0e5450 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1806 | Killing 0xa0e5438 with 1 refs, prog src/modules/Gmp/mpz_glue.c:2467 | Killing 0xa0e5408 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e5330 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e52e8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:936 | Killing 0xa178138 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa178120 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa178150 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa178180 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:936 | Killing 0xa1781e0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa1781f8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa177f40 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa177f28 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa177f10 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:936 | Killing 0xa177ef8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa177ee0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa177ec8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa177eb0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa177e98 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa177e80 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa177b50 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1064 | Killing 0xa177b20 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1095 | Killing 0xa177a48 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa177a00 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1779b8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1806 | Killing 0xa177988 with 1 refs, prog src/modules/Gmp/mpz_glue.c:2467 | Killing 0xa177958 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa177910 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1778c8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa177880 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa177838 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1777f0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1777a8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa177760 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1403 | Killing 0xa177718 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1776d0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa177688 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa177640 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa1775f8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1806 | Killing 0xa1775c8 with 1 refs, prog src/modules/Gmp/mpz_glue.c:2467 | Killing 0xa176830 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa176668 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa176650 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa176680 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1766b0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c7898 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:936 | Killing 0xa1c7880 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1806 | Killing 0xa1c78e0 with 1 refs, prog src/modules/Gmp/mpz_glue.c:2467 | Killing 0xa1c78f8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1806 | Killing 0xa1c7850 with 1 refs, prog src/modules/Gmp/mpz_glue.c:2467 | Killing 0xa1c7628 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa1c7610 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1806 | Killing 0xa1c7640 with 1 refs, prog src/modules/Gmp/mpz_glue.c:2467 | Killing 0xa1c7490 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1036 | Killing 0xa1c7460 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1036 | Killing 0xa1c7388 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1036 | Killing 0xa1c7340 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1036 | Killing 0xa1c6dd0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1806 | Killing 0xa1c6de8 with 1 refs, prog src/modules/Gmp/mpz_glue.c:2467 | Killing 0xa1c6c38 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1036 | Killing 0xa1c6c98 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1250 | Killing 0xa1c6c68 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c6cc8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1064 | Killing 0xa1c6c08 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:936 | Killing 0xa1c6bf0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1806 | Killing 0xa1c6bc0 with 1 refs, prog src/modules/Gmp/mpz_glue.c:2467 | Killing 0xa1c6ba8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c6bd8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c6b90 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c6b78 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c6b60 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa1c6b48 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:936 | Killing 0xa1c6b30 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa1c6b18 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:936 | Killing 0xa1c6b00 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa1c6ae8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:936 | Killing 0xa1c6ad0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1258 | Killing 0xa1c6ab8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1258 | Killing 0xa1c6aa0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c6a88 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa1c6a70 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa1c6a58 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa1c6a40 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa1c6a28 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c6a10 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c65f0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:936 | Killing 0xa1c65d8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa1c62d8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1806 | Killing 0xa1c62f0 with 1 refs, prog src/modules/Gmp/mpz_glue.c:2467 | Killing 0xa1c6320 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c62c0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c6398 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c6368 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c6350 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c6338 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c6290 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c6278 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa1c5c00 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:597 | Killing 0xa256518 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa2564e8 with 1 refs, prog /data/src/groxen/server/modules/database/sqltag.pike:899 | Killing 0xa256410 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa2563c8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa256380 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa256338 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:906 | Killing 0xa255f00 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1064 | Killing 0xa255ee8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1064 | Killing 0xa255f18 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1036 | Killing 0xa255f48 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa255cc0 with 1 refs, prog /usr/roxen/local/modules/custom/userdb_sql1.pike:346 | Killing 0xa255ba0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1250 | Killing 0xa255b88 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa255b70 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa255b58 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa255b40 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa255b28 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:801 | Killing 0xa255840 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/Variable.pmod/module.pmod:1994 | Killing 0xa0e6ba8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xb440ff80 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb440fef0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb440f068 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xb440ef90 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb440ef60 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb440eeb8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xb440ede0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb440edb0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb440e528 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xb440e450 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb440e420 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb444ae88 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xb444a7f8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xb444a660 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xb444a588 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb444a558 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb444a540 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xb444a3f0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa333548 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa3334a0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa333470 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa333038 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa332e88 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa333d58 with 2 refs, prog src/program.c:9997 | Killing 0xa333518 with 1 refs, prog src/modules/Parser/parser.c:181 | Killing 0xa333218 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa3332c0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa332cf0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa332480 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa3322b8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa332210 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa3321e0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa332378 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa3327c8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa333710 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb444ac78 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xb449f570 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xb449f360 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xb449f288 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb449f258 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa4212d8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa421020 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa420f00 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa420dc8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | kill: 205 objects killed, 939 things really freed | destruct: 37 things really freed done (1058 of 18550 were unreferenced), 36 ms Garbage collecting... | check: 149631 references in 30944 things, counted 744 weak refs | mark: 41809 markers referenced, 0 weak references freed, | 90 things to free, got 0 tricky weak refs | cycle: 3256 internal things visited, | 0 weak references freed, 1 more things to free, | 246 mark live visits, 0 frame rotations, | 515 links searched, used max 375 link frames, | 116 rec frames and 7 free extra frames | free: 3256 unreferenced, 1277 really freed, 1979 left with live references | Killing 0xb44a0260 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa421500 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa333698 with 2 refs, prog /usr/local/pike/7.8.409/lib/modules/Protocols.pmod/DNS.pmod:1355 | Killing 0xa3336c8 with 2 refs, prog /usr/local/pike/7.8.409/lib/modules/Protocols.pmod/HTTP.pmod/Query.pike:0 | Killing 0xa3334e8 with 1 refs, prog /usr/local/pike/7.8.409/lib/modules/Stdio.pmod/module.pmod:137 | Killing 0xa333440 with 1 refs, prog src/modules/files/file.c:4928 | Killing 0xa3336e0 with 1 refs, prog /usr/local/pike/7.8.409/lib/modules/Thread.pmod:278 | Killing 0xa4a08a0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa4a0d80 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa4a0c90 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa4a0c18 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa4a0828 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa4a07f8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa4a0e58 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa4a0eb8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa4a08d0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa4a0a08 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa4a0cc0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa4a0960 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa4a0870 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa4a0678 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa4a03c0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa4a0d68 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa49fe98 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa49fdf0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa49fdc0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa49fbb0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa49f9b8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa49f8f8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa49f820 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa49f7f0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa1775f8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa1776a0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa177700 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa0e7040 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa256428 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa2564d0 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xb449f600 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa177970 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa177ee0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa4a04b0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa49f8c8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa3b3590 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa3b3248 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa3b2cd8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa332210 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa177a48 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa548c08 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa422208 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa422100 with 10 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa421050 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa4a09f0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa3b3338 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa3b2d50 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa3b2b40 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa547618 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa4225f8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa422580 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa5472b8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xb440e5b8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb440e108 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa3b3398 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa3b3848 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa3b3410 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa3b3728 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa49f088 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa454b50 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa454970 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa454688 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa454598 with 3 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa454388 with 3 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa5f5038 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa7b5fc0 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa7b5a80 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa7b6a70 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa6f26f0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa6f2a38 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xb44492b0 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa547060 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa62fe08 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa64b320 with 2 refs, prog /usr/local/pike/7.8.409/lib/modules/Protocols.pmod/DNS.pmod:1355 | Killing 0xa64b2c0 with 2 refs, prog src/program.c:9997 | Killing 0xa64b2f0 with 1 refs, prog src/modules/Parser/parser.c:181 | Killing 0xa64b350 with 2 refs, prog /usr/local/pike/7.8.409/lib/modules/Protocols.pmod/HTTP.pmod/Query.pike:0 | Killing 0xa64b308 with 1 refs, prog /usr/local/pike/7.8.409/lib/modules/Stdio.pmod/module.pmod:137 | Killing 0xa64b2a8 with 1 refs, prog src/modules/files/file.c:4928 | Killing 0xa64b368 with 1 refs, prog /usr/local/pike/7.8.409/lib/modules/Thread.pmod:278 | Killing 0xa454538 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa53bcf8 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa53b578 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa53a978 with 3 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa53be60 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa53bab8 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa466710 with 5 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa421380 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xb440fc50 with 2 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:7865 | Killing 0xa465810 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PXml.pike:0 | Killing 0xa8bc0c8 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | Killing 0xa8bc098 with 1 refs, prog /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/PEnt.pike:0 | kill: 99 objects killed, 1702 things really freed | destruct: 9 things really freed done (3256 of 30944 were unreferenced), 69 ms Garbage collecting... | check: 184183 references in 42058 things, counted 746 weak refs /data/src/gpike/src/gc.c:2519: GC fatal: Marked a thing without refs. **Block: 0xa62ea10 Type: object Refs: 0 **Got gc marker at 0xaad8718: flags=0x00000 refs=0 weak=0 xrefs=0 saved=-1 frame=(nil) **Program id: 10 **The object is not on the object link list. **The object is destructed but program found from id. ******************* **Looking for references to 0xa62ea10: **Done looking for references to 0xa62ea10, found 0 refs. Pike was in GC stage 200 when this fatal occurred. Backtrace at time of fatal:

Any more verbosity I can/should add?

No, that was not the debug I expected. I think I confused this with the GC_CYCLE_DEBUG, but that one won't be helpful here. I'm sorry for the confusion.

Looking around in the code some more I actually think the C stack might provide some clues (didn't think so before because of the queue system in the mark pass). Could you please post a couple of gdb backtraces, preferably but not necessarily together with the stderr log?

The next core (I'm keeping all the cores, so more info is available):

#6 0x08112791 in debug_gc_fatal_2 (a=0xb4182918, type=3, flags=1, fmt=0x82efc13 "Marked a thing without refs.\n") at /data/src/gpike/src/gc.c:1041 #7 0x081128e9 in real_gc_mark (a=0xb4182918, type=3) at /data/src/gpike/src/gc.c:2519 #8 0x081687ec in gc_mark_object_as_referenced (o=0xb4182918) at /data/src/gpike/src/object.c:2035 #9 0x081c93a3 in run_queue (q=0x837893c) at /data/src/gpike/src/queue.c:48 #10 0x0811468c in do_gc (ignored=0x9d1b5b8, explicit_call=0) at /data/src/gpike/src/gc.c:3545 #11 0x080e08f0 in low_call_callback (lst=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece. ) at /data/src/gpike/src/callback.c:168 #12 0x080960b7 in low_mega_apply (type=APPLY_LOW, args=3, arg1=0xb3f9a748, ---Type <return> to continue, or q <return> to quit--- arg2=0xd) at /data/src/gpike/src/apply_low.h:296 #13 0x0809dd39 in jump_opcode_F_CALL_LFUN (arg1=13) at /data/src/gpike/src/interpret_functions.h:2061 #14 0xb6fa1174 in ?? () #15 0x080938e4 in eval_instruction ( pc=0xb6dc59bf "\213\r\020|7\b\270A\246#I\367؉A\034\350\034\366+Q\213\r\020|7\b\270,\246#I\367؉A\034\203A\034\020\307\004$\001") at /data/src/gpike/src/interpret.c:1421 #16 0x0809441f in catching_eval_instruction (pc=<value optimized out>) at /data/src/gpike/src/interpret.c:2245 #17 0x0809467b in jump_opcode_F_CATCH () at /data/src/gpike/src/interpret_functions.h:1287 #18 0xb6d0c62d in ?? () #19 0x080938e4 in eval_instruction ( pc=0xb6d0c580 "\213\r\020|7\b\270\200:/I\367؉A\034\307\004$\001") at /data/src/gpike/src/interpret.c:1421 #20 0x08097f28 in mega_apply (type=APPLY_STACK, args=2, arg1=0x0, arg2=0x0) at /data/src/gpike/src/interpret.c:2213 #21 0x0809a4a9 in f_call_function (args=2) at /data/src/gpike/src/interpret.c:2266 #22 0x081c88aa in new_thread_func (data=0xbf9e58ac) at /data/src/gpike/src/threads.c:1111 #23 0xb7f06585 in start_thread (arg=0xb57aeb70) at pthread_create.c:300 ---Type <return> to continue, or q <return> to quit--- #24 0xb7e552be in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

Stderr output: /data/src/gpike/src/gc.c:2519: GC fatal: Marked a thing without refs. **Block: 0xb4182918 Type: object Refs: 0 **Got gc marker at 0xb1d368f0: flags=0x00000 refs=0 weak=0 xrefs=0 saved=-1 frame=(nil) **Program id: 991604 **The object is not on the object link list. **The object is destructed and program not found from id. ******************* **Looking for references to 0xb4182918: **Done looking for references to 0xb4182918, found 0 refs. Pike was in GC stage 200 when this fatal occurred. Backtrace at time of fatal: /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:1826: Context()->set_var(({"vorigerewritetitle"}),"VERMIST-Mercedes-4x4.html") /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:1916: Context()->user_set_var("var.vorigerewritetitle","VERMIST-Mercedes-4x4.html",UNDEFINED) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:5234: RXML->user_set_var("var.vorigerewritetitle","VERMIST-Mercedes-4x4.html",UNDEFINED) modules/tags/rxmltags.pike:796: Frame(set)->do_return(RequestID(/extern/nieuws.rxml?id=2763)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4986: Frame(set)->_eval(Context(),RXML.PCode(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set)),RXML.t_html(text/html, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:9027: RXML.PCode(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set))->_eval(@0=Context(),0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4923: Frame(emit)->_eval(@0,RXML.PCode(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set)),@1=RXML.t_html(text/html, xml)) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:9027: RXML.PCode(RXML.t_html(text/html, xml),RXML.CompositeTagSet(emit+rxml_tag_set))->_eval(@0,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:4923: Frame(emit)->_eval(@0,RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set)),@1) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:9027: RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->_eval(@0,0) /usr/roxen/server-4.5.0/etc/modules/RXML.pmod/module.pmod:8435: RXML.PCode(RXML.t_html(text/html, xml),RXMLTagSet(entreeding.com,rxml_tag_set))->eval(@0,UNDEFINED) modules/tags/rxmlparse.pike:171: RoxenModule(entreeding.com/rxmlparse#0)->handle_file_extension(Stdio.File("/home/entreeding/extern/nieuws.rxml", "r", 666 /* fd=29 */),"rxml",@2=RequestID(/extern/nieuws.rxml?id=2763)) base_server/configuration.pike:2361: Configuration(entreeding.com)->low_get_file(@2,UNDEFINED) base_server/configuration.pike:2495: Configuration(entreeding.com)->get_file(@2,UNDEFINED,UNDEFINED) base_server/configuration.pike:2470: Configuration(entreeding.com)->handle_request(@2,UNDEFINED) protocols/http.pike:2827: RequestID(/extern/nieuws.rxml?id=2763)->handle_request() base_server/roxen.pike:849: roxen()->handler_thread(13)

And about 30 minutes later we get core4 (this one is odd and interesting, since it has a rather short Pike backtrace):

#6 0x08112791 in debug_gc_fatal_2 (a=0x92985d0, type=3, flags=1, fmt=0x82efc13 "Marked a thing without refs.\n") at /data/src/gpike/src/gc.c:1041 #7 0x081128e9 in real_gc_mark (a=0x92985d0, type=3) at /data/src/gpike/src/gc.c:2519 #8 0x081687ec in gc_mark_object_as_referenced (o=0x92985d0) at /data/src/gpike/src/object.c:2035 #9 0x081c93a3 in run_queue (q=0x837893c) at /data/src/gpike/src/queue.c:48 #10 0x0811468c in do_gc (ignored=0x89f7448, explicit_call=0) at /data/src/gpike/src/gc.c:3545 #11 0x080e08f0 in low_call_callback (lst=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece. ) at /data/src/gpike/src/callback.c:168 #12 0x0808408e in opcode_F_CALL_BUILTIN (arg1=36) ---Type <return> to continue, or q <return> to quit--- at /data/src/gpike/src/interpret_functions.h:2303 #13 0xb45e5c7b in ?? () #14 0x080938e4 in eval_instruction ( pc=0xb46097fc "\241\020|7\b\203@\034\017\307\004$\225") at /data/src/gpike/src/interpret.c:1421 #15 0x0809441f in catching_eval_instruction (pc=<value optimized out>) at /data/src/gpike/src/interpret.c:2245 #16 0x0809467b in jump_opcode_F_CATCH () at /data/src/gpike/src/interpret_functions.h:1287 #17 0xb46097f6 in ?? () #18 0x080938e4 in eval_instruction ( pc=0xb7a0a104 "\213\r\020|7\b\270\374^_H\367؉A\034\307D$\004;") at /data/src/gpike/src/interpret.c:1421 #19 0x08097f28 in mega_apply (type=APPLY_SVALUE, args=0, arg1=0x90d04cc, arg2=0x0) at /data/src/gpike/src/interpret.c:2213 #20 0x08097fe1 in apply_svalue (s=0x0, args=0) at /data/src/gpike/src/interpret.c:2632 #21 0x08261034 in got_fd_event (box=0x90d04b0, event=0) at /data/src/gpike/src/modules/files/file.c:275 #22 0x080db1b7 in backend_call_active_callbacks (fd_list=0xbfd2e21c, me=<value optimized out>) at /data/src/gpike/src/backend.cmod:1885 #23 0x080dfcbe in pdb_low_backend_once (pdb=<value optimized out>, timeout=<value optimized out>) at /data/src/gpike/src/backend.cmod:3360 ---Type <return> to continue, or q <return> to quit--- #24 0x080e047e in f_PollDeviceBackend_cq__backtick_28_29 (args=1) at /data/src/gpike/src/backend.cmod:3458 #25 0x08095a39 in low_mega_apply (type=APPLY_SVALUE, args=<value optimized out>, arg1=0x0, arg2=0x0) at /data/src/gpike/src/apply_low.h:226 #26 0x0809c393 in jump_opcode_F_CALL_FUNCTION_AND_POP () at /data/src/gpike/src/interpret_functions.h:2068 #27 0xb79570dc in ?? () #28 0x080938e4 in eval_instruction ( pc=0xb795585e "\241\020|7\b\203@\034\027\307\004$Z") at /data/src/gpike/src/interpret.c:1421 #29 0x0809441f in catching_eval_instruction (pc=<value optimized out>) at /data/src/gpike/src/interpret.c:2245 #30 0x0809467b in jump_opcode_F_CATCH () at /data/src/gpike/src/interpret_functions.h:1287 #31 0xb7955858 in ?? () #32 0x080938e4 in eval_instruction ( pc=0xb794d22e "\213\r\020|7\b\270\322-kH\367؉A\034\307D$\004") at /data/src/gpike/src/interpret.c:1421 #33 0x08097f28 in mega_apply (type=APPLY_LOW, args=1, arg1=0x896e108, arg2=0x38) at /data/src/gpike/src/interpret.c:2213 #34 0x080989d0 in apply (o=0x896e108, fun=0x82caaae "_main", args=1) at /data/src/gpike/src/interpret.c:2617 ---Type <return> to continue, or q <return> to quit--- #35 0x08071253 in main (argc=17, argv=0xbfd2e744) at /data/src/gpike/src/main.c:634

Stderr output:

/data/src/gpike/src/gc.c:2519: GC fatal: Marked a thing without refs. **Block: 0x92985d0 Type: object Refs: 0 **Got gc marker at 0xb2485198: flags=0x00000 refs=0 weak=0 xrefs=0 saved=-1 frame=(nil) **Program id: 65596 **The object is not on the object link list. **The object is destructed but program found from id. ******************* **Looking for references to 0x92985d0: **Done looking for references to 0x92985d0, found 0 refs. Pike was in GC stage 200 when this fatal occurred. Backtrace at time of fatal: protocols/http.pike:615: RequestID(/_internal/cimg!0/h5jcuo71yjrtsjn8xj0a3lna56zwh6j)->things_to_do_when_not_sending_from_cache() protocols/http.pike:3388: RequestID(/_internal/cimg!0/h5jcuo71yjrtsjn8xj0a3lna56zwh6j)->got_data(0,"GET /_internal/cimg!0/h5jcuo71yjrtsjn8xj0a3lna56zwh6j HTTP/1.1\r\nAccept: */*\r\nReferer: http://member.entreeding.com/(offer)/m/ember/showinventory3.rxml?photo=1%5Cr...: nl\r\nUA-CP"+[752],UNDEFINED) -:1: Pike.Backend(0)->`()(3600.0) /usr/local/pike/7.8.409/lib/master.pike:3849: master()->_main(({"/usr/roxen/server-4.5.0/bin/roxen","-DENABLE_THREADS","-DRAM_CACHE","-DHTTP_COMPRESSION","-DENABLE_THREADS","-DHTTP_COMPRESSION",,,10}))

The length of the C backtrace seems to be mainly because you have 8 levels of catch in the pike code.

Martin Stjernholm wrote:

Could you also in, say, this last coredump check the contents of the object struct? Something like this:

(gdb) fr 8 #8 0x081687ec in gc_mark_object_as_referenced (o=0x92985d0) at /data/src/gpike/src/object.c:2035 2035 if(gc_mark(o, T_OBJECT)) { (gdb) p *o $1 = {refs = 0, prog = 0x0, next = 0x9296878, prev = 0x0, program_id = 65596, storage = 0x0} (gdb) p *o->next $2 = {refs = 0, prog = 0x0, next = 0x0, prev = 0x0, program_id = 65596, storage = 0x0} (gdb)

Going back to the first coredump, we get: (gdb) fr 8 #8 0x081687ec in gc_mark_object_as_referenced (o=0xb35dbd40) at /data/src/gpike/src/object.c:2035 2035 if(gc_mark(o, T_OBJECT)) { (gdb) p *o $1 = {refs = 0, prog = 0x0, next = 0xb35dc2e0, prev = 0x0, program_id = 166647, storage = 0x0} (gdb) p *o->next $2 = {refs = 0, prog = 0x0, next = 0xb35dc730, prev = 0x0, program_id = 166648, storage = 0x0} (gdb) p *o->next->next $3 = {refs = 0, prog = 0x0, next = 0xb35dc370, prev = 0x0, program_id = 65540, storage = 0x0} (gdb) p *o->next->next->next $4 = {refs = 0, prog = 0x0, next = 0xb35dbe90, prev = 0x0, program_id = 66573, storage = 0x0} (gdb) p *o->next->next->next->next $5 = {refs = 0, prog = 0x0, next = 0xb35dcc70, prev = 0x0, program_id = 66179, storage = 0x0} (gdb)

Martin Stjernholm wrote:

Please use a pike fresh from cvs as I've made a few more tweaks there. I

Ok, picked up latest version.

Compile with the GC_MARK_DEBUG define (it's outcommented in gc.h). It

Done.

the queue. If you have the opportunity to debug in a live process, please try

(gdb) call describe(b->entries[e].in)

before you kill it.

Hmmm, that will be difficult. I usually get coredumps, but I'll give it a try to see if I can wait for a coredump while still in gdb.

You can also compile with GC_DEBUG (now outcommented in gc.c). That adds

Turned on. Waiting for results.

"Stephen R. van den Berg" srb@cuci.nl wrote:

/data/src/gpike/src/gc.c:1767: GC fatal: gc_mark_enqueue() called outside GC_ENTER. **Block: 0xb4227740 Type: object Refs: 1

Could be something. Can you post the gdb backtrace, please?

Stephen R. van den Berg wrote:

/data/src/gpike/src/gc.c:1767: GC fatal: gc_mark_enqueue() called outside GC_ENTER. **Block: 0x95812b8 Type: object Refs: 1 **Program id: 69849 **Object variables:

And the other one:

#6 0x08113663 in debug_gc_fatal (a=0x95812b8, flags=0, fmt=0x82f3038 "gc_mark_enqueue() called outside GC_ENTER.\n") at /data/src/gpike/src/gc.c:1034 #7 0x0811474d in gc_mark_enqueue ( call=0x8169560 <gc_mark_object_as_referenced>, data=0x95812b8) at /data/src/gpike/src/gc.c:1767 #8 0x081f0332 in real_gc_mark_svalues (s=0xb3e0b070, num=3) at /data/src/gpike/src/svalue.c:2354 #9 0x0818943c in gc_recurse_frame (o=0x9581b88) at /data/src/gpike/src/program.c:9920 #10 gc_recurse_trampoline (o=0x9581b88) at /data/src/gpike/src/program.c:9926 #11 0x081687e8 in visit_object (o=0x9581b88, action=<value optimized out>) at /data/src/gpike/src/object.c:2010 #12 0x0810ea21 in f_count_memory (args=2) at /data/src/gpike/src/gc.c:5532 #13 0x08095ea6 in low_mega_apply (type=APPLY_SVALUE, args=<value optimized out>, arg1=0x8d21058, arg2=0x0) at /data/src/gpike/src/apply_low.h:226 #14 0x0809d72f in jump_opcode_F_APPLY (arg1=2) at /data/src/gpike/src/interpret_functions.h:2065 #15 0xb6e262ca in ?? () #16 0x08093d04 in eval_instruction ( pc=0xb6cfa9df "\213\rp\235\067\b\270!V0I\367؉A\034蜨8Q\213\rp\235\067\b\270\fV0I\367؉A\034\203A\034\020\307\004$\001") at /data/src/gpike/src/interpret.c:1421 #17 0x0809483f in catching_eval_instruction (pc=<value optimized out>) at /data/src/gpike/src/interpret.c:2245 #18 0x08094aa8 in jump_opcode_F_CATCH () at /data/src/gpike/src/interpret_functions.h:1287 #19 0xb6c4162d in ?? () #20 0x08093d04 in eval_instruction ( ---Type <return> to continue, or q <return> to quit--- pc=0xb6c41580 "\213\rp\235\067\b\270\200\352;I\367؉A\034\307\004$\001") at /data/src/gpike/src/interpret.c:1421 #21 0x080984d8 in mega_apply (type=APPLY_STACK, args=2, arg1=0x0, arg2=0x0) at /data/src/gpike/src/interpret.c:2213 #22 0x0809aa59 in f_call_function (args=2) at /data/src/gpike/src/interpret.c:2266 #23 0x081c97da in new_thread_func (data=0xbfb1e9dc) at /data/src/gpike/src/threads.c:1111 #24 0xb7e3d585 in start_thread (arg=0xb6980b70) at pthread_create.c:300 #25 0xb7d8c2be in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

"Stephen R. van den Berg" srb@cuci.nl wrote:

Could you please put this a bit higher on the priority list, since I'd have to revert to an older version of the code otherwise?

Oh, I thought you already reverted it after catching those errors, considering it is a production system.

The debug shows a real problem, but I haven't worked out a fix yet. Just in case I need to see something more, please keep a core dump and the binary around (for gdb use).

"Stephen R. van den Berg" srb@cuci.nl wrote:

It just means that I need to compile from a different tree (which I wanted to avoid), since keeping the binaries, but not the accompanying sources tends to make gdb navigation unusable.

I usually don't bother - backtraces are still fairly sane, and you can still inspect data without problems. It's only line numbers and source listings that can be a little bit off.

Anyway, I've got good news. I've fixed the bug that GC_MARK_DEBUG caught, and I'm fairly certain that fixes the real issue too. Can you please check that by first updating gc.c only, so that you get the new assertions I've added there? If you get a fatal in the assertion at gc.c:3560 then you're hit by the bug I've found. Update gc.h and it should be gone.

The problem was that the gc_recurse callbacks used from Pike.count_memory didn't choose the correct code path in some cases, going into the gc mark code instead and thereby producing garbage in the gc mark queue. That garbage could then make the gc misbehave, but only in some rare circumstances to the extent that it crashes. In most cases I suspect the effect was that it just missed a little bit of garbage, nothing else.

So even though the error really was pretty big, it had very rarely any observable effect. Very insidious.

(Another effect of the bug was that Pike.count_memory missed some things and returned a little bit too low count.)

Martin Stjernholm wrote:

caught, and I'm fairly certain that fixes the real issue too. Can you please check that by first updating gc.c only, so that you get the new assertions I've added there? If you get a fatal in the assertion at gc.c:3560 then you're hit by the bug I've found. Update gc.h and it should be gone.

Looking good. Got this: roxen: /data/src/gpike/src/gc.c:3560: do_gc: Assertion `!gc_mark_queue.first' failed.

So I'm upgrading to bleeding edge with fix now. If this message-thread stops here, then it was the correct fix.

Some others:

#3 0xb7cebace in *__GI___assert_fail ( assertion=0x82f1303 "!gc_mark_queue.first", file=0x82f05d4 "/data/src/gpike/src/gc.c", line=3560, function=0x82f3dea "do_gc") at assert.c:78 #4 0x08116246 in do_gc (ignored=0x9cf61f8, explicit_call=0) at /data/src/gpike/src/gc.c:3560 #5 0x080e1320 in low_call_callback (lst=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece. ) at /data/src/gpike/src/callback.c:168 #6 0x08096491 in low_mega_apply (type=APPLY_LOW, args=1, arg1=0xa51ba68, arg2=0x5f) at /data/src/gpike/src/apply_low.h:296 #7 0x0809dffa in jump_opcode_F_CALL_LFUN_AND_POP (arg1=95) at /data/src/gpike/src/interpret_functions.h:2061 #8 0xb6cb8f14 in ?? () #9 0x08093c94 in eval_instruction (pc=0xb6f23a11 "\307D$\004\005") at /data/src/gpike/src/interpret.c:1421 #10 0x080947cf in catching_eval_instruction (pc=<value optimized out>) at /data/src/gpike/src/interpret.c:2245 #11 0x08094a38 in jump_opcode_F_CATCH () at /data/src/gpike/src/interpret_functions.h:1287 #12 0xb6f23a0b in ?? () ---Type <return> to continue, or q <return> to quit--- #13 0x08093c94 in eval_instruction ( pc=0xb6f23291 "\213\rp\235\067\b\270o\315\rI\367؉A\034\307\004$\023") at /data/src/gpike/src/interpret.c:1421 #14 0x08098468 in mega_apply (type=APPLY_STACK, args=4, arg1=0x0, arg2=0x0) at /data/src/gpike/src/interpret.c:2213 #15 0x0809a9e9 in f_call_function (args=4) at /data/src/gpike/src/interpret.c:2266 #16 0xb742a662 in container_callback (finished=<value optimized out>) at /data/src/gpike/src/modules/Parser/html.c:2755 #17 do_try_feed (finished=<value optimized out>) at /data/src/gpike/src/modules/Parser/html.c:3252 #18 try_feed (finished=<value optimized out>) at /data/src/gpike/src/modules/Parser/html.c:3771 #19 0xb742be36 in html_finish (args=1) at /data/src/gpike/src/modules/Parser/html.c:3966 #20 0x08095e36 in low_mega_apply (type=APPLY_LOW, args=<value optimized out>, arg1=0xa51a898, arg2=0x7f) at /data/src/gpike/src/apply_low.h:226 #21 0x0809dffa in jump_opcode_F_CALL_LFUN_AND_POP (arg1=127) at /data/src/gpike/src/interpret_functions.h:2061 #22 0xb6cb8584 in ?? () #23 0x08093c94 in eval_instruction (pc=0xb6f23a11 "\307D$\004\005") at /data/src/gpike/src/interpret.c:1421 #24 0x080947cf in catching_eval_instruction (pc=<value optimized out>) ---Type <return> to continue, or q <return> to quit--- at /data/src/gpike/src/interpret.c:2245 #25 0x08094a38 in jump_opcode_F_CATCH () at /data/src/gpike/src/interpret_functions.h:1287 #26 0xb6f23a0b in ?? () #27 0x08093c94 in eval_instruction ( pc=0xb6f23291 "\213\rp\235\067\b\270o\315\rI\367؉A\034\307\004$\023") at /data/src/gpike/src/interpret.c:1421 #28 0x08098468 in mega_apply (type=APPLY_STACK, args=4, arg1=0x0, arg2=0x0) at /data/src/gpike/src/interpret.c:2213 #29 0x0809a9e9 in f_call_function (args=4) at /data/src/gpike/src/interpret.c:2266 #30 0xb742a662 in container_callback (finished=<value optimized out>) at /data/src/gpike/src/modules/Parser/html.c:2755 #31 do_try_feed (finished=<value optimized out>) at /data/src/gpike/src/modules/Parser/html.c:3252 #32 try_feed (finished=<value optimized out>) at /data/src/gpike/src/modules/Parser/html.c:3771 #33 0xb742be36 in html_finish (args=1) at /data/src/gpike/src/modules/Parser/html.c:3966 #34 0x08095e36 in low_mega_apply (type=APPLY_LOW, args=<value optimized out>, arg1=0xa51a2c8, arg2=0x7f) at /data/src/gpike/src/apply_low.h:226 #35 0x0809dffa in jump_opcode_F_CALL_LFUN_AND_POP (arg1=127) at /data/src/gpike/src/interpret_functions.h:2061 ---Type <return> to continue, or q <return> to quit--- #36 0xb6cb8584 in ?? () #37 0x08093c94 in eval_instruction (pc=0xb6f23a11 "\307D$\004\005") at /data/src/gpike/src/interpret.c:1421 #38 0x080947cf in catching_eval_instruction (pc=<value optimized out>) at /data/src/gpike/src/interpret.c:2245 #39 0x08094a38 in jump_opcode_F_CATCH () at /data/src/gpike/src/interpret_functions.h:1287 #40 0xb6f23a0b in ?? () #41 0x08093c94 in eval_instruction ( pc=0xb6f23291 "\213\rp\235\067\b\270o\315\rI\367؉A\034\307\004$\023") at /data/src/gpike/src/interpret.c:1421 #42 0x08098468 in mega_apply (type=APPLY_STACK, args=4, arg1=0x0, arg2=0x0) at /data/src/gpike/src/interpret.c:2213 #43 0x0809a9e9 in f_call_function (args=4) at /data/src/gpike/src/interpret.c:2266 #44 0xb742a662 in container_callback (finished=<value optimized out>) at /data/src/gpike/src/modules/Parser/html.c:2755 #45 do_try_feed (finished=<value optimized out>) at /data/src/gpike/src/modules/Parser/html.c:3252 #46 try_feed (finished=<value optimized out>) at /data/src/gpike/src/modules/Parser/html.c:3771 #47 0xb742be36 in html_finish (args=1) at /data/src/gpike/src/modules/Parser/html.c:3966 ---Type <return> to continue, or q <return> to quit--- #48 0x08095e36 in low_mega_apply (type=APPLY_LOW, args=<value optimized out>, arg1=0xa51b030, arg2=0x7f) at /data/src/gpike/src/apply_low.h:226 #49 0x0809dffa in jump_opcode_F_CALL_LFUN_AND_POP (arg1=127) at /data/src/gpike/src/interpret_functions.h:2061 #50 0xb6cb8584 in ?? () #51 0x08093c94 in eval_instruction (pc=0xb6f23a11 "\307D$\004\005") at /data/src/gpike/src/interpret.c:1421 #52 0x080947cf in catching_eval_instruction (pc=<value optimized out>) at /data/src/gpike/src/interpret.c:2245 #53 0x08094a38 in jump_opcode_F_CATCH () at /data/src/gpike/src/interpret_functions.h:1287 #54 0xb6f23a0b in ?? () #55 0x08093c94 in eval_instruction ( pc=0xb6f23291 "\213\rp\235\067\b\270o\315\rI\367؉A\034\307\004$\023") at /data/src/gpike/src/interpret.c:1421 #56 0x08098468 in mega_apply (type=APPLY_STACK, args=4, arg1=0x0, arg2=0x0) at /data/src/gpike/src/interpret.c:2213 #57 0x0809a9e9 in f_call_function (args=4) at /data/src/gpike/src/interpret.c:2266 #58 0xb742a662 in container_callback (finished=<value optimized out>) at /data/src/gpike/src/modules/Parser/html.c:2755 #59 do_try_feed (finished=<value optimized out>) at /data/src/gpike/src/modules/Parser/html.c:3252 ---Type <return> to continue, or q <return> to quit--- #60 try_feed (finished=<value optimized out>) at /data/src/gpike/src/modules/Parser/html.c:3771 #61 0xb742be36 in html_finish (args=1) at /data/src/gpike/src/modules/Parser/html.c:3966 #62 0x08095e36 in low_mega_apply (type=APPLY_LOW, args=<value optimized out>, arg1=0xa51ba50, arg2=0x7f) at /data/src/gpike/src/apply_low.h:226 #63 0x0809dffa in jump_opcode_F_CALL_LFUN_AND_POP (arg1=127) at /data/src/gpike/src/interpret_functions.h:2061 #64 0xb6cb8584 in ?? () #65 0x08093c94 in eval_instruction ( pc=0xb6d059bf "\213\rp\235\067\b\270A\246/I\367؉A\034\350L\370\067Q\213\rp\235\067\b\270,\246/I\367؉A\034\203A\034\020\307\004$\001") at /data/src/gpike/src/interpret.c:1421 #66 0x080947cf in catching_eval_instruction (pc=<value optimized out>) at /data/src/gpike/src/interpret.c:2245 #67 0x08094a38 in jump_opcode_F_CATCH () at /data/src/gpike/src/interpret_functions.h:1287 #68 0xb6c4c62d in ?? () #69 0x08093c94 in eval_instruction ( pc=0xb6c4c580 "\213\rp\235\067\b\270\200:;I\367؉A\034\307\004$\001") at /data/src/gpike/src/interpret.c:1421 #70 0x08098468 in mega_apply (type=APPLY_STACK, args=2, arg1=0x0, arg2=0x0) at /data/src/gpike/src/interpret.c:2213 ---Type <return> to continue, or q <return> to quit--- #71 0x0809a9e9 in f_call_function (args=2) at /data/src/gpike/src/interpret.c:2266 #72 0x081c966a in new_thread_func (data=0xbfb291ec) at /data/src/gpike/src/threads.c:1111 #73 0xb7e48585 in start_thread (arg=0xb5199b70) at pthread_create.c:300 #74 0xb7d972be in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

And another one:

#3 0xb7de8ace in *__GI___assert_fail ( assertion=0x82f1303 "!gc_mark_queue.first", file=0x82f05d4 "/data/src/gpike/src/gc.c", line=3560, function=0x82f3dea "do_gc") at assert.c:78 #4 0x08116246 in do_gc (ignored=0xa383678, explicit_call=0) at /data/src/gpike/src/gc.c:3560 #5 0x080e1320 in low_call_callback (lst=DWARF-2 expression error: DW_OP_reg operations must be used either alone or in conjuction with DW_OP_piece. ) at /data/src/gpike/src/callback.c:168 #6 0x08096491 in low_mega_apply (type=APPLY_LOW, args=1, arg1=0xa84d388, arg2=0x101) at /data/src/gpike/src/apply_low.h:296 #7 0x0809e2a9 in jump_opcode_F_CALL_LFUN (arg1=257) at /data/src/gpike/src/interpret_functions.h:2061 #8 0xb6be2138 in ?? () #9 0x08093c94 in eval_instruction ( pc=0xb6e029bf "\213\rp\235\067\b\270A\326\037I\367؉A\034\350L((Q\213\rp\235\067\b\270,\326\037I\367؉A\034\203A\034\020\307\004$\001") at /data/src/gpike/src/interpret.c:1421 #10 0x080947cf in catching_eval_instruction (pc=<value optimized out>) at /data/src/gpike/src/interpret.c:2245 #11 0x08094a38 in jump_opcode_F_CATCH () ---Type <return> to continue, or q <return> to quit--- at /data/src/gpike/src/interpret_functions.h:1287 #12 0xb6d4962d in ?? () #13 0x08093c94 in eval_instruction ( pc=0xb6d49580 "\213\rp\235\067\b\270\200j+I\367؉A\034\307\004$\001") at /data/src/gpike/src/interpret.c:1421 #14 0x08098468 in mega_apply (type=APPLY_STACK, args=2, arg1=0x0, arg2=0x0) at /data/src/gpike/src/interpret.c:2213 #15 0x0809a9e9 in f_call_function (args=2) at /data/src/gpike/src/interpret.c:2266 #16 0x081c966a in new_thread_func (data=0xbff24dec) at /data/src/gpike/src/threads.c:1111 #17 0xb7f45585 in start_thread (arg=0xb6a88b70) at pthread_create.c:300 #18 0xb7e942be in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130

"Stephen R. van den Berg" srb@cuci.nl wrote:

Some others:

#3 0xb7cebace in *__GI___assert_fail ( assertion=0x82f1303 "!gc_mark_queue.first",

Ok, let's take a couple more steps from the mark queue garbage angle then. I've added a few more checks now which hopefully nails that bug at the source. It doesn't need GC_MARK_DEBUG, but if you can, please define that anyway since it might improve the debug info a little.

I think one hit is enough if you get it at one of the new checks. Then please feel free to back it out if it blows up all over..

"Stephen R. van den Berg" srb@cuci.nl wrote:

/.../

So it seems like it is solved after all.

Sorry for the confusion.

No problem, I'm just glad it's sorted out at last. I've now closed [bug 5072] as well.