I'm trying to track down a truly bizarre issue, and not even sure where to start. Deep in my HTTP handler, I have this code:

mapping args = ([]); // ... various conditionals that can set args->blah = something [array streams, mapping self] = await(Concurrent.all( get_helix_paginated("https://api.twitch.tv/helix/streams", args), twitch_api_request("https://api.twitch.tv/helix/streams?user_id=" + userid), ));

Basically, by looking at different aspects of req->variables, we set different arguments to be used in the API call.

What I'm seeing, though, is that - on my secondary server, but NOT my primary - the args mapping starts out with stuff in it!

mapping args = ([]); if (req->variables->categories == "DJs") werror("RAIDFINDER ARGS %O\n", args);

I assume that the issue is intermittent?

Logically, this should always and only print out an empty mapping, yet it doesn't. The contents of the mapping look like other meaningful mappings used elsewhere in the code, so this is hard to make a minimal test case for.

Do both servers use the same exact pike binary?

Sounds like it's likely a reference counting bug on struct mapping_data. I have never seen this issue. It could also be due to C object files having been compiled with different configure options.

Have you run make depend?

How are the mappings that you have identified created (eg mkmapping(), aggregate_mapping(), literal mappings, elements inserted one at a time, mapping addition, etc...)?

Do they have anything in common?

I was testing all this using a master branch build from May, and just now restarted the server on the latest from branch 9.0 (f920879). The problem hasn't yet recurred, but none of the commits between May and now seem to have any bearing on this, so it's possible it will come right back.

For now, I've stuck in a guard line:

if (sizeof(args)) {werror("RAIDFINDER ARGS %O\n", args); m_clear(args);}

so hopefully this won't actually cause HTTP 500s, but I'm at a loss as to how to even investigate this. Any thoughts?

/grubba