Pike-devel April 2024

pike-devel@lists.lysator.liu.se

2 participants
4 discussions

Understand the behaviour of Stdio.Buffer()->match()
by Chris Angelico 08 May '24

08 May '24

I'm a bit confused by this behaviour, and am unsure whether it needs to be better documented, or perhaps is a bug. int main() { object buf = Stdio.Buffer("#44"); write("Matches: %O %O\n", buf->sscanf("#%2[4]"), (string)buf); buf = Stdio.Buffer("#4"); write("Doesn't: %O %O\n", buf->sscanf("#%2[4]"), (string)buf); buf = Stdio.Buffer("#44"); write("Matches: %O %O\n", buf->match("#%2[4]"), (string)buf); buf = Stdio.Buffer("#4"); write("Doesn't: %O %O\n", buf->match("#%2[4]"), (string)buf); } When using sscanf, the successful match returns ({"44"}) and leaves the buffer empty, and the unsuccessful match returns an empty array and leaves the "4" in the buffer. This makes sense; the hash was matched and consumed, but the 2-character requirement meant that nothing was matched. Using match instead produces a somewhat odd result though. If there's enough text in the buffer, it returns "44" and leaves the buffer empty; this corresponds to sscanf's behaviour. In the other case, though, it returns "#%2[4]" (the pattern), and leaves "4" in the buffer. Is this intentional? (If the first character of the buffer isn't a hash, both return 0 and leave the buffer unchanged. This behaviour is also correct. It's only the partial match that seems curious.) ChrisA

3 3

Pike Meet-up 2024-Q2
by Henrik Grubbstr�m (Lysator) ＠ Pike (-) developers forum 07 May '24

07 May '24

Hi. At Pike Conference 2017 it was decided that we ought to hold a more informal meeting in Linköping ~ once every quarter. As the 6 month mark from the latest conference is coming up soon, it seems time to announce the second such meeting for 2024: Pike Meet-up 2024-Q2 Date: 2024-05-11 (Sat) Time: 10:00 - afternoon/evening Place: Roxen IS AB Storgatan 46 Linköping Welcome! /grubba

1 1

Re: File descriptor usage in Protocols.HTTP.Server.SSLPort
by Chris Angelico 07 Apr '24

07 Apr '24

On Fri, 5 Apr 2024 at 23:59, Henrik Grubbström (Lysator) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > Is there a better way to handle this than simply forcing garbage > > collection every request? > > The best way is to break the cycle before dropping the object on the > floor (or to not introduce it to begin with) or destruct the > problematic objects explicitly. There's no cycles in my code, and this phenomenon doesn't happen with a non-SSL connection, so I'm guessing it's something to do with encryption. > To identify the cycles you may want to log the cycles detected by the > gc; take a look at base_server/roxen.pike:reinstall_gc_callbacks(). Thanks, will investigate further. ChrisA

1 0

File descriptor usage in Protocols.HTTP.Server.SSLPort
by Chris Angelico 05 Apr '24

05 Apr '24

I've run into a strangeness with my TLS-based web server. It seems that, for every incoming request, three file descriptors are used (they all seem to be sockets), and they aren't immediately cleaned up. With keep-alive disabled, they CAN be disposed of, but it takes an explicit gc() call. With keep-alive active, they aren't garbage, so they stick around even with a gc() call. (Which is probably correct, but there might need to be a limit on how many get retained.) constant listen_addr = "::", listen_port = 9876; void http_handler(Protocols.HTTP.Server.Request req) { int before = sizeof(get_dir("/proc/self/fd")); int garbo = gc(); //Disabling this check results in FDs accumulating. int after = sizeof(get_dir("/proc/self/fd")); werror("Garbage %O, closed %d files, now %d open\n", garbo, before - after, after); //The "Connection: close" header is vital to the sockets becoming garbage. //Without it, they are retained pending a followup request, which doesn't change the //fundamental issue but does mean that a call to gc() doesn't clean them up. req->response_and_finish((["data": "OK", "extra_heads": (["Connection": "close"])])); //req->response_and_finish((["data": "OK"])); } int main() { //If you don't have a cert, the first request is slower b/c generating self-signed. //This ONLY happens with SSL connections. There are *three* file descriptors wasted //for every request. string cert = Stdio.read_file("certificate_local.pem"); string key = Stdio.read_file("privkey_local.pem"); array certs = cert && Standards.PEM.Messages(cert)->get_certificates(); string pk = key && Standards.PEM.simple_decode(key); Protocols.HTTP.Server.SSLPort(http_handler, listen_port, listen_addr, pk, certs); return -1; } What are the three FDs used for? Presumably one of them is the actual incoming socket, but the other two are less obvious. This phenomenon does not seem to happen with non-SSL ports, possibly relating to the simpler shutdown sequence for unencrypted sockets. This isn't usually a major problem, as the GC does get run eventually, but under certain types of workloads, it's easy to overload it and run out of FDs. This can be triggered by removing the gc() call from each request, and then any of these will spin up until the process FD limit is hit: // Pike: while (1) Protocols.HTTP.get_url_data("https://YOUR_SERVER.EXAMPLE:9876/"); # Python: import requests while 1: requests.get("https://YOUR_SERVER.EXAMPLE:9876/") : Bash: while wget -qO/dev/null https://YOUR_SERVER.EXAMPLE:9876/; do true; done // JavaScript in a web browser, if not blocked: while (1) await(fetch("https://YOUR_SERVER.EXAMPLE:9876/")); So I think it's probably not purely an issue with a client library misbehaving. Is there a better way to handle this than simply forcing garbage collection every request? ChrisA

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Pike-devel April 2024