Pike-devel

pike-devel@lists.lysator.liu.se

2 participants
3039 discussions

Understand the behaviour of Stdio.Buffer()->match()
by Chris Angelico 08 May '24

08 May '24

I'm a bit confused by this behaviour, and am unsure whether it needs to be better documented, or perhaps is a bug. int main() { object buf = Stdio.Buffer("#44"); write("Matches: %O %O\n", buf->sscanf("#%2[4]"), (string)buf); buf = Stdio.Buffer("#4"); write("Doesn't: %O %O\n", buf->sscanf("#%2[4]"), (string)buf); buf = Stdio.Buffer("#44"); write("Matches: %O %O\n", buf->match("#%2[4]"), (string)buf); buf = Stdio.Buffer("#4"); write("Doesn't: %O %O\n", buf->match("#%2[4]"), (string)buf); } When using sscanf, the successful match returns ({"44"}) and leaves the buffer empty, and the unsuccessful match returns an empty array and leaves the "4" in the buffer. This makes sense; the hash was matched and consumed, but the 2-character requirement meant that nothing was matched. Using match instead produces a somewhat odd result though. If there's enough text in the buffer, it returns "44" and leaves the buffer empty; this corresponds to sscanf's behaviour. In the other case, though, it returns "#%2[4]" (the pattern), and leaves "4" in the buffer. Is this intentional? (If the first character of the buffer isn't a hash, both return 0 and leave the buffer unchanged. This behaviour is also correct. It's only the partial match that seems curious.) ChrisA

3 3

Re: Understand the behaviour of Stdio.Buffer()->match()
by Chris Angelico 08 May '24

08 May '24

On Wed, 8 May 2024 at 17:57, Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > Yes, although I'm still not sure this follows the principle of least > astonishment: > > If no matches results in "", and the result is the sum of the matches, > wouldn't you expect the result in this case to be ""+4+7, i.e. the > string "47"? If an arithmetic sum is expected then you should get 0 > from a non-match IMO... Well, you do get zero for a non-match, so ... I guess so? :) > object buf = Stdio.Buffer("asdf"); > buf->match("qwer"); (1) Result: 0 But I don't think that's quite the same as a null sum. ChrisA

2 1

Re: Understand the behaviour of Stdio.Buffer()->match()
by Chris Angelico 08 May '24

08 May '24

On Wed, 8 May 2024 at 17:04, Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > The behaviour of match() when there is not exactly one thing matching > seems a bit weird in general: > > Pike v9.0 release 2 running Hilfe v3.5 (Incremental Pike Frontend) > > object buf = Stdio.Buffer("#4,7"); > > buf->match("#%d,%d"); > (1) Result: 11 > > That IS a bit weird, although not undocumented - if it matches multiple strings, it returns their concatenation. But since it's defined as returning the *sum*, this quirk happens. It's designed for a single match pattern and for that it generally does fine. Definitely an amusing consequence though! ChrisA

2 1

Re: Understand the behaviour of Stdio.Buffer()->match()
by Chris Angelico 08 May '24

08 May '24

On Wed, 8 May 2024 at 16:28, Henrik Grubbström (Lysator) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > Thanks for the report. > > Fixed in Pike master. match() now returns "" in this case. Ah! Thank you! ChrisA

1 0

Pike Meet-up 2024-Q2
by Henrik Grubbstr�m (Lysator) ＠ Pike (-) developers forum 07 May '24

07 May '24

Hi. At Pike Conference 2017 it was decided that we ought to hold a more informal meeting in Linköping ~ once every quarter. As the 6 month mark from the latest conference is coming up soon, it seems time to announce the second such meeting for 2024: Pike Meet-up 2024-Q2 Date: 2024-05-11 (Sat) Time: 10:00 - afternoon/evening Place: Roxen IS AB Storgatan 46 Linköping Welcome! /grubba

1 1

Re: Running an HTTP(S) server using an externally-provided file descriptor
by Chris Angelico 02 May '24

02 May '24

On Thu, 2 May 2024 at 19:15, Henrik Grubbström (Lysator) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > Hi again Chris. > > > On Thu, 2 May 2024 at 17:58, Henrik Grubbström (Lysator) @ Pike (-) > > developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > > > > > Does SSL.Port()->listen_fd() not work? > > > > Inside HTTP.Server.Port, it kinda does, I think? I didn't run into any > > actual problems. > > > > Inside HTTP.Server.SSLPort though, it doesn't, because omitting (or > > zeroing) the port number results in the default of 443 being used. So > > at best, you would have to pick a port and hope that it's not in use, > > listen on that, and then replace it with listen_fd(). I didn't try > > that but it seems rather inelegant. > > Hmm... With your fixes, wouldn't it it work to just give an > appropriate SSL.Port object to HTTP.Server.Port()? You would > miss out on the generation of the default certificate, but > otherwise it ought to behave the same. > Huh. That would be rather nice, if it works. Currently, the two codebases look pretty different. It would be rather neat if SSLPort could simply inherit Port and provide a default parameter. ChrisA

1 0

Re: Running an HTTP(S) server using an externally-provided file descriptor
by Chris Angelico 02 May '24

02 May '24

On Thu, 2 May 2024 at 17:58, Henrik Grubbström (Lysator) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > Hi Chris. > > > See branch rosuav/pre-listening-ports for a first attempt at this. > > > > With Stdio.Port(), it's possible to receive an open socket as a file > > descriptor and start accepting connections on it. (Notably, this > > allows systemd to provide a socket as FD 3, allowing unprivileged > > programs to listen on privileged ports.) Currently, this isn't > > possible with Protocols.HTTP.Server.Port or its friends. > > This reminds me of the original lpc4 Spider webserver installation > at Lysator. > > > For unencrypted connections, this isn't too hard. Instead of having it > > construct a Port, it can receive one already listening. For SSLPort > > though, it inherits SSL.Port, so this doesn't work. What would be a > > good API for making this possible? > > Does SSL.Port()->listen_fd() not work? Inside HTTP.Server.Port, it kinda does, I think? I didn't run into any actual problems. Inside HTTP.Server.SSLPort though, it doesn't, because omitting (or zeroing) the port number results in the default of 443 being used. So at best, you would have to pick a port and hope that it's not in use, listen on that, and then replace it with listen_fd(). I didn't try that but it seems rather inelegant. On Thu, 2 May 2024 at 18:21, Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > If I understand correctly, the problem is that while you can construct > an Stdio.Port or an SSL.Port from a file descriptor, you can't do so > with a Protocols.HTTP.Server.SSLPort, because the create function of > this class (taking 1-6 arguments) unconditionally calls > Port::socket::bind(). > > In which case I guess the solution would be to add a variant create() > taking either zero arguments or an SSL.Context (to mimick the one of > SSL.Port) while leaving the current one as it is for compatibility? This may be the solution to both problems. Let me see if I understand you correctly. The variant create would take just the Context, and then before the SSLPort becomes useful, you call listen_fd() and then set the callback? ChrisA

3 2

Running an HTTP(S) server using an externally-provided file descriptor
by Chris Angelico 02 May '24

02 May '24

See branch rosuav/pre-listening-ports for a first attempt at this. With Stdio.Port(), it's possible to receive an open socket as a file descriptor and start accepting connections on it. (Notably, this allows systemd to provide a socket as FD 3, allowing unprivileged programs to listen on privileged ports.) Currently, this isn't possible with Protocols.HTTP.Server.Port or its friends. For unencrypted connections, this isn't too hard. Instead of having it construct a Port, it can receive one already listening. For SSLPort though, it inherits SSL.Port, so this doesn't work. What would be a good API for making this possible? (Protocols.WebSocket.Port and Protocols.WebSocket.SSLPort inherit from their non-websocket equivalents, so they broadly "come along for the ride".) Incidentally, this actually means it is easier to use opportunistic TLS than mandated, since a regular Port can, with a handful of lines of code, be enhanced to call on SSL.File directly. But API-wise it would be a bit quirky if Port supports this feature and SSLPort doesn't. Recommendations? ChrisA

3 2

Re: File descriptor usage in Protocols.HTTP.Server.SSLPort
by Chris Angelico 08 Apr '24

08 Apr '24

On Fri, 5 Apr 2024 at 23:59, Henrik Grubbström (Lysator) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > Is there a better way to handle this than simply forcing garbage > > collection every request? > > The best way is to break the cycle before dropping the object on the > floor (or to not introduce it to begin with) or destruct the > problematic objects explicitly. There's no cycles in my code, and this phenomenon doesn't happen with a non-SSL connection, so I'm guessing it's something to do with encryption. > To identify the cycles you may want to log the cycles detected by the > gc; take a look at base_server/roxen.pike:reinstall_gc_callbacks(). Thanks, will investigate further. ChrisA

1 0

File descriptor usage in Protocols.HTTP.Server.SSLPort
by Chris Angelico 05 Apr '24

05 Apr '24

I've run into a strangeness with my TLS-based web server. It seems that, for every incoming request, three file descriptors are used (they all seem to be sockets), and they aren't immediately cleaned up. With keep-alive disabled, they CAN be disposed of, but it takes an explicit gc() call. With keep-alive active, they aren't garbage, so they stick around even with a gc() call. (Which is probably correct, but there might need to be a limit on how many get retained.) constant listen_addr = "::", listen_port = 9876; void http_handler(Protocols.HTTP.Server.Request req) { int before = sizeof(get_dir("/proc/self/fd")); int garbo = gc(); //Disabling this check results in FDs accumulating. int after = sizeof(get_dir("/proc/self/fd")); werror("Garbage %O, closed %d files, now %d open\n", garbo, before - after, after); //The "Connection: close" header is vital to the sockets becoming garbage. //Without it, they are retained pending a followup request, which doesn't change the //fundamental issue but does mean that a call to gc() doesn't clean them up. req->response_and_finish((["data": "OK", "extra_heads": (["Connection": "close"])])); //req->response_and_finish((["data": "OK"])); } int main() { //If you don't have a cert, the first request is slower b/c generating self-signed. //This ONLY happens with SSL connections. There are *three* file descriptors wasted //for every request. string cert = Stdio.read_file("certificate_local.pem"); string key = Stdio.read_file("privkey_local.pem"); array certs = cert && Standards.PEM.Messages(cert)->get_certificates(); string pk = key && Standards.PEM.simple_decode(key); Protocols.HTTP.Server.SSLPort(http_handler, listen_port, listen_addr, pk, certs); return -1; } What are the three FDs used for? Presumably one of them is the actual incoming socket, but the other two are less obvious. This phenomenon does not seem to happen with non-SSL ports, possibly relating to the simpler shutdown sequence for unencrypted sockets. This isn't usually a major problem, as the GC does get run eventually, but under certain types of workloads, it's easy to overload it and run out of FDs. This can be triggered by removing the gc() call from each request, and then any of these will spin up until the process FD limit is hit: // Pike: while (1) Protocols.HTTP.get_url_data("https://YOUR_SERVER.EXAMPLE:9876/"); # Python: import requests while 1: requests.get("https://YOUR_SERVER.EXAMPLE:9876/") : Bash: while wget -qO/dev/null https://YOUR_SERVER.EXAMPLE:9876/; do true; done // JavaScript in a web browser, if not blocked: while (1) await(fetch("https://YOUR_SERVER.EXAMPLE:9876/")); So I think it's probably not purely an issue with a client library misbehaving. Is there a better way to handle this than simply forcing garbage collection every request? ChrisA

2 1

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Pike-devel