Pike-devel

pike-devel@lists.lysator.liu.se

3046 discussions

Re: Understand the behaviour of Stdio.Buffer()->match()
by Chris Angelico 08 May '24

08 May '24

On Wed, 8 May 2024 at 17:57, Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > Yes, although I'm still not sure this follows the principle of least > astonishment: > > If no matches results in "", and the result is the sum of the matches, > wouldn't you expect the result in this case to be ""+4+7, i.e. the > string "47"? If an arithmetic sum is expected then you should get 0 > from a non-match IMO... Well, you do get zero for a non-match, so ... I guess so? :) > object buf = Stdio.Buffer("asdf"); > buf->match("qwer"); (1) Result: 0 But I don't think that's quite the same as a null sum. ChrisA

2 1

Re: Understand the behaviour of Stdio.Buffer()->match()
by Chris Angelico 08 May '24

08 May '24

On Wed, 8 May 2024 at 17:04, Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > The behaviour of match() when there is not exactly one thing matching > seems a bit weird in general: > > Pike v9.0 release 2 running Hilfe v3.5 (Incremental Pike Frontend) > > object buf = Stdio.Buffer("#4,7"); > > buf->match("#%d,%d"); > (1) Result: 11 > > That IS a bit weird, although not undocumented - if it matches multiple strings, it returns their concatenation. But since it's defined as returning the *sum*, this quirk happens. It's designed for a single match pattern and for that it generally does fine. Definitely an amusing consequence though! ChrisA

2 1

Re: Understand the behaviour of Stdio.Buffer()->match()
by Chris Angelico 08 May '24

08 May '24

On Wed, 8 May 2024 at 16:28, Henrik Grubbström (Lysator) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > Thanks for the report. > > Fixed in Pike master. match() now returns "" in this case. Ah! Thank you! ChrisA

1 0

Re: Running an HTTP(S) server using an externally-provided file descriptor
by Chris Angelico 02 May '24

02 May '24

On Thu, 2 May 2024 at 19:15, Henrik Grubbström (Lysator) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > Hi again Chris. > > > On Thu, 2 May 2024 at 17:58, Henrik Grubbström (Lysator) @ Pike (-) > > developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > > > > > Does SSL.Port()->listen_fd() not work? > > > > Inside HTTP.Server.Port, it kinda does, I think? I didn't run into any > > actual problems. > > > > Inside HTTP.Server.SSLPort though, it doesn't, because omitting (or > > zeroing) the port number results in the default of 443 being used. So > > at best, you would have to pick a port and hope that it's not in use, > > listen on that, and then replace it with listen_fd(). I didn't try > > that but it seems rather inelegant. > > Hmm... With your fixes, wouldn't it it work to just give an > appropriate SSL.Port object to HTTP.Server.Port()? You would > miss out on the generation of the default certificate, but > otherwise it ought to behave the same. > Huh. That would be rather nice, if it works. Currently, the two codebases look pretty different. It would be rather neat if SSLPort could simply inherit Port and provide a default parameter. ChrisA

1 0

Re: Running an HTTP(S) server using an externally-provided file descriptor
by Chris Angelico 02 May '24

02 May '24

On Thu, 2 May 2024 at 17:58, Henrik Grubbström (Lysator) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > Hi Chris. > > > See branch rosuav/pre-listening-ports for a first attempt at this. > > > > With Stdio.Port(), it's possible to receive an open socket as a file > > descriptor and start accepting connections on it. (Notably, this > > allows systemd to provide a socket as FD 3, allowing unprivileged > > programs to listen on privileged ports.) Currently, this isn't > > possible with Protocols.HTTP.Server.Port or its friends. > > This reminds me of the original lpc4 Spider webserver installation > at Lysator. > > > For unencrypted connections, this isn't too hard. Instead of having it > > construct a Port, it can receive one already listening. For SSLPort > > though, it inherits SSL.Port, so this doesn't work. What would be a > > good API for making this possible? > > Does SSL.Port()->listen_fd() not work? Inside HTTP.Server.Port, it kinda does, I think? I didn't run into any actual problems. Inside HTTP.Server.SSLPort though, it doesn't, because omitting (or zeroing) the port number results in the default of 443 being used. So at best, you would have to pick a port and hope that it's not in use, listen on that, and then replace it with listen_fd(). I didn't try that but it seems rather inelegant. On Thu, 2 May 2024 at 18:21, Marcus Comstedt (ACROSS) (Hail Ilpalazzo!) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > If I understand correctly, the problem is that while you can construct > an Stdio.Port or an SSL.Port from a file descriptor, you can't do so > with a Protocols.HTTP.Server.SSLPort, because the create function of > this class (taking 1-6 arguments) unconditionally calls > Port::socket::bind(). > > In which case I guess the solution would be to add a variant create() > taking either zero arguments or an SSL.Context (to mimick the one of > SSL.Port) while leaving the current one as it is for compatibility? This may be the solution to both problems. Let me see if I understand you correctly. The variant create would take just the Context, and then before the SSLPort becomes useful, you call listen_fd() and then set the callback? ChrisA

3 2

Running an HTTP(S) server using an externally-provided file descriptor
by Chris Angelico 02 May '24

02 May '24

See branch rosuav/pre-listening-ports for a first attempt at this. With Stdio.Port(), it's possible to receive an open socket as a file descriptor and start accepting connections on it. (Notably, this allows systemd to provide a socket as FD 3, allowing unprivileged programs to listen on privileged ports.) Currently, this isn't possible with Protocols.HTTP.Server.Port or its friends. For unencrypted connections, this isn't too hard. Instead of having it construct a Port, it can receive one already listening. For SSLPort though, it inherits SSL.Port, so this doesn't work. What would be a good API for making this possible? (Protocols.WebSocket.Port and Protocols.WebSocket.SSLPort inherit from their non-websocket equivalents, so they broadly "come along for the ride".) Incidentally, this actually means it is easier to use opportunistic TLS than mandated, since a regular Port can, with a handful of lines of code, be enhanced to call on SSL.File directly. But API-wise it would be a bit quirky if Port supports this feature and SSLPort doesn't. Recommendations? ChrisA

3 2

Re: File descriptor usage in Protocols.HTTP.Server.SSLPort
by Chris Angelico 08 Apr '24

08 Apr '24

On Fri, 5 Apr 2024 at 23:59, Henrik Grubbström (Lysator) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > Is there a better way to handle this than simply forcing garbage > > collection every request? > > The best way is to break the cycle before dropping the object on the > floor (or to not introduce it to begin with) or destruct the > problematic objects explicitly. There's no cycles in my code, and this phenomenon doesn't happen with a non-SSL connection, so I'm guessing it's something to do with encryption. > To identify the cycles you may want to log the cycles detected by the > gc; take a look at base_server/roxen.pike:reinstall_gc_callbacks(). Thanks, will investigate further. ChrisA

1 0

File descriptor usage in Protocols.HTTP.Server.SSLPort
by Chris Angelico 05 Apr '24

05 Apr '24

I've run into a strangeness with my TLS-based web server. It seems that, for every incoming request, three file descriptors are used (they all seem to be sockets), and they aren't immediately cleaned up. With keep-alive disabled, they CAN be disposed of, but it takes an explicit gc() call. With keep-alive active, they aren't garbage, so they stick around even with a gc() call. (Which is probably correct, but there might need to be a limit on how many get retained.) constant listen_addr = "::", listen_port = 9876; void http_handler(Protocols.HTTP.Server.Request req) { int before = sizeof(get_dir("/proc/self/fd")); int garbo = gc(); //Disabling this check results in FDs accumulating. int after = sizeof(get_dir("/proc/self/fd")); werror("Garbage %O, closed %d files, now %d open\n", garbo, before - after, after); //The "Connection: close" header is vital to the sockets becoming garbage. //Without it, they are retained pending a followup request, which doesn't change the //fundamental issue but does mean that a call to gc() doesn't clean them up. req->response_and_finish((["data": "OK", "extra_heads": (["Connection": "close"])])); //req->response_and_finish((["data": "OK"])); } int main() { //If you don't have a cert, the first request is slower b/c generating self-signed. //This ONLY happens with SSL connections. There are *three* file descriptors wasted //for every request. string cert = Stdio.read_file("certificate_local.pem"); string key = Stdio.read_file("privkey_local.pem"); array certs = cert && Standards.PEM.Messages(cert)->get_certificates(); string pk = key && Standards.PEM.simple_decode(key); Protocols.HTTP.Server.SSLPort(http_handler, listen_port, listen_addr, pk, certs); return -1; } What are the three FDs used for? Presumably one of them is the actual incoming socket, but the other two are less obvious. This phenomenon does not seem to happen with non-SSL ports, possibly relating to the simpler shutdown sequence for unencrypted sockets. This isn't usually a major problem, as the GC does get run eventually, but under certain types of workloads, it's easy to overload it and run out of FDs. This can be triggered by removing the gc() call from each request, and then any of these will spin up until the process FD limit is hit: // Pike: while (1) Protocols.HTTP.get_url_data("https://YOUR_SERVER.EXAMPLE:9876/"); # Python: import requests while 1: requests.get("https://YOUR_SERVER.EXAMPLE:9876/") : Bash: while wget -qO/dev/null https://YOUR_SERVER.EXAMPLE:9876/; do true; done // JavaScript in a web browser, if not blocked: while (1) await(fetch("https://YOUR_SERVER.EXAMPLE:9876/")); So I think it's probably not purely an issue with a client library misbehaving. Is there a better way to handle this than simply forcing garbage collection every request? ChrisA

2 1

Re: Pure Pike annotations?
by Chris Angelico 15 Feb '24

15 Feb '24

On Thu, 15 Feb 2024 at 23:39, Henrik Grubbström (Lysator) @ Pike (-) developers forum <10353(a)lyskom.lysator.liu.se> wrote: > > I'm trying to make good use of annotations, but all the existing ones > > are in cmod files, and I'm trying to work out how best to write them > > in Pike. My goal is to have an annotation that can be parameterized, > > something like this: > > > > class Anno(string label) { > > inherit Pike.Annotation; > > @constant; > > } > > > > @Anno("foobar"): > > void some_func() { } > > The above is supposed to work. > > > But while this does result in the annotation itself showing up, the > > label on that annotation doesn't. > > Try having the Anno class in a separate file. Ah yes, now it works. Thanks. Didn't think to try that. As long as Anno itself is added as a constant, the parameterized versions work correctly. > > What's the intended way to create parameterized annotations? > > This seems to be a compiler bug. Thanks. Not a major issue since there's a workaround (and I'll usually have them in separate files anyway). ChrisA

2 2

Pure Pike annotations?
by Chris Angelico 15 Feb '24

15 Feb '24

(Side note: This mailing list seems very dead lately. Is the LysKOM gateway down?) I'm trying to make good use of annotations, but all the existing ones are in cmod files, and I'm trying to work out how best to write them in Pike. My goal is to have an annotation that can be parameterized, something like this: class Anno(string label) { inherit Pike.Annotation; @constant; } @Anno("foobar"): void some_func() { } But while this does result in the annotation itself showing up, the label on that annotation doesn't. What's the intended way to create parameterized annotations? ChrisA

2 1

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Pike-devel