Hi Chris,
the SSL module supports passing client certificates, and the Sql.pgsql module uses SSL.File, however it doesn't directly provide any hooks for configuring the client certs. I think it would be easy to add, basically you just need a way to pass a properly configured SSL.Context object to the SSL.File object in (I think the relevant place is in Sql.pgsql_util.pmod). Once that's done, the handshake process should see the client certs and then pass them when requested by the server.
As a side note, I think anything that offers TLS support should provide a means for passing a Context, as that's where all of the configuration is done. I don't think that is currently the case, and might be a worthwhile side project.
I'm a bit swamped here, otherwise I'd offer some tested code, but I think it be fairly straightforward to add a constructor variant to Sql.pgsql and the associated wiring below it.
If you've got specific questions, I'm happy to try to answer them (I wrote the initial client cert support many, many years ago).
Bill
On 2023-12-03 19:51, Chris Angelico wrote:
Is it possible to use client SSL certificates for authentication with the Sql.pgsql or (the deprecated) Sql.postgres driver? I've been setting up a multihomed Pike program and would ideally like to be able to have all nodes connect to the same database, using their SSL certificates as proof of identity. In theory, this should work, but I've had no success. Anyone done this and if so, how?
The connection works in the command line psql interface, but not in Pike.
ChrisA
On Wed, 6 Dec 2023 at 04:28, william@welliver.org wrote:
Hi Chris,
the SSL module supports passing client certificates, and the Sql.pgsql module uses SSL.File, however it doesn't directly provide any hooks for configuring the client certs. I think it would be easy to add, basically you just need a way to pass a properly configured SSL.Context object to the SSL.File object in (I think the relevant place is in Sql.pgsql_util.pmod). Once that's done, the handshake process should see the client certs and then pass them when requested by the server.
That sounds pretty doable. The options mapping would be the place for it. I propose adding a third SSL-related option:
use_ssl - use SSL if available, else unencrypted force_ssl - use SSL or fail if not supported ssl_context - if SSL is enabled by either of the above options, use this SSL.Context
Sound good? I'll try to put together an implementation, although I'm not sure there's a viable way to add tests for it.
ChrisA
On Wed, 6 Dec 2023 at 05:10, Chris Angelico rosuav@gmail.com wrote:
Sound good? I'll try to put together an implementation, although I'm not sure there's a viable way to add tests for it.
In working on the implementation, I'm running into some trouble with SSL connections in general. As of Pike 8.1, the pgsql module uses Shuffler (this wasn't the case in Pike 8.0, but for unrelated reasons I'm not able to build Pike 8.0 on here at the moment). Non-SSL connections are fine, but SSL ones run into a problem due to sendcmd() calling shuffle->add_source(this) early in connectloop(), before the shuffler is created down below. Can anyone confirm whether or not Pike 9.0 is able to establish SSL encrypted connections in this way?
ChrisA
On Wed, 6 Dec 2023 at 08:32, Chris Angelico rosuav@gmail.com wrote:
On Wed, 6 Dec 2023 at 05:10, Chris Angelico rosuav@gmail.com wrote:
Sound good? I'll try to put together an implementation, although I'm not sure there's a viable way to add tests for it.
In working on the implementation, I'm running into some trouble with SSL connections in general. As of Pike 8.1, the pgsql module uses Shuffler (this wasn't the case in Pike 8.0, but for unrelated reasons I'm not able to build Pike 8.0 on here at the moment). Non-SSL connections are fine, but SSL ones run into a problem due to sendcmd() calling shuffle->add_source(this) early in connectloop(), before the shuffler is created down below. Can anyone confirm whether or not Pike 9.0 is able to establish SSL encrypted connections in this way?
Welp. Over the past few days, I have learned:
1. I don't know much about SSL's internals. 2. I REALLY don't know much about SSL's internals. 3. It's surprisingly hard to find tools that can help you debug client certificates.
Anyhow. I've created the branch rosuav/pgsql-ssl for this. Can someone review it please? Particularly with respect to the "DEBUG HACK" commit there, where I ripped out all of the Shuffler code and just went straight to the Stdio.Buffer; there's a lot going on with the Shuffler and I'm sure there's a way better way to make this work, but I wasn't able to figure that out.
But the upshot is that I was able to connect to PostgreSQL with a client cert, and get authenticated! Here's the script I used for testing:
https://github.com/Rosuav/StilleBot/blob/master/pgssl.pike https://github.com/Rosuav/StilleBot/blob/master/sslport.pike (mini server to show certs)
To get things to work, I had to force the Context to return a certificate (done with subclassing here), and for the mini server, I had to add the root cert to the end of the chain. More things that I don't fully understand. But after 48 hours with Wireshark, I've come to the conclusion that, if it works, it works...
If anyone has time to read over this and weigh in, I would very much appreciate it!
ChrisA
Chris Angelico wrote:
- It's surprisingly hard to find tools that can help you debug client
certificates.
It's a mess. It has been a mess for a long time. The few times I tried to use something like wireshark or tcpdump to peek into an SSL connection I gave up after trying for a while. It's a lot of work to get it right, way too much work compared to quickly starting tcpdump on a stream.
Anyhow. I've created the branch rosuav/pgsql-ssl for this. Can someone review it please? Particularly with respect to the "DEBUG HACK" commit there, where I ripped out all of the Shuffler code and just went straight to the Stdio.Buffer; there's a lot going on with the Shuffler and I'm sure there's a way better way to make this work, but I wasn't able to figure that out.
I'll have a look.
On Fri, 8 Dec 2023 at 20:17, Stephen R. van den Berg srb@cuci.nl wrote:
Chris Angelico wrote:
- It's surprisingly hard to find tools that can help you debug client
certificates.
It's a mess. It has been a mess for a long time. The few times I tried to use something like wireshark or tcpdump to peek into an SSL connection I gave up after trying for a while. It's a lot of work to get it right, way too much work compared to quickly starting tcpdump on a stream.
Hmm. Maybe I should polish the two scripts I was using, and then put them into the SSL module somewhere as examples. There are quite a few subtleties (like that you won't see client certs unless you set "ctx->auth_level = SSL.Constants.AUTHLEVEL_ask" on the server side) and I would really have appreciated an example like that.
Anyhow. I've created the branch rosuav/pgsql-ssl for this. Can someone review it please? Particularly with respect to the "DEBUG HACK" commit there, where I ripped out all of the Shuffler code and just went straight to the Stdio.Buffer; there's a lot going on with the Shuffler and I'm sure there's a way better way to make this work, but I wasn't able to figure that out.
I'll have a look.
Thanks! Appreciate it.
ChrisA
Hmm. Maybe I should polish the two scripts I was using, and then put them into the SSL module somewhere as examples. There are quite a few subtleties (like that you won't see client certs unless you set "ctx->auth_level = SSL.Constants.AUTHLEVEL_ask" on the server side) and I would really have appreciated an example like that.
There's a lot of black magic in TLS, and Client certificate handling is deceptively complicated, and there isn't really a one size fits all solution. A lot of clients (web browsers particularly) won't provide a certificate, even if they have one, unless the server auth level is set to require. In a situation where you plan to use the certificate for authentication, it's best to set the level to require, otherwise users may find themselves failing to authenticate despite having configured a certificate.
Bill
On Sat, 9 Dec 2023 at 03:57, william@welliver.org wrote:
Hmm. Maybe I should polish the two scripts I was using, and then put them into the SSL module somewhere as examples. There are quite a few subtleties (like that you won't see client certs unless you set "ctx->auth_level = SSL.Constants.AUTHLEVEL_ask" on the server side) and I would really have appreciated an example like that.
There's a lot of black magic in TLS, and Client certificate handling is deceptively complicated, and there isn't really a one size fits all solution. A lot of clients (web browsers particularly) won't provide a certificate, even if they have one, unless the server auth level is set to require. In a situation where you plan to use the certificate for authentication, it's best to set the level to require, otherwise users may find themselves failing to authenticate despite having configured a certificate.
Bill
Yeah, "ask" or "require", else the client won't offer it. It makes sense, I guess, but definitely means it would be worth having at least one example somewhere for people to refer to. I'll see what I can do to add different use-cases and annotations to the script before inclusion.
One part that I'm still very iffy on is certificate authorities. For my own usage, I hard-coded the one authority that signed the certs I'm using, but for a proper example, I'll have to look into how authority checking is done for server certs and imitate that.
ChrisA
Chris Angelico wrote:
In working on the implementation, I'm running into some trouble with SSL connections in general. As of Pike 8.1, the pgsql module uses Shuffler (this wasn't the case in Pike 8.0, but for unrelated reasons I'm not able to build Pike 8.0 on here at the moment). Non-SSL connections are fine, but SSL ones run into a problem due to sendcmd() calling shuffle->add_source(this) early in connectloop(), before the shuffler is created down below. Can anyone confirm whether or not Pike 9.0 is able to establish SSL encrypted connections in this way?
You still have problems with ssl and pgsql on 9.0 ?
On Fri, 8 Dec 2023 at 19:39, Stephen R. van den Berg srb@cuci.nl wrote:
Chris Angelico wrote:
In working on the implementation, I'm running into some trouble with SSL connections in general. As of Pike 8.1, the pgsql module uses Shuffler (this wasn't the case in Pike 8.0, but for unrelated reasons I'm not able to build Pike 8.0 on here at the moment). Non-SSL connections are fine, but SSL ones run into a problem due to sendcmd() calling shuffle->add_source(this) early in connectloop(), before the shuffler is created down below. Can anyone confirm whether or not Pike 9.0 is able to establish SSL encrypted connections in this way?
You still have problems with ssl and pgsql on 9.0 ?
Yeah, still unable to get a connection to succeed as of this week's latest master. I wanted to compare against Pike 8.0, but the Shuffler transformation came in since then, so I wasn't able to get a good comparison.
My best understanding of what's going on is that the opening needs to be done in blocking mode, but sendcmd() assumes that it's already nonblocking. This might be able to be solved by parameterizing it, or possibly by not using sendcmd) in the "please use SSL" packet.
There's another small issue which is that the processloop needs to wait until the SSL handshake completes, but that's easy enough (though I think I ended up making that change in the same "debug hack" commit since I wasn't sure), just needs a write callback to start the thread rather than doing it unconditionally.
ChrisA
Chris Angelico wrote:
On Fri, 8 Dec 2023 at 19:39, Stephen R. van den Berg srb@cuci.nl wrote: Yeah, still unable to get a connection to succeed as of this week's latest master. I wanted to compare against Pike 8.0, but the Shuffler transformation came in since then, so I wasn't able to get a good comparison.
Well, I can confirm that it does not work indeed. I'll check into it this weekend.
On 2023-12-08 04:00, Chris Angelico wrote:
My best understanding of what's going on is that the opening needs to be done in blocking mode, but sendcmd() assumes that it's already nonblocking. This might be able to be solved by parameterizing it, or possibly by not using sendcmd) in the "please use SSL" packet.
There's another small issue which is that the processloop needs to wait until the SSL handshake completes, but that's easy enough (though I think I ended up making that change in the same "debug hack" commit since I wasn't sure), just needs a write callback to start the thread rather than doing it unconditionally.
I'm pretty sure that SSL.File should be able to handshake in callback mode. Shuffler however, is a complication that can break the facade that SSL.File is just another Stdio.File object... I've definitely used shuffler with SSL.File objects in webservers, but in those situations, the shuffler isn't used until after the request has been parsed. The handshake, in those cases is performed immediately on accept() from the backend. I'm not familiar with how the pgsql module is set up, so there may be some magic going on that needs to be worked around. If you're trying to attempt some sort of conditional TLS ala STARTTLS, you might find that handshaking doesn't work until the underlying file thinks there's some data to read.
Bill
On Sat, 9 Dec 2023 at 04:06, william@welliver.org wrote:
I'm not familiar with how the pgsql module is set up, so there may be some magic going on that needs to be worked around. If you're trying to attempt some sort of conditional TLS ala STARTTLS, you might find that handshaking doesn't work until the underlying file thinks there's some data to read.
Yes, it is. After establishing a socket connection, the Postgres client either sends a version request (to initiate an unencrypted connection), or a special "please encrypt" packet, to which the server will either respond yea or nay. After that, a regular TLS handshake begins (starting with the ClientHello).
ChrisA
On Sat, 9 Dec 2023 at 04:06, william@welliver.org wrote:
I'm pretty sure that SSL.File should be able to handshake in callback mode. Shuffler however, is a complication that can break the facade that SSL.File is just another Stdio.File object... I've definitely used shuffler with SSL.File objects in webservers, but in those situations, the shuffler isn't used until after the request has been parsed. The handshake, in those cases is performed immediately on accept() from the backend. I'm not familiar with how the pgsql module is set up, so there may be some magic going on that needs to be worked around. If you're trying to attempt some sort of conditional TLS ala STARTTLS, you might find that handshaking doesn't work until the underlying file thinks there's some data to read.
Bill
Thanks for this info. I've tried to dig into exactly what's going on, and would love some example code for how you use Shuffler with SSL.File. For the moment (see branch rosuav/pgsql-ssl - just updated and rebased it), the Shuffler is disabled when using SSL, but still active for all other connections.
The problem that I'm seeing with Shuffler and PGSQL at the moment is that, after the SSL handshake is completed, further data is sent in the clear (directly to the FD), rather than passing through the encryption layer as would normally happen. The authentication packet (username, database, etc) is misinterpreted by the other end as a malformed TLS packet, resulting in an encrypted alert and a TCP RST.
From what I'm seeing in the Shuffler code, it does seem to support the idea of not writing directly to the FD, but I'm a bit unclear on how that's to be triggered.
For the moment, the simple hack of disabling Shuffler with SSL does work, but I'd love to properly understand what's going on, and to reenable the Shuffler.
ChrisA
Chris Angelico wrote:
use_ssl - use SSL if available, else unencrypted force_ssl - use SSL or fail if not supported ssl_context - if SSL is enabled by either of the above options, use this SSL.Context
Sounds good to me.
pike-devel@lists.lysator.liu.se
-
Chris Angelico -
Stephen R. van den Berg -
william＠welliver.org