Hi Niels,
On Apr 8, 2026 at 5:00:44 AM, Niels Möller nisse@lysator.liu.se wrote:
I've been hacking a bit on the sntrup code in recent days. See branch https://git.lysator.liu.se/nettle/nettle/-/tree/sntrup761
I have a couple of questions, both on api details and on the algorithm.
- According to the spec, the secret key includes a copy of the public
key. Should we stick to this for nettle's api, or would it make more sense to have the decrypt function taking a separate pubkey argument?
When generating a new keypair, I think you’ll want the API to return both the private key and public key. Both of these can be encoded as byte strings. You’ll want both keys to be returned here, as typically the owner of the keypair needs to share the public key in some way with whoever wants to send a secret.
When decrypting, you don’t need the public key. The decrypt call takes ciphertext and a private key and returns the original shared secret, assuming it was encrypted using the corresponding public_key.
The public key is used in the encrypt phase. It takes a shared secret and a public key and returns ciphertext when can be decrypted using the private key.
2. Would it be useful with an api where public and private keys can be
decoded from byte strings to an internal representation, to not have to redo the decoding on each operation? Or stick to only bytestring input and output?
I would stick to byte strings here. It’d be messy to be converting back and forth between the two representations, and you will need the byte string version to be able to send it over the network.
3. For private key decoding, it may happen that the private key is
invalid. The private key includes lots of mod 3 coefficients, where each coefficient is represented as two bits with valid values 00, 01, 10. What if caller passes 11? Hitting some undefined behaviour or an assertion failure for this isn't that nice. One could return an error (which would be somewhat natural if one has a private key decode function as above), or silently replace 11 values with 00? Neither the spec or the current code includes any error handling as far as I've found.
I haven’t looked at the internals, but if a caller passes in an invalid key (public or private), I would recommend returning an error and not trying to translate the key bytes into a “valid” key. I’m not sure it is even possible for a caller with an invalid public key and no access to the private key to figure out what the public key was supposed to be.
4. I'm confused by the extra steps taken to get "implicit rejection".
Decryption may fail if fed a random (or maliciously chosen) ciphertext. It makes intuitive sense to me to hide all details of where during decryption something looked wrong. But why go to extra effort to not return any error at all? When the system is attacked, why does it matter if the attacker gets an explicit error at key exchange time, or an error (typically a MAC or AEAD decrypt failure) at first use of the resulting session key? I understand argument is likely subtle, but I'd appreciate any layman explanation for this design.
In the case of a bad key, the decrypt call probably wouldn’t be able to return any plaintext, so I think it pretty much has to return an error in this case to indicate that the plaintext buffer is not valid. We wouldn’t want the caller to try and access uninitialized memory.
To summarize, I think the API should look something like:
keypair() -> (private_key, public_key) encaps(shared_secret, public_key) -> ciphertext decaps(ciphertext, private_key) -> shared_secret
Buffers will need to be allocated to an appropriate size depending on the algorithm, and in C I would expect a mix of “in” and “out” parameters passed as arguments, probably with the return value of all the functions being an error enum indicating any error which occurred.
Here’s an example C API, modeled after liboqs:
#define sntrup761_length_public_key 1158 #define sntrup761_length_private_key 1763 #define sntrup761_length_ciphertext 1039 #define sntrup761_length_shared_secret 32
STATUS sntrup761_keypair(uint8_t *public_key, uint8_t *private_key);
STATUS sntrup761_encaps(uint8_t *ciphertext, uint8_t *shared_secret, const uint8_t *public_key);
STATUS sntrup761_decaps(uint8_t *shared_secret, const uint8_t *ciphertext, const uint8_t *private_key);
With the exception of the function names and buffer sizes, this exact same API should be usable for the various flavors of ML-KEM as well. To see an example of this in actual use, you can take a look at the PQ key exchange code from AsyncSSH at https://github.com/ronf/asyncssh/blob/develop/asyncssh/crypto/pq.py. -- Ron Frederick ronf@timeheart.net