I've been hacking a bit on the sntrup code in recent days. See branch https://git.lysator.liu.se/nettle/nettle/-/tree/sntrup761
I have a couple of questions, both on api details and on the algorithm.
1. According to the spec, the secret key includes a copy of the public key. Should we stick to this for nettle's api, or would it make more sense to have the decrypt function taking a separate pubkey argument?
2. Would it be useful with an api where public and private keys can be decoded from byte strings to an internal representation, to not have to redo the decoding on each operation? Or stick to only bytestring input and output?
3. For private key decoding, it may happen that the private key is invalid. The private key includes lots of mod 3 coefficients, where each coefficient is represented as two bits with valid values 00, 01, 10. What if caller passes 11? Hitting some undefined behaviour or an assertion failure for this isn't that nice. One could return an error (which would be somewhat natural if one has a private key decode function as above), or silently replace 11 values with 00? Neither the spec or the current code includes any error handling as far as I've found.
4. I'm confused by the extra steps taken to get "implicit rejection". Decryption may fail if fed a random (or maliciously chosen) ciphertext. It makes intuitive sense to me to hide all details of where during decryption something looked wrong. But why go to extra effort to not return any error at all? When the system is attacked, why does it matter if the attacker gets an explicit error at key exchange time, or an error (typically a MAC or AEAD decrypt failure) at first use of the resulting session key? I understand argument is likely subtle, but I'd appreciate any layman explanation for this design.
Regards, /Niels