Daiki Ueno ueno@gnu.org writes:
Thank you; I have addressed those issues. As for the merging, I think it is ready now.
Thanks, merged.
Thanks to the doc update, I now noticed the possibility of failure from the encryption functions. Failure is propagated from _oaep_encode_mgf1, which does
assert (key_size >= 2 * hash->digest_size - 2);
if (message_length > key_size - 2 * hash->digest_size - 2) return 0;
Why is the first an assert (it could be triggered by using an unusually small RSA key with a large hash function, say rsa_oaep_sha512_encrypt with an old 512-bit RSA key, which from the docs isn't an obviously invalid usage), and the other a return value to indicate failure?
One alternative could be to instead check
if (message_length > key_size || message_length + 2 + 2*hash->digest_size > key_size) return 0;
(with two tests, to not trigger overflow in case message_length is close to the maximum size_t value; maybe that is more defensive than necessary since message_length large enough to trigger overflow can't be the size of properly allocated memory area).
The opposite alternative would to be to have a documented way for the application to get the maximum message size, and have an assertion failure for both cases. That would have the advantage that no return value is needed, simplifying the api (at least very locally).
Another doc detail: The docs for the decrypt functions don't say explicitly that *length is both an input and output argument. The text for the older function rsa_decrypt could be reused (or possibly improved).
Regards, /Niels