Hello Niels,
Thank you for merging it and the suggestions.
Niels Möller nisse@lysator.liu.se writes:
Thanks to the doc update, I now noticed the possibility of failure from the encryption functions. Failure is propagated from _oaep_encode_mgf1, which does
assert (key_size >= 2 * hash->digest_size - 2);
if (message_length > key_size - 2 * hash->digest_size - 2) return 0;
Why is the first an assert (it could be triggered by using an unusually small RSA key with a large hash function, say rsa_oaep_sha512_encrypt with an old 512-bit RSA key, which from the docs isn't an obviously invalid usage), and the other a return value to indicate failure?
I think I split these conditions that way, as the first condition is more about the algorithm setup, i.e., the user should choose RSA key size and hash digest size appropriately, before using any of the rsa_oaep_*_encrypt functions; otherwise treat it as a programming error.
That said, I agree that it would be more user friendly to combine them and treat it as a regular error, as we do in pss_encode_mgf1.
One alternative could be to instead check
if (message_length > key_size || message_length + 2 + 2*hash->digest_size > key_size) return 0;
(with two tests, to not trigger overflow in case message_length is close to the maximum size_t value; maybe that is more defensive than necessary since message_length large enough to trigger overflow can't be the size of properly allocated memory area).
I made this change in the attached patch.
The opposite alternative would to be to have a documented way for the application to get the maximum message size, and have an assertion failure for both cases. That would have the advantage that no return value is needed, simplifying the api (at least very locally).
It is tempting, though for now I just documented the size restriction.
Another doc detail: The docs for the decrypt functions don't say explicitly that *length is both an input and output argument. The text for the older function rsa_decrypt could be reused (or possibly improved).
Updated the documentation.
Regards,