On Wed, Dec 11, 2013 at 2:47 PM, Niels Möller nisse@lysator.liu.se wrote:
Ok didn't notice that. However, the bignum functions like pkcs1_decrypt, pkcs1_encrypt, pkcs1_rsa_digest_encode, nettle_mpz_random_size seem to have no such limits.
They work under the assumption that key_size is less than 10000 bits (see NETTLE_MAX_BIGNUM_BITS, 10000, and NETTLE_MAX_BIGNUM_SIZE, 1250). I agree this is a bit more questionable.
As far as I understand that size assumption is only enforced on systems without alloca using an assert. In systems with alloca there is no such check. In both cases, it seems to be easy to abuse them for a denial of service.
I think it would be better for these functions to fail rather than abort() if parameters are out of supported range.
regards, Nikos