Daiki Ueno ueno@gnu.org writes:
nisse@lysator.liu.se (Niels Möller) writes:
if (mpz_sizeinbase(m, 2) > bits) goto cleanup;
(one might also move initial size checks before the allocations).
I think the above check is too rigid, since it is based on bit-length, it wouldn't tolerate some cases such as m is 1016 bits and bits is 1015, where both can be represented in 127 octets.
I see, a bit odd to tolerate that case. Can it ever happen for a valid signature created according to spec? Section 8.1.1 on signing says
1. EMSA-PSS encoding: Apply the EMSA-PSS encoding operation (Section 9.1.1) to the message M to produce an encoded message EM of length \ceil ((modBits - 1)/8) octets such that the bit length of the integer OS2IP (EM) (see Section 4.2) is at most modBits - 1, where modBits is the length in bits of the RSA modulus n
If this EM is the same EM recovered when verifying the signature, then it must still correspond to an integer of size at most modBits - 1. And if it happens that verification sees an EM in the range
2^{modBits - 1} <= O2ISP(EM) < n,
can the signature possibly be valid? I imagine it will be rejected by EMSA-PSS-VERIFY? It's not trivial to follow the notation, but I guess sec. 9.1.2, steps
5. Let maskedDB be the leftmost emLen - hLen - 1 octets of EM, and let H be the next hLen octets.
6. If the leftmost 8emLen - emBits bits of the leftmost octet in maskedDB are not all equal to zero, output "inconsistent" and stop.
implies a strict check of the bit size of m, and that step (6) would be redundant with a strict check of the bit size earlier.
Nevertheless, I see that there's also some value in having the implementation closely following the way things are described in the spec, with no cleverness.
Regards, /Niels