Testing on the debian build machines revealed a bug in the new ecc code. Curiously enough, causing failures only on 32-bit sparc (see https://buildd.debian.org/status/fetch.php?pkg=nettle&arch=sparc&ver...), but the bug is not really platform-specific.
It turned out that ecc_j_to_a called GMP:s mpn_mul_n (via ecc_modp_mul) with overlapping input and output arguments, which is not supported.
The following patch seems to solve the problem:
diff --git a/ecc-j-to-a.c b/ecc-j-to-a.c index df8b876..26c1a03 100644 --- a/ecc-j-to-a.c +++ b/ecc-j-to-a.c @@ -46,6 +46,7 @@ ecc_j_to_a (const struct ecc_curve *ecc, #define up (scratch + ecc->size) #define iz2p (scratch + ecc->size) #define iz3p (scratch + 2*ecc->size) +#define izBp (scratch + 3*ecc->size) #define tp scratch
mp_limb_t cy; @@ -72,11 +73,11 @@ ecc_j_to_a (const struct ecc_curve *ecc, if (flags & 1) { /* Divide this common factor by B */ - mpn_copyi (iz3p, izp, ecc->size); - mpn_zero (iz3p + ecc->size, ecc->size); - ecc->redc (ecc, iz3p); - - ecc_modp_mul (ecc, iz2p, izp, iz3p); + mpn_copyi (izBp, izp, ecc->size); + mpn_zero (izBp + ecc->size, ecc->size); + ecc->redc (ecc, izBp); + + ecc_modp_mul (ecc, iz2p, izp, izBp); } else ecc_modp_sqr (ecc, iz2p, izp);
Thanks to Magnus Holmgren for tracking this down.
Note that this bug causes overwrites *within* a properly allocated limb array, which gives incorrect results, but it does not overwrite any pointers or code.
I should plan for a bugfix release reasonably soon.
Testing with a GMP library compiled with --enable-assert would have revealed the bug earlier.
Regards, /Niels