nisse@lysator.liu.se (Niels Möller) writes:
I'm puzzled, because I'm fairly sure I got the test vectors from serpent's submission package (I could try to double check that), which if I understand correctly ought to be serpent-1. I vaguely remember I had some difficulty understanding the organization of the test data, though.
The files defining these testvectors use an input and output format with bytes in the reverse order. At the time (2001-06-17, almost ten years ago), I thought this reversal was part of the cipher operation, rather than a peculiarity with test programs and test vectors in the serpent submission package, and incorrectly added byte reversal to the functions in serpent.c.
I've now reverted this old change, which means that from now on serpent in Nettle ought to agree with other implementations. And this naturally also breaks compatibility for anyone using the serpent implementation of all (or almost all) earlier versions of Nettle.
I've fixed the serpent-test.c to reverse inputs and outputs for the testvectors copied from the serpent submission package, and added a few additional test vectors (from http://www.cs.technion.ac.il/~biham/Reports/Serpent/) using a sane byte order.
Credits go to Eli Biham and Lars Knudsen for finally putting me on the right track.
Regards, /Niels