Taken from https://github.com/floodyberry/chacha-opt (released by author as public-domain-or-MIT, so I guess ok to borrow).
On x86/sse2 and x86_64: 80 to 100% faster.
Passes regression test on linux/debian/stretch x86 and x86_64, benchmarks ran with patched nettle-3.4.1 (due to abi break in 3.5). *Not* tested on win{32,64} (important: win64 ABI difference).
chacha-opt also contains x86{,_64}-{ssse3,avx{,2},xop} optimized code, but I don't have hardware to test (and there are difference in structure/argument layout that need to be corrected and tested).
WIP, will add armv6 and arm/neon a bit later.
P.S. Then I will probably take a look at poly1305 and likely try to borrow license-compatible arm asm somewhere (current nettle code is painfully slow); gcrypt is somewhat faster than nettle and LGPLv2.1+; cryptograms has definitely fastest crypto, but it is BSD-3-clause-or-GPLv2+; while it is, AFAIK, compatible with LGPL, but not sure if that's acceptable for nettle inclusion.
P.S. previously posted arm neon gcm patch breaks x86_64 compilation, will post trivial fix later.
On 12.03.2019 15:02, Yuriy M. Kaminskiy wrote:
Then I will probably take a look at poly1305
... and it looks problematic; porting poly1305/armv6 is possible, but there won't be much improvement over generic C code:
$ poly1305-opt/bin/poly1305-util bench 8192 byte(s): neon, 15114.40 ns per call, 1.8 ns/byte armv6, 31944.33 ns per call, 3.9 ns/byte generic/32, 39088.50 ns per call, 4.8 ns/byte
neon (and other simd implementations) is much faster, but requires larger state; current nettle's struct poly1305_ctx is 56 bytes, optimized versions requires up to 192 bytes.
And it is embedded in struct chacha_poly1305_ctx and poly1305_aes_ctx, which looks like part of public (and used) low-level ABI.
(nettle-meta.h interface would be safe wrt struct size changes, but so far everything I've looked at - including gnutls - was not using it :-()
On Thu, 2019-03-14 at 00:25 +0300, Yuriy M. Kaminskiy wrote:
On 12.03.2019 15:02, Yuriy M. Kaminskiy wrote:
Then I will probably take a look at poly1305
... and it looks problematic; porting poly1305/armv6 is possible, but there won't be much improvement over generic C code:
$ poly1305-opt/bin/poly1305-util bench 8192 byte(s): neon, 15114.40 ns per call, 1.8 ns/byte armv6, 31944.33 ns per call, 3.9 ns/byte generic/32, 39088.50 ns per call, 4.8 ns/byte
neon (and other simd implementations) is much faster, but requires larger state; current nettle's struct poly1305_ctx is 56 bytes, optimized versions requires up to 192 bytes.
And it is embedded in struct chacha_poly1305_ctx and poly1305_aes_ctx, which looks like part of public (and used) low-level ABI.
(nettle-meta.h interface would be safe wrt struct size changes, but so far everything I've looked at - including gnutls - was not using it :-()
FWIW, I wouldn't feel blocked by an ABI break in Nettle. Simo.
On 14.03.2019 00:31, Simo Sorce wrote:
On Thu, 2019-03-14 at 00:25 +0300, Yuriy M. Kaminskiy wrote:
On 12.03.2019 15:02, Yuriy M. Kaminskiy wrote: And it is embedded in struct chacha_poly1305_ctx and poly1305_aes_ctx, which looks like part of public (and used) low-level ABI.
(nettle-meta.h interface would be safe wrt struct size changes, but so far everything I've looked at - including gnutls - was not using it :-()
FWIW, I wouldn't feel blocked by an ABI break in Nettle.
Breaking ABI in the library that is used in another libraries is always problematic.
Scenario: $app links to libgnutls.so.1 and libnettle.so.1 (and libgnutls.so.1 linked against libnettle.so.1; then libnettle.so.2 installed and libgnutls.so.1 rebuilt against new nettle; what will happen with $app?
(Especially since nettle does not use versioned symbols)
So, you either bump libgnutls soname too (and you must rebuild all apps to take advantage of it) [also it triggers same problem with libraries that uses libgnutls], or you add Conflict/Breaks in libnettle2 (and you must rebuild all libraries and apps to be able to even install libnettle2).
(And both renders new libnettle unusable for stable-backports.)
When you are forced to break ABI, it is good point to think: can it be avoided, and how can this be prevented in the future?
(poly1305 is not only algo that may require altering context structure for optimized implementation [e.g. bitsliced or vectorized aes]).
E.g. openssl made all structures opaque, and I believe it is correct long-term solution.
(Well, I've thought about a way, although not very nice: keep old version internally, add separate {chacha,aes}_poly1305_encrypt_v2, #define $foo $foo_v2 in headers; you'll need to rebuild all directly dependent libraries and apps to take advantage of new implementation, but not necessarily whole system; also it is not 146% safe [lib$a built against old version, lib$b built against new version, lib$a allocates struct chacha20_poly13_ctx and passes pointer to lib$b; libb calls $foo_v2, BOOM], but I doubt anyone uses libnettle this way in practice).
On Sun, 2019-03-17 at 12:56 +0300, Yuriy M. Kaminskiy wrote:
On 14.03.2019 00:31, Simo Sorce wrote:
On Thu, 2019-03-14 at 00:25 +0300, Yuriy M. Kaminskiy wrote:
On 12.03.2019 15:02, Yuriy M. Kaminskiy wrote: And it is embedded in struct chacha_poly1305_ctx and poly1305_aes_ctx, which looks like part of public (and used) low-level ABI.
(nettle-meta.h interface would be safe wrt struct size changes, but so far everything I've looked at - including gnutls - was not using it :-()
FWIW, I wouldn't feel blocked by an ABI break in Nettle.
Breaking ABI in the library that is used in another libraries is always problematic.
Scenario: $app links to libgnutls.so.1 and libnettle.so.1 (and libgnutls.so.1 linked against libnettle.so.1; then libnettle.so.2 installed and libgnutls.so.1 rebuilt against new nettle; what will happen with $app?
(Especially since nettle does not use versioned symbols)
Hi, Actually it does (see libnettle.map.in and libhogweed.map.in), and they were added (recently) mainly for the scenario you describe above. That is, the versioned symbols name contain to soname and thus any previous or newer version of the library don't conflict.
(And both renders new libnettle unusable for stable-backports.)
When you are forced to break ABI, it is good point to think: can it be avoided, and how can this be prevented in the future?
The nature of nettle (very low-level with structures exposed) brings often the need for abi breakage. The approach gnutls follows is to keep the gnutls abi stable and expose crypto algorithms in a more high level and stable ABI for applications to use. On certain cases it also overrides nettle code with Andy Polyakov's assembly implementation (e.g., aes-gcm on aarch64 and x86-64) for performance.
Having all crypto in nettle and optimized would be the ideal scenario for gnutls.
regards, Nikos
On Sun, 2019-03-17 at 12:56 +0300, Yuriy M. Kaminskiy wrote:
On 14.03.2019 00:31, Simo Sorce wrote:
On Thu, 2019-03-14 at 00:25 +0300, Yuriy M. Kaminskiy wrote:
On 12.03.2019 15:02, Yuriy M. Kaminskiy wrote: And it is embedded in struct chacha_poly1305_ctx and poly1305_aes_ctx, which looks like part of public (and used) low-level ABI.
(nettle-meta.h interface would be safe wrt struct size changes, but so far everything I've looked at - including gnutls - was not using it :-()
FWIW, I wouldn't feel blocked by an ABI break in Nettle.
Breaking ABI in the library that is used in another libraries is always problematic.
Scenario: $app links to libgnutls.so.1 and libnettle.so.1 (and libgnutls.so.1 linked against libnettle.so.1; then libnettle.so.2 installed and libgnutls.so.1 rebuilt against new nettle; what will happen with $app?
(Especially since nettle does not use versioned symbols)
So, you either bump libgnutls soname too (and you must rebuild all apps to take advantage of it) [also it triggers same problem with libraries that uses libgnutls], or you add Conflict/Breaks in libnettle2 (and you must rebuild all libraries and apps to be able to even install libnettle2).
No, if app is built against nettle all you need to do is bump nettle's so name and rebuild gnutls, the app will have to be rebuilt against the new nettle only as gnutls completely masks nettle behind it's apis and offers a stable ABI even when nettle ABI changes.
(And both renders new libnettle unusable for stable-backports.)
Yes, backports may be an issue, but nettle broke the ABI previously, and it offers no ABI guarntee.
When you are forced to break ABI, it is good point to think: can it be avoided, and how can this be prevented in the future?
This is not possible with Nettle as it uses explicit structures exported to the caller, to have a stable ABI nettle would need to change how it deals with context structures by allocating them on the heap and making them opaque. I do not see this happening.
(poly1305 is not only algo that may require altering context structure for optimized implementation [e.g. bitsliced or vectorized aes]).
E.g. openssl made all structures opaque, and I believe it is correct long-term solution.
Openssl is a higher level library. We can discuss a wrapper library around nettle that offers a stable API/ABI and could be used by GnuTLS perhaps.
(Well, I've thought about a way, although not very nice: keep old version internally, add separate {chacha,aes}_poly1305_encrypt_v2, #define $foo $foo_v2 in headers; you'll need to rebuild all directly dependent libraries and apps to take advantage of new implementation, but not necessarily whole system; also it is not 146% safe [lib$a built against old version, lib$b built against new version, lib$a allocates struct chacha20_poly13_ctx and passes pointer to lib$b; libb calls $foo_v2, BOOM], but I doubt anyone uses libnettle this way in practice).
Yes, as you described this method is broken, you would have to have explicit different APIs or introduce symbol versioning.
Simo.
On 17.03.2019 17:22, Simo Sorce wrote:
On Sun, 2019-03-17 at 12:56 +0300, Yuriy M. Kaminskiy wrote:
On 14.03.2019 00:31, Simo Sorce wrote:
On Thu, 2019-03-14 at 00:25 +0300, Yuriy M. Kaminskiy wrote:
On 12.03.2019 15:02, Yuriy M. Kaminskiy wrote: And it is embedded in struct chacha_poly1305_ctx and poly1305_aes_ctx, which looks like part of public (and used) low-level ABI.
(nettle-meta.h interface would be safe wrt struct size changes, but so far everything I've looked at - including gnutls - was not using it :-()
FWIW, I wouldn't feel blocked by an ABI break in Nettle.
Breaking ABI in the library that is used in another libraries is always problematic.
Scenario: $app links to libgnutls.so.1 and libnettle.so.1 (and libgnutls.so.1 linked against libnettle.so.1; then libnettle.so.2 installed and libgnutls.so.1 rebuilt against new nettle; what will happen with $app?
(Especially since nettle does not use versioned symbols)
(I was wrong here, nettle uses versioned symbols, so nothing terminally bad happens in this scenario: both libraries loaded, but $app uses @VER1 symbols, while libgnutls @VER2 symbols).
So, you either bump libgnutls soname too (and you must rebuild all apps to take advantage of it) [also it triggers same problem with libraries that uses libgnutls], or you add Conflict/Breaks in libnettle2 (and you must rebuild all libraries and apps to be able to even install libnettle2).
No, if app is built against nettle all you need to do is bump nettle's so name and rebuild gnutls, the app will have to be rebuilt against the new nettle only as gnutls completely masks nettle behind it's apis and offers a stable ABI even when nettle ABI changes.
(And both renders new libnettle unusable for stable-backports.)
Yes, backports may be an issue, but nettle broke the ABI previously, and it offers no ABI guarntee.
When you are forced to break ABI, it is good point to think: can it be avoided, and how can this be prevented in the future?
This is not possible with Nettle as it uses explicit structures exported to the caller, to have a stable ABI nettle would need to change how it deals with context structures by allocating them on the heap and making them opaque. I do not see this happening.
Actually, nettle provides (almost) everything needed already (except for umac): in nettle-meta.h.
User can obtain correct context structure size, allocate it with malloc or alloca, and use it safely with matching set of functions.
It's just that nettle *also* provides another, fragile, interface, and don't encourage or force library users to use the safe one.
(What's annoying, "fragile" interface is not even much more efficient; it is just a tiny bit more easy to use - `struct chacha_poly1305 ctx;` vs `void *pctx = alloca(nettle_chacha_poly1305.context_size);`).
(poly1305 is not only algo that may require altering context structure for optimized implementation [e.g. bitsliced or vectorized aes]).
E.g. openssl made all structures opaque, and I believe it is correct long-term solution.
Openssl is a higher level library. We can discuss a wrapper library around nettle that offers a stable API/ABI and could be used by GnuTLS perhaps.
Openssl provides both higher-level (libssl) and low-level (libcrypto) primitives. It is even easier for low-level primitives (you don't need accessors for e.g. internals of md5 context struct; it is much more complex for tls session struct).
(Well, I've thought about a way, although not very nice: keep old version internally, add separate {chacha,aes}_poly1305_encrypt_v2, #define $foo $foo_v2 in headers; you'll need to rebuild all directly dependent libraries and apps to take advantage of new implementation, but not necessarily whole system; also it is not 146% safe [lib$a built against old version, lib$b built against new version, lib$a allocates struct chacha20_poly13_ctx and passes pointer to lib$b; libb calls $foo_v2, BOOM], but I doubt anyone uses libnettle this way in practice).
Yes, as you described this method is broken, you would have to have explicit different APIs or introduce symbol versioning.
Symbol versioning won't help in this breakage scenario. At all.
lib$a allocate structure for use foo@VER1, passes it to lib$b, lib$b calls foo@VER2, BOOM.
Only way around this issue - use nettle-meta.h interface and pass matching struct nettle_{cipher,aead,...}. Then lib$b can call matching function set, regardless of which version of libnettle it was linked to.
nettle-bugs@lists.lysator.liu.se
-
Nikos Mavrogiannopoulos -
Simo Sorce -
Yuriy M. Kaminskiy