I used to think that next Nettle release should be focused on performance and add a few more features, no incompatible changes.
But since the gcm changes breaks gnutls (fix in https://gitlab.com/gnutls/gnutls/commit/176aa191380e5d63a981f08f537deffdd78f...), best way to reduce damage might be to bump the nettle soname. And if we do that, there are a couple of other things that could be done at the same time. Some changes to consider:
1. Remove the the symbols nettle_hashes and nettle_secp_384r1 and friends from the interface. They should be renamed with leading underscore. Important to be able to merge ed448 work without breaking the abi again.
2. Delete the old aes_* interface, in favor of aes128_, aes192_* and aes256_*. This could help aesni performance, where it might be nice with separately completely unrolled code for each key size.
3. Change struct nettle_aead to be more message-oriented.
4. Hide undocumented and internal symbols (the ones with leading underscore) more. Move declarations to internal uninstalled headers, possible tweak linker script to not expose at all in the shared library.
5. Revamp hmac and underlying hash functions with a separate state struct. Probably low priority, but it is a bit silly that, e.g., hmac_sha512_ctx includes three 128-byte large block buffers.
Regards, /Niels