Hi Nettle folks--
In the course of writing Crypt::Nettle, it occurs to me that i'm going to need to update the list of available ciphers and digests for each new version of nettle that comes out. This also means that newer versions of the perl bindings won't work by default against older versions of libnettle, which seems like it might make the perl bindings less useful for people who are stuck (for whatever reason) with an older version of libnettle.
The attached patch would present an interface to dynamically enumerate the available ciphers and digest algorithms (and armor methods) supported by the current version of libnettle. bindings built against this meta-interface wouldn't need to be closely tied to any particular version of nettle, and so could be both forward- and backward-compatible.
I'd be happy with any other interface that would give me the same effect, of course, and i'm not convinced that this is the best way to do it -- it just seemed the simplest in the current implementation.
It would also be nice to make a similar interface to genericize cipher modes (ECB, CBC, CTR, and GCM), if that's possible; i'm not sure what the best approach is for doing that.
Feedback welcome, of course.
--dkg
One comment is that it is more difficult to export variables portably than functions (especially Windows), so sometimes it is preferred to export a function that returns a static array than exporting the static array itself. However, I think nettle already relies on exporting variables so then this point is moot. I support the patch otherwise.
/Simon
Daniel Kahn Gillmor dkg@fifthhorseman.net writes:
This also means that newer versions of the perl bindings won't work by default against older versions of libnettle,
I don't know how the build system for the perl bindings is setup, but using something like autoconf should make it easier if you really want to support multiple nettle versions, with or without public key support.
The attached patch would present an interface to dynamically enumerate the available ciphers and digest algorithms (and armor methods) supported by the current version of libnettle.
The need for this hasn't occured to me, but I see the point. What do others think?
bindings built against this meta-interface wouldn't need to be closely tied to any particular version of nettle, and so could be both forward- and backward-compatible.
But some other parts of the nettle interfaces have not been as stable between releases, so it doesn't solve the problem completely.
I'd be happy with any other interface that would give me the same effect, of course, and i'm not convinced that this is the best way to do it -- it just seemed the simplest in the current implementation.
Right, it's about as simple as one can get, and I like that. I think one level of indirection like you do is desirable, for linking reasons.
One may want to think about fancy hardware accelerators (currently not supported at all), which would require some runtime check on what's available.
It would also be nice to make a similar interface to genericize cipher modes (ECB, CBC, CTR, and GCM), if that's possible; i'm not sure what the best approach is for doing that.
I'm not sure this is as useful. The modes have different properties and generally don't work well as substitutes for each other. And at least, the list is expected to be expanded less frequently than the list of hashes or block ciphers.
Some comments on the implementation:
+const struct nettle_armor *nettle_armors[] = {
- &nettle_base64,
- &nettle_base16,
- NULL
+};
The typing should be something like
const struct nettle_armor * const nettle_armors[]
to say that both the list itself and the pointed to structures are const.
Also, I think there should be one compilation unit and object file per list, so that it's, for example, possible to statically link with the list of all hash algorithms without also dragging in all ciphers.
Regards, /Niels
On 03/21/2011 03:04 PM, Niels Möller wrote:
I don't know how the build system for the perl bindings is setup, but using something like autoconf should make it easier if you really want to support multiple nettle versions, with or without public key support.
that's an interesting suggestion -- i'm afraid i don't know enough about integrating autoconf with perlxs to know how to go down that path without a more research, but i'll consider it for the future. It would only allow backward-compatibility, though -- not forward-compatibility. That is: if you built Crypt::Nettle against libnettle version X, and then upgraded to libnettle version Y, you'd have to also re-build Crypt::Nettle.
That's not a horrible tradeoff, i guess.
But some other parts of the nettle interfaces have not been as stable between releases, so it doesn't solve the problem completely.
yes, that's true -- i'm proposing this just for the ciphers and hashes at the moment.
One may want to think about fancy hardware accelerators (currently not supported at all), which would require some runtime check on what's available.
that's an interesting part i hadn't considered. I'm not certain how to represent such a change, but i like the idea that one could discover available/supported hardware dynamically somehow. Would supporting that use case suggest a different API?
I'm not sure this is as useful. The modes have different properties and generally don't work well as substitutes for each other. And at least, the list is expected to be expanded less frequently than the list of hashes or block ciphers.
OK. Maybe this is a good target for the autoconf approach you mentioned above.
Some comments on the implementation:
+const struct nettle_armor *nettle_armors[] = {
- &nettle_base64,
- &nettle_base16,
- NULL
+};
The typing should be something like
const struct nettle_armor * const nettle_armors[]
to say that both the list itself and the pointed to structures are const.
Yes, you are right, thanks!
Also, I think there should be one compilation unit and object file per list, so that it's, for example, possible to statically link with the list of all hash algorithms without also dragging in all ciphers.
Is there any reason to use this approach for statically-linked tools? Maybe for UI or hardware discoverability, i guess. It certainly wouldn't help in terms of software discoverability or forward/backward compatibility in a statically-linked library.
Do you want me to submit a modified patch with these changes?
--dkg
Daniel Kahn Gillmor dkg@fifthhorseman.net writes:
On 03/21/2011 03:04 PM, Niels Möller wrote:
One may want to think about fancy hardware accelerators (currently not supported at all), which would require some runtime check on what's available.
that's an interesting part i hadn't considered. I'm not certain how to represent such a change, but i like the idea that one could discover available/supported hardware dynamically somehow. Would supporting that use case suggest a different API?
To add hardware acceleration, one would need to either detect hardware and build a list automatically at loadtime, or have a function to build and return the list. It can no longer be a compile-time constant.
But I think the API you suggested is good enough for now.
It's hard to guess what really interface is needed before the hardware support exists, and I'd rather not define something overly complicated just because it might possibly be needed later.
Also, I think there should be one compilation unit and object file per list, so that it's, for example, possible to statically link with the list of all hash algorithms without also dragging in all ciphers.
Is there any reason to use this approach for statically-linked tools?
The general principle is that independent functionality should be in separate object files. I don't think this case is an exception. To be a little more concrete, say you want a tool that can compute a hash sum using any algorithm supported by nettle, and you then want that tool on some constrained system with statically linked binaries, or included in a statically linked busybox executable.
Do you want me to submit a modified patch with these changes?
That would be nice. Documentation would also be appreciated (it should probably mention why des is not on the list, but I don't think it needs to be terribly many words). Test cases are perhaps not essential for this, but maybe you can think of some relevant test?
It may also be good to review the naming. Until now, the name attribute in nettle_cipher and similar structs has been intended to be displayed for informational purposes only. With your interface, users will be invited to use it as an identifier to look up an algorithm. How do you intend to use that name in the perl bindings?
Regards, /Niels
On 03/21/2011 04:07 PM, Niels Möller wrote:
But I think the API you suggested is good enough for now.
ok.
The general principle is that independent functionality should be in separate object files. I don't think this case is an exception. To be a little more concrete, say you want a tool that can compute a hash sum using any algorithm supported by nettle, and you then want that tool on some constrained system with statically linked binaries, or included in a statically linked busybox executable.
Gotcha, this is a useful way to think about things. Thanks for your explanation.
That would be nice. Documentation would also be appreciated (it should probably mention why des is not on the list, but I don't think it needs to be terribly many words). Test cases are perhaps not essential for this, but maybe you can think of some relevant test?
i'll work on a revised patch with docs and a test if i can figure out a test as well.
It may also be good to review the naming. Until now, the name attribute in nettle_cipher and similar structs has been intended to be displayed for informational purposes only. With your interface, users will be invited to use it as an identifier to look up an algorithm. How do you intend to use that name in the perl bindings?
The perl bindings already expose an enumeration + lookup interface, but i've implemented it myself outside of nettle. I'll be happy to keep the existing perl API but drop the Crypt::Nettle implementation of it in favor of one in the main library, though.
fwiw, i have no objections to the current naming scheme. I'll probably do a case-insensitive match against the exported names.
Your earlier suggestion that the user should be able to say "aes" and then we would select an algorithm based on the supplied keysize suggests a particular simple naming convention, where the keysize is always a strict suffix to the algorithm name.
That is, a library might use a function by comparing the requested algorithm name as a strict prefix with the size of the keylength.
I suppose that selection function could be part of libnettle as well, instead of making each binding report it. something like:
const nettle_cipher* nettle_select_cipher(const char* algoname, int keylength = 0);
This would not remove the need for the other API, though, since it doesn't provide algorithm enumeration.
I can try to include such an implementation in the revised patch if y'all think it would be useful.
--dkg
Daniel Kahn Gillmor dkg@fifthhorseman.net writes:
That is, a library might use a function by comparing the requested algorithm name as a strict prefix with the size of the keylength.
If that works with all algorithms, that's good. I recall I changed or was about to change these things for the CAST cipher a while back. "arctwo_gutmann" may also be a corner case.
const nettle_cipher* nettle_select_cipher(const char* algoname, int keylength = 0);
I think this function might fit better as an example in the documentation. (And you can't do default arguments like that in C).
There are so many queries one might want to do. Beside this example, "give me all variants of aes", or "give me all block ciphers with a given block size", so I think it's a bit premature to decide which query functions to add.
Regards, /Niels
On 03/21/2011 04:32 PM, Daniel Kahn Gillmor wrote:
i'll work on a revised patch with docs and a test if i can figure out a test as well.
Attached is a patch that breaks out separate compilation units, const-ifies the arrays, and adjusts the docs and the test suite.
I did not see nettle_armor in the docs at all, so i didn't add any docs about nettle_armors[]. Maybe that whole section needs to be written?
Any further comments?
--dkg
Daniel Kahn Gillmor dkg@fifthhorseman.net writes:
Attached is a patch that breaks out separate compilation units, const-ifies the arrays, and adjusts the docs and the test suite.
Thanks. I'm going to commit this. I noticed some incorrect file names in the file headers (which I'll fix), and there's no copyright line. I guess I should simply add
Copyright (C) 2011 Daniel Kahn Gillmor
at the top of the new files.
I did not see nettle_armor in the docs at all, so i didn't add any docs about nettle_armors[]. Maybe that whole section needs to be written?
The reason these functionality is undocumented is most likely that I wasn't confident that the interface is right, it's not been used very much.
Regards, /Niels
nisse@lysator.liu.se (Niels Möller) writes:
Thanks. I'm going to commit this.
Checked in now.
I also wrote a program tools/nettle-hash to take advantage of the new interface. Comments appreciated.
Should output be more compatible with md5sum, sha1sum et al? But I think I like to also include information on the algorithm used.
Should it have a -c (check) mode as well?
Regards, /Niels
On 03/23/2011 06:12 AM, Niels Möller wrote:
Thanks. I'm going to commit this. I noticed some incorrect file names in the file headers (which I'll fix), and there's no copyright line. I guess I should simply add
Copyright (C) 2011 Daniel Kahn Gillmor
at the top of the new files.
This is fine, thanks. sorry about the incorrect file names -- i hate being caught copying and pasting! :P
I did not see nettle_armor in the docs at all, so i didn't add any docs about nettle_armors[]. Maybe that whole section needs to be written?
The reason these functionality is undocumented is most likely that I wasn't confident that the interface is right, it's not been used very much.
That makes sense to me. I looked at trying to make a Crypt::Nettle::Armor in the perl bindings to take advantage of it, and decided that (a) i wasn't exactly sure how i wanted to use it, and (b) it probably wasn't worth it because perl has good native base64 and base16 encoding/decoding anyway.
--dkg
Daniel Kahn Gillmor dkg@fifthhorseman.net writes:
Any further comments?
The following (in nettle.texinfo) looks a bit strange.
+@deftypevrx {Constant Struct * Constant} {struct nettle_hash *} nettle_hashes[]
+@deftypevrx {Constant Struct * const} {struct nettle_cipher *} nettle_ciphers[]
I'm no texinfo guru, but I think the first {...} is supposed to be just a category (Function, Macro, Variable, etc), and the second {...} the type. I thik
@deftypevrx {Constant Struct} {struct nettle_hash **} nettle_hashes
would be more correct (unless we want a new categories "Algorithm List" which I think is a bit overkill), but I haven't yet checked the generated output.
Regards, /Niels
On 03/23/2011 06:25 AM, Niels Möller wrote:
Daniel Kahn Gillmor dkg@fifthhorseman.net writes:
Any further comments?
The following (in nettle.texinfo) looks a bit strange.
+@deftypevrx {Constant Struct * Constant} {struct nettle_hash *} nettle_hashes[]
+@deftypevrx {Constant Struct * const} {struct nettle_cipher *} nettle_ciphers[]
I'm no texinfo guru, but I think the first {...} is supposed to be just a category (Function, Macro, Variable, etc), and the second {...} the type. I thik
@deftypevrx {Constant Struct} {struct nettle_hash **} nettle_hashes
would be more correct (unless we want a new categories "Algorithm List" which I think is a bit overkill), but I haven't yet checked the generated output.
Ah, ok. I'm no texinfo guru either -- i just put in what made the pdf output look reasonable. Your changes make at least as much sense to me as what i put in there.
Thanks for fixing it,
--dkg
nettle-bugs@lists.lysator.liu.se