On Dec 21, 2016, at 1:01 PM, Niels Möller nisse@lysator.liu.se wrote:
Assuming there aren’t Python bindings yet, I would be tempted to try and craft my own (at least for UMAC for now). However, I’d like to be able to do it using the Python ‘ctypes’ library, and one difficulty I can see with doing that is knowing the amount of memory I should allocate for some of the structures used by this code. Specifically, for UMAC, I’d need to know the size things like “struct umac32_ctx”, “struct umac64_ctx”, “struct umac96_ctx”, and “struct umac128_ctx”. These sizes are trivial to get from C code using sizeof(), but I don’t believe there’s any good way to get them from Python using ctypes unless there’s an API call that can be made to a C function in the library which returns this information.
These sizes are arhitecture dependent. And may change between nettle release (with an ABI break, new soname, etc). It's technically possible to add functions you suggest, but it seems both ugly and unusual to me. I hope you can solve the problem in another way.
[Ron] Thanks very much for getting back to me, Niels!
Rather than adding a function to return the structure size here, what do you think about adding functions such as umac32_new() which would do the allocation of the structure and return a pointer to it? This would address the issue I’m seeing with ctypes in Python and remain completely compatible with the existing API. I understand the distaste for having libraries like this allocate memory, but the only options I know of to provide cross-language support like this is to either provide an allocator function that knows how much space to allocate or to provide a function which returns the amount of space needed to pass to an external allocator.
To make sure there weren’t any other issues in doing this, I did a proof of concept where I simply did a fixed-size allocation of 4 KB for the contexts (since the largest context is currently only about 2800 bytes on the system I’m running on) and I was able to make everything after that work fine. Here’s a snippet of some Python 3 test code (without a pretty wrapper yet):
import binascii import ctypes
UMAC_CTX_SIZE = 4096
UMAC32_DIGEST_SIZE = 4
_nettle_lib = ctypes.cdll.LoadLibrary('/opt/local/lib/libnettle.dylib')
_umac32_set_key = _nettle_lib.nettle_umac32_set_key _umac32_set_nonce = _nettle_lib.nettle_umac32_set_nonce _umac32_update = _nettle_lib.nettle_umac32_update _umac32_digest = _nettle_lib.nettle_umac32_digest
ctx = ctypes.create_string_buffer(UMAC_CTX_SIZE)
key = b'abcdefghijklmnop' nonce = b'bcdefghi' digest = ctypes.create_string_buffer(UMAC32_DIGEST_SIZE)
for msg in (b'', 3*b'a', 1024*b'a', 32768*b'a', 1024*1024*b'a', 32768*1024*b'a', b'abc', 500*b'abc'): _umac32_set_key(ctx, key) _umac32_set_nonce(ctx, ctypes.c_size_t(len(nonce)), nonce) _umac32_update(ctx, ctypes.c_size_t(len(msg)), msg) _umac32_digest(ctx, ctypes.c_size_t(len(digest)), digest) print(binascii.b2a_hex(digest.raw).decode())
I really don’t want to rely on a hard-coded size like this, though, in case future versions of the context structure grow much larger for some reason. Also, even if they don’t, rounding up like this is a waste of memory. Can you think of any other options here?
As an example of this, libsodium provides functions like crypto_stream_chacha20_keybytes() and crypto_stream_chacha20_noncebytes() which can be called to return the size of a Chacha20 key and nonce, respectively. In the C header file, these look like:
These are different; these constants are part of the algorithm specification. The defines in the header files are for convenience, but not the ultimate definition of the values.
[Ron] Fair enough. For these, I could potentially just replicate the definitions in the Python code, without worry that they would change. It was just nice in the libsodium case that I didn’t have to.