I'm having a second look at dealing with the length argument to ocb_digest and ccm_digest, which I left as is when deleting all other digest size arguments a while ago.

The alternative I see is to have the _set_nonce function store the intended digest size in the context, and use that at _digest time. That makes the digest functions a bit more consistent with the rest of nettle, and it also makes it harder to use inconsistent digest size to the two calls. One drawback is that the size becomes less obvious from the _digest call site. I'm trying this out in the branch https://git.lysator.liu.se/nettle/nettle/-/tree/delete-ocb-ccm-digest-size-a...

Some questions related to this:

1. For both functions, I would like to minimize the risk of large memory overwrites in the case of bugs, say the tag_length member has been clobbered by bugs elsewhere. Currently, I have an

assert (ctx->tag_length <= 16);

just prior to the call (memxor3 or ctr_crypt, respectively) that writes to digest. Which should be safe when asserts are enabled. A different option not depending on asserts would be an explicit clamping, say

unsigned length = 1 + ((15 & (ctx->tag_length - 1)));

Maybe that is overly defensive; after all, if there's random memory clobber bugs, one could just as well get random large lengths for the _encrypt or _decrypt calls.

Regardless of the level of defensive programming, the intention is that if you don't know locally at the _digest call site which tag_length was used, it should always be correct to call _digest with an output buffer of size 16, and extract the tag of appropriate size later.

2. For CCM, I also put in some more checks on the tag_length, which according to RFC 3610 is limited to 4, 6, 8, 10, 12, 14, 16 (maybe 2 would also work technically). But then I had to delete one test, labeled "From IEEE 802.15.4-2011 ... C.2.2 MAC data frame", which uses tag_length 0. Are there relevant usecases for CCM with no authentication tag (which I think boils down to CTR mode with some special way to assign initial counter)?

3. For OCB, size of both plaintext and the auxilliary authenticated data is "unlimited", according to RFC 7253. Currently, Nettle uses a size_t to count number of blocks for each input, which will wrap around and produce out-of-spec output when size exceeds 2^36 bytes (with 32-bit size_t) or 2^68 bytes (with 64-bit size_t).

It would make some sense to replace size_t with uint64_t, to get the same limit regardless of platform.

If we go to 64-bit counts, I also wonder if it makes sense to steal 4 of the high bits to store the tag_length (otherwise, due to padding, an extra struct field for the tag_length increases the ocb_ctx from 80 bytes to 96 (on a 64-bit platform).

4. The reason I was looking into this now, is that I'm thinking about adding support for blake2 (RFC 7693), which also similarly has variable digest size, and that size encoded into a parameter word at init time.

All opinions welcome.

Regards, /Niels