- nettle-bugs - lists.lysator.liu.se

Arcfour status
by nisse＠lysator.liu.se 14 Feb '21

14 Feb '21

I've had a report (from Matthew Kempe) about another problem with the openssl benchmarking code. It fails on FreeBSD, because there (and possible in other environments too) openssl has been configured without RC4 (aka arcfour) support. I'm considering just deleting code to benchmark openssl arcfour; I don't plan any improvements of Nettle's arcfour performance, and I would be surprised if the openssl people do. I do intend to keep arcfour support in Nettle for the foreseeable future, to support old protocols and applications. But I'm thinking that maybe the arcfour assembly code could be deleted? Nettle currently includes arcfour assembly implementations for x86 (32-bit) and sparc (both 32-bit and 64-bit), which as far as I remember gave a modest speedup when added. But the code hasn't been tuned or benchmarked recently. And we have nothing for more relevant platforms. Regards, /Niels -- Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677. Internet email is subject to wholesale government surveillance.

2 1

GnuTLS testsuite error on ppc64 after nettle upgrade
by Andreas Metzler 10 Feb '21

10 Feb '21

Hello, Upgrading nettle from 3.6 to 3.7 triggers a GnuTLS 3.7.0 testsuite error on both ppc64 and ppc64el: (sid_ppc64el-dchroot)ametzler@plummer:~/GNUTLS/gnutls28-3.7.0/b4deb/tests$ ./min i-record-2 testing aes-cbc testing aes-cbc-sha256 testing aes-gcm testing aes-ccm testing aes-ccm-8 testing null-sha1 testing arcfour-sha1 testing arcfour-md5 testing chacha20-poly1305 testing tls13-chacha20-poly1305 server:330: client: Error: An unexpected TLS packet was received. Running the same binary dynamically (with LD_LIBRARY_PATH setting) linked against nettle 3.6 succeeds. --verbose logs are huge (5-7 MB xz-compressed), I have uploaded them to https://people.debian.org/~ametzler/tmp/ cu Andreas -- `What a good friend you are to him, Dr. Maturin. His other friends are so grateful to you.' `I sew his ears on from time to time, sure'

4 10

Bug fix release Nettle-3.7.1 ?
by nisse＠lysator.liu.se 10 Feb '21

10 Feb '21

Hi, I think the chacha bug is a severe enough regression to warrant a bugfix release pretty soon. I'll aim to get it out in a week from now. I think it should be fine to do a 3.7.1 release from the master branch, rather than cherry-picking selected bugfixes. I've looked through git history and ChangeLog since the release one and a half month ago, and I think this is a accurate summary for the NEWS file: Bug fixes: * Fix bug in chacha counter update logic. The problem affected ppc64 and ppc64el, with the new altivec assembly code enabled. Reported by Andreas Metzler, after breakage in GnuTLS tests on ppc64. * Support for big-endian ARM platforms has been restored. Fixes contributed by Michael Weiser. * Fix build problem on OpenBSD/powerpc64, reported by Jasper Lievisse Adriaanse. * Fix corner case bug in ECDSA verify, it would produce incorrect result in the unlikely case of an all-zero message hash. Reported by Guido Vranken. New features: * Support for pbkdf2_hmac_sha384 and pbkdf2_hmac_sha512, contributed by Nicolas Mora. Miscellaneous: * Poorly performing ARM Neon code for doing single-block Salsa20 and Chacha has been deleted. The code to do two or three blocks in parallel, introduced in Nettle-3.7, is unchanged. Sonames will be unchanged. libnettle.so should get an incremented minor number, for the addition of the new pbkdf2 function. I don't think libhogweed.so strictly needs an incremented minor number, but maybe it's less confusing to increment it anyway. Anything I'm missing? Any easy in-progress changes that should also get into the release? Regards, /Niels -- Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677. Internet email is subject to wholesale government surveillance.

1 0

Re: Release of Nettle-3.7?
by nisse＠lysator.liu.se 06 Feb '21

06 Feb '21

Michael Weiser <michael(a)weiser.dinsnail.net> writes: > Porting over the basic > IF_[LB]E mechanism from chacha-core-internal was easy and fixed up the > first of the three interleaved blocks right away. For the other two I am > still in the process of wrapping my head around how the interleaving > works and how it would need some adjustment for BE. The 3-way functions don't do anything fancy, just each of the three blocks represented in separate registers, and same instruction sequence as for the 1-way version, duplicated threee times and interleaved. The 2-way version (for ARM, that's salsa only) tries to be a bit more clever, with registers representing either odd or even words from both blocks. Not sure how endianness affects the code to move words around. Byte swapping should go close to the final stores, but after the addition of the initial state. Regards, /Niels -- Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677. Internet email is subject to wholesale government surveillance.

3 16

[AArch64] Optimize GHASH
by Maamoun TK 06 Feb '21

06 Feb '21

I made a merge request in the main repo that enables optimized GHASH on AArch64 architecture. The implementation is based on Niels Möller's enhanced algorithm which yields more speedup on AArch64 arch in comparison with intel algorithm. Using the Karatsuba algorithm with Intel algorithm yielded an overhead so I dropped its benchmark result. I'll attach the file of Intel algorithm implementation here since it's not include in the MR. Here is the benchmark result on AArch64: *---------------------------------------------------------------------------------------------* | C version | Intel algorithm | Niels Möller's enhanced algorithm | | 208 Mbyte/s | 2781 Mbyte/s | 3255 Mbyte/s | *---------------------------------------------------------------------------------------------* This is +17% performance boost of the enhanced algorithm over the Intel algorithm, it's not as impressive as PowerPC benchmark result but it did a great job on AArch64 considering PMULL instruction doesn't have the assistance that vpmsumd offers by multiply four polynomials then summing. I tried to avoid using the stack in this implementation so I wrote a procedure to handle leftovers by just using the registers, let me know if there's a room for improvement here. regards, Mamone C arm/v8/gcm-hash.asm ifelse(` Copyright (C) 2020 Niels Möller and Mamone Tarsha This file is part of GNU Nettle. GNU Nettle is free software: you can redistribute it and/or modify it under the terms of either: * the GNU Lesser General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. or * the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. or both in parallel, as here. GNU Nettle is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received copies of the GNU General Public License and the GNU Lesser General Public License along with this program. If not, see http://www.gnu.org/licenses/. ') C gcm_set_key() assigns H value in the middle element of the table define(`H_Idx', `128') .file "gcm-hash.asm" .text C void gcm_init_key (union gcm_block *table) C This function populates the gcm table as the following layout C ******************************************************************************* C | H1M = (H1 div x⁶⁴)||((H1 mod x⁶⁴) × (x⁶⁴+x⁶³+x⁶²+x⁵⁷)) div x⁶⁴ | C | H1L = (H1 mod x⁶⁴)||(((H1 mod x⁶⁴) × (x⁶³+x⁶²+x⁵⁷)) mod x⁶⁴) + (H1 div x⁶⁴) | C | | C | H2M = (H2 div x⁶⁴)||((H2 mod x⁶⁴) × (x⁶⁴+x⁶³+x⁶²+x⁵⁷)) div x⁶⁴ | C | H2L = (H2 mod x⁶⁴)||(((H2 mod x⁶⁴) × (x⁶³+x⁶²+x⁵⁷)) mod x⁶⁴) + (H2 div x⁶⁴) | C | | C | H3M = (H3 div x⁶⁴)||((H3 mod x⁶⁴) × (x⁶⁴+x⁶³+x⁶²+x⁵⁷)) div x⁶⁴ | C | H3L = (H3 mod x⁶⁴)||(((H3 mod x⁶⁴) × (x⁶³+x⁶²+x⁵⁷)) mod x⁶⁴) + (H3 div x⁶⁴) | C | | C | H4M = (H3 div x⁶⁴)||((H4 mod x⁶⁴) × (x⁶⁴+x⁶³+x⁶²+x⁵⁷)) div x⁶⁴ | C | H4L = (H3 mod x⁶⁴)||(((H4 mod x⁶⁴) × (x⁶³+x⁶²+x⁵⁷)) mod x⁶⁴) + (H4 div x⁶⁴) | C ******************************************************************************* define(`TABLE', `x0') define(`ZERO', `v0') define(`EMSB', `v1') define(`POLY', `v2') define(`B', `v3') define(`H', `v4') define(`HQ', `q4') define(`H_t', `v5') define(`H2', `v6') define(`H2_t', `v7') define(`H3', `v16') define(`H3_t', `v17') define(`H4', `v18') define(`H4_t', `v19') define(`H_m', `v20') define(`H_m1', `v21') define(`H_h', `v22') define(`H_l', `v23') define(`RP', `v24') define(`Ml', `v25') define(`Mh', `v26') PROLOGUE(_nettle_gcm_init_key) ldr HQ,[TABLE,#16*H_Idx] dup EMSB.16b,H.b[0] rev64 H.16b,H.16b mov x9,#0xC200000000000000 mov x10,#1 mov POLY.d[0],x9 mov POLY.d[1],x10 sshr EMSB.16b,EMSB.16b,#7 and EMSB.16b,EMSB.16b,POLY.16b ushr B.2d,H.2d,#63 and B.16b,B.16b,POLY.16b ext B.16b,B.16b,B.16b,#8 shl H.2d,H.2d,#1 orr H.16b,H.16b,B.16b eor H.16b,H.16b,EMSB.16b eor ZERO.16b,ZERO.16b,ZERO.16b dup POLY.2d,POLY.d[0] ext H_t.16b,H.16b,H.16b,#8 pmull H_m.1q,H.1d,H_t.1d pmull2 H_m1.1q,H.2d,H_t.2d pmull H_h.1q,H.1d,H.1d pmull2 H_l.1q,H.2d,H.2d eor H_m.16b,H_m.16b,H_m1.16b pmull RP.1q,H_l.1d,POLY.1d ext Ml.16b,ZERO.16b,H_m.16b,#8 ext Mh.16b,H_m.16b,ZERO.16b,#8 ext RP.16b,RP.16b,RP.16b,#8 eor H_l.16b,H_l.16b,Ml.16b eor H_h.16b,H_h.16b,Mh.16b eor H_l.16b,H_l.16b,RP.16b pmull2 RP.1q,H_l.2d,POLY.2d eor H_h.16b,H_h.16b,H_l.16b eor H2_t.16b,H_h.16b,RP.16b ext H2.16b,H2_t.16b,H2_t.16b,#8 st1 {H.16b,H_t.16b,H2.16b,H2_t.16b},[TABLE],#64 pmull H_m.1q,H.1d,H2_t.1d pmull2 H_m1.1q,H.2d,H2_t.2d pmull H_h.1q,H.1d,H2.1d pmull2 H_l.1q,H.2d,H2.2d eor H_m.16b,H_m.16b,H_m1.16b pmull RP.1q,H_l.1d,POLY.1d ext Ml.16b,ZERO.16b,H_m.16b,#8 ext Mh.16b,H_m.16b,ZERO.16b,#8 ext RP.16b,RP.16b,RP.16b,#8 eor H_l.16b,H_l.16b,Ml.16b eor H_h.16b,H_h.16b,Mh.16b eor H_l.16b,H_l.16b,RP.16b pmull2 RP.1q,H_l.2d,POLY.2d eor H_h.16b,H_h.16b,H_l.16b eor H3_t.16b,H_h.16b,RP.16b ext H3.16b,H3_t.16b,H3_t.16b,#8 pmull H_m.1q,H2.1d,H2_t.1d pmull2 H_m1.1q,H2.2d,H2_t.2d pmull H_h.1q,H2.1d,H2.1d pmull2 H_l.1q,H2.2d,H2.2d eor H_m.16b,H_m.16b,H_m1.16b pmull RP.1q,H_l.1d,POLY.1d ext Ml.16b,ZERO.16b,H_m.16b,#8 ext Mh.16b,H_m.16b,ZERO.16b,#8 ext RP.16b,RP.16b,RP.16b,#8 eor H_l.16b,H_l.16b,Ml.16b eor H_h.16b,H_h.16b,Mh.16b eor H_l.16b,H_l.16b,RP.16b pmull2 RP.1q,H_l.2d,POLY.2d eor H_h.16b,H_h.16b,H_l.16b eor H4_t.16b,H_h.16b,RP.16b ext H4.16b,H4_t.16b,H4_t.16b,#8 st1 {H3.16b,H3_t.16b,H4.16b,H4_t.16b},[TABLE] ret EPILOGUE(_nettle_gcm_init_key) define(`TABLE', `x0') define(`X', `x1') define(`LENGTH', `x2') define(`DATA', `x3') define(`POLY', `v0') define(`ZERO', `v1') define(`D', `v2') define(`C0', `v3') define(`C0D', `d3') define(`C1', `v4') define(`C2', `v5') define(`C3', `v6') define(`RP', `v7') define(`H', `v16') define(`H_t', `v17') define(`H2', `v18') define(`H2_t', `v19') define(`H3', `v20') define(`H3_t', `v21') define(`H4', `v22') define(`H4_t', `v23') define(`H_m', `v24') define(`H_m1', `v25') define(`H_h', `v26') define(`H_l', `v27') define(`H_m2', `v28') define(`H_m3', `v29') define(`H_h2', `v30') define(`H_l2', `v31') define(`Ml', `v4') define(`Mh', `v5') C void gcm_hash (const struct gcm_key *key, union gcm_block *x, C size_t length, const uint8_t *data) PROLOGUE(_nettle_gcm_hash) mov x10,#0xC200000000000000 mov POLY.d[0],x10 dup POLY.2d,POLY.d[0] eor ZERO.16b,ZERO.16b,ZERO.16b ld1 {D.16b},[X] rev64 D.16b,D.16b ands x10,LENGTH,#-64 b.eq L2x add x9,TABLE,64 ld1 {H.16b,H_t.16b,H2.16b,H2_t.16b},[TABLE] ld1 {H3.16b,H3_t.16b,H4.16b,H4_t.16b},[x9] L4x_loop: ld1 {C0.16b,C1.16b,C2.16b,C3.16b},[DATA],#64 rev64 C0.16b,C0.16b rev64 C1.16b,C1.16b rev64 C2.16b,C2.16b rev64 C3.16b,C3.16b eor C0.16b,C0.16b,D.16b pmull H_m.1q,C1.1d,H3_t.1d pmull2 H_m1.1q,C1.2d,H3_t.2d pmull H_h.1q,C1.1d,H3.1d pmull2 H_l.1q,C1.2d,H3.2d pmull H_m2.1q,C2.1d,H2_t.1d pmull2 H_m3.1q,C2.2d,H2_t.2d pmull H_h2.1q,C2.1d,H2.1d pmull2 H_l2.1q,C2.2d,H2.2d eor H_m.16b,H_m.16b,H_m2.16b eor H_m1.16b,H_m1.16b,H_m3.16b eor H_h.16b,H_h.16b,H_h2.16b eor H_l.16b,H_l.16b,H_l2.16b pmull H_m2.1q,C3.1d,H_t.1d pmull2 H_m3.1q,C3.2d,H_t.2d pmull H_h2.1q,C3.1d,H.1d pmull2 H_l2.1q,C3.2d,H.2d eor H_m.16b,H_m.16b,H_m2.16b eor H_m1.16b,H_m1.16b,H_m3.16b eor H_h.16b,H_h.16b,H_h2.16b eor H_l.16b,H_l.16b,H_l2.16b pmull H_m2.1q,C0.1d,H4_t.1d pmull2 H_m3.1q,C0.2d,H4_t.2d pmull H_h2.1q,C0.1d,H4.1d pmull2 H_l2.1q,C0.2d,H4.2d eor H_m.16b,H_m.16b,H_m2.16b eor H_m1.16b,H_m1.16b,H_m3.16b eor H_h.16b,H_h.16b,H_h2.16b eor H_l.16b,H_l.16b,H_l2.16b eor H_m.16b,H_m.16b,H_m1.16b pmull RP.1q,H_l.1d,POLY.1d ext Ml.16b,ZERO.16b,H_m.16b,#8 ext Mh.16b,H_m.16b,ZERO.16b,#8 ext RP.16b,RP.16b,RP.16b,#8 eor H_l.16b,H_l.16b,Ml.16b eor H_h.16b,H_h.16b,Mh.16b eor H_l.16b,H_l.16b,RP.16b pmull2 RP.1q,H_l.2d,POLY.2d eor H_h.16b,H_h.16b,H_l.16b eor D.16b,H_h.16b,RP.16b ext D.16b,D.16b,D.16b,#8 subs x10,x10,64 b.ne L4x_loop and LENGTH,LENGTH,#63 L2x: tst LENGTH,#-32 b.eq L1x ld1 {H.16b,H_t.16b,H2.16b,H2_t.16b},[TABLE] ld1 {C0.16b,C1.16b},[DATA],#32 rev64 C0.16b,C0.16b rev64 C1.16b,C1.16b eor C0.16b,C0.16b,D.16b pmull H_m.1q,C1.1d,H_t.1d pmull2 H_m1.1q,C1.2d,H_t.2d pmull H_h.1q,C1.1d,H.1d pmull2 H_l.1q,C1.2d,H.2d pmull H_m2.1q,C0.1d,H2_t.1d pmull2 H_m3.1q,C0.2d,H2_t.2d pmull H_h2.1q,C0.1d,H2.1d pmull2 H_l2.1q,C0.2d,H2.2d eor H_m.16b,H_m.16b,H_m2.16b eor H_m1.16b,H_m1.16b,H_m3.16b eor H_h.16b,H_h.16b,H_h2.16b eor H_l.16b,H_l.16b,H_l2.16b eor H_m.16b,H_m.16b,H_m1.16b pmull RP.1q,H_l.1d,POLY.1d ext Ml.16b,ZERO.16b,H_m.16b,#8 ext Mh.16b,H_m.16b,ZERO.16b,#8 ext RP.16b,RP.16b,RP.16b,#8 eor H_l.16b,H_l.16b,Ml.16b eor H_h.16b,H_h.16b,Mh.16b eor H_l.16b,H_l.16b,RP.16b pmull2 RP.1q,H_l.2d,POLY.2d eor H_h.16b,H_h.16b,H_l.16b eor D.16b,H_h.16b,RP.16b ext D.16b,D.16b,D.16b,#8 and LENGTH,LENGTH,#31 L1x: tst LENGTH,#-16 b.eq Lmod ld1 {H.16b,H_t.16b},[TABLE] ld1 {C0.16b},[DATA],#16 rev64 C0.16b,C0.16b eor C0.16b,C0.16b,D.16b pmull H_m.1q,C0.1d,H_t.1d pmull2 H_m1.1q,C0.2d,H_t.2d pmull H_h.1q,C0.1d,H.1d pmull2 H_l.1q,C0.2d,H.2d eor H_m.16b,H_m.16b,H_m1.16b pmull RP.1q,H_l.1d,POLY.1d ext Ml.16b,ZERO.16b,H_m.16b,#8 ext Mh.16b,H_m.16b,ZERO.16b,#8 ext RP.16b,RP.16b,RP.16b,#8 eor H_l.16b,H_l.16b,Ml.16b eor H_h.16b,H_h.16b,Mh.16b eor H_l.16b,H_l.16b,RP.16b pmull2 RP.1q,H_l.2d,POLY.2d eor H_h.16b,H_h.16b,H_l.16b eor D.16b,H_h.16b,RP.16b ext D.16b,D.16b,D.16b,#8 Lmod: tst LENGTH,#15 b.eq Ldone ld1 {H.16b,H_t.16b},[TABLE] tbz LENGTH,3,Lmod_8 ldr C0D,[DATA],#8 rev64 C0.16b,C0.16b mov x10,#0 mov C0.d[1],x10 Lmod_8: tst LENGTH,#7 b.eq Lmod_8_done mov x9,#0 mov x8,#64 and x7,LENGTH,#7 Lmod_8_loop: mov x10,#0 ldrb w10,[DATA],#1 sub x8,x8,#8 lsl x10,x10,x8 orr x9,x9,x10 subs x7,x7,#1 b.ne Lmod_8_loop tbz LENGTH,3,Lmod_8_load mov C0.d[1],x9 b Lmod_8_done Lmod_8_load: mov x10,#0 mov C0.d[0],x9 mov C0.d[1],x10 Lmod_8_done: eor C0.16b,C0.16b,D.16b pmull H_m.1q,C0.1d,H_t.1d pmull2 H_m1.1q,C0.2d,H_t.2d pmull H_h.1q,C0.1d,H.1d pmull2 H_l.1q,C0.2d,H.2d eor H_m.16b,H_m.16b,H_m1.16b pmull RP.1q,H_l.1d,POLY.1d ext Ml.16b,ZERO.16b,H_m.16b,#8 ext Mh.16b,H_m.16b,ZERO.16b,#8 ext RP.16b,RP.16b,RP.16b,#8 eor H_l.16b,H_l.16b,Ml.16b eor H_h.16b,H_h.16b,Mh.16b eor H_l.16b,H_l.16b,RP.16b pmull2 RP.1q,H_l.2d,POLY.2d eor H_h.16b,H_h.16b,H_l.16b eor D.16b,H_h.16b,RP.16b ext D.16b,D.16b,D.16b,#8 Ldone: rev64 D.16b,D.16b st1 {D.16b},[X] ret EPILOGUE(_nettle_gcm_hash)

6 62

Add pbkdf2_hmac_sha384 and pbkdf2_hmac_sha512 to Nettle
by Nicolas Mora 01 Feb '21

01 Feb '21

Hello, I just opened a merge request [1] to add pbkdf2_hmac_sha384 and pbkdf2_hmac_sha512 to the Nettle library. These pbkdf2 functions are required to implement pbes2-* key management algorithms defined in the JSON Web Encryption (JWE) and JSON Web Algorithms (JWA) specifications [2], [3]. I added the new pbkdf2 functions, one test case per function, based on the same key derivation as the pbkdf2_hmac_sha256 test case. I also updated the documentation. Is it possible to get a code review for this patch? Thanks in advance /Nicolas [1] https://git.lysator.liu.se/nettle/nettle/-/merge_requests/18 [2] https://tools.ietf.org/html/rfc7516 [3] https://tools.ietf.org/html/rfc7518

2 1

ANNOUNCE: Nettle-3.7
by nisse＠lysator.liu.se 04 Jan '21

04 Jan '21

I'm happy to announce a new release of GNU Nettle, a low-level cryptographics library. This includes one new feature, and several optimizations, see NEWS entries below. The Nettle home page can be found at https://www.lysator.liu.se/~nisse/nettle/, and the manual at https://www.lysator.liu.se/~nisse/nettle/nettle.html. The release can be downloaded from https://ftp.gnu.org/gnu/nettle/nettle-3.7.tar.gz ftp://ftp.gnu.org/gnu/nettle/nettle-3.7.tar.gz https://www.lysator.liu.se/~nisse/archive/nettle-3.7.tar.gz Happy hacking, /Niels Möller NEWS for the Nettle 3.7 release This release adds one new feature, the bcrypt password hashing function, and lots of optimizations. There's also one important change to how Nettle is configured: Fat builds are now on by default. The release adds PowerPC64 assembly for a few algorithms, resulting in great speedups. Benchmarked on a Power9 machine, speedup was 13 times for AES256-CTR and AES256-GCM, and 3.5 times for Chacha. For fat builds (now the default), the new code is used automatically, on processors supporting the needed instruction set extensions. The new version is intended to be fully source and binary compatible with Nettle-3.6. The shared library names are libnettle.so.8.1 and libhogweed.so.6.1, with sonames libnettle.so.8 and libhogweed.so.6. New features: * Support for bcrypt, contributed by Stephen R. van den Berg. Optimizations: * Much faster AES and GCM on PowerPC64 processors supporting the corresponding crypto extensions. Contributed by Mamone Tarsha. * Speed of Chacha improved on PowerPC64, x86_64 and ARM Neon. * Speed of Salsa20 improved on x86_64 and ARM Neon. * Overhaul of some elliptic curve primitives, improving ECDSA signature speed. Configure: * Fat builds are enabled by default on the architectures where it is supported (x86_64, arm and powerpc64). To disable runtime selection, and instead specify the processor flavor at configure time, you need to pass --disable-fat to the configure script. Known issues: * The ARM assembly code in this release doesn't work correctly on big-endian ARM systems. This will hopefully be fixed in a later release. Miscellaneous: * Use a few more gmp-6.1 functions: mpn_cnd_add_n, mpn_cnd_sub_n, mpn_cnd_swap. Delete corresponding internal Nettle functions. * Convert all assembly files to use the default m4 quote characters. -- Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677. Internet email is subject to wholesale government surveillance.

1 0

Release of Nettle-3.7?
by nisse＠lysator.liu.se 02 Jan '21

02 Jan '21

Hi, I wonder if it would make sense to try to cut a release pretty soon (and without any arm64 changes)? Previous release was made end of April, and there's been quite a few improvements since then. I wonder if it is possible to make a release in time for the upcoming debian release? https://lists.debian.org/debian-devel-announce/2020/11/msg00002.html Nettle-3.7 should be abi-compatible, and with unchanged soname, so I'm not sure if it would count as a "transition" in the debian world. Regards, /Niels -- Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677. Internet email is subject to wholesale government surveillance.

6 16

Failing gnutls tests
by nisse＠lysator.liu.se 28 Dec '20

28 Dec '20

Hi, recent gnutls tests on the gitlab ci system all fail the test "testpkcs11.sh". See e.g., https://gitlab.com/gnutls/nettle/-/jobs/932664781. First failure was an a merge commit with a minor ppc-related fix. And it looks like gnutls' own pipelines have been failing for a month or so too, see https://gitlab.com/gnutls/gnutls/-/pipelines. Any gnutls people on the list who could have a look? Regards, /Niels -- Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677. Internet email is subject to wholesale government surveillance.

3 3

Re: Release of Nettle-3.7?
by Maamoun TK 19 Dec '20

19 Dec '20

Hi Michael, On Fri, Dec 18, 2020 at 8:00 PM Michael Weiser <michael(a)weiser.dinsnail.net> wrote: > qemu-user works nicely for aarch64_be. I used it to semi-natively > compile a whole aarch64 userland. I could dust off pine64 board that is > running that userland now for real-world testing if you like. > Thank you, I will give qemu-user a shot for aarch64_be testing. Also, I will let you know if I reach a point where real-world testing is needed. regards, Mamone

1 0

PPC chacha
by nisse＠lysator.liu.se 02 Dec '20

02 Dec '20

I'm trying to learn a bit of ppc assembly. Below is an implementation of _chacha_core. Seems to work, when tested on gcc112.fsffrance.org (just put the file in the powerpc64 directory and reconfigure). This machine is little-endian, I haven't yet tested on big-endian. Unfortunately I don't get any accurate benchmark numbers on that machine, but I think speedup may be on the order of 50%. It could likely be speedup further by processing 2, 3 or 4 blocks in parallel, similar to recent improvements for arm and x86_64. I'd like to do that after the simpler single-block function is properly merged. I'm not sure where it fits under powerpc64. The code doesn't need any cryptographic extensions, but it depends on vector instructions as well as VSX registers (for the unaligned load and store instructions). So I'd need advice both on the directory hierarchy and compile time configuration, and appropriate runtime tests for fat builds. Comments on the code highly appreciated! It's the first ppc code I've written, and the reference manual isn't that easy to navigate. The vector instructions seem very nice to work with, and makes for a shorter QROUND than both x86_64 SSE and ARM Neon (these suffer a bit from missing vector rotate instruction). Help with additional benchmarking would also be useful. Regards, /Niels C powerpc64/chacha-core-internal.asm ifelse(` Copyright (C) 2020 Niels Möller and Torbjörn Granlund This file is part of GNU Nettle. GNU Nettle is free software: you can redistribute it and/or modify it under the terms of either: * the GNU Lesser General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. or * the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. or both in parallel, as here. GNU Nettle is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received copies of the GNU General Public License and the GNU Lesser General Public License along with this program. If not, see http://www.gnu.org/licenses/. ') C Register usage: C Argments define(`DST', `r3') define(`SRC', `r4') define(`ROUNDS', `r5') C Working state define(`X0', `v0') define(`X1', `v1') define(`X2', `v2') define(`X3', `v3') define(`ROT16', `v4') define(`ROT12', `v5') define(`ROT8', `v6') define(`ROT7', `v7') C Original input state define(`S0', `v8') define(`S1', `v9') define(`S2', `v10') define(`S3', `v11') C QROUND(X0, X1, X2, X3) define(`QROUND', ` C x0 += x1, x3 ^= x0, x3 lrot 16 C x2 += x3, x1 ^= x2, x1 lrot 12 C x0 += x1, x3 ^= x0, x3 lrot 8 C x2 += x3, x1 ^= x2, x1 lrot 7 vadduwm $1, $1, $2 vxor $4, $4, $1 vrlw $4, $4, ROT16 vadduwm $3, $3, $4 vxor $2, $2, $3 vrlw $2, $2, ROT12 vadduwm $1, $1, $2 vxor $4, $4, $1 vrlw $4, $4, ROT8 vadduwm $3, $3, $4 vxor $2, $2, $3 vrlw $2, $2, ROT7 ') .text .align 4 C _chacha_core(uint32_t *dst, const uint32_t *src, unsigned rounds) PROLOGUE(_nettle_chacha_core) li r6, 0x10 C set up some... li r7, 0x20 C ...useful... li r8, 0x30 C ...offsets vspltisw ROT16, -16 C -16 instead of 16 actually works! vspltisw ROT12, 12 vspltisw ROT8, 8 vspltisw ROT7, 7 lxvw4x VSR(X0), 0, SRC lxvw4x VSR(X1), r6, SRC lxvw4x VSR(X2), r7, SRC lxvw4x VSR(X3), r8, SRC vor S0, X0, X0 vor S1, X1, X1 vor S2, X2, X2 vor S3, X3, X3 srdi ROUNDS, ROUNDS, 1 mtctr ROUNDS .Loop: QROUND(X0, X1, X2, X3) C Rotate rows, to get C 0 1 2 3 C 5 6 7 4 <<< 1 C 10 11 8 9 <<< 2 C 15 12 13 14 <<< 3 vsldoi X1, X1, X1, 4 vsldoi X2, X2, X2, 8 vsldoi X3, X3, X3, 12 QROUND(X0, X1, X2, X3) C Inverse rotation vsldoi X1, X1, X1, 12 vsldoi X2, X2, X2, 8 vsldoi X3, X3, X3, 4 bdnz .Loop vadduwm X0, X0, S0 vadduwm X1, X1, S1 vadduwm X2, X2, S2 vadduwm X3, X3, S3 stxvw4x VSR(X0), 0, DST stxvw4x VSR(X1), r6, DST stxvw4x VSR(X2), r7, DST stxvw4x VSR(X3), r8, DST blr EPILOGUE(_nettle_chacha_core) -- Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677. Internet email is subject to wholesale government surveillance.

5 43

Fwd: [PowerPC] GCM optimization
by Maamoun TK 02 Dec '20

02 Dec '20

---------- Forwarded message --------- From: Maamoun TK <maamoun.tk(a)googlemail.com> Date: Thu, Nov 12, 2020 at 7:42 PM Subject: Re: [PowerPC] GCM optimization To: Niels Möller <nisse(a)lysator.liu.se> On Thu, Nov 12, 2020 at 6:40 PM Niels Möller <nisse(a)lysator.liu.se> wrote: > I gave it a test run on gcc112 in the gcc compile farm, and speedup of > gcm update seems to be 26 times(!) compared to the C version. > That's reasonable, I got similar speedup on more stable POWER instances than gcc compile farm. > Where would that documentation be published? In the Nettle manual, as > some IBM white paper, or as a more-or-less academic paper, e.g., on > arxiv? I will not be able to spend much time on writing, but I'd be > happy to review. > I'll start writing the papers once I got more details from IBM, similar to intel documents, the document will be academic and practical at the same time, I'll dive into finite field equations to demonstrate how we get there as well as I'll add a practical example to clarify the preference of this method in addition to the expected speedup of this method. My intention that other crypto libraries could take advantage of this document or maybe be a starting point for further improvements to the algorithm so I'm checking if IBM would publish or approve such a document the same as intel. > I have a sketch of ARM Neon code doing the equivalent of two vpmsumd, > with reasonable parallelism. Quite a lot of instructions needed. > If you don't have much time, you can send it here and I'll continue from that point. I'm planning to compare the new method with the usual method with and without the karatsuba algorithm. > +C Alignment of gcm_key table elements, which is declared in gcm.h > > +define(`TableElemAlign', `0x100') > > I still find this large constant puzzling. If I try > > struct gcm_key key; > printf("sizeof (key): %zd, sizeof(key.h[0]): %zd\n", sizeof(key), > sizeof(key.h[0])); > > (I added it to the start of test_main in gcm-test.c) and run on the > gcc112 machine, I get > > sizeof (key): 4096, sizeof(key.h[0]): 16 > > Which is what I'd expect, with elements of size 16 bytes, not 256 bytes. > > I haven't yet had the time to read the code carefully. > You see, the alignment of each element is 0x100 (256). The table has 16 elements and you got the size of the table 4096 which is reasonable because 16*256=4096 regards, Mamone

3 5

[PowerPC] GCM optimization
by Maamoun TK 28 Nov '20

28 Nov '20

This implementation takes advantage of research made by Niels Möller to optimize GCM on PowerPC, this optimization yields a +27.7% performance boost on POWER8 over the previous implementation that was based on intel documents. The performance comparison is made by processing 4 blocks per loop without any further optimizations. I made some documentations between the lines but I suggest writing a document similar to the intel ones that go into more details and clarify the preference of this method. I'm also curious if this method can also make a difference in other architectures like ARM, I'm planning to try it out for ARM to figure that out. --- configure.ac | 6 +- gcm.c | 49 +++-- powerpc64/p8/gcm-hash.asm | 502 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 542 insertions(+), 15 deletions(-) create mode 100644 powerpc64/p8/gcm-hash.asm diff --git a/configure.ac b/configure.ac index 2a47f940..20f7cf74 100644 --- a/configure.ac +++ b/configure.ac @@ -497,7 +497,7 @@ asm_replace_list="aes-encrypt-internal.asm aes-decrypt-internal.asm \ sha3-permute.asm umac-nh.asm umac-nh-n.asm machine.m4" # Assembler files which generate additional object files if they are used. -asm_nettle_optional_list="gcm-hash8.asm cpuid.asm \ +asm_nettle_optional_list="gcm-hash.asm gcm-hash8.asm cpuid.asm \ aes-encrypt-internal-2.asm aes-decrypt-internal-2.asm memxor-2.asm \ chacha-3core.asm chacha-core-internal-2.asm salsa20-2core.asm \ salsa20-core-internal-2.asm sha1-compress-2.asm sha256-compress-2.asm \ @@ -621,9 +621,9 @@ AH_VERBATIM([HAVE_NATIVE], #undef HAVE_NATIVE_ecc_secp384r1_redc #undef HAVE_NATIVE_ecc_secp521r1_modp #undef HAVE_NATIVE_ecc_secp521r1_redc -#undef HAVE_NATIVE_gcm_init_key8 +#undef HAVE_NATIVE_gcm_init_key +#undef HAVE_NATIVE_gcm_hash #undef HAVE_NATIVE_gcm_hash8 -#undef HAVE_NATIVE_gcm_fill #undef HAVE_NATIVE_salsa20_core #undef HAVE_NATIVE_salsa20_2core #undef HAVE_NATIVE_fat_salsa20_2core diff --git a/gcm.c b/gcm.c index 48b3e75a..81981c1c 100644 --- a/gcm.c +++ b/gcm.c @@ -140,6 +140,19 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) memcpy (x->b, Z.b, sizeof(Z)); } # elif GCM_TABLE_BITS == 8 +# if HAVE_NATIVE_gcm_init_key + +#define gcm_init_key _nettle_gcm_init_key +void +_nettle_gcm_init_key (union nettle_block16 *table); +# endif /* HAVE_NATIVE_gcm_init_key */ +# if HAVE_NATIVE_gcm_hash + +#define gcm_hash _nettle_gcm_hash +void +_nettle_gcm_hash (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data); +# endif /* HAVE_NATIVE_gcm_hash */ # if HAVE_NATIVE_gcm_hash8 #define gcm_hash _nettle_gcm_hash8 @@ -228,6 +241,29 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) /* Increment the rightmost 32 bits. */ #define INC32(block) INCREMENT(4, (block.b) + GCM_BLOCK_SIZE - 4) +#ifndef gcm_init_key +static void +gcm_init_key(union nettle_block16 *table) +{ +#if GCM_TABLE_BITS + /* Middle element if GCM_TABLE_BITS > 0, otherwise the first + element */ + unsigned i = (1<<GCM_TABLE_BITS)/2; + + /* Algorithm 3 from the gcm paper. First do powers of two, then do + the rest by adding. */ + while (i /= 2) + block16_mulx_ghash(&table[i], &table[2*i]); + for (i = 2; i < 1<<GCM_TABLE_BITS; i *= 2) + { + unsigned j; + for (j = 1; j < i; j++) + block16_xor3(&table[i+j], &table[i], &table[j]); + } +#endif +} +#endif /* !gcm_init_key */ + /* Initialization of GCM. * @ctx: The context of GCM * @cipher: The context of the underlying block cipher @@ -245,18 +281,7 @@ gcm_set_key(struct gcm_key *key, memset(key->h[0].b, 0, GCM_BLOCK_SIZE); f (cipher, GCM_BLOCK_SIZE, key->h[i].b, key->h[0].b); -#if GCM_TABLE_BITS - /* Algorithm 3 from the gcm paper. First do powers of two, then do - the rest by adding. */ - while (i /= 2) - block16_mulx_ghash(&key->h[i], &key->h[2*i]); - for (i = 2; i < 1<<GCM_TABLE_BITS; i *= 2) - { - unsigned j; - for (j = 1; j < i; j++) - block16_xor3(&key->h[i+j], &key->h[i],&key->h[j]); - } -#endif + gcm_init_key(key->h); } #ifndef gcm_hash diff --git a/powerpc64/p8/gcm-hash.asm b/powerpc64/p8/gcm-hash.asm new file mode 100644 index 00000000..e79fbdc2 --- /dev/null +++ b/powerpc64/p8/gcm-hash.asm @@ -0,0 +1,502 @@ +C powerpc64/p8/gcm-hash.asm + +ifelse(` + Copyright (C) 2020 Niels Möller and Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +') + +C Alignment of gcm_key table elements, which is declared in gcm.h +define(`TableElemAlign', `0x100') + +C Register usage: + +define(`SP', `r1') +define(`TOCP', `r2') + +define(`TABLE', `r3') + +define(`ZERO', `v0') +define(`B1', `v1') +define(`EMSB', `v16') +define(`POLY', `v17') +define(`POLY_L', `v1') + +define(`H', `v2') +define(`H2', `v3') +define(`H3', `v4') +define(`H4', `v5') +define(`H1M', `v6') +define(`H1L', `v7') +define(`H2M', `v8') +define(`H2L', `v9') +define(`Hl', `v10') +define(`Hm', `v11') +define(`Hp', `v12') +define(`Hl2', `v13') +define(`Hm2', `v14') +define(`Hp2', `v15') +define(`R', `v13') +define(`F', `v14') +define(`T', `v15') +define(`R2', `v16') +define(`F2', `v17') +define(`T2', `v18') + +define(`LE_TEMP', `v18') +define(`LE_MASK', `v19') + +.file "gcm-hash.asm" + +.text + + C void gcm_init_key (union gcm_block *table) + +C This function populates the gcm table as the following layout +C ******************************************************************************* +C | H1M = (H1 div x⁶⁴)||((H1 mod x⁶⁴) × (x⁶⁴+x⁶³+x⁶²+x⁵⁷)) div x⁶⁴ | +C | H1L = (H1 mod x⁶⁴)||(((H1 mod x⁶⁴) × (x⁶³+x⁶²+x⁵⁷)) mod x⁶⁴) + (H1 div x⁶⁴) | +C | | +C | H2M = (H2 div x⁶⁴)||((H2 mod x⁶⁴) × (x⁶⁴+x⁶³+x⁶²+x⁵⁷)) div x⁶⁴ | +C | H2L = (H2 mod x⁶⁴)||(((H2 mod x⁶⁴) × (x⁶³+x⁶²+x⁵⁷)) mod x⁶⁴) + (H2 div x⁶⁴) | +C | | +C | H3M = (H3 div x⁶⁴)||((H3 mod x⁶⁴) × (x⁶⁴+x⁶³+x⁶²+x⁵⁷)) div x⁶⁴ | +C | H3L = (H3 mod x⁶⁴)||(((H3 mod x⁶⁴) × (x⁶³+x⁶²+x⁵⁷)) mod x⁶⁴) + (H3 div x⁶⁴) | +C | | +C | H4M = (H3 div x⁶⁴)||((H4 mod x⁶⁴) × (x⁶⁴+x⁶³+x⁶²+x⁵⁷)) div x⁶⁴ | +C | H4L = (H3 mod x⁶⁴)||(((H4 mod x⁶⁴) × (x⁶³+x⁶²+x⁵⁷)) mod x⁶⁴) + (H4 div x⁶⁴) | +C ******************************************************************************* + +define(`FUNC_ALIGN', `5') +PROLOGUE(_nettle_gcm_init_key) + DATA_LOAD_VEC(POLY,.polynomial,r7) C 0xC2000000000000000000000000000001 +IF_LE(` + li r8,0 + lvsl LE_MASK,0,r8 C 0x000102030405060708090A0B0C0D0E0F + vspltisb LE_TEMP,0x07 C 0x07070707070707070707070707070707 + vxor LE_MASK,LE_MASK,LE_TEMP C 0x07060504030201000F0E0D0C0B0A0908 +') + + C 'H' is assigned by gcm_set_key() to the middle element of the table + li r10,8*TableElemAlign + lxvd2x VSR(H),r10,TABLE C load 'H' + C byte-reverse of each doubleword permuting on little-endian mode +IF_LE(` + vperm H,H,H,LE_MASK +') + + C --- calculate H = H << 1 mod P(X), P(X) = (x¹²⁸+x¹²⁷+x¹²⁶+x¹²¹+1) --- + + vupkhsb EMSB,H C extend most significant bit to first byte + vspltisb B1,1 C 0x01010101010101010101010101010101 + vspltb EMSB,EMSB,0 C first byte quadword-extend + vsl H,H,B1 C H = H << 1 + vand EMSB,EMSB,POLY C EMSB &= 0xC2000000000000000000000000000001 + vxor ZERO,ZERO,ZERO C 0x00000000000000000000000000000000 + vxor H,H,EMSB C H ^= EMSB + + C --- calculate H^2 = H*H --- + + xxmrghd VSR(POLY_L),VSR(ZERO),VSR(POLY) C 0x0000000000000000C200000000000000 + + C --- Hp = (H mod x⁶⁴) / x⁶⁴ mod P(X) --- + C --- Hp = (H mod x⁶⁴) × (x⁶⁴+x⁶³+x⁶²+x⁵⁷) mod P(X), deg(Hp) ≤ 127 --- + C --- Hp = (H mod x⁶⁴) × (x⁶⁴+x⁶³+x⁶²+x⁵⁷) --- + vpmsumd Hp,H,POLY_L C Hp = (H mod x⁶⁴) × (x⁶³+x⁶²+x⁵⁷) + xxmrgld VSR(Hl),VSR(H),VSR(ZERO) C Hl = (H mod x⁶⁴) × x⁶⁴ + xxswapd VSR(Hm),VSR(H) + vxor Hl,Hl,Hp C Hl = Hl + Hp + vxor Hm,Hm,Hp C Hm = Hm + Hp + xxmrghd VSR(H1M),VSR(H),VSR(Hl) C H1M = (H div x⁶⁴)||(Hl div x⁶⁴) + xxmrgld VSR(H1L),VSR(H),VSR(Hm) C H1L = (H mod x⁶⁴)||(Hl mod x⁶⁴) + + vpmsumd F,H1L,H C F = (H1Lh × Hh) + (H1Ll × Hl) + vpmsumd R,H1M,H C R = (H1Mh × Hh) + (H1Ml × Hl) + + C --- rduction --- + vpmsumd T,F,POLY_L C T = (F mod x⁶⁴) × (x⁶³+x⁶²+x⁵⁷) + xxswapd VSR(H2),VSR(F) + vxor R,R,T C R = R + T + vxor H2,R,H2 + + xxmrgld VSR(Hl),VSR(H2),VSR(ZERO) + xxswapd VSR(Hm),VSR(H2) + vpmsumd Hp,H2,POLY_L + vxor Hl,Hl,Hp + vxor Hm,Hm,Hp + xxmrghd VSR(H2M),VSR(H2),VSR(Hl) + xxmrgld VSR(H2L),VSR(H2),VSR(Hm) + + C store H1M, H1L, H2M, H2L + li r8,1*TableElemAlign + li r9,2*TableElemAlign + li r10,3*TableElemAlign + stxvd2x VSR(H1M),0,TABLE + stxvd2x VSR(H1L),r8,TABLE + stxvd2x VSR(H2M),r9,TABLE + stxvd2x VSR(H2L),r10,TABLE + + C --- calculate H^3 = H^1*H^2, H^4 = H^2*H^2 --- + + vpmsumd F,H1L,H2 + vpmsumd F2,H2L,H2 + vpmsumd R,H1M,H2 + vpmsumd R2,H2M,H2 + + vpmsumd T,F,POLY_L + vpmsumd T2,F2,POLY_L + xxswapd VSR(H3),VSR(F) + xxswapd VSR(H4),VSR(F2) + vxor R,R,T + vxor R2,R2,T2 + vxor H3,R,H3 + vxor H4,R2,H4 + + xxmrgld VSR(Hl),VSR(H3),VSR(ZERO) + xxmrgld VSR(Hl2),VSR(H4),VSR(ZERO) + xxswapd VSR(Hm),VSR(H3) + xxswapd VSR(Hm2),VSR(H4) + vpmsumd Hp,H3,POLY_L + vpmsumd Hp2,H4,POLY_L + vxor Hl,Hl,Hp + vxor Hl2,Hl2,Hp2 + vxor Hm,Hm,Hp + vxor Hm2,Hm2,Hp2 + xxmrghd VSR(H1M),VSR(H3),VSR(Hl) + xxmrghd VSR(H2M),VSR(H4),VSR(Hl2) + xxmrgld VSR(H1L),VSR(H3),VSR(Hm) + xxmrgld VSR(H2L),VSR(H4),VSR(Hm2) + + C store H3M, H3L, H4M, H4L + li r7,4*TableElemAlign + li r8,5*TableElemAlign + li r9,6*TableElemAlign + li r10,7*TableElemAlign + stxvd2x VSR(H1M),r7,TABLE + stxvd2x VSR(H1L),r8,TABLE + stxvd2x VSR(H2M),r9,TABLE + stxvd2x VSR(H2L),r10,TABLE + + blr +EPILOGUE(_nettle_gcm_init_key) + +define(`TABLE', `r3') +define(`X', `r4') +define(`LENGTH', `r5') +define(`DATA', `r6') + +define(`ZERO', `v16') +define(`POLY', `v17') +define(`POLY_L', `v0') + +define(`D', `v1') +define(`C0', `v2') +define(`C1', `v3') +define(`C2', `v4') +define(`C3', `v5') +define(`H1M', `v6') +define(`H1L', `v7') +define(`H2M', `v8') +define(`H2L', `v9') +define(`H3M', `v10') +define(`H3L', `v11') +define(`H4M', `v12') +define(`H4L', `v13') +define(`R', `v14') +define(`F', `v15') +define(`R2', `v16') +define(`F2', `v17') +define(`R3', `v18') +define(`F3', `v20') +define(`R4', `v21') +define(`F4', `v22') +define(`T', `v23') + +define(`LE_TEMP', `v18') +define(`LE_MASK', `v19') + + C void gcm_hash (const struct gcm_key *key, union gcm_block *x, + C size_t length, const uint8_t *data) + +define(`FUNC_ALIGN', `5') +PROLOGUE(_nettle_gcm_hash) + DATA_LOAD_VEC(POLY,.polynomial,r7) +IF_LE(` + li r8,0 + lvsl LE_MASK,0,r8 + vspltisb LE_TEMP,0x07 + vxor LE_MASK,LE_MASK,LE_TEMP +') + vxor ZERO,ZERO,ZERO + xxmrghd VSR(POLY_L),VSR(ZERO),VSR(POLY) + + lxvd2x VSR(D),0,X C load 'X' pointer + C byte-reverse of each doubleword permuting on little-endian mode +IF_LE(` + vperm D,D,D,LE_MASK +') + + C --- process 4 blocks '128-bit each' per one loop --- + + srdi r7,LENGTH,6 C 4-blocks loop count 'LENGTH / (4 * 16)' + cmpldi r7,0 + beq L2x + + mtctr r7 C assign counter register to loop count + + C store non-volatile vector registers + addi r8,SP,-64 + stvx 20,0,r8 + addi r8,r8,16 + stvx 21,0,r8 + addi r8,r8,16 + stvx 22,0,r8 + addi r8,r8,16 + stvx 23,0,r8 + + C load table elements + li r8,1*TableElemAlign + li r9,2*TableElemAlign + li r10,3*TableElemAlign + lxvd2x VSR(H1M),0,TABLE + lxvd2x VSR(H1L),r8,TABLE + lxvd2x VSR(H2M),r9,TABLE + lxvd2x VSR(H2L),r10,TABLE + li r7,4*TableElemAlign + li r8,5*TableElemAlign + li r9,6*TableElemAlign + li r10,7*TableElemAlign + lxvd2x VSR(H3M),r7,TABLE + lxvd2x VSR(H3L),r8,TABLE + lxvd2x VSR(H4M),r9,TABLE + lxvd2x VSR(H4L),r10,TABLE + + li r8,0x10 + li r9,0x20 + li r10,0x30 +.align 5 +L4x_loop: + C input loading + lxvd2x VSR(C0),0,DATA C load C0 + lxvd2x VSR(C1),r8,DATA C load C1 + lxvd2x VSR(C2),r9,DATA C load C2 + lxvd2x VSR(C3),r10,DATA C load C3 + +IF_LE(` + vperm C0,C0,C0,LE_MASK + vperm C1,C1,C1,LE_MASK + vperm C2,C2,C2,LE_MASK + vperm C3,C3,C3,LE_MASK +') + + C previous digest combining + vxor C0,C0,D + + C polynomial multiplication + vpmsumd F2,H3L,C1 + vpmsumd R2,H3M,C1 + vpmsumd F3,H2L,C2 + vpmsumd R3,H2M,C2 + vpmsumd F4,H1L,C3 + vpmsumd R4,H1M,C3 + vpmsumd F,H4L,C0 + vpmsumd R,H4M,C0 + + C deferred recombination of partial products + vxor F3,F3,F4 + vxor R3,R3,R4 + vxor F,F,F2 + vxor R,R,R2 + vxor F,F,F3 + vxor R,R,R3 + + C reduction + vpmsumd T,F,POLY_L + xxswapd VSR(D),VSR(F) + vxor R,R,T + vxor D,R,D + + addi DATA,DATA,0x40 + bdnz L4x_loop + + C restore non-volatile vector registers + addi r8,SP,-64 + lvx 20,0,r8 + addi r8,r8,16 + lvx 21,0,r8 + addi r8,r8,16 + lvx 22,0,r8 + addi r8,r8,16 + lvx 23,0,r8 + + clrldi LENGTH,LENGTH,58 C 'set the high-order 58 bits to zeros' +L2x: + C --- process 2 blocks --- + + srdi r7,LENGTH,5 C 'LENGTH / (2 * 16)' + cmpldi r7,0 + beq L1x + + C load table elements + li r8,1*TableElemAlign + li r9,2*TableElemAlign + li r10,3*TableElemAlign + lxvd2x VSR(H1M),0,TABLE + lxvd2x VSR(H1L),r8,TABLE + lxvd2x VSR(H2M),r9,TABLE + lxvd2x VSR(H2L),r10,TABLE + + C input loading + li r10,0x10 + lxvd2x VSR(C0),0,DATA C load C0 + lxvd2x VSR(C1),r10,DATA C load C1 + +IF_LE(` + vperm C0,C0,C0,LE_MASK + vperm C1,C1,C1,LE_MASK +') + + C previous digest combining + vxor C0,C0,D + + C polynomial multiplication + vpmsumd F2,H1L,C1 + vpmsumd R2,H1M,C1 + vpmsumd F,H2L,C0 + vpmsumd R,H2M,C0 + + C deferred recombination of partial products + vxor F,F,F2 + vxor R,R,R2 + + C reduction + vpmsumd T,F,POLY_L + xxswapd VSR(D),VSR(F) + vxor R,R,T + vxor D,R,D + + addi DATA,DATA,0x20 + clrldi LENGTH,LENGTH,59 C 'set the high-order 59 bits to zeros' +L1x: + C --- process 1 block --- + + srdi r7,LENGTH,4 C 'LENGTH / (1 * 16)' + cmpldi r7,0 + beq Lmod + + C load table elements + li r8,1*TableElemAlign + lxvd2x VSR(H1M),0,TABLE + lxvd2x VSR(H1L),r8,TABLE + + C input loading + lxvd2x VSR(C0),0,DATA C load C0 + +IF_LE(` + vperm C0,C0,C0,LE_MASK +') + + C previous digest combining + vxor C0,C0,D + + C polynomial multiplication + vpmsumd F,H1L,C0 + vpmsumd R,H1M,C0 + + C reduction + vpmsumd T,F,POLY_L + xxswapd VSR(D),VSR(F) + vxor R,R,T + vxor D,R,D + + addi DATA,DATA,0x10 + clrldi LENGTH,LENGTH,60 C 'set the high-order 60 bits to zeros' +Lmod: + C --- process the modulo bytes, padding the low-order bytes with zeros --- + + cmpldi LENGTH,0 + beq Ldone + + C load table elements + li r8,1*TableElemAlign + lxvd2x VSR(H1M),0,TABLE + lxvd2x VSR(H1L),r8,TABLE + + C push every modulo byte to the stack and load them with padding into vector register + vxor ZERO,ZERO,ZERO + addi r8,SP,-16 + stvx ZERO,0,r8 +Lstb_loop: + subic. LENGTH,LENGTH,1 + lbzx r7,LENGTH,DATA + stbx r7,LENGTH,r8 + bne Lstb_loop + lxvd2x VSR(C0),0,r8 + +IF_LE(` + vperm C0,C0,C0,LE_MASK +') + + C previous digest combining + vxor C0,C0,D + + C polynomial multiplication + vpmsumd F,H1L,C0 + vpmsumd R,H1M,C0 + + C reduction + vpmsumd T,F,POLY_L + xxswapd VSR(D),VSR(F) + vxor R,R,T + vxor D,R,D + +Ldone: + C byte-reverse of each doubleword permuting on little-endian mode +IF_LE(` + vperm D,D,D,LE_MASK +') + stxvd2x VSR(D),0,X C store digest 'D' + + blr +EPILOGUE(_nettle_gcm_hash) + +.data + C 0xC2000000000000000000000000000001 +.polynomial: +.align 4 +IF_BE(` +.byte 0xC2 +.rept 14 +.byte 0x00 +.endr +.byte 0x01 +',` +.byte 0x01 +.rept 14 +.byte 0x00 +.endr +.byte 0xC2 +') -- 2.17.1

3 28

[PATCH] "PowerPC" Detect VSX support on AIX and FreeBSD
by Maamoun TK 11 Nov '20

11 Nov '20

--- fat-ppc.c | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/fat-ppc.c b/fat-ppc.c index 2bfd649f..ec971706 100644 --- a/fat-ppc.c +++ b/fat-ppc.c @@ -43,8 +43,13 @@ #if defined(_AIX) # include <sys/systemcfg.h> #elif defined(__linux__) +# include <asm/cputable.h> # include <sys/auxv.h> #elif defined(__FreeBSD__) +# include <machine/cpu.h> +# ifdef PPC_FEATURE2_HAS_VEC_CRYPTO +# define PPC_FEATURE2_VEC_CRYPTO PPC_FEATURE2_HAS_VEC_CRYPTO +# endif # if __FreeBSD__ >= 12 # include <sys/auxv.h> # else @@ -58,7 +63,13 @@ #include "gcm.h" #include "fat-setup.h" -/* Define from arch/powerpc/include/uapi/asm/cputable.h in Linux kernel */ +/* Defines from arch/powerpc/include/uapi/asm/cputable.h in Linux kernel */ +#ifndef PPC_FEATURE_HAS_ALTIVEC +#define PPC_FEATURE_HAS_ALTIVEC 0x10000000 +#endif +#ifndef PPC_FEATURE_HAS_VSX +#define PPC_FEATURE_HAS_VSX 0x00000080 +#endif #ifndef PPC_FEATURE2_VEC_CRYPTO #define PPC_FEATURE2_VEC_CRYPTO 0x02000000 #endif @@ -96,8 +107,10 @@ get_ppc_features (struct ppc_features *features) } else { -#if defined(_AIX) && defined(__power_8_andup) - features->have_crypto_ext = __power_8_andup() != 0 ? 1 : 0; +#if defined(_AIX) + features->have_crypto_ext + = _system_configuration.implementation >= 0x10000u; + features->have_altivec = _system_configuration.vmx_version > 1; #else unsigned long hwcap = 0; unsigned long hwcap2 = 0; @@ -106,9 +119,13 @@ get_ppc_features (struct ppc_features *features) hwcap2 = getauxval(AT_HWCAP2); # elif defined(__FreeBSD__) # if __FreeBSD__ >= 12 + elf_aux_info(AT_HWCAP, &hwcap, sizeof(hwcap)); elf_aux_info(AT_HWCAP2, &hwcap2, sizeof(hwcap2)); # else - size_t len = sizeof(hwcap2); + size_t len; + len = sizeof(hwcap); + sysctlbyname("hw.cpu_features", &hwcap, &len, NULL, 0); + len = sizeof(hwcap2); sysctlbyname("hw.cpu_features2", &hwcap2, &len, NULL, 0); # endif # endif -- 2.17.1

2 1

Modular inversion via powering
by nisse＠lysator.liu.se 09 Nov '20

09 Nov '20

Hi, Nettle includes a function for side-channel silent modular inversion, which asymptotically is O(n^2), like binary gcd but slower by a pretty large constant factor. For prime moduli, inversion can also be done via a^{-1} = a^{p-2} (mod p). That's asymptotically O(n^3) (if the underlying multiplies are done with the basic O(n^2) algorithm), but it may nevertheless be faster than the other method for numbers of the size used for Nettle's elliptic curves. The code for curve25519 and curve448 has been using powering to invert for a long time. I've now spent some time writing specific powering code for the five secp curves as well. I've found fairly efficient addition chains where powering for a prime of n bits needs n-1 squarings and about a dozen multiplies. (I don't know what the *optimal* addition chains are, if you know of tools for that, let me know). Code is on the branch optimize-ecc-invert. However, there's some risk the new code is slower on some platforms, in particular platforms with slow multiplication. The main benchmark is ./examples/hogweed-benchmark ecdsa To get numbers for just the changed function, one can run ./examples-ecc-benchmark and look at the modinv column. Please compare performance between this branch and master, on the platforms that are important to you. And to get relevant numbers, make sure to build using a recent GMP library; performance when built with mini-gmp is not so important. I will merge these changes to master in a week or two, if no problems show up I've benchmarked on one recent and one older x86_64 machine, and on raspberry-pi version 1 (the slowest ARM machine I had easy access to). I've seen improvements in ecdsa signing performance for all the secp curves, ranging from 5% up to 20% depending on curve and platform. Not all inversions are rewritten. I haven't changed the modq inversion which is needed for ecdsa, and I haven't changed the code for the two supported gost curves. Regards, /Niels -- Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677. Internet email is subject to wholesale government surveillance.

2 1

[PATCH] "PowerPC64" chacha-core big-endian support "Shorter version"
by Maamoun TK 07 Nov '20

07 Nov '20

The last patch follows the C implementation but I just figured out a decent way to do it. --- powerpc64/p7/chacha-core-internal.asm | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/powerpc64/p7/chacha-core-internal.asm b/powerpc64/p7/chacha-core-internal.asm index 33c721c1..76ca0d45 100644 --- a/powerpc64/p7/chacha-core-internal.asm +++ b/powerpc64/p7/chacha-core-internal.asm @@ -53,6 +53,10 @@ define(`S1', `v9') define(`S2', `v10') define(`S3', `v11') +C Big-endian working state +define(`LE_MASK', `v12') +define(`LE_TEMP', `v13') + C QROUND(X0, X1, X2, X3) define(`QROUND', ` C x0 += x1, x3 ^= x0, x3 lrot 16 @@ -77,10 +81,18 @@ define(`QROUND', ` vrlw $2, $2, ROT7 ') +C LE_SWAP32(X0, X1, X2, X3) +define(`LE_SWAP32', `IF_BE(` + vperm X0, X0, X0, LE_MASK + vperm X1, X1, X1, LE_MASK + vperm X2, X2, X2, LE_MASK + vperm X3, X3, X3, LE_MASK +')') + .text - .align 4 C _chacha_core(uint32_t *dst, const uint32_t *src, unsigned rounds) +define(`FUNC_ALIGN', `5') PROLOGUE(_nettle_chacha_core) li r6, 0x10 C set up some... @@ -91,6 +103,12 @@ PROLOGUE(_nettle_chacha_core) vspltisw ROT12, 12 vspltisw ROT8, 8 vspltisw ROT7, 7 +IF_BE(` + li r9, 0 + lvsl LE_MASK, r9, r9 + vspltisb LE_TEMP, 0x03 + vxor LE_MASK, LE_MASK, LE_TEMP +') lxvw4x VSR(X0), 0, SRC lxvw4x VSR(X1), r6, SRC @@ -131,6 +149,8 @@ PROLOGUE(_nettle_chacha_core) vadduwm X2, X2, S2 vadduwm X3, X3, S3 + LE_SWAP32(X0, X1, X2, X3) + stxvw4x VSR(X0), 0, DST stxvw4x VSR(X1), r6, DST stxvw4x VSR(X2), r7, DST -- 2.17.1

3 4

SHA1 Collision Detection
by Neal H. Walfield 04 Nov '20

04 Nov '20

Hi, It's well known that SHA-1 is broken. I don't want to save it. But, particularly when dealing with data at rest, there are cases where one has to use SHA-1. It would be nice if Nettle integrated SHA-1 collision detection to make that a tiny bit safer: https://github.com/cr-marcstevens/sha1collisiondetection That library is under the MIT license, and apparently detects known attacks against SHA-1: [The routines] will compute the SHA-1 hash of any given file and additionally will detect cryptanalytic collision attacks against SHA-1 present in each file. It is very fast and takes less than twice the amount of time as regular SHA-1. More specifically they will detect any cryptanalytic collision attack against SHA-1 using any of the top 32 SHA-1 disturbance vectors with probability 1: ... The possibility of false positives can be neglected as the probability is smaller than 2^-90. Thanks, :) Neal

4 8

Blowfish integer overshift (undefined behavior)
by Guido Vranken 29 Oct '20

29 Oct '20

Hi all, My project is Cryptofuzz (https://github.com/guidovranken/cryptofuzz) which uses differential fuzzing to find correctness bugs (and memory bugs as well) in popular cryptographic libraries. It has bindings for Nettle and it tests many of the library's features: https://github.com/guidovranken/cryptofuzz/blob/master/modules/nettle/modul… I've been running this on Google's OSS-Fuzz ( https://github.com/google/oss-fuzz) for a while, and today it found a bug in the blowfish key setter function. Niels and other maintainers (if any), if you would like to be notified by e-mail of bugs found by Cryptofuzz, please send me your e-mail address and I will add you to the project. The e-mail address needs to be linked to a Google account in order to access the dashboard at oss-fuzz.com. Bug reproducer below: ------- #include <nettle/blowfish.h> int main(void) { const unsigned char key[] = {0xec, 0x00, 0x3a, 0x06, 0x73, 0x61, 0x74, 0x20, 0x74, 0xab, 0xe2, 0xc6, 0x61, 0x8b, 0x98, 0x89}; struct blowfish_ctx ctx; blowfish_set_key(&ctx, sizeof(key), key); return 0; } ------- If you compile with Clang and -fsanitize=undefined, this will print: blowfish.c:388:22: runtime error: left shift of 236 by 24 places cannot be represented in type 'int' Explicit casting around the shifted values will fix this.

2 1

[PATCH] "PowerPC64" GCM support
by Maamoun TK 11 Oct '20

11 Oct '20

--- configure.ac | 6 +- gcm.c | 49 +++- powerpc64/p8/gcm-hash.asm | 607 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 647 insertions(+), 15 deletions(-) create mode 100644 powerpc64/p8/gcm-hash.asm diff --git a/configure.ac b/configure.ac index e9983697..0129f950 100644 --- a/configure.ac +++ b/configure.ac @@ -488,7 +488,7 @@ asm_replace_list="aes-encrypt-internal.asm aes-decrypt-internal.asm \ sha3-permute.asm umac-nh.asm umac-nh-n.asm machine.m4" # Assembler files which generate additional object files if they are used. -asm_nettle_optional_list="gcm-hash8.asm cpuid.asm \ +asm_nettle_optional_list="gcm-hash.asm gcm-hash8.asm cpuid.asm \ aes-encrypt-internal-2.asm aes-decrypt-internal-2.asm memxor-2.asm \ chacha-3core.asm chacha-core-internal-2.asm salsa20-2core.asm \ salsa20-core-internal-2.asm sha1-compress-2.asm sha256-compress-2.asm \ @@ -612,9 +612,9 @@ AH_VERBATIM([HAVE_NATIVE], #undef HAVE_NATIVE_ecc_secp384r1_redc #undef HAVE_NATIVE_ecc_secp521r1_modp #undef HAVE_NATIVE_ecc_secp521r1_redc -#undef HAVE_NATIVE_gcm_init_key8 +#undef HAVE_NATIVE_gcm_init_key +#undef HAVE_NATIVE_gcm_hash #undef HAVE_NATIVE_gcm_hash8 -#undef HAVE_NATIVE_gcm_fill #undef HAVE_NATIVE_salsa20_core #undef HAVE_NATIVE_salsa20_2core #undef HAVE_NATIVE_fat_salsa20_2core diff --git a/gcm.c b/gcm.c index 48b3e75a..81981c1c 100644 --- a/gcm.c +++ b/gcm.c @@ -140,6 +140,19 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) memcpy (x->b, Z.b, sizeof(Z)); } # elif GCM_TABLE_BITS == 8 +# if HAVE_NATIVE_gcm_init_key + +#define gcm_init_key _nettle_gcm_init_key +void +_nettle_gcm_init_key (union nettle_block16 *table); +# endif /* HAVE_NATIVE_gcm_init_key */ +# if HAVE_NATIVE_gcm_hash + +#define gcm_hash _nettle_gcm_hash +void +_nettle_gcm_hash (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data); +# endif /* HAVE_NATIVE_gcm_hash */ # if HAVE_NATIVE_gcm_hash8 #define gcm_hash _nettle_gcm_hash8 @@ -228,6 +241,29 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) /* Increment the rightmost 32 bits. */ #define INC32(block) INCREMENT(4, (block.b) + GCM_BLOCK_SIZE - 4) +#ifndef gcm_init_key +static void +gcm_init_key(union nettle_block16 *table) +{ +#if GCM_TABLE_BITS + /* Middle element if GCM_TABLE_BITS > 0, otherwise the first + element */ + unsigned i = (1<<GCM_TABLE_BITS)/2; + + /* Algorithm 3 from the gcm paper. First do powers of two, then do + the rest by adding. */ + while (i /= 2) + block16_mulx_ghash(&table[i], &table[2*i]); + for (i = 2; i < 1<<GCM_TABLE_BITS; i *= 2) + { + unsigned j; + for (j = 1; j < i; j++) + block16_xor3(&table[i+j], &table[i], &table[j]); + } +#endif +} +#endif /* !gcm_init_key */ + /* Initialization of GCM. * @ctx: The context of GCM * @cipher: The context of the underlying block cipher @@ -245,18 +281,7 @@ gcm_set_key(struct gcm_key *key, memset(key->h[0].b, 0, GCM_BLOCK_SIZE); f (cipher, GCM_BLOCK_SIZE, key->h[i].b, key->h[0].b); -#if GCM_TABLE_BITS - /* Algorithm 3 from the gcm paper. First do powers of two, then do - the rest by adding. */ - while (i /= 2) - block16_mulx_ghash(&key->h[i], &key->h[2*i]); - for (i = 2; i < 1<<GCM_TABLE_BITS; i *= 2) - { - unsigned j; - for (j = 1; j < i; j++) - block16_xor3(&key->h[i+j], &key->h[i],&key->h[j]); - } -#endif + gcm_init_key(key->h); } #ifndef gcm_hash diff --git a/powerpc64/p8/gcm-hash.asm b/powerpc64/p8/gcm-hash.asm new file mode 100644 index 00000000..540c9f97 --- /dev/null +++ b/powerpc64/p8/gcm-hash.asm @@ -0,0 +1,607 @@ +C powerpc64/p8/gcm-hash.asm + +ifelse(` + Copyright (D) 2020 Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +') + +C Alignment of gcm_key table elements, which is declared in gcm.h +define(`TableElemAlign', `0x100') + +C Register usage: + +define(`SP', `r1') +define(`TOCP', `r2') + +define(`TABLE', `r3') + +define(`ZERO', `v0') +define(`B1', `v1') +define(`EMSB', `v2') +define(`POLY', `v7') +define(`POLY_L', `v1') +define(`POLY_H', `v2') + +define(`H', `v3') +define(`H3', `v3') +define(`Hl', `v4') +define(`Hm', `v5') +define(`Hh', `v6') +define(`RP', `v7') +define(`Mh', `v8') +define(`Ml', `v9') +define(`H2', `v7') +define(`H4', `v7') +define(`H2m', `v8') +define(`H2l', `v9') +define(`H2h', `v10') +define(`RP2', `v11') +define(`M2h', `v12') +define(`M2l', `v13') + +define(`H2_l', `v14') +define(`H2_m', `v15') +define(`H2_h', `v16') +define(`H3_l', `v14') +define(`H3_m', `v15') +define(`H3_h', `v16') +define(`H4_l', `v17') +define(`H4_m', `v18') +define(`H4_h', `v19') + +define(`H21l', `v16') +define(`H21h', `v17') +define(`H3m', `v16') +define(`H4m', `v17') +define(`H43l', `v18') +define(`H43h', `v19') + +define(`LE_TEMP', `v18') +define(`LE_MASK', `v19') + +.file "gcm-hash.asm" + +.text + + C void gcm_init_key (union gcm_block *table) + +C This function populates the gcm table as the following layout +C ******************************************************************** +C | Hm = low-order doubleword of H^1:high-order doubleword of H^1 | +C | Hl = 64-bits zeros:low-order doubleword of H^1 | +C | Hh = high-order doubleword of H^1:64-bits zeros | +C | | +C | H2m = low-order doubleword of H^2:high-order doubleword of H^2 | +C | H21l = low-order doubleword of H^2:low-order doubleword of H^1 | +C | H21h = high-order doubleword of H^2:high-order doubleword of H^1 | +C | | +C | H3m = low-order doubleword of H^3:high-order doubleword of H^3 | +C | H4m = low-order doubleword of H^4:high-order doubleword of H^4 | +C | H43l = low-order doubleword of H^4:low-order doubleword of H^3 | +C | H43h = high-order doubleword of H^4:high-order doubleword of H^3 | +C ******************************************************************** + +define(`FUNC_ALIGN', `5') +PROLOGUE(_nettle_gcm_init_key) + DATA_LOAD_VEC(POLY,.polynomial,r7) C 0xC2000000000000000000000000000001 +IF_LE(` + li r8,0 + lvsl LE_MASK,0,r8 C 0x000102030405060708090A0B0C0D0E0F + vspltisb LE_TEMP,0x07 C 0x07070707070707070707070707070707 + vxor LE_MASK,LE_MASK,LE_TEMP C 0x07060504030201000F0E0D0C0B0A0908 +') + + C 'H' is assigned by gcm_set_key() to the middle element of the table + li r10,8*TableElemAlign + lxvd2x VSR(H),r10,TABLE C load 'H' + C byte-reverse of each doubleword permuting on little-endian mode +IF_LE(` + vperm H,H,H,LE_MASK +') + + C --- calculate [H = H << 1 modulo polynomial] --- + + vupkhsb EMSB,H C extend most significant bit to first byte + vspltisb B1,1 C 0x01010101010101010101010101010101 + vspltb EMSB,EMSB,0 C first byte quadword-extend + vsl H,H,B1 C H = H << 1 + vand EMSB,EMSB,POLY C EMSB &= 0xC2000000000000000000000000000001 + vxor ZERO,ZERO,ZERO C 0x00000000000000000000000000000000 + vxor H,H,EMSB C H ^= EMSB + + C calculate [Hl = 0:H^1l, Hm = H^1l:H^1h, Hh = H^1h:0] + xxmrgld VSR(Hl),VSR(ZERO),VSR(H) + xxswapd VSR(Hm),VSR(H) + xxmrghd VSR(Hh),VSR(H),VSR(ZERO) + + C --- calculate H^2 = H*H --- + + C reduction pre-processing + xxmrghd VSR(POLY_H),VSR(POLY),VSR(ZERO) C 0xC2000000000000000000000000000000 + xxmrghd VSR(POLY_L),VSR(ZERO),VSR(POLY) C 0x0000000000000000C200000000000000 + + C polynomial multiplication "classical" + vpmsumd H2_l,H,Hl C H^1l*H^1l + vpmsumd H2_m,H,Hm C H^1h*H^1l⊕H^1l*H^1h + vpmsumd H2_h,H,Hh C H^1h*H^1h + + C reduction first phase [1] + vpmsumd RP,H2_l,POLY_L C [1] + + C polynomial multiplication post-processing [2] + xxmrghd VSR(Mh),VSR(ZERO),VSR(H2_m) C [2] + xxmrgld VSR(Ml),VSR(H2_m),VSR(ZERO) C [2] + xxswapd VSR(RP),VSR(RP) C [1] + vxor H2_h,H2_h,Mh C [2] + vxor H2_l,H2_l,Ml C [2] + vxor H2_l,H2_l,RP C [1] + + C reduction second phase + vpmsumd RP,H2_l,POLY_H + vxor H2_h,H2_h,H2_l + vxor H2,H2_h,RP + + C store [H2m = H^2l:H^2h, H2l = 0:H^2l, H2h = H^2h:0] + xxswapd VSR(H2m),VSR(H2) + xxmrgld VSR(H2l),VSR(ZERO),VSR(H2) + xxmrghd VSR(H2h),VSR(H2),VSR(ZERO) + + C calculate [H21l = H^2l:H^1l, H21h = H^2h:H^1h] + xxmrgld VSR(H21l),VSR(H2),VSR(H) + xxmrghd VSR(H21h),VSR(H2),VSR(H) + + C store [Hm, Hl, Hh] + li r9,1*TableElemAlign + li r10,2*TableElemAlign + stxvd2x VSR(Hm),0,TABLE + stxvd2x VSR(Hl),r9,TABLE + stxvd2x VSR(Hh),r10,TABLE + + C store [H2m, H21l, H21h] + li r8,3*TableElemAlign + li r9,4*TableElemAlign + li r10,5*TableElemAlign + stxvd2x VSR(H2m),r8,TABLE + stxvd2x VSR(H21l),r9,TABLE + stxvd2x VSR(H21h),r10,TABLE + + C --- calculate H^3 = H^1*H^2, H^4 = H^2*H^2 --- + + C polynomial multiplication "classical" + vpmsumd H3_l,H,H2l C H^1l*H^2l + vpmsumd H4_l,H2,H2l C H^2l*H^2l + vpmsumd H3_m,H,H2m C H^1h*H^2l⊕H^1l*H^2h + vpmsumd H4_m,H2,H2m C H^2h*H^2l⊕H^2l*H^2h + vpmsumd H3_h,H,H2h C H^1h*H^2h + vpmsumd H4_h,H2,H2h C H^2h*H^2h + + C reduction first phase [1] + vpmsumd RP,H3_l,POLY_L C [1] H^3 + vpmsumd RP2,H4_l,POLY_L C [1] H^4 + + C polynomial multiplication post-processing [2] + xxmrghd VSR(Mh),VSR(ZERO),VSR(H3_m) C [2] H^3 + xxmrghd VSR(M2h),VSR(ZERO),VSR(H4_m) C [2] H^4 + xxmrgld VSR(Ml),VSR(H3_m),VSR(ZERO) C [2] H^3 + xxmrgld VSR(M2l),VSR(H4_m),VSR(ZERO) C [2] H^4 + xxswapd VSR(RP),VSR(RP) C [1] H^3 + xxswapd VSR(RP2),VSR(RP2) C [1] H^4 + vxor H3_h,H3_h,Mh C [2] H^3 + vxor H4_h,H4_h,M2h C [2] H^4 + vxor H3_l,H3_l,Ml C [2] H^3 + vxor H4_l,H4_l,M2l C [2] H^4 + vxor H3_l,H3_l,RP C [1] H^3 + vxor H4_l,H4_l,RP2 C [1] H^4 + + C reduction second phase + vpmsumd RP,H3_l,POLY_H C H^3 + vpmsumd RP2,H4_l,POLY_H C H^4 + vxor H3_h,H3_h,H3_l C H^3 + vxor H4_h,H4_h,H4_l C H^4 + vxor H3,H3_h,RP C H^3 + vxor H4,H4_h,RP2 C H^4 + + C calculate [H3m = H^3l:H^3h, H4m = H^4l:H^4h, H43l = H^4l:H^3l, H43h = H^4h:H^3h] + xxswapd VSR(H3m),VSR(H3) + xxswapd VSR(H4m),VSR(H4) + xxmrgld VSR(H43l),VSR(H4),VSR(H3) + xxmrghd VSR(H43h),VSR(H4),VSR(H3) + + C store [H3m, H4m, H43l, H43h] + li r7,6*TableElemAlign + li r8,7*TableElemAlign + li r9,8*TableElemAlign + li r10,9*TableElemAlign + stxvd2x VSR(H3m),r7,TABLE + stxvd2x VSR(H4m),r8,TABLE + stxvd2x VSR(H43l),r9,TABLE + stxvd2x VSR(H43h),r10,TABLE + + blr +EPILOGUE(_nettle_gcm_init_key) + +define(`TABLE', `r3') +define(`X', `r4') +define(`LENGTH', `r5') +define(`DATA', `r6') + +define(`ZERO', `v0') +define(`POLY', `v3') +define(`POLY_L', `v1') +define(`POLY_H', `v2') + +define(`D', `v3') +define(`C0', `v4') +define(`C1', `v5') +define(`C2', `v6') +define(`C3', `v7') +define(`Mh', `v8') +define(`Ml', `v9') +define(`RP', `v10') +define(`C01h', `v11') +define(`C01l', `v12') +define(`C23h', `v13') +define(`C23l', `v14') + +define(`H1', `v15') +define(`H2', `v16') +define(`H21l', `v17') +define(`H21h', `v18') +define(`H3', `v20') +define(`H4', `v21') +define(`H43l', `v22') +define(`H43h', `v23') + +define(`Cl', `v5') +define(`Cm', `v6') +define(`Ch', `v7') +define(`Hl', `v15') +define(`H', `v16') +define(`Hh', `v17') + +define(`LE_TEMP', `v18') +define(`LE_MASK', `v19') + + C void gcm_hash (const struct gcm_key *key, union gcm_block *x, + C size_t length, const uint8_t *data) + +define(`FUNC_ALIGN', `5') +PROLOGUE(_nettle_gcm_hash) + DATA_LOAD_VEC(POLY,.polynomial,r7) +IF_LE(` + li r8,0 + lvsl LE_MASK,0,r8 + vspltisb LE_TEMP,0x07 + vxor LE_MASK,LE_MASK,LE_TEMP +') + vxor ZERO,ZERO,ZERO + + xxmrghd VSR(POLY_L),VSR(ZERO),VSR(POLY) + xxmrghd VSR(POLY_H),VSR(POLY),VSR(ZERO) + + lxvd2x VSR(D),0,X C load 'X' pointer + C byte-reverse of each doubleword permuting on little-endian mode +IF_LE(` + vperm D,D,D,LE_MASK +') + + C --- process 4 blocks '128-bit each' per one loop --- + + srdi r7,LENGTH,6 C 4-blocks loop count 'LENGTH / (4 * 16)' + cmpldi r7,0 + beq L2x + + mtctr r7 C assign counter register to loop count + + C backup non-volatile vector registers + addi r8,SP,-64 + stvx 20,0,r8 + addi r8,r8,16 + stvx 21,0,r8 + addi r8,r8,16 + stvx 22,0,r8 + addi r8,r8,16 + stvx 23,0,r8 + + C load table elements + li r8,3*TableElemAlign + li r9,4*TableElemAlign + li r10,5*TableElemAlign + lxvd2x VSR(H1),0,TABLE + lxvd2x VSR(H2),r8,TABLE + lxvd2x VSR(H21l),r9,TABLE + lxvd2x VSR(H21h),r10,TABLE + li r7,6*TableElemAlign + li r8,7*TableElemAlign + li r9,8*TableElemAlign + li r10,9*TableElemAlign + lxvd2x VSR(H3),r7,TABLE + lxvd2x VSR(H4),r8,TABLE + lxvd2x VSR(H43l),r9,TABLE + lxvd2x VSR(H43h),r10,TABLE + + li r8,0x10 + li r9,0x20 + li r10,0x30 +.align 5 +L4x_loop: + C input loading + lxvd2x VSR(C0),0,DATA C load C0 + lxvd2x VSR(C1),r8,DATA C load C1 + lxvd2x VSR(C2),r9,DATA C load C2 + lxvd2x VSR(C3),r10,DATA C load C3 + +IF_LE(` + vperm C0,C0,C0,LE_MASK + vperm C1,C1,C1,LE_MASK + vperm C2,C2,C2,LE_MASK + vperm C3,C3,C3,LE_MASK +') + + C previous digest combining + vxor C0,C0,D + + C polynomial multiplication "classical" pre-processing + xxmrghd VSR(C23h),VSR(C2),VSR(C3) + xxmrgld VSR(C23l),VSR(C2),VSR(C3) + xxmrghd VSR(C01h),VSR(C0),VSR(C1) + xxmrgld VSR(C01l),VSR(C0),VSR(C1) + + C polynomial multiplication "classical" + vpmsumd C3,C3,H1 C M3 = H^1l*C3h⊕H^1h*C3l + vpmsumd C2,C2,H2 C M2 = H^2l*C2h⊕H^2h*C2l + vpmsumd C1,C1,H3 C M1 = H^3l*C1h⊕H^3h*C1l + vpmsumd C0,C0,H4 C M0 = H^4l*C0h⊕H^4h*C0l + vpmsumd C23h,C23h,H21h C H23 = H^2h*C2h⊕H^1h*C3h + vpmsumd C23l,C23l,H21l C L23 = H^2l*C2l⊕H^1l*C3l + vpmsumd C01h,C01h,H43h C H01 = H^4h*C0h⊕H^4h*C1h + vpmsumd C01l,C01l,H43l C L01 = H^4l*C0l⊕H^4l*C1l + + C polynomial multiplication "classical" post-processing + vxor C2,C2,C3 C M2 = M2⊕M3 + vxor C0,C0,C1 C M0 = M0⊕M1 + + C deferred recombination of partial products + vxor C01h,C01h,C23h C H0 = H01⊕H23 + vxor C01l,C01l,C23l C L0 = L01⊕L23 + vxor C0,C0,C2 C M0 = M0⊕M2 + + C reduction first phase [1] + vpmsumd RP,C01l,POLY_L C [1] + + C polynomial multiplication post-processing [2] + xxmrghd VSR(Mh),VSR(ZERO),VSR(C0) C [2] + xxmrgld VSR(Ml),VSR(C0),VSR(ZERO) C [2] + xxswapd VSR(RP),VSR(RP) C [1] + vxor C01h,C01h,Mh C [2] + vxor C01l,C01l,Ml C [2] + vxor C01l,C01l,RP C [1] + + C reduction second phase + vpmsumd RP,C01l,POLY_H + vxor C01h,C01l,C01h + vxor D,C01h,RP + + addi DATA,DATA,0x40 + bdnz L4x_loop + + C restore non-volatile vector registers + addi r8,SP,-64 + lvx 20,0,r8 + addi r8,r8,16 + lvx 21,0,r8 + addi r8,r8,16 + lvx 22,0,r8 + addi r8,r8,16 + lvx 23,0,r8 + + clrldi LENGTH,LENGTH,58 C 'set the high-order 58 bits to zeros' +L2x: + C --- process 2 blocks --- + + srdi r7,LENGTH,5 C 'LENGTH / (2 * 16)' + cmpldi r7,0 + beq L1x + + C load table elements + li r8,3*TableElemAlign + li r9,4*TableElemAlign + li r10,5*TableElemAlign + lxvd2x VSR(H1),0,TABLE + lxvd2x VSR(H2),r8,TABLE + lxvd2x VSR(H21l),r9,TABLE + lxvd2x VSR(H21h),r10,TABLE + + C input loading + li r10,0x10 + lxvd2x VSR(C0),0,DATA C load C0 + lxvd2x VSR(C1),r10,DATA C load C1 + +IF_LE(` + vperm C0,C0,C0,LE_MASK + vperm C1,C1,C1,LE_MASK +') + + C previous digest combining + vxor C0,C0,D + + C polynomial multiplication "classical" pre-processing + xxmrghd VSR(C01h),VSR(C0),VSR(C1) + xxmrgld VSR(C01l),VSR(C0),VSR(C1) + + C polynomial multiplication "classical" + vpmsumd C1,C1,H1 C M1 = H^1l*C1h⊕H^1h*C1l + vpmsumd C0,C0,H2 C M0 = H^2l*C0h⊕H^2h*C0l + vpmsumd C01h,C01h,H21h C H01 = H^2h*C0h⊕H^1h*C1h + vpmsumd C01l,C01l,H21l C L01 = H^2l*C0l⊕H^1l*C1l + + C deferred recombination of partial products + vxor C0,C0,C1 C M0 = M0⊕M1 + + C reduction first phase [1] + vpmsumd RP,C01l,POLY_L C [1] + + C polynomial multiplication post-processing [2] + xxmrghd VSR(Mh),VSR(ZERO),VSR(C0) C [2] + xxmrgld VSR(Ml),VSR(C0),VSR(ZERO) C [2] + xxswapd VSR(RP),VSR(RP) C [1] + vxor C01h,C01h,Mh C [2] + vxor C01l,C01l,Ml C [2] + vxor C01l,C01l,RP C [1] + + C reduction second phase + vpmsumd RP,C01l,POLY_H + vxor C01h,C01l,C01h + vxor D,C01h,RP + + addi DATA,DATA,0x20 + clrldi LENGTH,LENGTH,59 C 'set the high-order 59 bits to zeros' +L1x: + C --- process 1 block --- + + srdi r7,LENGTH,4 C 'LENGTH / (1 * 16)' + cmpldi r7,0 + beq Lmod + + C load table elements + li r9,1*TableElemAlign + li r10,2*TableElemAlign + lxvd2x VSR(H),0,TABLE + lxvd2x VSR(Hl),r9,TABLE + lxvd2x VSR(Hh),r10,TABLE + + C input loading + lxvd2x VSR(C0),0,DATA C load C0 + +IF_LE(` + vperm C0,C0,C0,LE_MASK +') + + C previous digest combining + vxor C0,C0,D + + C polynomial multiplication "classical" + vpmsumd Cl,C0,Hl C L = Hl*Cl + vpmsumd Cm,C0,H C M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh C H = Hh*Ch + + C reduction first phase C [1] + vpmsumd RP,Cl,POLY_L C [1] + + C polynomial multiplication post-processing [2] + xxmrghd VSR(Mh),VSR(ZERO),VSR(Cm) C [2] + xxmrgld VSR(Ml),VSR(Cm),VSR(ZERO) C [2] + xxswapd VSR(RP),VSR(RP) C [1] + vxor Ch,Ch,Mh C [2] + vxor Cl,Cl,Ml C [2] + vxor Cl,Cl,RP C [1] + + C reduction second phase + vpmsumd RP,Cl,POLY_H + vxor Ch,Cl,Ch + vxor D,Ch,RP + + addi DATA,DATA,0x10 + clrldi LENGTH,LENGTH,60 C 'set the high-order 60 bits to zeros' +Lmod: + C --- process the modulo bytes, padding the low-order bytes with zeros --- + + cmpldi LENGTH,0 + beq Ldone + + C load table elements + li r9,1*TableElemAlign + li r10,2*TableElemAlign + lxvd2x VSR(H),0,TABLE + lxvd2x VSR(Hl),r9,TABLE + lxvd2x VSR(Hh),r10,TABLE + + C push every modulo byte to the stack and load them with padding into vector register + addi r8,SP,-16 + stvx ZERO,0,r8 +Lstb_loop: + subic. LENGTH,LENGTH,1 + lbzx r7,LENGTH,DATA + stbx r7,LENGTH,r8 + bne Lstb_loop + lxvd2x VSR(C0),0,r8 + +IF_LE(` + vperm C0,C0,C0,LE_MASK +') + + C previous digest combining + vxor C0,C0,D + + C polynomial multiplication "classical" + vpmsumd Cl,C0,Hl C L = Hl*Cl + vpmsumd Cm,C0,H C M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh C H = Hh*Ch + + C reduction first phase [1] + vpmsumd RP,Cl,POLY_L C [1] + + C polynomial multiplication post-processing [2] + xxmrghd VSR(Mh),VSR(ZERO),VSR(Cm) C [2] + xxmrgld VSR(Ml),VSR(Cm),VSR(ZERO) C [2] + xxswapd VSR(RP),VSR(RP) C [1] + vxor Ch,Ch,Mh C [2] + vxor Cl,Cl,Ml C [2] + vxor Cl,Cl,RP C [1] + + C reduction second phase + vpmsumd RP,Cl,POLY_H + vxor Ch,Cl,Ch + vxor D,Ch,RP + +Ldone: + C byte-reverse of each doubleword permuting on little-endian mode +IF_LE(` + vperm D,D,D,LE_MASK +') + stxvd2x VSR(D),0,X C store digest 'D' + + blr +EPILOGUE(_nettle_gcm_hash) + +.data + C 0xC2000000000000000000000000000001 +.polynomial: +.align 4 +IF_BE(` +.byte 0xC2 +.rept 14 +.byte 0x00 +.endr +.byte 0x01 +',` +.byte 0x01 +.rept 14 +.byte 0x00 +.endr +.byte 0xC2 +') -- 2.17.1

3 11

[PATCH] "PowerPC64" GCM support
by Maamoun TK 04 Oct '20

04 Oct '20

--- configure.ac | 6 +- fat-ppc.c | 33 ++ fat-setup.h | 7 + gcm.c | 69 +++- powerpc64/fat/gcm-hash.asm | 39 +++ powerpc64/p8/gcm-hash.asm | 773 +++++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 909 insertions(+), 18 deletions(-) create mode 100644 powerpc64/fat/gcm-hash.asm create mode 100644 powerpc64/p8/gcm-hash.asm diff --git a/configure.ac b/configure.ac index e9983697..0129f950 100644 --- a/configure.ac +++ b/configure.ac @@ -488,7 +488,7 @@ asm_replace_list="aes-encrypt-internal.asm aes-decrypt-internal.asm \ sha3-permute.asm umac-nh.asm umac-nh-n.asm machine.m4" # Assembler files which generate additional object files if they are used. -asm_nettle_optional_list="gcm-hash8.asm cpuid.asm \ +asm_nettle_optional_list="gcm-hash.asm gcm-hash8.asm cpuid.asm \ aes-encrypt-internal-2.asm aes-decrypt-internal-2.asm memxor-2.asm \ chacha-3core.asm chacha-core-internal-2.asm salsa20-2core.asm \ salsa20-core-internal-2.asm sha1-compress-2.asm sha256-compress-2.asm \ @@ -612,9 +612,9 @@ AH_VERBATIM([HAVE_NATIVE], #undef HAVE_NATIVE_ecc_secp384r1_redc #undef HAVE_NATIVE_ecc_secp521r1_modp #undef HAVE_NATIVE_ecc_secp521r1_redc -#undef HAVE_NATIVE_gcm_init_key8 +#undef HAVE_NATIVE_gcm_init_key +#undef HAVE_NATIVE_gcm_hash #undef HAVE_NATIVE_gcm_hash8 -#undef HAVE_NATIVE_gcm_fill #undef HAVE_NATIVE_salsa20_core #undef HAVE_NATIVE_salsa20_2core #undef HAVE_NATIVE_fat_salsa20_2core diff --git a/fat-ppc.c b/fat-ppc.c index 2bc50481..d73b45b0 100644 --- a/fat-ppc.c +++ b/fat-ppc.c @@ -120,6 +120,16 @@ DECLARE_FAT_FUNC(_nettle_aes_decrypt, aes_crypt_internal_func) DECLARE_FAT_FUNC_VAR(aes_decrypt, aes_crypt_internal_func, c) DECLARE_FAT_FUNC_VAR(aes_decrypt, aes_crypt_internal_func, ppc64) +#if GCM_TABLE_BITS == 8 +DECLARE_FAT_FUNC(_nettle_gcm_init_key, gcm_init_key_func) +DECLARE_FAT_FUNC_VAR(gcm_init_key, gcm_init_key_func, c) +DECLARE_FAT_FUNC_VAR(gcm_init_key, gcm_init_key_func, ppc64) + +DECLARE_FAT_FUNC(_nettle_gcm_hash, gcm_hash_func) +DECLARE_FAT_FUNC_VAR(gcm_hash, gcm_hash_func, c) +DECLARE_FAT_FUNC_VAR(gcm_hash, gcm_hash_func, ppc64) +#endif /* GCM_TABLE_BITS == 8 */ + static void CONSTRUCTOR fat_init (void) { @@ -139,11 +149,23 @@ fat_init (void) fprintf (stderr, "libnettle: enabling arch 2.07 code.\n"); _nettle_aes_encrypt_vec = _nettle_aes_encrypt_ppc64; _nettle_aes_decrypt_vec = _nettle_aes_decrypt_ppc64; +#if GCM_TABLE_BITS == 8 + /* Make sure _nettle_gcm_init_key_vec function is compatible + with _nettle_gcm_hash_vec function e.g. _nettle_gcm_init_key_c() + fills gcm_key table with values that are incompatible with + _nettle_gcm_hash_ppc64() */ + _nettle_gcm_init_key_vec = _nettle_gcm_init_key_ppc64; + _nettle_gcm_hash_vec = _nettle_gcm_hash_ppc64; +#endif /* GCM_TABLE_BITS == 8 */ } else { _nettle_aes_encrypt_vec = _nettle_aes_encrypt_c; _nettle_aes_decrypt_vec = _nettle_aes_decrypt_c; +#if GCM_TABLE_BITS == 8 + _nettle_gcm_init_key_vec = _nettle_gcm_init_key_c; + _nettle_gcm_hash_vec = _nettle_gcm_hash_c; +#endif /* GCM_TABLE_BITS == 8 */ } } @@ -160,3 +182,14 @@ DEFINE_FAT_FUNC(_nettle_aes_decrypt, void, size_t length, uint8_t *dst, const uint8_t *src), (rounds, keys, T, length, dst, src)) + +#if GCM_TABLE_BITS == 8 +DEFINE_FAT_FUNC(_nettle_gcm_init_key, void, + (union nettle_block16 *table), + (table)) + +DEFINE_FAT_FUNC(_nettle_gcm_hash, void, + (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data), + (key, x, length, data)) +#endif /* GCM_TABLE_BITS == 8 */ diff --git a/fat-setup.h b/fat-setup.h index 99f1ea67..4a9f7969 100644 --- a/fat-setup.h +++ b/fat-setup.h @@ -162,6 +162,13 @@ typedef void aes_crypt_internal_func (unsigned rounds, const uint32_t *keys, size_t length, uint8_t *dst, const uint8_t *src); +#if GCM_TABLE_BITS == 8 +typedef void gcm_init_key_func (union nettle_block16 *table); + +typedef void gcm_hash_func (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data); +#endif /* GCM_TABLE_BITS == 8 */ + typedef void *(memxor_func)(void *dst, const void *src, size_t n); typedef void salsa20_core_func (uint32_t *dst, const uint32_t *src, unsigned rounds); diff --git a/gcm.c b/gcm.c index 48b3e75a..787c303e 100644 --- a/gcm.c +++ b/gcm.c @@ -140,6 +140,26 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) memcpy (x->b, Z.b, sizeof(Z)); } # elif GCM_TABLE_BITS == 8 +# if HAVE_NATIVE_gcm_init_key + +#define gcm_init_key _nettle_gcm_init_key +void +_nettle_gcm_init_key (union nettle_block16 *table); +/* For fat builds */ +void +_nettle_gcm_init_key_c (union nettle_block16 *table); +# endif /* HAVE_NATIVE_gcm_init_key */ +# if HAVE_NATIVE_gcm_hash + +#define gcm_hash _nettle_gcm_hash +void +_nettle_gcm_hash (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data); +/* For fat builds */ +void +_nettle_gcm_hash_c (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data); +# endif /* HAVE_NATIVE_gcm_hash */ # if HAVE_NATIVE_gcm_hash8 #define gcm_hash _nettle_gcm_hash8 @@ -228,6 +248,32 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) /* Increment the rightmost 32 bits. */ #define INC32(block) INCREMENT(4, (block.b) + GCM_BLOCK_SIZE - 4) +#ifdef gcm_init_key +void +_nettle_gcm_init_key_c(union nettle_block16 *table) +#else +static void +gcm_init_key(union nettle_block16 *table) +#endif /* !gcm_init_key */ +{ +#if GCM_TABLE_BITS + /* Middle element if GCM_TABLE_BITS > 0, otherwise the first + element */ + unsigned i = (1<<GCM_TABLE_BITS)/2; + + /* Algorithm 3 from the gcm paper. First do powers of two, then do + the rest by adding. */ + while (i /= 2) + block16_mulx_ghash(&table[i], &table[2*i]); + for (i = 2; i < 1<<GCM_TABLE_BITS; i *= 2) + { + unsigned j; + for (j = 1; j < i; j++) + block16_xor3(&table[i+j], &table[i], &table[j]); + } +#endif +} + /* Initialization of GCM. * @ctx: The context of GCM * @cipher: The context of the underlying block cipher @@ -244,25 +290,19 @@ gcm_set_key(struct gcm_key *key, /* H */ memset(key->h[0].b, 0, GCM_BLOCK_SIZE); f (cipher, GCM_BLOCK_SIZE, key->h[i].b, key->h[0].b); - -#if GCM_TABLE_BITS - /* Algorithm 3 from the gcm paper. First do powers of two, then do - the rest by adding. */ - while (i /= 2) - block16_mulx_ghash(&key->h[i], &key->h[2*i]); - for (i = 2; i < 1<<GCM_TABLE_BITS; i *= 2) - { - unsigned j; - for (j = 1; j < i; j++) - block16_xor3(&key->h[i+j], &key->h[i],&key->h[j]); - } -#endif + + gcm_init_key(key->h); } -#ifndef gcm_hash +#ifdef gcm_hash +void +_nettle_gcm_hash_c(const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data) +#else static void gcm_hash(const struct gcm_key *key, union nettle_block16 *x, size_t length, const uint8_t *data) +#endif /* !gcm_hash */ { for (; length >= GCM_BLOCK_SIZE; length -= GCM_BLOCK_SIZE, data += GCM_BLOCK_SIZE) @@ -276,7 +316,6 @@ gcm_hash(const struct gcm_key *key, union nettle_block16 *x, gcm_gf_mul (x, key->h); } } -#endif /* !gcm_hash */ static void gcm_hash_sizes(const struct gcm_key *key, union nettle_block16 *x, diff --git a/powerpc64/fat/gcm-hash.asm b/powerpc64/fat/gcm-hash.asm new file mode 100644 index 00000000..7fd77223 --- /dev/null +++ b/powerpc64/fat/gcm-hash.asm @@ -0,0 +1,39 @@ +C powerpc64/fat/gcm-hash.asm + + +ifelse(` + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +') + +dnl picked up by configure +dnl PROLOGUE(_nettle_gcm_init_key) +dnl PROLOGUE(_nettle_gcm_hash) + +define(`fat_transform', `$1_ppc64') +include_src(`powerpc64/p8/gcm-hash.asm') diff --git a/powerpc64/p8/gcm-hash.asm b/powerpc64/p8/gcm-hash.asm new file mode 100644 index 00000000..f987f2ca --- /dev/null +++ b/powerpc64/p8/gcm-hash.asm @@ -0,0 +1,773 @@ +C powerpc64/p8/gcm-hash.asm + +ifelse(` + Copyright (C) 2020 Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +') + +C Alignment of gcm_key table elements, which is declared in gcm.h +define(`TableElemAlign', `0x100') + +C Register usage: + +define(`SP', `r1') +define(`TOCP', `r2') + +define(`TABLE', `r3') +define(`X', `r4') +define(`LENGTH', `r5') +define(`DATA', `r6') + +define(`zero', `v0') +define(`swap_mask', `v1') +define(`hidw_mask', `v2') +define(`lodw_mask', `v3') +define(`poly', `v4') +define(`poly_h', `v4') +define(`poly_l', `v5') +define(`RP', `v6') +define(`Mh', `v7') +define(`Ml', `v8') +define(`H', `v9') +define(`Hh', `v10') +define(`Hl', `v11') +define(`RP2', `v9') +define(`M2h', `v10') +define(`M2l', `v11') + +define(`sl1', `v1') +define(`msb', `v5') +define(`H2', `v6') +define(`H2h', `v7') +define(`H2l', `v8') +define(`H_h', `v12') +define(`H_m', `v13') +define(`H_l', `v14') +define(`H_Hh', `v12') +define(`H_H', `v13') +define(`H_Hl', `v14') +define(`H_t', `v15') +define(`H2_h', `v16') +define(`H2_m', `v17') +define(`H2_l', `v18') +define(`H2_t', `v19') + +define(`C0', `v6') +define(`C1', `v7') +define(`C2', `v8') +define(`C3', `v12') +define(`C4', `v6') +define(`C5', `v7') +define(`C6', `v8') +define(`C7', `v12') + +define(`C', `v13') + +define(`Ch', `v14') +define(`Cl', `v15') +define(`Cm', `v16') + +define(`C01h', `v14') +define(`C01l', `v15') +define(`C01', `v16') +define(`C23h', `v17') +define(`C23l', `v18') +define(`C23', `v19') +define(`C45h', `v20') +define(`C45l', `v21') +define(`C45', `v22') +define(`C67h', `v6') +define(`C67l', `v7') +define(`C67', `v8') + +define(`H21', `v9') +define(`H21h', `v10') +define(`H21l', `v11') +define(`H43', `v23') +define(`H43h', `v24') +define(`H43l', `v25') +define(`H65', `v26') +define(`H65h', `v27') +define(`H65l', `v28') +define(`H87', `v29') +define(`H87h', `v30') +define(`H87l', `v31') + +.file "gcm-hash.asm" + +.text + + # void gcm_init_key (union gcm_block *table) + +define(`FUNC_ALIGN', `5') +PROLOGUE(_nettle_gcm_init_key) + DATA_LOAD_VEC(poly,.polynomial,r7) +IF_LE(`DATA_LOAD_VEC(swap_mask,.swap_mask,r7)') + DATA_LOAD_VEC(hidw_mask,.hidw_mask,r7) + DATA_LOAD_VEC(lodw_mask,.lodw_mask,r7) + + li r10,8*TableElemAlign + lxvd2x VSR(H),r10,TABLE # load H +IF_LE(`vperm H,H,H,swap_mask') + + # --- calculate H = H shift left 1 modulo polynomial --- + + vupkhsw msb,H # most significant bit word-extend + vspltisb sl1,1 # splat 1 for shift left + vspltw msb,msb,0 # most significant bit extend + vsl H,H,sl1 # H shift left 1 + vand msb,msb,poly + vxor zero,zero,zero + vxor H_t,H,msb + + vsldoi H,H_t,H_t,8 # doubleword swap + vsldoi Hh,H,zero,8 + vsldoi Hl,zero,H,8 + + # --- calculate H^2 = H*H --- + + # reduction pre-processing + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + # polynomial multiplication "classical" + vpmsumd H_h,H_t,Hh # H^1h*H^1h + vpmsumd H_l,H_t,Hl # H^1l*H^1l + vpmsumd H_m,H_t,H # H^1h*H^1l⊕H^1l*H^1h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] + vsldoi Ml,H_m,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor H_h,H_h,Mh # [2] + vxor H_l,H_l,Ml # [2] + vxor H_l,H_l,RP # [1] + + # reduction second phase + vpmsumd RP,H_l,poly_l + vxor H_h,H_l,H_h + vxor H2_t,H_h,RP + + vsldoi H2,H2_t,H2_t,8 + vsldoi H2h,H2,zero,8 + vsldoi H2l,zero,H2,8 + + # --- calculate [H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store H,[H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + li r8,0*TableElemAlign + li r9,1*TableElemAlign + li r10,2*TableElemAlign + stxvd2x VSR(Hl),r8,TABLE + stxvd2x VSR(H),r9,TABLE + stxvd2x VSR(Hh),r10,TABLE + + li r8,3*TableElemAlign + li r9,4*TableElemAlign + li r10,5*TableElemAlign + stxvd2x VSR(H_Hh),r8,TABLE + stxvd2x VSR(H_H),r9,TABLE + stxvd2x VSR(H_Hl),r10,TABLE + + # --- calculate H^3,H^4 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^2l + vpmsumd H_m,H_t,H2 # H^1h*H^2l⊕H^1l*H^2h + vpmsumd H_h,H_t,H2h # H^1h*H^2h + vpmsumd H2_l,H2_t,H2l # H^2l*H^2l + vpmsumd H2_m,H2_t,H2 # H^2h*H^2l⊕H^2l*H^2h + vpmsumd H2_h,H2_t,H2h # H^2h*H^2h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^3 + vpmsumd RP2,H2_l,poly_h # [1] H^4 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^3 + vsldoi M2h,zero,H2_m,8 # [2] H^4 + vsldoi Ml,H_m,zero,8 # [2] H^3 + vsldoi M2l,H2_m,zero,8 # [2] H^4 + vsldoi RP,RP,RP,8 # [1] H^3 + vsldoi RP2,RP2,RP2,8 # [1] H^4 + vxor H_h,H_h,Mh # [2] H^3 + vxor H2_h,H2_h,M2h # [2] H^4 + vxor H_l,H_l,Ml # [2] H^3 + vxor H2_l,H2_l,M2l # [2] H^4 + vxor H_l,H_l,RP # [1] H^3 + vxor H2_l,H2_l,RP2 # [1] H^4 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^3 + vpmsumd RP2,H2_l,poly_l # H^4 + vxor H_h,H_l,H_h # H^3 + vxor H2_h,H2_l,H2_h # H^4 + vxor H_h,H_h,RP # H^3 + vxor H2_h,H2_h,RP2 # H^4 + + vsldoi H2,H2_h,H2_h,8 # H^4 + vsldoi H,H_h,H_h,8 # H^3 + vsldoi H2l,zero,H2,8 # H^4 + vsldoi H2h,H2,zero,8 # H^4 + + # --- calculate [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + li r8,6*TableElemAlign + li r9,7*TableElemAlign + li r10,8*TableElemAlign + stxvd2x VSR(H_Hh),r8,TABLE + stxvd2x VSR(H_H),r9,TABLE + stxvd2x VSR(H_Hl),r10,TABLE + + # --- calculate H^5,H^6 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^4l + vpmsumd H_m,H_t,H2 # H^1h*H^4l⊕H^1l*H^4h + vpmsumd H_h,H_t,H2h # H^1h*H^4h + vpmsumd H2_l,H2_t,H2l # H^2l*H^4l + vpmsumd H2_m,H2_t,H2 # H^2h*H^4l⊕H^2l*H^4h + vpmsumd H2_h,H2_t,H2h # H^2h*H^4h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^5 + vpmsumd RP2,H2_l,poly_h # [1] H^6 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^5 + vsldoi M2h,zero,H2_m,8 # [2] H^6 + vsldoi Ml,H_m,zero,8 # [2] H^5 + vsldoi M2l,H2_m,zero,8 # [2] H^6 + vsldoi RP,RP,RP,8 # [1] H^5 + vsldoi RP2,RP2,RP2,8 # [1] H^6 + vxor H_h,H_h,Mh # [2] H^5 + vxor H2_h,H2_h,M2h # [2] H^6 + vxor H_l,H_l,Ml # [2] H^5 + vxor H2_l,H2_l,M2l # [2] H^6 + vxor H_l,H_l,RP # [1] H^5 + vxor H2_l,H2_l,RP2 # [1] H^6 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^5 + vpmsumd RP2,H2_l,poly_l # H^6 + vxor H_h,H_l,H_h # H^5 + vxor H2_h,H2_l,H2_h # H^6 + vxor H_h,H_h,RP # H^5 + vxor H2_h,H2_h,RP2 # H^6 + + vsldoi H2,H2_h,H2_h,8 # H^6 + vsldoi H,H_h,H_h,8 # H^5 + vsldoi H2l,zero,H2,8 # H^6 + vsldoi H2h,H2,zero,8 # H^6 + + # --- calculate [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + li r8,9*TableElemAlign + li r9,10*TableElemAlign + li r10,11*TableElemAlign + stxvd2x VSR(H_Hh),r8,TABLE + stxvd2x VSR(H_H),r9,TABLE + stxvd2x VSR(H_Hl),r10,TABLE + + # --- calculate H^7,H^8 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^6l + vpmsumd H_m,H_t,H2 # H^1h*H^6l⊕H^1l*H^6h + vpmsumd H_h,H_t,H2h # H^1h*H^6h + vpmsumd H2_l,H2_t,H2l # H^2l*H^6l + vpmsumd H2_m,H2_t,H2 # H^2h*H^6l⊕H^2l*H^6h + vpmsumd H2_h,H2_t,H2h # H^2h*H^6h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^7 + vpmsumd RP2,H2_l,poly_h # [1] H^8 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^7 + vsldoi M2h,zero,H2_m,8 # [2] H^8 + vsldoi Ml,H_m,zero,8 # [2] H^7 + vsldoi M2l,H2_m,zero,8 # [2] H^8 + vsldoi RP,RP,RP,8 # [1] H^7 + vsldoi RP2,RP2,RP2,8 # [1] H^8 + vxor H_h,H_h,Mh # [2] H^7 + vxor H2_h,H2_h,M2h # [2] H^8 + vxor H_l,H_l,Ml # [2] H^7 + vxor H2_l,H2_l,M2l # [2] H^8 + vxor H_l,H_l,RP # [1] H^7 + vxor H2_l,H2_l,RP2 # [1] H^8 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^7 + vpmsumd RP2,H2_l,poly_l # H^8 + vxor H_h,H_l,H_h # H^7 + vxor H2_h,H2_l,H2_h # H^8 + vxor H_h,H_h,RP # H^7 + vxor H2_h,H2_h,RP2 # H^8 + + vsldoi H,H_h,H_h,8 # H^7 + vsldoi H2,H2_h,H2_h,8 # H^8 + + # --- calculate [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + li r8,12*TableElemAlign + li r9,13*TableElemAlign + li r10,14*TableElemAlign + stxvd2x VSR(H_Hh),r8,TABLE + stxvd2x VSR(H_H),r9,TABLE + stxvd2x VSR(H_Hl),r10,TABLE + + blr +EPILOGUE(_nettle_gcm_init_key) + + # void gcm_hash (const struct gcm_key *key, union gcm_block *x, + # size_t length, const uint8_t *data) + +define(`FUNC_ALIGN', `5') +PROLOGUE(_nettle_gcm_hash) + vxor zero,zero,zero + + DATA_LOAD_VEC(poly,.polynomial,r7) +IF_LE(`DATA_LOAD_VEC(swap_mask,.swap_mask,r7)') + DATA_LOAD_VEC(hidw_mask,.hidw_mask,r7) + DATA_LOAD_VEC(lodw_mask,.lodw_mask,r7) + + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + lxvd2x VSR(C),0,X # load X +IF_LE(`vperm C,C,C,swap_mask') + + srdi r7,LENGTH,7 # 8x loop count + cmpldi r7,0 + beq L2x + + # backup registers + stdu SP,-224(SP) + std r28,216(SP) + std r29,208(SP) + std r30,200(SP) + std r31,192(SP) + li 8,176 + stvx 20,8,SP + subi 8,8,16 + stvx 21,8,SP + subi 8,8,16 + stvx 22,8,SP + subi 8,8,16 + stvx 23,8,SP + subi 8,8,16 + stvx 24,8,SP + subi 8,8,16 + stvx 25,8,SP + subi 8,8,16 + stvx 26,8,SP + subi 8,8,16 + stvx 27,8,SP + subi 8,8,16 + stvx 28,8,SP + subi 8,8,16 + stvx 29,8,SP + subi 8,8,16 + stvx 30,8,SP + subi 8,8,16 + stvx 31,8,SP + + # table loading + li r8,3*TableElemAlign + li r9,4*TableElemAlign + li r10,5*TableElemAlign + lxvd2x VSR(H21h),r8,TABLE + lxvd2x VSR(H21),r9,TABLE + lxvd2x VSR(H21l),r10,TABLE + li r8,6*TableElemAlign + li r9,7*TableElemAlign + li r10,8*TableElemAlign + lxvd2x VSR(H43h),r8,TABLE + lxvd2x VSR(H43),r9,TABLE + lxvd2x VSR(H43l),r10,TABLE + li r8,9*TableElemAlign + li r9,10*TableElemAlign + li r10,11*TableElemAlign + lxvd2x VSR(H65h),r8,TABLE + lxvd2x VSR(H65),r9,TABLE + lxvd2x VSR(H65l),r10,TABLE + li r8,12*TableElemAlign + li r9,13*TableElemAlign + li r10,14*TableElemAlign + lxvd2x VSR(H87h),r8,TABLE + lxvd2x VSR(H87),r9,TABLE + lxvd2x VSR(H87l),r10,TABLE + + li r8,0x10 + li r9,0x20 + li r10,0x30 + li r28,0x40 + li r29,0x50 + li r30,0x60 + li r31,0x70 + + mtctr r7 +.align 5 +L8x_loop: + # input loading + lxvd2x VSR(C0),0,DATA # load C0 + lxvd2x VSR(C1),r8,DATA # load C1 + lxvd2x VSR(C2),r9,DATA # load C2 + lxvd2x VSR(C3),r10,DATA # load C3 + + # swap permuting +IF_LE(`vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask + vperm C2,C2,C2,swap_mask + vperm C3,C3,C3,swap_mask') + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C23h,C2,C3,hidw_mask + vperm C23l,C2,C3,lodw_mask + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + + # input loading + lxvd2x VSR(C4),r28,DATA # load C4 + lxvd2x VSR(C5),r29,DATA # load C5 + lxvd2x VSR(C6),r30,DATA # load C6 + lxvd2x VSR(C7),r31,DATA # load C7 + + # swap permuting +IF_LE(`vperm C4,C4,C4,swap_mask + vperm C5,C5,C5,swap_mask + vperm C6,C6,C6,swap_mask + vperm C7,C7,C7,swap_mask') + + # polynomial multiplication "karatsuba" pre-processing + vperm C45h,C4,C5,hidw_mask + vperm C45l,C4,C5,lodw_mask + vperm C67h,C6,C7,hidw_mask + vperm C67l,C6,C7,lodw_mask + vxor C23,C23h,C23l + vxor C01,C01h,C01l + vxor C45,C45h,C45l + vxor C67,C67h,C67l + + # polynomial multiplication "karatsuba" + vpmsumd C23h,C23h,H65h # H23 = H^6h*C2h⊕H^5h*C3h + vpmsumd C23l,C23l,H65l # L23 = H^6l*C2l⊕H^5l*C3l + vpmsumd C01h,C01h,H87h # H01 = H^8h*C0h⊕H^7h*C1h + vpmsumd C01l,C01l,H87l # L01 = H^8l*C0l⊕H^7l*C1l + vpmsumd C67h,C67h,H21h # H67 = H^2h*C6h⊕H^1h*C7h + vpmsumd C67l,C67l,H21l # L67 = H^2l*C6l⊕H^1l*C7l + vpmsumd C45h,C45h,H43h # H45 = H^4h*C4h⊕H^3h*C5h + vpmsumd C45l,C45l,H43l # L45 = H^4l*C4l⊕H^3l*C5l + vpmsumd C23,C23,H65 # M23 = (H^6h⊕H^5h)*(C2h⊕C3h)⊕(H^6l⊕H^5l)*(C2l⊕C3l) + vpmsumd C01,C01,H87 # M01 = (H^8h⊕H^7h)*(C0h⊕C1h)⊕(H^8l⊕H^7l)*(C0l⊕C1l) + vpmsumd C45,C45,H43 # M45 = (H^4h⊕H^3h)*(C4h⊕C5h)⊕(H^4l⊕H^3l)*(C4l⊕C5l) + vpmsumd C67,C67,H21 # M67 = (H^2h⊕H^1h)*(C6h⊕C7h)⊕(H^2l⊕H^1l)*(C6l⊕C7l) + + # polynomial multiplication "karatsuba" post-processing + vxor C23,C23,C23h + vxor C01,C01,C01h + vxor C45,C45,C45h + vxor C67,C67,C67h + vxor C23,C23,C23l + vxor C01,C01,C01l + vxor C45,C45,C45l + vxor C67,C67,C67l + + # deferred recombination of partial products + vxor C01h,C01h,C23h # H0 = H01⊕H23 + vxor C45h,C45h,C67h # H1 = H45⊕H67 + vxor C01l,C01l,C23l # L0 = L01⊕L23 + vxor C45l,C45l,C67l # L1 = L45⊕L45 + vxor C01,C01,C23 # M0 = M01⊕M23 + vxor C45,C45,C67 # M1 = M45⊕M45 + vxor C01h,C01h,C45h # H = H0⊕H1 + vxor C01l,C01l,C45l # L = L0⊕L1 + vxor C01,C01,C45 # M = M0⊕M1 + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x80 + bdnz L8x_loop + + # restore registers + li 8,0 + lvx 31,8,SP + addi 8,8,16 + lvx 30,8,SP + addi 8,8,16 + lvx 29,8,SP + addi 8,8,16 + lvx 28,8,SP + addi 8,8,16 + lvx 27,8,SP + addi 8,8,16 + lvx 26,8,SP + addi 8,8,16 + lvx 25,8,SP + addi 8,8,16 + lvx 24,8,SP + addi 8,8,16 + lvx 23,8,SP + addi 8,8,16 + lvx 22,8,SP + addi 8,8,16 + lvx 21,8,SP + addi 8,8,16 + lvx 20,8,SP + ld r31,192(SP) + ld r30,200(SP) + ld r29,208(SP) + ld r28,216(SP) + addi SP,SP,224 + + clrldi LENGTH,LENGTH,57 +L2x: + srdi r7,LENGTH,5 + cmpldi r7,0 + beq L1x + + # table loading + li r8,3*TableElemAlign + li r9,4*TableElemAlign + li r10,5*TableElemAlign + lxvd2x VSR(H21h),r8,TABLE + lxvd2x VSR(H21),r9,TABLE + lxvd2x VSR(H21l),r10,TABLE + + li r10,0x10 + + mtctr r7 +.align 5 +L2x_loop: + # input loading + lxvd2x VSR(C0),0,DATA # load C0 + lxvd2x VSR(C1),r10,DATA # load C1 + + # swap permuting +IF_LE(`vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask') + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + vxor C01,C01h,C01l + + # polynomial multiplication "karatsuba" + vpmsumd C01h,C01h,H21h # H01 = H^2h*C0h⊕H^1h*C1h + vpmsumd C01l,C01l,H21l # L01 = H^2l*C0l⊕H^1l*C1l + vpmsumd C01,C01,H21 # M01 = (H^2h⊕H^1h)*(C0h⊕C1h)⊕(H^2l⊕H^1l)*(C0l⊕C1l) + + # polynomial multiplication "karatsuba" post-processing + vxor C01,C01,C01h + vxor C01,C01,C01l + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x20 + bdnz L2x_loop + + clrldi LENGTH,LENGTH,59 +L1x: + srdi r7,LENGTH,4 + cmpldi r7,0 + beq Lrem + + # table loading + li r9,1*TableElemAlign + li r10,2*TableElemAlign + lxvd2x VSR(Hl),0,TABLE + lxvd2x VSR(H), r9,TABLE + lxvd2x VSR(Hh),r10,TABLE + + # input loading + lxvd2x VSR(C0),0,DATA # load C0 + + # swap permuting +IF_LE(`vperm C0,C0,C0,swap_mask') + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + + addi DATA,DATA,0x10 + clrldi LENGTH,LENGTH,60 +Lrem: + cmpldi LENGTH,0 + beq Ldone + + # table loading + li r9,1*TableElemAlign + li r10,2*TableElemAlign + lxvd2x VSR(Hl),0,TABLE + lxvd2x VSR(H), r9,TABLE + lxvd2x VSR(Hh),r10,TABLE + + # input loading + stdu SP,-16(SP) + stvx zero,0,SP +Lst_loop: + subic. LENGTH,LENGTH,1 + lbzx r7,LENGTH,DATA + stbx r7,LENGTH,SP + bne Lst_loop + lxvd2x VSR(C0),0,SP + addi SP,SP,16 + + # swap permuting +IF_LE(`vperm C0,C0,C0,swap_mask') + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + +Ldone: +IF_LE(`vperm C,C,C,swap_mask') + stxvd2x VSR(C),0,X # store C + blr +EPILOGUE(_nettle_gcm_hash) + + .data +IF_LE(`.align 4 +.polynomial: + .byte 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0xc2 + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 + .align 4 +.hidw_mask: + .byte 23,22,21,20,19,18,17,16,7,6,5,4,3,2,1,0 + .align 4 +.lodw_mask: + .byte 31,30,29,28,27,26,25,24,15,14,13,12,11,10,9,8') +IF_BE(`.align 4 +.polynomial: + .byte 0xc2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1 + .align 4 +.hidw_mask: + .byte 0,1,2,3,4,5,6,7,16,17,18,19,20,21,22,23 + .align 4 +.lodw_mask: + .byte 8,9,10,11,12,13,14,15,24,25,26,27,28,29,30,31') -- 2.17.1

4 9

[PATCH] "PowerPC64" chacha-core big-endian support
by Maamoun TK 25 Sep '20

25 Sep '20

--- powerpc64/p7/chacha-core-internal.asm | 55 ++++++++++++++++++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/powerpc64/p7/chacha-core-internal.asm b/powerpc64/p7/chacha-core-internal.asm index 33c721c1..922050ff 100644 --- a/powerpc64/p7/chacha-core-internal.asm +++ b/powerpc64/p7/chacha-core-internal.asm @@ -53,6 +53,18 @@ define(`S1', `v9') define(`S2', `v10') define(`S3', `v11') +C Big-endian working state +define(`ROT24', `v12') +define(`ODD', `v13') +define(`EVEN', `v14') +define(`ZERO', `v15') +define(`NEG', `v16') + +define(`XR0', `v15') +define(`XR1', `v16') +define(`XR2', `v17') +define(`XR3', `v18') + C QROUND(X0, X1, X2, X3) define(`QROUND', ` C x0 += x1, x3 ^= x0, x3 lrot 16 @@ -77,10 +89,42 @@ define(`QROUND', ` vrlw $2, $2, ROT7 ') +C LE_SWAP32(X0, X1, X2, X3) +define(`LE_SWAP32', `IF_BE(` + C xr = x lrot 8, xr &= 0x00FF00FF + C x = x lrot 24, x &= 0xFF00FF00 + C x |= xr + + vrlw XR0, X0, ROT8 + vrlw XR1, X1, ROT8 + vrlw XR2, X2, ROT8 + vrlw XR3, X3, ROT8 + + vand XR0, XR0, ODD + vand XR1, XR1, ODD + vand XR2, XR2, ODD + vand XR3, XR3, ODD + + vrlw X0, X0, ROT24 + vrlw X1, X1, ROT24 + vrlw X2, X2, ROT24 + vrlw X3, X3, ROT24 + + vand X0, X0, EVEN + vand X1, X1, EVEN + vand X2, X2, EVEN + vand X3, X3, EVEN + + vor X0, X0, XR0 + vor X1, X1, XR1 + vor X2, X2, XR2 + vor X3, X3, XR3 +')') + .text - .align 4 C _chacha_core(uint32_t *dst, const uint32_t *src, unsigned rounds) +define(`FUNC_ALIGN', `5') PROLOGUE(_nettle_chacha_core) li r6, 0x10 C set up some... @@ -91,6 +135,13 @@ PROLOGUE(_nettle_chacha_core) vspltisw ROT12, 12 vspltisw ROT8, 8 vspltisw ROT7, 7 +IF_BE(` + vspltisw ZERO, 0 + vspltisw NEG, -1 + vmrghb ODD, ZERO, NEG + vmrghb EVEN, NEG, ZERO + vadduwm ROT24, ROT12, ROT12 +') lxvw4x VSR(X0), 0, SRC lxvw4x VSR(X1), r6, SRC @@ -131,6 +182,8 @@ PROLOGUE(_nettle_chacha_core) vadduwm X2, X2, S2 vadduwm X3, X3, S3 + LE_SWAP32(X0, X1, X2, X3) + stxvw4x VSR(X0), 0, DST stxvw4x VSR(X1), r6, DST stxvw4x VSR(X2), r7, DST -- 2.17.1

1 0

[PATCH] "PowerPC64" GCM support
by Maamoun TK 25 Sep '20

25 Sep '20

This is a stand-alone patch that applies all the previous patches to the optimized GCM implementation. This patch is based on the master upstream so it can be merged directly. It passes the testsuite and yields the expected performance. --- configure.ac | 5 +- fat-ppc.c | 44 +++ fat-setup.h | 9 + gcm.c | 82 +++- powerpc64/README | 2 +- powerpc64/fat/gcm-hash.asm | 40 ++ powerpc64/p8/gcm-hash.asm | 956 +++++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 1121 insertions(+), 17 deletions(-) create mode 100644 powerpc64/fat/gcm-hash.asm create mode 100644 powerpc64/p8/gcm-hash.asm diff --git a/configure.ac b/configure.ac index e9983697..13cfe33a 100644 --- a/configure.ac +++ b/configure.ac @@ -488,7 +488,7 @@ asm_replace_list="aes-encrypt-internal.asm aes-decrypt-internal.asm \ sha3-permute.asm umac-nh.asm umac-nh-n.asm machine.m4" # Assembler files which generate additional object files if they are used. -asm_nettle_optional_list="gcm-hash8.asm cpuid.asm \ +asm_nettle_optional_list="gcm-hash.asm gcm-hash8.asm cpuid.asm \ aes-encrypt-internal-2.asm aes-decrypt-internal-2.asm memxor-2.asm \ chacha-3core.asm chacha-core-internal-2.asm salsa20-2core.asm \ salsa20-core-internal-2.asm sha1-compress-2.asm sha256-compress-2.asm \ @@ -612,7 +612,8 @@ AH_VERBATIM([HAVE_NATIVE], #undef HAVE_NATIVE_ecc_secp384r1_redc #undef HAVE_NATIVE_ecc_secp521r1_modp #undef HAVE_NATIVE_ecc_secp521r1_redc -#undef HAVE_NATIVE_gcm_init_key8 +#undef HAVE_NATIVE_gcm_init_key +#undef HAVE_NATIVE_gcm_hash #undef HAVE_NATIVE_gcm_hash8 #undef HAVE_NATIVE_gcm_fill #undef HAVE_NATIVE_salsa20_core diff --git a/fat-ppc.c b/fat-ppc.c index 2bc50481..e69dff95 100644 --- a/fat-ppc.c +++ b/fat-ppc.c @@ -120,6 +120,20 @@ DECLARE_FAT_FUNC(_nettle_aes_decrypt, aes_crypt_internal_func) DECLARE_FAT_FUNC_VAR(aes_decrypt, aes_crypt_internal_func, c) DECLARE_FAT_FUNC_VAR(aes_decrypt, aes_crypt_internal_func, ppc64) +#if GCM_TABLE_BITS == 8 +DECLARE_FAT_FUNC(_nettle_gcm_init_key, gcm_init_key_func) +DECLARE_FAT_FUNC_VAR(gcm_init_key, gcm_init_key_func, c) +DECLARE_FAT_FUNC_VAR(gcm_init_key, gcm_init_key_func, ppc64) + +DECLARE_FAT_FUNC(_nettle_gcm_hash, gcm_hash_func) +DECLARE_FAT_FUNC_VAR(gcm_hash, gcm_hash_func, c) +DECLARE_FAT_FUNC_VAR(gcm_hash, gcm_hash_func, ppc64) +#endif /* GCM_TABLE_BITS == 8 */ + +DECLARE_FAT_FUNC(_nettle_gcm_fill, gcm_fill_func) +DECLARE_FAT_FUNC_VAR(gcm_fill, gcm_fill_func, c) +DECLARE_FAT_FUNC_VAR(gcm_fill, gcm_fill_func, ppc64) + static void CONSTRUCTOR fat_init (void) { @@ -139,11 +153,25 @@ fat_init (void) fprintf (stderr, "libnettle: enabling arch 2.07 code.\n"); _nettle_aes_encrypt_vec = _nettle_aes_encrypt_ppc64; _nettle_aes_decrypt_vec = _nettle_aes_decrypt_ppc64; +#if GCM_TABLE_BITS == 8 + /* Make sure _nettle_gcm_init_key_vec function is compatible + with _nettle_gcm_hash_vec function e.g. _nettle_gcm_init_key_c() + fills gcm_key table with values that are incompatible with + _nettle_gcm_hash_ppc64() */ + _nettle_gcm_init_key_vec = _nettle_gcm_init_key_ppc64; + _nettle_gcm_hash_vec = _nettle_gcm_hash_ppc64; +#endif /* GCM_TABLE_BITS == 8 */ + _nettle_gcm_fill_vec = _nettle_gcm_fill_ppc64; } else { _nettle_aes_encrypt_vec = _nettle_aes_encrypt_c; _nettle_aes_decrypt_vec = _nettle_aes_decrypt_c; +#if GCM_TABLE_BITS == 8 + _nettle_gcm_init_key_vec = _nettle_gcm_init_key_c; + _nettle_gcm_hash_vec = _nettle_gcm_hash_c; +#endif /* GCM_TABLE_BITS == 8 */ + _nettle_gcm_fill_vec = _nettle_gcm_fill_c; } } @@ -160,3 +188,19 @@ DEFINE_FAT_FUNC(_nettle_aes_decrypt, void, size_t length, uint8_t *dst, const uint8_t *src), (rounds, keys, T, length, dst, src)) + + #if GCM_TABLE_BITS == 8 +DEFINE_FAT_FUNC(_nettle_gcm_init_key, void, + (union nettle_block16 *table), + (table)) + +DEFINE_FAT_FUNC(_nettle_gcm_hash, void, + (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data), + (key, x, length, data)) +#endif /* GCM_TABLE_BITS == 8 */ + +DEFINE_FAT_FUNC(_nettle_gcm_fill, void, + (uint8_t *ctr, size_t blocks, + union nettle_block16 *buffer), + (ctr, blocks, buffer)) diff --git a/fat-setup.h b/fat-setup.h index 99f1ea67..623f9579 100644 --- a/fat-setup.h +++ b/fat-setup.h @@ -162,6 +162,15 @@ typedef void aes_crypt_internal_func (unsigned rounds, const uint32_t *keys, size_t length, uint8_t *dst, const uint8_t *src); +#if GCM_TABLE_BITS == 8 +typedef void gcm_init_key_func (union nettle_block16 *table); + +typedef void gcm_hash_func (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data); +#endif /* GCM_TABLE_BITS == 8 */ + +typedef void gcm_fill_func (uint8_t *ctr, size_t blocks, union nettle_block16 *buffer); + typedef void *(memxor_func)(void *dst, const void *src, size_t n); typedef void salsa20_core_func (uint32_t *dst, const uint32_t *src, unsigned rounds); diff --git a/gcm.c b/gcm.c index cf615daf..935d4420 100644 --- a/gcm.c +++ b/gcm.c @@ -140,6 +140,26 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) memcpy (x->b, Z.b, sizeof(Z)); } # elif GCM_TABLE_BITS == 8 +# if HAVE_NATIVE_gcm_init_key + +#define gcm_init_key _nettle_gcm_init_key +void +_nettle_gcm_init_key (union nettle_block16 *table); +/* For fat builds */ +void +_nettle_gcm_init_key_c (union nettle_block16 *table); +# endif /* HAVE_NATIVE_gcm_init_key */ +# if HAVE_NATIVE_gcm_hash + +#define gcm_hash _nettle_gcm_hash +void +_nettle_gcm_hash (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data); +/* For fat builds */ +void +_nettle_gcm_hash_c (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data); +# endif /* HAVE_NATIVE_gcm_hash */ # if HAVE_NATIVE_gcm_hash8 #define gcm_hash _nettle_gcm_hash8 @@ -225,9 +245,45 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) #endif /* GCM_TABLE_BITS */ +#if HAVE_NATIVE_gcm_fill + +#define gcm_fill _nettle_gcm_fill +void +_nettle_gcm_fill (uint8_t *ctr, size_t blocks, union nettle_block16 *buffer); +/* For fat builds */ +void +_nettle_gcm_fill_c (uint8_t *ctr, size_t blocks, union nettle_block16 *buffer); +#endif /* HAVE_NATIVE_gcm_fill */ + /* Increment the rightmost 32 bits. */ #define INC32(block) INCREMENT(4, (block.b) + GCM_BLOCK_SIZE - 4) +#ifdef gcm_init_key +void +_nettle_gcm_init_key_c(union nettle_block16 *table) +#else +static void +gcm_init_key(union nettle_block16 *table) +#endif /* !gcm_init_key */ +{ +#if GCM_TABLE_BITS + /* Middle element if GCM_TABLE_BITS > 0, otherwise the first + element */ + unsigned i = (1<<GCM_TABLE_BITS)/2; + + /* Algorithm 3 from the gcm paper. First do powers of two, then do + the rest by adding. */ + while (i /= 2) + block16_mulx_ghash(&table[i], &table[2*i]); + for (i = 2; i < 1<<GCM_TABLE_BITS; i *= 2) + { + unsigned j; + for (j = 1; j < i; j++) + block16_xor3(&table[i+j], &table[i], &table[j]); + } +#endif +} + /* Initialization of GCM. * @ctx: The context of GCM * @cipher: The context of the underlying block cipher @@ -245,24 +301,18 @@ gcm_set_key(struct gcm_key *key, memset(key->h[0].b, 0, GCM_BLOCK_SIZE); f (cipher, GCM_BLOCK_SIZE, key->h[i].b, key->h[0].b); -#if GCM_TABLE_BITS - /* Algorithm 3 from the gcm paper. First do powers of two, then do - the rest by adding. */ - while (i /= 2) - block16_mulx_ghash(&key->h[i], &key->h[2*i]); - for (i = 2; i < 1<<GCM_TABLE_BITS; i *= 2) - { - unsigned j; - for (j = 1; j < i; j++) - block16_xor3(&key->h[i+j], &key->h[i],&key->h[j]); - } -#endif + gcm_init_key(key->h); } -#ifndef gcm_hash +#ifdef gcm_hash +void +_nettle_gcm_hash_c(const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data) +#else static void gcm_hash(const struct gcm_key *key, union nettle_block16 *x, size_t length, const uint8_t *data) +#endif /* !gcm_hash */ { for (; length >= GCM_BLOCK_SIZE; length -= GCM_BLOCK_SIZE, data += GCM_BLOCK_SIZE) @@ -276,7 +326,6 @@ gcm_hash(const struct gcm_key *key, union nettle_block16 *x, gcm_gf_mul (x, key->h); } } -#endif /* !gcm_hash */ static void gcm_hash_sizes(const struct gcm_key *key, union nettle_block16 *x, @@ -333,9 +382,14 @@ gcm_update(struct gcm_ctx *ctx, const struct gcm_key *key, ctx->auth_size += length; } +#ifdef gcm_fill +void +_nettle_gcm_fill_c(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) +#else static nettle_fill16_func gcm_fill; static void gcm_fill(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) +#endif /* !gcm_fill */ { uint32_t c; diff --git a/powerpc64/README b/powerpc64/README index 7301953b..fac1108b 100644 --- a/powerpc64/README +++ b/powerpc64/README @@ -52,7 +52,7 @@ storage operands, refer to "6.4.1 Accessing Unaligned Storage Operands" in [3] to see an example of accessing unaligned storage operands. "lxvd2x/stxvd2x" can be used to load/store data into unaligned storage operands but permuting is needed for loading and storing data in -little-endian mode VSX registers are defined with "X" suffix +little-endian mode Function Prologue diff --git a/powerpc64/fat/gcm-hash.asm b/powerpc64/fat/gcm-hash.asm new file mode 100644 index 00000000..2d9f281e --- /dev/null +++ b/powerpc64/fat/gcm-hash.asm @@ -0,0 +1,40 @@ +C powerpc64/fat/gcm-hash.asm + + +ifelse(` + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +') + +dnl picked up by configure +dnl PROLOGUE(_nettle_gcm_init_key) +dnl PROLOGUE(_nettle_gcm_hash) +dnl PROLOGUE(_nettle_gcm_fill) + +define(`fat_transform', `$1_ppc64') +include_src(`powerpc64/p8/gcm-hash.asm') diff --git a/powerpc64/p8/gcm-hash.asm b/powerpc64/p8/gcm-hash.asm new file mode 100644 index 00000000..f57cbfe4 --- /dev/null +++ b/powerpc64/p8/gcm-hash.asm @@ -0,0 +1,956 @@ +C powerpc64/p8/gcm-hash.asm + +ifelse(` + Copyright (C) 2020 Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +') + +C Alignment of gcm_key table elements, which is declared in gcm.h +define(`TableElemAlign', `0x100') + +C Register usage: + +define(`SP', `r1') +define(`TOCP', `r2') + +define(`TABLE', `r3') +define(`X', `r4') +define(`LENGTH', `r5') +define(`DATA', `r6') + +define(`zero', `v0') +define(`swap_mask', `v1') +define(`hidw_mask', `v2') +define(`lodw_mask', `v3') +define(`poly', `v4') +define(`poly_h', `v4') +define(`poly_l', `v5') +define(`RP', `v6') +define(`Mh', `v7') +define(`Ml', `v8') +define(`H', `v9') +define(`Hh', `v10') +define(`Hl', `v11') +define(`RP2', `v9') +define(`M2h', `v10') +define(`M2l', `v11') + +define(`sl1', `v1') +define(`msb', `v5') +define(`H2', `v6') +define(`H2h', `v7') +define(`H2l', `v8') +define(`H_h', `v12') +define(`H_m', `v13') +define(`H_l', `v14') +define(`H_Hh', `v12') +define(`H_H', `v13') +define(`H_Hl', `v14') +define(`H_t', `v15') +define(`H2_h', `v16') +define(`H2_m', `v17') +define(`H2_l', `v18') +define(`H2_t', `v19') + +define(`C0', `v6') +define(`C1', `v7') +define(`C2', `v8') +define(`C3', `v12') +define(`C4', `v6') +define(`C5', `v7') +define(`C6', `v8') +define(`C7', `v12') + +define(`C', `v13') + +define(`Ch', `v14') +define(`Cl', `v15') +define(`Cm', `v16') + +define(`C01h', `v14') +define(`C01l', `v15') +define(`C01', `v16') +define(`C23h', `v17') +define(`C23l', `v18') +define(`C23', `v19') +define(`C45h', `v20') +define(`C45l', `v21') +define(`C45', `v22') +define(`C67h', `v6') +define(`C67l', `v7') +define(`C67', `v8') + +define(`H21', `v9') +define(`H21h', `v10') +define(`H21l', `v11') +define(`H43', `v23') +define(`H43h', `v24') +define(`H43l', `v25') +define(`H65', `v26') +define(`H65h', `v27') +define(`H65l', `v28') +define(`H87', `v29') +define(`H87h', `v30') +define(`H87l', `v31') + +# gcm_fill registers: + +define(`CTR', `r3') +define(`BLOCKS', `r4') +define(`BUFFER', `r5') + +define(`CTR0', `v2') +define(`CTR0S', `v3') +define(`CTR1', `v4') +define(`CTR2', `v5') +define(`CTR3', `v6') +define(`CTR4', `v7') +define(`CTR5', `v8') +define(`CTR6', `v9') +define(`CTR7', `v10') + +define(`I1', `v11') +define(`I2', `v12') +define(`I3', `v13') +define(`I4', `v14') +define(`I5', `v15') +define(`I6', `v16') +define(`I7', `v17') +define(`I8', `v18') + +.file "gcm-hash.asm" + +.text + + # void gcm_init_key (union gcm_block *table) + +define(`FUNC_ALIGN', `5') +PROLOGUE(_nettle_gcm_init_key) + DATA_LOAD_VEC(poly,.polynomial,r7) +IF_LE(`DATA_LOAD_VEC(swap_mask,.swap_mask,r7)') + DATA_LOAD_VEC(hidw_mask,.hidw_mask,r7) + DATA_LOAD_VEC(lodw_mask,.lodw_mask,r7) + + li r10,8*TableElemAlign + lxvd2x VSR(H),r10,TABLE # load H +IF_LE(`vperm H,H,H,swap_mask') + + # --- calculate H = H shift left 1 modulo polynomial --- + + vupkhsw msb,H # most significant bit word-extend + vspltisb sl1,1 # splat 1 for shift left + vspltw msb,msb,0 # most significant bit extend + vsl H,H,sl1 # H shift left 1 + vand msb,msb,poly + vxor zero,zero,zero + vxor H_t,H,msb + + vsldoi H,H_t,H_t,8 # doubleword swap + vsldoi Hh,H,zero,8 + vsldoi Hl,zero,H,8 + + # --- calculate H^2 = H*H --- + + # reduction pre-processing + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + # polynomial multiplication "classical" + vpmsumd H_h,H_t,Hh # H^1h*H^1h + vpmsumd H_l,H_t,Hl # H^1l*H^1l + vpmsumd H_m,H_t,H # H^1h*H^1l⊕H^1l*H^1h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] + vsldoi Ml,H_m,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor H_h,H_h,Mh # [2] + vxor H_l,H_l,Ml # [2] + vxor H_l,H_l,RP # [1] + + # reduction second phase + vpmsumd RP,H_l,poly_l + vxor H_h,H_l,H_h + vxor H2_t,H_h,RP + + vsldoi H2,H2_t,H2_t,8 + vsldoi H2h,H2,zero,8 + vsldoi H2l,zero,H2,8 + + # --- calculate [H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store H,[H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + li r8,0*TableElemAlign + li r9,1*TableElemAlign + li r10,2*TableElemAlign + stxvd2x VSR(Hl),r8,TABLE + stxvd2x VSR(H),r9,TABLE + stxvd2x VSR(Hh),r10,TABLE + + li r8,3*TableElemAlign + li r9,4*TableElemAlign + li r10,5*TableElemAlign + stxvd2x VSR(H_Hh),r8,TABLE + stxvd2x VSR(H_H),r9,TABLE + stxvd2x VSR(H_Hl),r10,TABLE + + # --- calculate H^3,H^4 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^2l + vpmsumd H_m,H_t,H2 # H^1h*H^2l⊕H^1l*H^2h + vpmsumd H_h,H_t,H2h # H^1h*H^2h + vpmsumd H2_l,H2_t,H2l # H^2l*H^2l + vpmsumd H2_m,H2_t,H2 # H^2h*H^2l⊕H^2l*H^2h + vpmsumd H2_h,H2_t,H2h # H^2h*H^2h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^3 + vpmsumd RP2,H2_l,poly_h # [1] H^4 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^3 + vsldoi M2h,zero,H2_m,8 # [2] H^4 + vsldoi Ml,H_m,zero,8 # [2] H^3 + vsldoi M2l,H2_m,zero,8 # [2] H^4 + vsldoi RP,RP,RP,8 # [1] H^3 + vsldoi RP2,RP2,RP2,8 # [1] H^4 + vxor H_h,H_h,Mh # [2] H^3 + vxor H2_h,H2_h,M2h # [2] H^4 + vxor H_l,H_l,Ml # [2] H^3 + vxor H2_l,H2_l,M2l # [2] H^4 + vxor H_l,H_l,RP # [1] H^3 + vxor H2_l,H2_l,RP2 # [1] H^4 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^3 + vpmsumd RP2,H2_l,poly_l # H^4 + vxor H_h,H_l,H_h # H^3 + vxor H2_h,H2_l,H2_h # H^4 + vxor H_h,H_h,RP # H^3 + vxor H2_h,H2_h,RP2 # H^4 + + vsldoi H2,H2_h,H2_h,8 # H^4 + vsldoi H,H_h,H_h,8 # H^3 + vsldoi H2l,zero,H2,8 # H^4 + vsldoi H2h,H2,zero,8 # H^4 + + # --- calculate [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + li r8,6*TableElemAlign + li r9,7*TableElemAlign + li r10,8*TableElemAlign + stxvd2x VSR(H_Hh),r8,TABLE + stxvd2x VSR(H_H),r9,TABLE + stxvd2x VSR(H_Hl),r10,TABLE + + # --- calculate H^5,H^6 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^4l + vpmsumd H_m,H_t,H2 # H^1h*H^4l⊕H^1l*H^4h + vpmsumd H_h,H_t,H2h # H^1h*H^4h + vpmsumd H2_l,H2_t,H2l # H^2l*H^4l + vpmsumd H2_m,H2_t,H2 # H^2h*H^4l⊕H^2l*H^4h + vpmsumd H2_h,H2_t,H2h # H^2h*H^4h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^5 + vpmsumd RP2,H2_l,poly_h # [1] H^6 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^5 + vsldoi M2h,zero,H2_m,8 # [2] H^6 + vsldoi Ml,H_m,zero,8 # [2] H^5 + vsldoi M2l,H2_m,zero,8 # [2] H^6 + vsldoi RP,RP,RP,8 # [1] H^5 + vsldoi RP2,RP2,RP2,8 # [1] H^6 + vxor H_h,H_h,Mh # [2] H^5 + vxor H2_h,H2_h,M2h # [2] H^6 + vxor H_l,H_l,Ml # [2] H^5 + vxor H2_l,H2_l,M2l # [2] H^6 + vxor H_l,H_l,RP # [1] H^5 + vxor H2_l,H2_l,RP2 # [1] H^6 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^5 + vpmsumd RP2,H2_l,poly_l # H^6 + vxor H_h,H_l,H_h # H^5 + vxor H2_h,H2_l,H2_h # H^6 + vxor H_h,H_h,RP # H^5 + vxor H2_h,H2_h,RP2 # H^6 + + vsldoi H2,H2_h,H2_h,8 # H^6 + vsldoi H,H_h,H_h,8 # H^5 + vsldoi H2l,zero,H2,8 # H^6 + vsldoi H2h,H2,zero,8 # H^6 + + # --- calculate [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + li r8,9*TableElemAlign + li r9,10*TableElemAlign + li r10,11*TableElemAlign + stxvd2x VSR(H_Hh),r8,TABLE + stxvd2x VSR(H_H),r9,TABLE + stxvd2x VSR(H_Hl),r10,TABLE + + # --- calculate H^7,H^8 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^6l + vpmsumd H_m,H_t,H2 # H^1h*H^6l⊕H^1l*H^6h + vpmsumd H_h,H_t,H2h # H^1h*H^6h + vpmsumd H2_l,H2_t,H2l # H^2l*H^6l + vpmsumd H2_m,H2_t,H2 # H^2h*H^6l⊕H^2l*H^6h + vpmsumd H2_h,H2_t,H2h # H^2h*H^6h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^7 + vpmsumd RP2,H2_l,poly_h # [1] H^8 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^7 + vsldoi M2h,zero,H2_m,8 # [2] H^8 + vsldoi Ml,H_m,zero,8 # [2] H^7 + vsldoi M2l,H2_m,zero,8 # [2] H^8 + vsldoi RP,RP,RP,8 # [1] H^7 + vsldoi RP2,RP2,RP2,8 # [1] H^8 + vxor H_h,H_h,Mh # [2] H^7 + vxor H2_h,H2_h,M2h # [2] H^8 + vxor H_l,H_l,Ml # [2] H^7 + vxor H2_l,H2_l,M2l # [2] H^8 + vxor H_l,H_l,RP # [1] H^7 + vxor H2_l,H2_l,RP2 # [1] H^8 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^7 + vpmsumd RP2,H2_l,poly_l # H^8 + vxor H_h,H_l,H_h # H^7 + vxor H2_h,H2_l,H2_h # H^8 + vxor H_h,H_h,RP # H^7 + vxor H2_h,H2_h,RP2 # H^8 + + vsldoi H,H_h,H_h,8 # H^7 + vsldoi H2,H2_h,H2_h,8 # H^8 + + # --- calculate [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + li r8,12*TableElemAlign + li r9,13*TableElemAlign + li r10,14*TableElemAlign + stxvd2x VSR(H_Hh),r8,TABLE + stxvd2x VSR(H_H),r9,TABLE + stxvd2x VSR(H_Hl),r10,TABLE + + blr +EPILOGUE(_nettle_gcm_init_key) + + # void gcm_hash (const struct gcm_key *key, union gcm_block *x, + # size_t length, const uint8_t *data) + +define(`FUNC_ALIGN', `5') +PROLOGUE(_nettle_gcm_hash) + vxor zero,zero,zero + + DATA_LOAD_VEC(poly,.polynomial,r7) +IF_LE(`DATA_LOAD_VEC(swap_mask,.swap_mask,r7)') + DATA_LOAD_VEC(hidw_mask,.hidw_mask,r7) + DATA_LOAD_VEC(lodw_mask,.lodw_mask,r7) + + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + lxvd2x VSR(C),0,X # load X +IF_LE(`vperm C,C,C,swap_mask') + + srdi r7,LENGTH,7 # 8x loop count + cmpldi r7,0 + beq L2x + + # backup registers + stdu SP,-224(SP) + std r28,216(SP) + std r29,208(SP) + std r30,200(SP) + std r31,192(SP) + li 8,176 + stvx 20,8,SP + subi 8,8,16 + stvx 21,8,SP + subi 8,8,16 + stvx 22,8,SP + subi 8,8,16 + stvx 23,8,SP + subi 8,8,16 + stvx 24,8,SP + subi 8,8,16 + stvx 25,8,SP + subi 8,8,16 + stvx 26,8,SP + subi 8,8,16 + stvx 27,8,SP + subi 8,8,16 + stvx 28,8,SP + subi 8,8,16 + stvx 29,8,SP + subi 8,8,16 + stvx 30,8,SP + subi 8,8,16 + stvx 31,8,SP + + # table loading + li r8,3*TableElemAlign + li r9,4*TableElemAlign + li r10,5*TableElemAlign + lxvd2x VSR(H21h),r8,TABLE + lxvd2x VSR(H21),r9,TABLE + lxvd2x VSR(H21l),r10,TABLE + li r8,6*TableElemAlign + li r9,7*TableElemAlign + li r10,8*TableElemAlign + lxvd2x VSR(H43h),r8,TABLE + lxvd2x VSR(H43),r9,TABLE + lxvd2x VSR(H43l),r10,TABLE + li r8,9*TableElemAlign + li r9,10*TableElemAlign + li r10,11*TableElemAlign + lxvd2x VSR(H65h),r8,TABLE + lxvd2x VSR(H65),r9,TABLE + lxvd2x VSR(H65l),r10,TABLE + li r8,12*TableElemAlign + li r9,13*TableElemAlign + li r10,14*TableElemAlign + lxvd2x VSR(H87h),r8,TABLE + lxvd2x VSR(H87),r9,TABLE + lxvd2x VSR(H87l),r10,TABLE + + li r8,0x10 + li r9,0x20 + li r10,0x30 + li r28,0x40 + li r29,0x50 + li r30,0x60 + li r31,0x70 + + mtctr r7 +.align 5 +L8x_loop: + # input loading + lxvd2x VSR(C0),0,DATA # load C0 + lxvd2x VSR(C1),r8,DATA # load C1 + lxvd2x VSR(C2),r9,DATA # load C2 + lxvd2x VSR(C3),r10,DATA # load C3 + + # swap permuting +IF_LE(`vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask + vperm C2,C2,C2,swap_mask + vperm C3,C3,C3,swap_mask') + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C23h,C2,C3,hidw_mask + vperm C23l,C2,C3,lodw_mask + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + + # input loading + lxvd2x VSR(C4),r28,DATA # load C4 + lxvd2x VSR(C5),r29,DATA # load C5 + lxvd2x VSR(C6),r30,DATA # load C6 + lxvd2x VSR(C7),r31,DATA # load C7 + + # swap permuting +IF_LE(`vperm C4,C4,C4,swap_mask + vperm C5,C5,C5,swap_mask + vperm C6,C6,C6,swap_mask + vperm C7,C7,C7,swap_mask') + + # polynomial multiplication "karatsuba" pre-processing + vperm C45h,C4,C5,hidw_mask + vperm C45l,C4,C5,lodw_mask + vperm C67h,C6,C7,hidw_mask + vperm C67l,C6,C7,lodw_mask + vxor C23,C23h,C23l + vxor C01,C01h,C01l + vxor C45,C45h,C45l + vxor C67,C67h,C67l + + # polynomial multiplication "karatsuba" + vpmsumd C23h,C23h,H65h # H23 = H^6h*C2h⊕H^5h*C3h + vpmsumd C23l,C23l,H65l # L23 = H^6l*C2l⊕H^5l*C3l + vpmsumd C01h,C01h,H87h # H01 = H^8h*C0h⊕H^7h*C1h + vpmsumd C01l,C01l,H87l # L01 = H^8l*C0l⊕H^7l*C1l + vpmsumd C67h,C67h,H21h # H67 = H^2h*C6h⊕H^1h*C7h + vpmsumd C67l,C67l,H21l # L67 = H^2l*C6l⊕H^1l*C7l + vpmsumd C45h,C45h,H43h # H45 = H^4h*C4h⊕H^3h*C5h + vpmsumd C45l,C45l,H43l # L45 = H^4l*C4l⊕H^3l*C5l + vpmsumd C23,C23,H65 # M23 = (H^6h⊕H^5h)*(C2h⊕C3h)⊕(H^6l⊕H^5l)*(C2l⊕C3l) + vpmsumd C01,C01,H87 # M01 = (H^8h⊕H^7h)*(C0h⊕C1h)⊕(H^8l⊕H^7l)*(C0l⊕C1l) + vpmsumd C45,C45,H43 # M45 = (H^4h⊕H^3h)*(C4h⊕C5h)⊕(H^4l⊕H^3l)*(C4l⊕C5l) + vpmsumd C67,C67,H21 # M67 = (H^2h⊕H^1h)*(C6h⊕C7h)⊕(H^2l⊕H^1l)*(C6l⊕C7l) + + # polynomial multiplication "karatsuba" post-processing + vxor C23,C23,C23h + vxor C01,C01,C01h + vxor C45,C45,C45h + vxor C67,C67,C67h + vxor C23,C23,C23l + vxor C01,C01,C01l + vxor C45,C45,C45l + vxor C67,C67,C67l + + # deferred recombination of partial products + vxor C01h,C01h,C23h # H0 = H01⊕H23 + vxor C45h,C45h,C67h # H1 = H45⊕H67 + vxor C01l,C01l,C23l # L0 = L01⊕L23 + vxor C45l,C45l,C67l # L1 = L45⊕L45 + vxor C01,C01,C23 # M0 = M01⊕M23 + vxor C45,C45,C67 # M1 = M45⊕M45 + vxor C01h,C01h,C45h # H = H0⊕H1 + vxor C01l,C01l,C45l # L = L0⊕L1 + vxor C01,C01,C45 # M = M0⊕M1 + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x80 + bdnz L8x_loop + + # restore registers + li 8,0 + lvx 31,8,SP + addi 8,8,16 + lvx 30,8,SP + addi 8,8,16 + lvx 29,8,SP + addi 8,8,16 + lvx 28,8,SP + addi 8,8,16 + lvx 27,8,SP + addi 8,8,16 + lvx 26,8,SP + addi 8,8,16 + lvx 25,8,SP + addi 8,8,16 + lvx 24,8,SP + addi 8,8,16 + lvx 23,8,SP + addi 8,8,16 + lvx 22,8,SP + addi 8,8,16 + lvx 21,8,SP + addi 8,8,16 + lvx 20,8,SP + ld r31,192(SP) + ld r30,200(SP) + ld r29,208(SP) + ld r28,216(SP) + addi SP,SP,224 + + clrldi LENGTH,LENGTH,57 +L2x: + srdi r7,LENGTH,5 + cmpldi r7,0 + beq L1x + + # table loading + li r8,3*TableElemAlign + li r9,4*TableElemAlign + li r10,5*TableElemAlign + lxvd2x VSR(H21h),r8,TABLE + lxvd2x VSR(H21),r9,TABLE + lxvd2x VSR(H21l),r10,TABLE + + li r10,0x10 + + mtctr r7 +.align 5 +L2x_loop: + # input loading + lxvd2x VSR(C0),0,DATA # load C0 + lxvd2x VSR(C1),r10,DATA # load C1 + + # swap permuting +IF_LE(`vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask') + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + vxor C01,C01h,C01l + + # polynomial multiplication "karatsuba" + vpmsumd C01h,C01h,H21h # H01 = H^2h*C0h⊕H^1h*C1h + vpmsumd C01l,C01l,H21l # L01 = H^2l*C0l⊕H^1l*C1l + vpmsumd C01,C01,H21 # M01 = (H^2h⊕H^1h)*(C0h⊕C1h)⊕(H^2l⊕H^1l)*(C0l⊕C1l) + + # polynomial multiplication "karatsuba" post-processing + vxor C01,C01,C01h + vxor C01,C01,C01l + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x20 + bdnz L2x_loop + + clrldi LENGTH,LENGTH,59 +L1x: + srdi r7,LENGTH,4 + cmpldi r7,0 + beq Lrem + + # table loading + li r9,1*TableElemAlign + li r10,2*TableElemAlign + lxvd2x VSR(Hl),0,TABLE + lxvd2x VSR(H), r9,TABLE + lxvd2x VSR(Hh),r10,TABLE + + # input loading + lxvd2x VSR(C0),0,DATA # load C0 + + # swap permuting +IF_LE(`vperm C0,C0,C0,swap_mask') + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + + addi DATA,DATA,0x10 + clrldi LENGTH,LENGTH,60 +Lrem: + cmpldi LENGTH,0 + beq Ldone + + # table loading + li r9,1*TableElemAlign + li r10,2*TableElemAlign + lxvd2x VSR(Hl),0,TABLE + lxvd2x VSR(H), r9,TABLE + lxvd2x VSR(Hh),r10,TABLE + + # input loading + stdu SP,-16(SP) + stvx zero,0,SP +Lst_loop: + subic. LENGTH,LENGTH,1 + lbzx r7,LENGTH,DATA + stbx r7,LENGTH,SP + bne Lst_loop + lxvd2x VSR(C0),0,SP + addi SP,SP,16 + + # swap permuting +IF_LE(`vperm C0,C0,C0,swap_mask') + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + +Ldone: +IF_LE(`vperm C,C,C,swap_mask') + stxvd2x VSR(C),0,X # store C + blr +EPILOGUE(_nettle_gcm_hash) + + # gcm_fill (uint8_t *ctr, size_t blocks, union gcm_block *buffer) + +define(`FUNC_ALIGN', `5') +PROLOGUE(_nettle_gcm_fill) +IF_LE(`DATA_LOAD_VEC(swap_mask,.swap_mask,r6)') + + vxor zero,zero,zero + vspltisb I1,1 + vspltisb I2,2 + vspltisb I3,3 + vspltisb I4,4 + vspltisb I5,5 + vspltisb I6,6 + vspltisb I7,7 + vspltisb I8,8 + vsldoi I1,zero,I1,1 + vsldoi I2,zero,I2,1 + vsldoi I3,zero,I3,1 + vsldoi I4,zero,I4,1 + vsldoi I5,zero,I5,1 + vsldoi I6,zero,I6,1 + vsldoi I7,zero,I7,1 + vsldoi I8,zero,I8,1 + + lxvd2x VSR(CTR0),0,CTR + IF_LE(`vperm CTR0,CTR0,CTR0,swap_mask') + + srdi r6,BLOCKS,3 # 8x loop count + cmpldi r6,0 + beq Lfill_4x + + std r25,-56(SP); + std r26,-48(SP); + std r27,-40(SP); + std r28,-32(SP); + std r29,-24(SP); + std r30,-16(SP); + std r31,-8(SP); + + li r25,0x10 + li r26,0x20 + li r27,0x30 + li r28,0x40 + li r29,0x50 + li r30,0x60 + li r31,0x70 + + mtctr r6 + L8x_fill_loop: + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + vadduwm CTR4,CTR0,I4 + vadduwm CTR5,CTR0,I5 + vadduwm CTR6,CTR0,I6 + vadduwm CTR7,CTR0,I7 + + IF_LE(`vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask + vperm CTR4,CTR4,CTR4,swap_mask + vperm CTR5,CTR5,CTR5,swap_mask + vperm CTR6,CTR6,CTR6,swap_mask + vperm CTR7,CTR7,CTR7,swap_mask') + + IF_LE(`stxvd2x VSR(CTR0S),0,BUFFER') + IF_BE(`stxvd2x VSR(CTR0),0,BUFFER') + stxvd2x VSR(CTR1),r25,BUFFER + stxvd2x VSR(CTR2),r26,BUFFER + stxvd2x VSR(CTR3),r27,BUFFER + stxvd2x VSR(CTR4),r28,BUFFER + stxvd2x VSR(CTR5),r29,BUFFER + stxvd2x VSR(CTR6),r30,BUFFER + stxvd2x VSR(CTR7),r31,BUFFER + + vadduwm CTR0,CTR0,I8 + addi BUFFER,BUFFER,0x80 + bdnz L8x_fill_loop + + ld r25,-56(SP); + ld r26,-48(SP); + ld r27,-40(SP); + ld r28,-32(SP); + ld r29,-24(SP); + ld r30,-16(SP); + ld r31,-8(SP); + + clrldi BLOCKS,BLOCKS,61 + + Lfill_4x: + srdi r6,BLOCKS,2 + cmpldi r6,0 + beq Lfill_2x + + li r8,0x10 + li r9,0x20 + li r10,0x30 + + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + + IF_LE(`vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask') + + IF_LE(`stxvd2x VSR(CTR0S),0,BUFFER') + IF_BE(`stxvd2x VSR(CTR0),0,BUFFER') + stxvd2x VSR(CTR1),r8,BUFFER + stxvd2x VSR(CTR2),r9,BUFFER + stxvd2x VSR(CTR3),r10,BUFFER + + vadduwm CTR0,CTR0,I4 + addi BUFFER,BUFFER,0x40 + + clrldi BLOCKS,BLOCKS,62 + + Lfill_2x: + srdi r6,BLOCKS,1 + cmpldi r6,0 + beq Lfill_1x + + li r10,0x10 + + vadduwm CTR1,CTR0,I1 + + IF_LE(`vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask') + + IF_LE(`stxvd2x VSR(CTR0S),0,BUFFER') + IF_BE(`stxvd2x VSR(CTR0),0,BUFFER') + stxvd2x VSR(CTR1),r10,BUFFER + + vadduwm CTR0,CTR0,I2 + addi BUFFER,BUFFER,0x20 + + clrldi BLOCKS,BLOCKS,63 + + Lfill_1x: + cmpldi BLOCKS,0 + beq Lfill_done + + IF_LE(`vperm CTR0S,CTR0,CTR0,swap_mask') + + IF_LE(`stxvd2x VSR(CTR0S),0,BUFFER') + IF_BE(`stxvd2x VSR(CTR0),0,BUFFER') + + vadduwm CTR0,CTR0,I1 + + Lfill_done: + IF_LE(`vperm CTR0,CTR0,CTR0,swap_mask') + stxvd2x VSR(CTR0),0,CTR + + blr +EPILOGUE(_nettle_gcm_fill) + + .data +IF_LE(`.align 4 +.polynomial: + .byte 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0xc2 + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 + .align 4 +.hidw_mask: + .byte 23,22,21,20,19,18,17,16,7,6,5,4,3,2,1,0 + .align 4 +.lodw_mask: + .byte 31,30,29,28,27,26,25,24,15,14,13,12,11,10,9,8') +IF_BE(`.align 4 +.polynomial: + .byte 0xc2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1 + .align 4 +.hidw_mask: + .byte 0,1,2,3,4,5,6,7,16,17,18,19,20,21,22,23 + .align 4 +.lodw_mask: + .byte 8,9,10,11,12,13,14,15,24,25,26,27,28,29,30,31') -- 2.17.1

2 4

[PATCH] "PowerPC64" Use explicit register names
by Maamoun TK 23 Sep '20

23 Sep '20

This patch is built upon ppc-m4-macrology.patch. Using explicit register names is working as expected now. --- powerpc64/machine.m4 | 11 +- powerpc64/p8/aes-decrypt-internal.asm | 194 +++++++++++++++++----------------- powerpc64/p8/aes-encrypt-internal.asm | 192 ++++++++++++++++----------------- 3 files changed, 200 insertions(+), 197 deletions(-) diff --git a/powerpc64/machine.m4 b/powerpc64/machine.m4 index ae161d79..f867ec01 100644 --- a/powerpc64/machine.m4 +++ b/powerpc64/machine.m4 @@ -24,7 +24,10 @@ define(`EPILOGUE', C Get vector-scalar register from vector register C VSR(VR) -define(`VSR',`32+$1') +define(`VSR',`ifelse(ASM_PPC_WANT_R_REGISTERS,no, +`eval(32+$1)', +``vs'eval(32+substr($1,1,len($1)))' +)') C Load the quadword in DATA_SRC storage into C VEC_DST. GPR is general-purpose register @@ -32,19 +35,19 @@ C used to obtain the effective address of C DATA_SRC storage. C DATA_LOAD_VEC(VEC_DST, DATA_SRC, GPR) define(`DATA_LOAD_VEC', -`ld $3,$2@got(2) +`ld $3,$2@got(r2) lvx $1,0,$3') dnl Usage: r0 ... r31, cr0 ... cr7 dnl dnl Registers names, either left as "r0" etc or mapped to plain 0 etc, -dnl according to the result of the GMP_ASM_POWERPC_REGISTERS configure +dnl according to the result of the ASM_PPC_WANT_R_REGISTERS configure dnl test. ifelse(ASM_PPC_WANT_R_REGISTERS,no,` forloop(i,0,31,`deflit(`r'i,i)') forloop(i,0,31,`deflit(`v'i,i)') +forloop(i,0,63,`deflit(`vs'i,i)') forloop(i,0,31,`deflit(`f'i,i)') forloop(i,0,7, `deflit(`cr'i,i)') ') - diff --git a/powerpc64/p8/aes-decrypt-internal.asm b/powerpc64/p8/aes-decrypt-internal.asm index acdbc1bd..7c79ffcb 100644 --- a/powerpc64/p8/aes-decrypt-internal.asm +++ b/powerpc64/p8/aes-decrypt-internal.asm @@ -31,32 +31,32 @@ ifelse(` C Register usage: -define(`SP', `1') -define(`TOCP', `2') - -define(`ROUNDS', `3') -define(`KEYS', `4') -define(`LENGTH', `6') -define(`DST', `7') -define(`SRC', `8') - -define(`swap_mask', `0') - -define(`K', `1') -define(`S0', `2') -define(`S1', `3') -define(`S2', `4') -define(`S3', `5') -define(`S4', `6') -define(`S5', `7') -define(`S6', `8') -define(`S7', `9') +define(`SP', `r1') +define(`TOCP', `r2') + +define(`ROUNDS', `r3') +define(`KEYS', `r4') +define(`LENGTH', `r6') +define(`DST', `r7') +define(`SRC', `r8') + +define(`swap_mask', `v0') + +define(`K', `v1') +define(`S0', `v2') +define(`S1', `v3') +define(`S2', `v4') +define(`S3', `v5') +define(`S4', `v6') +define(`S5', `v7') +define(`S6', `v8') +define(`S7', `v9') C ZERO vector register is used in place of RoundKey C for vncipher instruction because the order of InvMixColumns C and Xor processes are flipped in that instruction. C The Xor process with RoundKey is executed afterward. -define(`ZERO', `10') +define(`ZERO', `v10') .file "aes-decrypt-internal.asm" @@ -71,30 +71,30 @@ define(`FUNC_ALIGN', `5') PROLOGUE(_nettle_aes_decrypt) vxor ZERO,ZERO,ZERO - DATA_LOAD_VEC(swap_mask,.swap_mask,5) + DATA_LOAD_VEC(swap_mask,.swap_mask,r5) subi ROUNDS,ROUNDS,1 srdi LENGTH,LENGTH,4 - srdi 5,LENGTH,3 #8x loop count - cmpldi 5,0 + srdi r5,LENGTH,3 #8x loop count + cmpldi r5,0 beq L4x - std 25,-56(SP); - std 26,-48(SP); - std 27,-40(SP); - std 28,-32(SP); - std 29,-24(SP); - std 30,-16(SP); - std 31,-8(SP); - - li 25,0x10 - li 26,0x20 - li 27,0x30 - li 28,0x40 - li 29,0x50 - li 30,0x60 - li 31,0x70 + std r25,-56(SP); + std r26,-48(SP); + std r27,-40(SP); + std r28,-32(SP); + std r29,-24(SP); + std r30,-16(SP); + std r31,-8(SP); + + li r25,0x10 + li r26,0x20 + li r27,0x30 + li r28,0x40 + li r29,0x50 + li r30,0x60 + li r31,0x70 .align 5 Lx8_loop: @@ -102,13 +102,13 @@ Lx8_loop: vperm K,K,K,swap_mask lxvd2x VSR(S0),0,SRC - lxvd2x VSR(S1),25,SRC - lxvd2x VSR(S2),26,SRC - lxvd2x VSR(S3),27,SRC - lxvd2x VSR(S4),28,SRC - lxvd2x VSR(S5),29,SRC - lxvd2x VSR(S6),30,SRC - lxvd2x VSR(S7),31,SRC + lxvd2x VSR(S1),r25,SRC + lxvd2x VSR(S2),r26,SRC + lxvd2x VSR(S3),r27,SRC + lxvd2x VSR(S4),r28,SRC + lxvd2x VSR(S5),r29,SRC + lxvd2x VSR(S6),r30,SRC + lxvd2x VSR(S7),r31,SRC IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask @@ -129,10 +129,10 @@ IF_LE(`vperm S0,S0,S0,swap_mask vxor S7,S7,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L8x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipher S0,S0,ZERO vncipher S1,S1,ZERO @@ -150,10 +150,10 @@ L8x_round_loop: vxor S5,S5,K vxor S6,S6,K vxor S7,S7,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L8x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipherlast S0,S0,K vncipherlast S1,S1,K @@ -174,44 +174,44 @@ IF_LE(`vperm S0,S0,S0,swap_mask vperm S7,S7,S7,swap_mask') stxvd2x VSR(S0),0,DST - stxvd2x VSR(S1),25,DST - stxvd2x VSR(S2),26,DST - stxvd2x VSR(S3),27,DST - stxvd2x VSR(S4),28,DST - stxvd2x VSR(S5),29,DST - stxvd2x VSR(S6),30,DST - stxvd2x VSR(S7),31,DST + stxvd2x VSR(S1),r25,DST + stxvd2x VSR(S2),r26,DST + stxvd2x VSR(S3),r27,DST + stxvd2x VSR(S4),r28,DST + stxvd2x VSR(S5),r29,DST + stxvd2x VSR(S6),r30,DST + stxvd2x VSR(S7),r31,DST addi SRC,SRC,0x80 addi DST,DST,0x80 - subic. 5,5,1 + subic. r5,r5,1 bne Lx8_loop - ld 25,-56(SP); - ld 26,-48(SP); - ld 27,-40(SP); - ld 28,-32(SP); - ld 29,-24(SP); - ld 30,-16(SP); - ld 31,-8(SP); + ld r25,-56(SP); + ld r26,-48(SP); + ld r27,-40(SP); + ld r28,-32(SP); + ld r29,-24(SP); + ld r30,-16(SP); + ld r31,-8(SP); clrldi LENGTH,LENGTH,61 L4x: - srdi 5,LENGTH,2 - cmpldi 5,0 + srdi r5,LENGTH,2 + cmpldi r5,0 beq L2x lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask lxvd2x VSR(S0),0,SRC - li 9,0x10 - lxvd2x VSR(S1),9,SRC - addi 9,9,0x10 - lxvd2x VSR(S2),9,SRC - addi 9,9,0x10 - lxvd2x VSR(S3),9,SRC + li r9,0x10 + lxvd2x VSR(S1),r9,SRC + addi r9,r9,0x10 + lxvd2x VSR(S2),r9,SRC + addi r9,r9,0x10 + lxvd2x VSR(S3),r9,SRC IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask @@ -224,10 +224,10 @@ IF_LE(`vperm S0,S0,S0,swap_mask vxor S3,S3,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L4x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipher S0,S0,ZERO vncipher S1,S1,ZERO @@ -237,10 +237,10 @@ L4x_round_loop: vxor S1,S1,K vxor S2,S2,K vxor S3,S3,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L4x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipherlast S0,S0,K vncipherlast S1,S1,K @@ -253,12 +253,12 @@ IF_LE(`vperm S0,S0,S0,swap_mask vperm S3,S3,S3,swap_mask') stxvd2x VSR(S0),0,DST - li 9,0x10 - stxvd2x VSR(S1),9,DST - addi 9,9,0x10 - stxvd2x VSR(S2),9,DST - addi 9,9,0x10 - stxvd2x VSR(S3),9,DST + li r9,0x10 + stxvd2x VSR(S1),r9,DST + addi r9,r9,0x10 + stxvd2x VSR(S2),r9,DST + addi r9,r9,0x10 + stxvd2x VSR(S3),r9,DST addi SRC,SRC,0x40 addi DST,DST,0x40 @@ -266,16 +266,16 @@ IF_LE(`vperm S0,S0,S0,swap_mask clrldi LENGTH,LENGTH,62 L2x: - srdi 5,LENGTH,1 - cmpldi 5,0 + srdi r5,LENGTH,1 + cmpldi r5,0 beq L1x lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask lxvd2x VSR(S0),0,SRC - li 9,0x10 - lxvd2x VSR(S1),9,SRC + li r9,0x10 + lxvd2x VSR(S1),r9,SRC IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask') @@ -284,19 +284,19 @@ IF_LE(`vperm S0,S0,S0,swap_mask vxor S1,S1,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L2x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipher S0,S0,ZERO vncipher S1,S1,ZERO vxor S0,S0,K vxor S1,S1,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L2x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipherlast S0,S0,K vncipherlast S1,S1,K @@ -305,8 +305,8 @@ IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask') stxvd2x VSR(S0),0,DST - li 9,0x10 - stxvd2x VSR(S1),9,DST + li r9,0x10 + stxvd2x VSR(S1),r9,DST addi SRC,SRC,0x20 addi DST,DST,0x20 @@ -327,17 +327,17 @@ IF_LE(`vperm S0,S0,S0,swap_mask') vxor S0,S0,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L1x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipher S0,S0,ZERO vxor S0,S0,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L1x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipherlast S0,S0,K diff --git a/powerpc64/p8/aes-encrypt-internal.asm b/powerpc64/p8/aes-encrypt-internal.asm index 482dff25..3dd6e7b5 100644 --- a/powerpc64/p8/aes-encrypt-internal.asm +++ b/powerpc64/p8/aes-encrypt-internal.asm @@ -31,26 +31,26 @@ ifelse(` C Register usage: -define(`SP', `1') -define(`TOCP', `2') - -define(`ROUNDS', `3') -define(`KEYS', `4') -define(`LENGTH', `6') -define(`DST', `7') -define(`SRC', `8') - -define(`swap_mask', `0') - -define(`K', `1') -define(`S0', `2') -define(`S1', `3') -define(`S2', `4') -define(`S3', `5') -define(`S4', `6') -define(`S5', `7') -define(`S6', `8') -define(`S7', `9') +define(`SP', `r1') +define(`TOCP', `r2') + +define(`ROUNDS', `r3') +define(`KEYS', `r4') +define(`LENGTH', `r6') +define(`DST', `r7') +define(`SRC', `r8') + +define(`swap_mask', `v0') + +define(`K', `v1') +define(`S0', `v2') +define(`S1', `v3') +define(`S2', `v4') +define(`S3', `v5') +define(`S4', `v6') +define(`S5', `v7') +define(`S6', `v8') +define(`S7', `v9') .file "aes-encrypt-internal.asm" @@ -63,30 +63,30 @@ define(`S7', `9') define(`FUNC_ALIGN', `5') PROLOGUE(_nettle_aes_encrypt) - DATA_LOAD_VEC(swap_mask,.swap_mask,5) + DATA_LOAD_VEC(swap_mask,.swap_mask,r5) subi ROUNDS,ROUNDS,1 srdi LENGTH,LENGTH,4 - srdi 5,LENGTH,3 #8x loop count - cmpldi 5,0 + srdi r5,LENGTH,3 #8x loop count + cmpldi r5,0 beq L4x - std 25,-56(SP); - std 26,-48(SP); - std 27,-40(SP); - std 28,-32(SP); - std 29,-24(SP); - std 30,-16(SP); - std 31,-8(SP); - - li 25,0x10 - li 26,0x20 - li 27,0x30 - li 28,0x40 - li 29,0x50 - li 30,0x60 - li 31,0x70 + std r25,-56(SP); + std r26,-48(SP); + std r27,-40(SP); + std r28,-32(SP); + std r29,-24(SP); + std r30,-16(SP); + std r31,-8(SP); + + li r25,0x10 + li r26,0x20 + li r27,0x30 + li r28,0x40 + li r29,0x50 + li r30,0x60 + li r31,0x70 .align 5 Lx8_loop: @@ -94,13 +94,13 @@ Lx8_loop: vperm K,K,K,swap_mask lxvd2x VSR(S0),0,SRC - lxvd2x VSR(S1),25,SRC - lxvd2x VSR(S2),26,SRC - lxvd2x VSR(S3),27,SRC - lxvd2x VSR(S4),28,SRC - lxvd2x VSR(S5),29,SRC - lxvd2x VSR(S6),30,SRC - lxvd2x VSR(S7),31,SRC + lxvd2x VSR(S1),r25,SRC + lxvd2x VSR(S2),r26,SRC + lxvd2x VSR(S3),r27,SRC + lxvd2x VSR(S4),r28,SRC + lxvd2x VSR(S5),r29,SRC + lxvd2x VSR(S6),r30,SRC + lxvd2x VSR(S7),r31,SRC IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask @@ -121,10 +121,10 @@ IF_LE(`vperm S0,S0,S0,swap_mask vxor S7,S7,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L8x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipher S0,S0,K vcipher S1,S1,K @@ -134,10 +134,10 @@ L8x_round_loop: vcipher S5,S5,K vcipher S6,S6,K vcipher S7,S7,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L8x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipherlast S0,S0,K vcipherlast S1,S1,K @@ -158,44 +158,44 @@ IF_LE(`vperm S0,S0,S0,swap_mask vperm S7,S7,S7,swap_mask') stxvd2x VSR(S0),0,DST - stxvd2x VSR(S1),25,DST - stxvd2x VSR(S2),26,DST - stxvd2x VSR(S3),27,DST - stxvd2x VSR(S4),28,DST - stxvd2x VSR(S5),29,DST - stxvd2x VSR(S6),30,DST - stxvd2x VSR(S7),31,DST + stxvd2x VSR(S1),r25,DST + stxvd2x VSR(S2),r26,DST + stxvd2x VSR(S3),r27,DST + stxvd2x VSR(S4),r28,DST + stxvd2x VSR(S5),r29,DST + stxvd2x VSR(S6),r30,DST + stxvd2x VSR(S7),r31,DST addi SRC,SRC,0x80 addi DST,DST,0x80 - subic. 5,5,1 + subic. r5,r5,1 bne Lx8_loop - ld 25,-56(SP); - ld 26,-48(SP); - ld 27,-40(SP); - ld 28,-32(SP); - ld 29,-24(SP); - ld 30,-16(SP); - ld 31,-8(SP); + ld r25,-56(SP); + ld r26,-48(SP); + ld r27,-40(SP); + ld r28,-32(SP); + ld r29,-24(SP); + ld r30,-16(SP); + ld r31,-8(SP); clrldi LENGTH,LENGTH,61 L4x: - srdi 5,LENGTH,2 - cmpldi 5,0 + srdi r5,LENGTH,2 + cmpldi r5,0 beq L2x lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask lxvd2x VSR(S0),0,SRC - li 9,0x10 - lxvd2x VSR(S1),9,SRC - addi 9,9,0x10 - lxvd2x VSR(S2),9,SRC - addi 9,9,0x10 - lxvd2x VSR(S3),9,SRC + li r9,0x10 + lxvd2x VSR(S1),r9,SRC + addi r9,r9,0x10 + lxvd2x VSR(S2),r9,SRC + addi r9,r9,0x10 + lxvd2x VSR(S3),r9,SRC IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask @@ -208,19 +208,19 @@ IF_LE(`vperm S0,S0,S0,swap_mask vxor S3,S3,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L4x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipher S0,S0,K vcipher S1,S1,K vcipher S2,S2,K vcipher S3,S3,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L4x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipherlast S0,S0,K vcipherlast S1,S1,K @@ -233,12 +233,12 @@ IF_LE(`vperm S0,S0,S0,swap_mask vperm S3,S3,S3,swap_mask') stxvd2x VSR(S0),0,DST - li 9,0x10 - stxvd2x VSR(S1),9,DST - addi 9,9,0x10 - stxvd2x VSR(S2),9,DST - addi 9,9,0x10 - stxvd2x VSR(S3),9,DST + li r9,0x10 + stxvd2x VSR(S1),r9,DST + addi r9,r9,0x10 + stxvd2x VSR(S2),r9,DST + addi r9,r9,0x10 + stxvd2x VSR(S3),r9,DST addi SRC,SRC,0x40 addi DST,DST,0x40 @@ -246,16 +246,16 @@ IF_LE(`vperm S0,S0,S0,swap_mask clrldi LENGTH,LENGTH,62 L2x: - srdi 5,LENGTH,1 - cmpldi 5,0 + srdi r5,LENGTH,1 + cmpldi r5,0 beq L1x lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask lxvd2x VSR(S0),0,SRC - li 9,0x10 - lxvd2x VSR(S1),9,SRC + li r9,0x10 + lxvd2x VSR(S1),r9,SRC IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask') @@ -264,17 +264,17 @@ IF_LE(`vperm S0,S0,S0,swap_mask vxor S1,S1,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L2x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipher S0,S0,K vcipher S1,S1,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L2x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipherlast S0,S0,K vcipherlast S1,S1,K @@ -283,8 +283,8 @@ IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask') stxvd2x VSR(S0),0,DST - li 9,0x10 - stxvd2x VSR(S1),9,DST + li r9,0x10 + stxvd2x VSR(S1),r9,DST addi SRC,SRC,0x20 addi DST,DST,0x20 @@ -305,16 +305,16 @@ IF_LE(`vperm S0,S0,S0,swap_mask') vxor S0,S0,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L1x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipher S0,S0,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L1x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipherlast S0,S0,K -- 2.17.1

2 1

Re: NAPLES standup
by Jaak Ristioja 21 Sep '20

21 Sep '20

On esmaspäev, 21. september 2020 13:33:10 EEST Angela Sahk wrote: > Ma olen hetkel Zoomi VMRis... > > > On September 21, 2020 at 1:32 PM Angela Sahk <angela.sahk(a)cyber.ee> wrote: > > Hei, > > > > > > Kuna meie vahel Matrix ei toimi, siis kirjutan siia... kas sa liitud > > täna? > > > > > > Angela Meanwhile in matrix: [13:32:01] <J> hei! Naples-vormsi standup ikka toimub? ma ei näe VMR02-s kedagi [13:33:19] <J> muidugi seal taskis on hoopis mingi muu pexipi link, mis annab HTTP 404: https://pexip.me/meet/53256948/ J

1 1

[PATCH] "PowerPC64" Use same register convention in VSR macro
by Maamoun TK 19 Sep '20

19 Sep '20

--- powerpc64/machine.m4 | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/powerpc64/machine.m4 b/powerpc64/machine.m4 index f867ec01..e2383201 100644 --- a/powerpc64/machine.m4 +++ b/powerpc64/machine.m4 @@ -24,10 +24,9 @@ define(`EPILOGUE', C Get vector-scalar register from vector register C VSR(VR) -define(`VSR',`ifelse(ASM_PPC_WANT_R_REGISTERS,no, -`eval(32+$1)', -``vs'eval(32+substr($1,1,len($1)))' -)') +define(`VSR',`ifelse(substr($1,0,1),`v', +``vs'eval(32+substr($1,1,len($1)))', +`eval(32+$1)')') C Load the quadword in DATA_SRC storage into C VEC_DST. GPR is general-purpose register -- 2.17.1

1 0

[PATCH] "PowerPC64" Use register names explicitly
by Maamoun TK 18 Sep '20

18 Sep '20

Use explicit register names to improve the syntax of assembly files and pass -mregnames to the assembler to allow building the assembly files. I will make a stand-alone patch for GCM which brings all the accumulated modifications so it can be directly merged. --- configure.ac | 4 +- powerpc64/machine.m4 | 4 +- powerpc64/p8/aes-decrypt-internal.asm | 194 +++++++++++++++++----------------- powerpc64/p8/aes-encrypt-internal.asm | 192 ++++++++++++++++----------------- 4 files changed, 198 insertions(+), 196 deletions(-) diff --git a/configure.ac b/configure.ac index 666b2f4a..6ab32f03 100644 --- a/configure.ac +++ b/configure.ac @@ -458,10 +458,12 @@ if test "x$enable_assembler" = xyes ; then if test "$ABI" = 64 ; then asm_path="powerpc64" if test "x$enable_fat" = xyes ; then - asm_path="powerpc64/fat $asm_path" + CFLAGS="$CFLAGS -Wa,-mregnames" + asm_path="powerpc64/fat $asm_path" OPT_NETTLE_SOURCES="fat-ppc.c $OPT_NETTLE_SOURCES" FAT_TEST_LIST="none crypto_ext" elif test "x$enable_power_crypto_ext" = xyes ; then + CFLAGS="$CFLAGS -Wa,-mregnames" asm_path="powerpc64/p8 $asm_path" fi fi diff --git a/powerpc64/machine.m4 b/powerpc64/machine.m4 index 221fa523..cefabc9b 100644 --- a/powerpc64/machine.m4 +++ b/powerpc64/machine.m4 @@ -24,7 +24,7 @@ define(`EPILOGUE', C Get vector-scalar register from vector register C VSR(VR) -define(`VSR',`32+$1') +define(`VSR',``vs'eval(32+substr($1,1,len($1)))') C Load the quadword in DATA_SRC storage into C VEC_DST. GPR is general-purpose register @@ -32,5 +32,5 @@ C used to obtain the effective address of C DATA_SRC storage. C DATA_LOAD_VEC(VEC_DST, DATA_SRC, GPR) define(`DATA_LOAD_VEC', -`ld $3,$2@got(2) +`ld $3,$2@got(r2) lvx $1,0,$3') diff --git a/powerpc64/p8/aes-decrypt-internal.asm b/powerpc64/p8/aes-decrypt-internal.asm index acdbc1bd..7c79ffcb 100644 --- a/powerpc64/p8/aes-decrypt-internal.asm +++ b/powerpc64/p8/aes-decrypt-internal.asm @@ -31,32 +31,32 @@ ifelse(` C Register usage: -define(`SP', `1') -define(`TOCP', `2') - -define(`ROUNDS', `3') -define(`KEYS', `4') -define(`LENGTH', `6') -define(`DST', `7') -define(`SRC', `8') - -define(`swap_mask', `0') - -define(`K', `1') -define(`S0', `2') -define(`S1', `3') -define(`S2', `4') -define(`S3', `5') -define(`S4', `6') -define(`S5', `7') -define(`S6', `8') -define(`S7', `9') +define(`SP', `r1') +define(`TOCP', `r2') + +define(`ROUNDS', `r3') +define(`KEYS', `r4') +define(`LENGTH', `r6') +define(`DST', `r7') +define(`SRC', `r8') + +define(`swap_mask', `v0') + +define(`K', `v1') +define(`S0', `v2') +define(`S1', `v3') +define(`S2', `v4') +define(`S3', `v5') +define(`S4', `v6') +define(`S5', `v7') +define(`S6', `v8') +define(`S7', `v9') C ZERO vector register is used in place of RoundKey C for vncipher instruction because the order of InvMixColumns C and Xor processes are flipped in that instruction. C The Xor process with RoundKey is executed afterward. -define(`ZERO', `10') +define(`ZERO', `v10') .file "aes-decrypt-internal.asm" @@ -71,30 +71,30 @@ define(`FUNC_ALIGN', `5') PROLOGUE(_nettle_aes_decrypt) vxor ZERO,ZERO,ZERO - DATA_LOAD_VEC(swap_mask,.swap_mask,5) + DATA_LOAD_VEC(swap_mask,.swap_mask,r5) subi ROUNDS,ROUNDS,1 srdi LENGTH,LENGTH,4 - srdi 5,LENGTH,3 #8x loop count - cmpldi 5,0 + srdi r5,LENGTH,3 #8x loop count + cmpldi r5,0 beq L4x - std 25,-56(SP); - std 26,-48(SP); - std 27,-40(SP); - std 28,-32(SP); - std 29,-24(SP); - std 30,-16(SP); - std 31,-8(SP); - - li 25,0x10 - li 26,0x20 - li 27,0x30 - li 28,0x40 - li 29,0x50 - li 30,0x60 - li 31,0x70 + std r25,-56(SP); + std r26,-48(SP); + std r27,-40(SP); + std r28,-32(SP); + std r29,-24(SP); + std r30,-16(SP); + std r31,-8(SP); + + li r25,0x10 + li r26,0x20 + li r27,0x30 + li r28,0x40 + li r29,0x50 + li r30,0x60 + li r31,0x70 .align 5 Lx8_loop: @@ -102,13 +102,13 @@ Lx8_loop: vperm K,K,K,swap_mask lxvd2x VSR(S0),0,SRC - lxvd2x VSR(S1),25,SRC - lxvd2x VSR(S2),26,SRC - lxvd2x VSR(S3),27,SRC - lxvd2x VSR(S4),28,SRC - lxvd2x VSR(S5),29,SRC - lxvd2x VSR(S6),30,SRC - lxvd2x VSR(S7),31,SRC + lxvd2x VSR(S1),r25,SRC + lxvd2x VSR(S2),r26,SRC + lxvd2x VSR(S3),r27,SRC + lxvd2x VSR(S4),r28,SRC + lxvd2x VSR(S5),r29,SRC + lxvd2x VSR(S6),r30,SRC + lxvd2x VSR(S7),r31,SRC IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask @@ -129,10 +129,10 @@ IF_LE(`vperm S0,S0,S0,swap_mask vxor S7,S7,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L8x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipher S0,S0,ZERO vncipher S1,S1,ZERO @@ -150,10 +150,10 @@ L8x_round_loop: vxor S5,S5,K vxor S6,S6,K vxor S7,S7,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L8x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipherlast S0,S0,K vncipherlast S1,S1,K @@ -174,44 +174,44 @@ IF_LE(`vperm S0,S0,S0,swap_mask vperm S7,S7,S7,swap_mask') stxvd2x VSR(S0),0,DST - stxvd2x VSR(S1),25,DST - stxvd2x VSR(S2),26,DST - stxvd2x VSR(S3),27,DST - stxvd2x VSR(S4),28,DST - stxvd2x VSR(S5),29,DST - stxvd2x VSR(S6),30,DST - stxvd2x VSR(S7),31,DST + stxvd2x VSR(S1),r25,DST + stxvd2x VSR(S2),r26,DST + stxvd2x VSR(S3),r27,DST + stxvd2x VSR(S4),r28,DST + stxvd2x VSR(S5),r29,DST + stxvd2x VSR(S6),r30,DST + stxvd2x VSR(S7),r31,DST addi SRC,SRC,0x80 addi DST,DST,0x80 - subic. 5,5,1 + subic. r5,r5,1 bne Lx8_loop - ld 25,-56(SP); - ld 26,-48(SP); - ld 27,-40(SP); - ld 28,-32(SP); - ld 29,-24(SP); - ld 30,-16(SP); - ld 31,-8(SP); + ld r25,-56(SP); + ld r26,-48(SP); + ld r27,-40(SP); + ld r28,-32(SP); + ld r29,-24(SP); + ld r30,-16(SP); + ld r31,-8(SP); clrldi LENGTH,LENGTH,61 L4x: - srdi 5,LENGTH,2 - cmpldi 5,0 + srdi r5,LENGTH,2 + cmpldi r5,0 beq L2x lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask lxvd2x VSR(S0),0,SRC - li 9,0x10 - lxvd2x VSR(S1),9,SRC - addi 9,9,0x10 - lxvd2x VSR(S2),9,SRC - addi 9,9,0x10 - lxvd2x VSR(S3),9,SRC + li r9,0x10 + lxvd2x VSR(S1),r9,SRC + addi r9,r9,0x10 + lxvd2x VSR(S2),r9,SRC + addi r9,r9,0x10 + lxvd2x VSR(S3),r9,SRC IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask @@ -224,10 +224,10 @@ IF_LE(`vperm S0,S0,S0,swap_mask vxor S3,S3,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L4x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipher S0,S0,ZERO vncipher S1,S1,ZERO @@ -237,10 +237,10 @@ L4x_round_loop: vxor S1,S1,K vxor S2,S2,K vxor S3,S3,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L4x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipherlast S0,S0,K vncipherlast S1,S1,K @@ -253,12 +253,12 @@ IF_LE(`vperm S0,S0,S0,swap_mask vperm S3,S3,S3,swap_mask') stxvd2x VSR(S0),0,DST - li 9,0x10 - stxvd2x VSR(S1),9,DST - addi 9,9,0x10 - stxvd2x VSR(S2),9,DST - addi 9,9,0x10 - stxvd2x VSR(S3),9,DST + li r9,0x10 + stxvd2x VSR(S1),r9,DST + addi r9,r9,0x10 + stxvd2x VSR(S2),r9,DST + addi r9,r9,0x10 + stxvd2x VSR(S3),r9,DST addi SRC,SRC,0x40 addi DST,DST,0x40 @@ -266,16 +266,16 @@ IF_LE(`vperm S0,S0,S0,swap_mask clrldi LENGTH,LENGTH,62 L2x: - srdi 5,LENGTH,1 - cmpldi 5,0 + srdi r5,LENGTH,1 + cmpldi r5,0 beq L1x lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask lxvd2x VSR(S0),0,SRC - li 9,0x10 - lxvd2x VSR(S1),9,SRC + li r9,0x10 + lxvd2x VSR(S1),r9,SRC IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask') @@ -284,19 +284,19 @@ IF_LE(`vperm S0,S0,S0,swap_mask vxor S1,S1,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L2x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipher S0,S0,ZERO vncipher S1,S1,ZERO vxor S0,S0,K vxor S1,S1,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L2x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipherlast S0,S0,K vncipherlast S1,S1,K @@ -305,8 +305,8 @@ IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask') stxvd2x VSR(S0),0,DST - li 9,0x10 - stxvd2x VSR(S1),9,DST + li r9,0x10 + stxvd2x VSR(S1),r9,DST addi SRC,SRC,0x20 addi DST,DST,0x20 @@ -327,17 +327,17 @@ IF_LE(`vperm S0,S0,S0,swap_mask') vxor S0,S0,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L1x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipher S0,S0,ZERO vxor S0,S0,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L1x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vncipherlast S0,S0,K diff --git a/powerpc64/p8/aes-encrypt-internal.asm b/powerpc64/p8/aes-encrypt-internal.asm index 482dff25..3dd6e7b5 100644 --- a/powerpc64/p8/aes-encrypt-internal.asm +++ b/powerpc64/p8/aes-encrypt-internal.asm @@ -31,26 +31,26 @@ ifelse(` C Register usage: -define(`SP', `1') -define(`TOCP', `2') - -define(`ROUNDS', `3') -define(`KEYS', `4') -define(`LENGTH', `6') -define(`DST', `7') -define(`SRC', `8') - -define(`swap_mask', `0') - -define(`K', `1') -define(`S0', `2') -define(`S1', `3') -define(`S2', `4') -define(`S3', `5') -define(`S4', `6') -define(`S5', `7') -define(`S6', `8') -define(`S7', `9') +define(`SP', `r1') +define(`TOCP', `r2') + +define(`ROUNDS', `r3') +define(`KEYS', `r4') +define(`LENGTH', `r6') +define(`DST', `r7') +define(`SRC', `r8') + +define(`swap_mask', `v0') + +define(`K', `v1') +define(`S0', `v2') +define(`S1', `v3') +define(`S2', `v4') +define(`S3', `v5') +define(`S4', `v6') +define(`S5', `v7') +define(`S6', `v8') +define(`S7', `v9') .file "aes-encrypt-internal.asm" @@ -63,30 +63,30 @@ define(`S7', `9') define(`FUNC_ALIGN', `5') PROLOGUE(_nettle_aes_encrypt) - DATA_LOAD_VEC(swap_mask,.swap_mask,5) + DATA_LOAD_VEC(swap_mask,.swap_mask,r5) subi ROUNDS,ROUNDS,1 srdi LENGTH,LENGTH,4 - srdi 5,LENGTH,3 #8x loop count - cmpldi 5,0 + srdi r5,LENGTH,3 #8x loop count + cmpldi r5,0 beq L4x - std 25,-56(SP); - std 26,-48(SP); - std 27,-40(SP); - std 28,-32(SP); - std 29,-24(SP); - std 30,-16(SP); - std 31,-8(SP); - - li 25,0x10 - li 26,0x20 - li 27,0x30 - li 28,0x40 - li 29,0x50 - li 30,0x60 - li 31,0x70 + std r25,-56(SP); + std r26,-48(SP); + std r27,-40(SP); + std r28,-32(SP); + std r29,-24(SP); + std r30,-16(SP); + std r31,-8(SP); + + li r25,0x10 + li r26,0x20 + li r27,0x30 + li r28,0x40 + li r29,0x50 + li r30,0x60 + li r31,0x70 .align 5 Lx8_loop: @@ -94,13 +94,13 @@ Lx8_loop: vperm K,K,K,swap_mask lxvd2x VSR(S0),0,SRC - lxvd2x VSR(S1),25,SRC - lxvd2x VSR(S2),26,SRC - lxvd2x VSR(S3),27,SRC - lxvd2x VSR(S4),28,SRC - lxvd2x VSR(S5),29,SRC - lxvd2x VSR(S6),30,SRC - lxvd2x VSR(S7),31,SRC + lxvd2x VSR(S1),r25,SRC + lxvd2x VSR(S2),r26,SRC + lxvd2x VSR(S3),r27,SRC + lxvd2x VSR(S4),r28,SRC + lxvd2x VSR(S5),r29,SRC + lxvd2x VSR(S6),r30,SRC + lxvd2x VSR(S7),r31,SRC IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask @@ -121,10 +121,10 @@ IF_LE(`vperm S0,S0,S0,swap_mask vxor S7,S7,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L8x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipher S0,S0,K vcipher S1,S1,K @@ -134,10 +134,10 @@ L8x_round_loop: vcipher S5,S5,K vcipher S6,S6,K vcipher S7,S7,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L8x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipherlast S0,S0,K vcipherlast S1,S1,K @@ -158,44 +158,44 @@ IF_LE(`vperm S0,S0,S0,swap_mask vperm S7,S7,S7,swap_mask') stxvd2x VSR(S0),0,DST - stxvd2x VSR(S1),25,DST - stxvd2x VSR(S2),26,DST - stxvd2x VSR(S3),27,DST - stxvd2x VSR(S4),28,DST - stxvd2x VSR(S5),29,DST - stxvd2x VSR(S6),30,DST - stxvd2x VSR(S7),31,DST + stxvd2x VSR(S1),r25,DST + stxvd2x VSR(S2),r26,DST + stxvd2x VSR(S3),r27,DST + stxvd2x VSR(S4),r28,DST + stxvd2x VSR(S5),r29,DST + stxvd2x VSR(S6),r30,DST + stxvd2x VSR(S7),r31,DST addi SRC,SRC,0x80 addi DST,DST,0x80 - subic. 5,5,1 + subic. r5,r5,1 bne Lx8_loop - ld 25,-56(SP); - ld 26,-48(SP); - ld 27,-40(SP); - ld 28,-32(SP); - ld 29,-24(SP); - ld 30,-16(SP); - ld 31,-8(SP); + ld r25,-56(SP); + ld r26,-48(SP); + ld r27,-40(SP); + ld r28,-32(SP); + ld r29,-24(SP); + ld r30,-16(SP); + ld r31,-8(SP); clrldi LENGTH,LENGTH,61 L4x: - srdi 5,LENGTH,2 - cmpldi 5,0 + srdi r5,LENGTH,2 + cmpldi r5,0 beq L2x lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask lxvd2x VSR(S0),0,SRC - li 9,0x10 - lxvd2x VSR(S1),9,SRC - addi 9,9,0x10 - lxvd2x VSR(S2),9,SRC - addi 9,9,0x10 - lxvd2x VSR(S3),9,SRC + li r9,0x10 + lxvd2x VSR(S1),r9,SRC + addi r9,r9,0x10 + lxvd2x VSR(S2),r9,SRC + addi r9,r9,0x10 + lxvd2x VSR(S3),r9,SRC IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask @@ -208,19 +208,19 @@ IF_LE(`vperm S0,S0,S0,swap_mask vxor S3,S3,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L4x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipher S0,S0,K vcipher S1,S1,K vcipher S2,S2,K vcipher S3,S3,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L4x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipherlast S0,S0,K vcipherlast S1,S1,K @@ -233,12 +233,12 @@ IF_LE(`vperm S0,S0,S0,swap_mask vperm S3,S3,S3,swap_mask') stxvd2x VSR(S0),0,DST - li 9,0x10 - stxvd2x VSR(S1),9,DST - addi 9,9,0x10 - stxvd2x VSR(S2),9,DST - addi 9,9,0x10 - stxvd2x VSR(S3),9,DST + li r9,0x10 + stxvd2x VSR(S1),r9,DST + addi r9,r9,0x10 + stxvd2x VSR(S2),r9,DST + addi r9,r9,0x10 + stxvd2x VSR(S3),r9,DST addi SRC,SRC,0x40 addi DST,DST,0x40 @@ -246,16 +246,16 @@ IF_LE(`vperm S0,S0,S0,swap_mask clrldi LENGTH,LENGTH,62 L2x: - srdi 5,LENGTH,1 - cmpldi 5,0 + srdi r5,LENGTH,1 + cmpldi r5,0 beq L1x lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask lxvd2x VSR(S0),0,SRC - li 9,0x10 - lxvd2x VSR(S1),9,SRC + li r9,0x10 + lxvd2x VSR(S1),r9,SRC IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask') @@ -264,17 +264,17 @@ IF_LE(`vperm S0,S0,S0,swap_mask vxor S1,S1,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L2x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipher S0,S0,K vcipher S1,S1,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L2x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipherlast S0,S0,K vcipherlast S1,S1,K @@ -283,8 +283,8 @@ IF_LE(`vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask') stxvd2x VSR(S0),0,DST - li 9,0x10 - stxvd2x VSR(S1),9,DST + li r9,0x10 + stxvd2x VSR(S1),r9,DST addi SRC,SRC,0x20 addi DST,DST,0x20 @@ -305,16 +305,16 @@ IF_LE(`vperm S0,S0,S0,swap_mask') vxor S0,S0,K mtctr ROUNDS - li 10,0x10 + li r10,0x10 .align 5 L1x_round_loop: - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipher S0,S0,K - addi 10,10,0x10 + addi r10,r10,0x10 bdnz L1x_round_loop - lxvd2x VSR(K),10,KEYS + lxvd2x VSR(K),r10,KEYS vperm K,K,K,swap_mask vcipherlast S0,S0,K -- 2.17.1

2 2

The m4 quote characters
by nisse＠lysator.liu.se 18 Sep '20

18 Sep '20

To adopt the m4 macrology used in GMP, for optionally substituting r0-->0, r1-->1, etc, I tried to copy the m4 macros, including an m4 forloop. But that didn't work out of the box, and it seems that one complication is that Nettle's uses <> as m4 quote characters. I've used them since I introduced m4 preprocessing back in 2002, and I honestly don't remember why I thought that was a good idea at the time. I guess I found `' ugly, and the autoconf choice of [] didn't work well with assembly syntax (in particular on sparc). In a few places nettle needs to use < as an operator in m4 eval, and there are workarounds for that using local changequote, with the curious hack define(<W64_ENTRY>, < changequote([,])dnl ifelse(<<<<<<<<<<<<<<<<<< ignored; only for balancing) ifelse(W64_ABI,yes,[ dnl unconditionally push %rdi, making %rsp 16-byte aligned push %rdi dnl Save %xmm6, ..., if needed ifelse(eval($2 > 6), 1, [ sub [$]eval(16*($2 - 6)), %rsp movdqa %xmm6, 0(%rsp) ]) ifelse(eval($2 > 7), 1, [ movdqa %xmm7, 16(%rsp) ]) ... So I'm considering changing that, and instead stick to m4 default quotes. Opinions? Regards, /Niels -- Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677. Internet email is subject to wholesale government surveillance.

2 3

[PATCH] "PowerPC64" AES improve syntax
by Maamoun TK 08 Sep '20

08 Sep '20

This patch adds "VSR" macro to improve the syntax of assembly code, I will create a separate patch for gcm-hash since it hasn't merged yet to the master. I also removed the TODO from README because I tried to use "lxv/stxv" in POWER9 instead of "lxvd2x/stxvd2x" but gcc produced "lxvd2x/stxvd2x" in the binary. I'm not sure if it's variant issue of gcc but this will be problematic since "lxvd2x/stxvd2x" need permuting in little-endian mode while "lxv/stxv" is endianness aware. --- powerpc64/README | 2 - powerpc64/machine.m4 | 4 ++ powerpc64/p8/aes-decrypt-internal.asm | 97 ++++++++++++++++------------------- powerpc64/p8/aes-encrypt-internal.asm | 97 ++++++++++++++++------------------- 4 files changed, 90 insertions(+), 110 deletions(-) diff --git a/powerpc64/README b/powerpc64/README index 5410791f..7301953b 100644 --- a/powerpc64/README +++ b/powerpc64/README @@ -53,8 +53,6 @@ in [3] to see an example of accessing unaligned storage operands. "lxvd2x/stxvd2x" can be used to load/store data into unaligned storage operands but permuting is needed for loading and storing data in little-endian mode VSX registers are defined with "X" suffix -TODO: use architecture 3.0 instructions "lxv/stxv" instead for POWER9 - and newer Function Prologue diff --git a/powerpc64/machine.m4 b/powerpc64/machine.m4 index 2f91adec..b76bb8b1 100644 --- a/powerpc64/machine.m4 +++ b/powerpc64/machine.m4 @@ -22,6 +22,10 @@ define(<EPILOGUE>, <.size .C_NAME($1), . - .C_NAME($1) .size C_NAME($1), . - .C_NAME($1)>)>) +C Get vector-scalar register from vector register +C VSR(VR) +define(<VSR>,<32+$1>) + C Load the quadword in DATA_SRC storage into C VEC_DST. GPR is general-purpose register C used to obtain the effective address of diff --git a/powerpc64/p8/aes-decrypt-internal.asm b/powerpc64/p8/aes-decrypt-internal.asm index 7d518cd9..bfedb32b 100644 --- a/powerpc64/p8/aes-decrypt-internal.asm +++ b/powerpc64/p8/aes-decrypt-internal.asm @@ -1,4 +1,4 @@ -C powerpc64/P8/aes-decrypt-internal.asm +C powerpc64/p8/aes-decrypt-internal.asm ifelse(< Copyright (C) 2020 Mamone Tarsha @@ -52,16 +52,6 @@ define(<S5>, <7>) define(<S6>, <8>) define(<S7>, <9>) -define(<KX>, <33>) -define(<S0X>, <34>) -define(<S1X>, <35>) -define(<S2X>, <36>) -define(<S3X>, <37>) -define(<S4X>, <38>) -define(<S5X>, <39>) -define(<S6X>, <40>) -define(<S7X>, <41>) - C ZERO vector register is used in place of RoundKey C for vncipher instruction because the order of InvMixColumns C and Xor processes are flipped in that instruction. @@ -70,7 +60,6 @@ define(<ZERO>, <10>) .file "aes-decrypt-internal.asm" -IF_LE(<.abiversion 2>) .text C _aes_decrypt(unsigned rounds, const uint32_t *keys, @@ -109,17 +98,17 @@ PROLOGUE(_nettle_aes_decrypt) .align 5 Lx8_loop: - lxvd2x KX,0,KEYS + lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask - lxvd2x S0X,0,SRC - lxvd2x S1X,25,SRC - lxvd2x S2X,26,SRC - lxvd2x S3X,27,SRC - lxvd2x S4X,28,SRC - lxvd2x S5X,29,SRC - lxvd2x S6X,30,SRC - lxvd2x S7X,31,SRC + lxvd2x VSR(S0),0,SRC + lxvd2x VSR(S1),25,SRC + lxvd2x VSR(S2),26,SRC + lxvd2x VSR(S3),27,SRC + lxvd2x VSR(S4),28,SRC + lxvd2x VSR(S5),29,SRC + lxvd2x VSR(S6),30,SRC + lxvd2x VSR(S7),31,SRC IF_LE(<vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask @@ -143,7 +132,7 @@ IF_LE(<vperm S0,S0,S0,swap_mask li 10,0x10 .align 5 L8x_round_loop: - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vncipher S0,S0,ZERO vncipher S1,S1,ZERO @@ -164,7 +153,7 @@ L8x_round_loop: addi 10,10,0x10 bdnz L8x_round_loop - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vncipherlast S0,S0,K vncipherlast S1,S1,K @@ -184,14 +173,14 @@ IF_LE(<vperm S0,S0,S0,swap_mask vperm S6,S6,S6,swap_mask vperm S7,S7,S7,swap_mask>) - stxvd2x S0X,0,DST - stxvd2x S1X,25,DST - stxvd2x S2X,26,DST - stxvd2x S3X,27,DST - stxvd2x S4X,28,DST - stxvd2x S5X,29,DST - stxvd2x S6X,30,DST - stxvd2x S7X,31,DST + stxvd2x VSR(S0),0,DST + stxvd2x VSR(S1),25,DST + stxvd2x VSR(S2),26,DST + stxvd2x VSR(S3),27,DST + stxvd2x VSR(S4),28,DST + stxvd2x VSR(S5),29,DST + stxvd2x VSR(S6),30,DST + stxvd2x VSR(S7),31,DST addi SRC,SRC,0x80 addi DST,DST,0x80 @@ -213,16 +202,16 @@ L4x: cmpldi 5,0 beq L2x - lxvd2x KX,0,KEYS + lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask - lxvd2x S0X,0,SRC + lxvd2x VSR(S0),0,SRC li 9,0x10 - lxvd2x S1X,9,SRC + lxvd2x VSR(S1),9,SRC addi 9,9,0x10 - lxvd2x S2X,9,SRC + lxvd2x VSR(S2),9,SRC addi 9,9,0x10 - lxvd2x S3X,9,SRC + lxvd2x VSR(S3),9,SRC IF_LE(<vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask @@ -238,7 +227,7 @@ IF_LE(<vperm S0,S0,S0,swap_mask li 10,0x10 .align 5 L4x_round_loop: - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vncipher S0,S0,ZERO vncipher S1,S1,ZERO @@ -251,7 +240,7 @@ L4x_round_loop: addi 10,10,0x10 bdnz L4x_round_loop - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vncipherlast S0,S0,K vncipherlast S1,S1,K @@ -263,13 +252,13 @@ IF_LE(<vperm S0,S0,S0,swap_mask vperm S2,S2,S2,swap_mask vperm S3,S3,S3,swap_mask>) - stxvd2x S0X,0,DST + stxvd2x VSR(S0),0,DST li 9,0x10 - stxvd2x S1X,9,DST + stxvd2x VSR(S1),9,DST addi 9,9,0x10 - stxvd2x S2X,9,DST + stxvd2x VSR(S2),9,DST addi 9,9,0x10 - stxvd2x S3X,9,DST + stxvd2x VSR(S3),9,DST addi SRC,SRC,0x40 addi DST,DST,0x40 @@ -281,12 +270,12 @@ L2x: cmpldi 5,0 beq L1x - lxvd2x KX,0,KEYS + lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask - lxvd2x S0X,0,SRC + lxvd2x VSR(S0),0,SRC li 9,0x10 - lxvd2x S1X,9,SRC + lxvd2x VSR(S1),9,SRC IF_LE(<vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask>) @@ -298,7 +287,7 @@ IF_LE(<vperm S0,S0,S0,swap_mask li 10,0x10 .align 5 L2x_round_loop: - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vncipher S0,S0,ZERO vncipher S1,S1,ZERO @@ -307,7 +296,7 @@ L2x_round_loop: addi 10,10,0x10 bdnz L2x_round_loop - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vncipherlast S0,S0,K vncipherlast S1,S1,K @@ -315,9 +304,9 @@ L2x_round_loop: IF_LE(<vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask>) - stxvd2x S0X,0,DST + stxvd2x VSR(S0),0,DST li 9,0x10 - stxvd2x S1X,9,DST + stxvd2x VSR(S1),9,DST addi SRC,SRC,0x20 addi DST,DST,0x20 @@ -328,10 +317,10 @@ L1x: cmpldi LENGTH,0 beq Ldone - lxvd2x KX,0,KEYS + lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask - lxvd2x S0X,0,SRC + lxvd2x VSR(S0),0,SRC IF_LE(<vperm S0,S0,S0,swap_mask>) @@ -341,20 +330,20 @@ IF_LE(<vperm S0,S0,S0,swap_mask>) li 10,0x10 .align 5 L1x_round_loop: - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vncipher S0,S0,ZERO vxor S0,S0,K addi 10,10,0x10 bdnz L1x_round_loop - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vncipherlast S0,S0,K IF_LE(<vperm S0,S0,S0,swap_mask>) - stxvd2x S0X,0,DST + stxvd2x VSR(S0),0,DST Ldone: blr diff --git a/powerpc64/p8/aes-encrypt-internal.asm b/powerpc64/p8/aes-encrypt-internal.asm index c696a4a3..67c7e597 100644 --- a/powerpc64/p8/aes-encrypt-internal.asm +++ b/powerpc64/p8/aes-encrypt-internal.asm @@ -1,4 +1,4 @@ -C powerpc64/P8/aes-encrypt-internal.asm +C powerpc64/p8/aes-encrypt-internal.asm ifelse(< Copyright (C) 2020 Mamone Tarsha @@ -52,19 +52,8 @@ define(<S5>, <7>) define(<S6>, <8>) define(<S7>, <9>) -define(<KX>, <33>) -define(<S0X>, <34>) -define(<S1X>, <35>) -define(<S2X>, <36>) -define(<S3X>, <37>) -define(<S4X>, <38>) -define(<S5X>, <39>) -define(<S6X>, <40>) -define(<S7X>, <41>) - .file "aes-encrypt-internal.asm" -IF_LE(<.abiversion 2>) .text C _aes_encrypt(unsigned rounds, const uint32_t *keys, @@ -101,17 +90,17 @@ PROLOGUE(_nettle_aes_encrypt) .align 5 Lx8_loop: - lxvd2x KX,0,KEYS + lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask - lxvd2x S0X,0,SRC - lxvd2x S1X,25,SRC - lxvd2x S2X,26,SRC - lxvd2x S3X,27,SRC - lxvd2x S4X,28,SRC - lxvd2x S5X,29,SRC - lxvd2x S6X,30,SRC - lxvd2x S7X,31,SRC + lxvd2x VSR(S0),0,SRC + lxvd2x VSR(S1),25,SRC + lxvd2x VSR(S2),26,SRC + lxvd2x VSR(S3),27,SRC + lxvd2x VSR(S4),28,SRC + lxvd2x VSR(S5),29,SRC + lxvd2x VSR(S6),30,SRC + lxvd2x VSR(S7),31,SRC IF_LE(<vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask @@ -135,7 +124,7 @@ IF_LE(<vperm S0,S0,S0,swap_mask li 10,0x10 .align 5 L8x_round_loop: - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vcipher S0,S0,K vcipher S1,S1,K @@ -148,7 +137,7 @@ L8x_round_loop: addi 10,10,0x10 bdnz L8x_round_loop - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vcipherlast S0,S0,K vcipherlast S1,S1,K @@ -168,14 +157,14 @@ IF_LE(<vperm S0,S0,S0,swap_mask vperm S6,S6,S6,swap_mask vperm S7,S7,S7,swap_mask>) - stxvd2x S0X,0,DST - stxvd2x S1X,25,DST - stxvd2x S2X,26,DST - stxvd2x S3X,27,DST - stxvd2x S4X,28,DST - stxvd2x S5X,29,DST - stxvd2x S6X,30,DST - stxvd2x S7X,31,DST + stxvd2x VSR(S0),0,DST + stxvd2x VSR(S1),25,DST + stxvd2x VSR(S2),26,DST + stxvd2x VSR(S3),27,DST + stxvd2x VSR(S4),28,DST + stxvd2x VSR(S5),29,DST + stxvd2x VSR(S6),30,DST + stxvd2x VSR(S7),31,DST addi SRC,SRC,0x80 addi DST,DST,0x80 @@ -197,16 +186,16 @@ L4x: cmpldi 5,0 beq L2x - lxvd2x KX,0,KEYS + lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask - lxvd2x S0X,0,SRC + lxvd2x VSR(S0),0,SRC li 9,0x10 - lxvd2x S1X,9,SRC + lxvd2x VSR(S1),9,SRC addi 9,9,0x10 - lxvd2x S2X,9,SRC + lxvd2x VSR(S2),9,SRC addi 9,9,0x10 - lxvd2x S3X,9,SRC + lxvd2x VSR(S3),9,SRC IF_LE(<vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask @@ -222,7 +211,7 @@ IF_LE(<vperm S0,S0,S0,swap_mask li 10,0x10 .align 5 L4x_round_loop: - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vcipher S0,S0,K vcipher S1,S1,K @@ -231,7 +220,7 @@ L4x_round_loop: addi 10,10,0x10 bdnz L4x_round_loop - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vcipherlast S0,S0,K vcipherlast S1,S1,K @@ -243,13 +232,13 @@ IF_LE(<vperm S0,S0,S0,swap_mask vperm S2,S2,S2,swap_mask vperm S3,S3,S3,swap_mask>) - stxvd2x S0X,0,DST + stxvd2x VSR(S0),0,DST li 9,0x10 - stxvd2x S1X,9,DST + stxvd2x VSR(S1),9,DST addi 9,9,0x10 - stxvd2x S2X,9,DST + stxvd2x VSR(S2),9,DST addi 9,9,0x10 - stxvd2x S3X,9,DST + stxvd2x VSR(S3),9,DST addi SRC,SRC,0x40 addi DST,DST,0x40 @@ -261,12 +250,12 @@ L2x: cmpldi 5,0 beq L1x - lxvd2x KX,0,KEYS + lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask - lxvd2x S0X,0,SRC + lxvd2x VSR(S0),0,SRC li 9,0x10 - lxvd2x S1X,9,SRC + lxvd2x VSR(S1),9,SRC IF_LE(<vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask>) @@ -278,14 +267,14 @@ IF_LE(<vperm S0,S0,S0,swap_mask li 10,0x10 .align 5 L2x_round_loop: - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vcipher S0,S0,K vcipher S1,S1,K addi 10,10,0x10 bdnz L2x_round_loop - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vcipherlast S0,S0,K vcipherlast S1,S1,K @@ -293,9 +282,9 @@ L2x_round_loop: IF_LE(<vperm S0,S0,S0,swap_mask vperm S1,S1,S1,swap_mask>) - stxvd2x S0X,0,DST + stxvd2x VSR(S0),0,DST li 9,0x10 - stxvd2x S1X,9,DST + stxvd2x VSR(S1),9,DST addi SRC,SRC,0x20 addi DST,DST,0x20 @@ -306,10 +295,10 @@ L1x: cmpldi LENGTH,0 beq Ldone - lxvd2x KX,0,KEYS + lxvd2x VSR(K),0,KEYS vperm K,K,K,swap_mask - lxvd2x S0X,0,SRC + lxvd2x VSR(S0),0,SRC IF_LE(<vperm S0,S0,S0,swap_mask>) @@ -319,19 +308,19 @@ IF_LE(<vperm S0,S0,S0,swap_mask>) li 10,0x10 .align 5 L1x_round_loop: - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vcipher S0,S0,K addi 10,10,0x10 bdnz L1x_round_loop - lxvd2x KX,10,KEYS + lxvd2x VSR(K),10,KEYS vperm K,K,K,swap_mask vcipherlast S0,S0,K IF_LE(<vperm S0,S0,S0,swap_mask>) - stxvd2x S0X,0,DST + stxvd2x VSR(S0),0,DST Ldone: blr -- 2.17.1

2 4

[Patch] "PowerPC64" Add README (Reformatted)
by Maamoun TK 29 Aug '20

29 Aug '20

--- powerpc64/README | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 powerpc64/README diff --git a/powerpc64/README b/powerpc64/README new file mode 100644 index 00000000..19351be8 --- /dev/null +++ b/powerpc64/README @@ -0,0 +1,73 @@ +General-Purpose Register Conventions + +Register Status Use + +GPR0 volatile In function prologs. +GPR1 dedicated Stack pointer. +GPR2 dedicated Table of Contents (TOC) pointer. +GPR3 volatile First word of a function's argument list; + first word of a scalar function return. +GPR4 volatile Second word of a function's argument list; + second word of a scalar function return. +GPR5 volatile Third word of a function's argument list. +GPR6 volatile Fourth word of a function's argument list. +GPR7 volatile Fifth word of a function's argument list. +GPR8 volatile Sixth word of a function's argument list. +GPR9 volatile Seventh word of a function's argument list. +GPR10 volatile Eighth word of a function's argument list. +GPR11 volatile In calls by pointer and as an environment pointer + for languages that require it (for example, PASCAL). +GPR12 volatile For special exception handling required by certain + languages and in glink code. +GPR13 reserved Reserved under 64-bit environment; + not restored across system calls. +GPR14:GPR31 nonvolatile These registers must be preserved across + a function call. + +Vector Register Conventions + +Register Status + +VR0:V19 Volatile +VR20:VR31 Nonvolatile (extended ABI mode) their values are preserved + across function calls + +Addressing memory + +There are many ways to reference data, in the sake of writing +position-independent code the current implementation uses GOT-indirect +addressing (Accessing data through the global offset table): +1. Define data in .data section +2. Load the address of data into register from the global offset table + e.g. ld 7, my_var@got(2) +3. Use the address to load the value of data into register + e.g. ld 3, 0(7) +Refer to [2] for more information about referencing data + +VSX instructions "lxvd2x/stxvd2x" are used to load and store data to +memory instead of VR instructions "lvx/stvx" as it produces a fewer +instructions "lvx/stvx" can be used to load/store data into storage +operands but additional instructions are needed to access unaligned +storage operands, refer to "6.4.1 Accessing Unaligned Storage Operands" +in [3] to see an example of accessing unaligned storage operands. +"lxvd2x/stxvd2x" can be used to load/store data into unaligned storage +operands but permuting is needed for loading and storing data in +little-endian mode VSX registers are defined with "X" suffix +TODO: use architecture 3.0 instructions "lxv/stxv" instead for POWER9 + and newer + +Function Prologue + +Big-endian systems only support ELFv1 ABI which requires the following +steps in the function prologue: +1. Write the "official procedure descriptor" in ".opd","aw" section +2. Write procedure description for .my_func in my_func label +3. Switch back to ".text" section for program code +4. Label the beginning of the code .my_func +Refer to [1] for more information +Little-endian systems are compatible with ELFv2 ABI, an example of +function prologue for ELFv2 ABI can be seen in [2] + +[1] http://www.ibm.com/developerworks/linux/library/l-powasm1.html +[2] https://openpowerfoundation.org/?resource_lib=64-bit-elf-v2-abi-specificati… +[3] https://openpowerfoundation.org/?resource_lib=ibm-power-isa-version-2-07-b -- 2.17.1

3 5

[PATCH 4/6] "PowerPC64" Add fat build
by Maamoun TK 20 Aug '20

20 Aug '20

--- aes-decrypt-internal.c | 10 ++ aes-encrypt-internal.c | 10 ++ fat-ppc.c | 173 +++++++++++++++++++++++++++++++ fat-setup.h | 9 ++ powerpc64/fat/aes-decrypt-internal-2.asm | 37 +++++++ powerpc64/fat/aes-encrypt-internal-2.asm | 37 +++++++ powerpc64/fat/gcm-hash.asm | 40 +++++++ 7 files changed, 316 insertions(+) create mode 100644 fat-ppc.c create mode 100644 powerpc64/fat/aes-decrypt-internal-2.asm create mode 100644 powerpc64/fat/aes-encrypt-internal-2.asm create mode 100644 powerpc64/fat/gcm-hash.asm diff --git a/aes-decrypt-internal.c b/aes-decrypt-internal.c index 709c52f9..6326befb 100644 --- a/aes-decrypt-internal.c +++ b/aes-decrypt-internal.c @@ -40,6 +40,16 @@ #include "aes-internal.h" #include "macros.h" +/* For fat builds */ +#if HAVE_NATIVE_aes_decrypt +void +_nettle_aes_decrypt_c(unsigned rounds, const uint32_t *keys, + const struct aes_table *T, + size_t length, uint8_t *dst, + const uint8_t *src); +#define _nettle_aes_decrypt _nettle_aes_decrypt_c +#endif + void _nettle_aes_decrypt(unsigned rounds, const uint32_t *keys, const struct aes_table *T, diff --git a/aes-encrypt-internal.c b/aes-encrypt-internal.c index 9f61386d..7ff4ca40 100644 --- a/aes-encrypt-internal.c +++ b/aes-encrypt-internal.c @@ -40,6 +40,16 @@ #include "aes-internal.h" #include "macros.h" +/* For fat builds */ +#if HAVE_NATIVE_aes_encrypt +void +_nettle_aes_encrypt_c(unsigned rounds, const uint32_t *keys, + const struct aes_table *T, + size_t length, uint8_t *dst, + const uint8_t *src); +#define _nettle_aes_encrypt _nettle_aes_encrypt_c +#endif + void _nettle_aes_encrypt(unsigned rounds, const uint32_t *keys, const struct aes_table *T, diff --git a/fat-ppc.c b/fat-ppc.c new file mode 100644 index 00000000..e09b2097 --- /dev/null +++ b/fat-ppc.c @@ -0,0 +1,173 @@ +/* fat-ppc.c + + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +*/ + +#define _GNU_SOURCE + +#if HAVE_CONFIG_H +# include "config.h" +#endif + +#include <assert.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#if defined(__FreeBSD__) && __FreeBSD__ < 12 +#include <sys/sysctl.h> +#else +#include <sys/auxv.h> +#endif + +#include "nettle-types.h" + +#include "aes-internal.h" +#include "gcm.h" +#include "fat-setup.h" + +/* Define from arch/powerpc/include/uapi/asm/cputable.h in Linux kernel */ +#ifndef PPC_FEATURE2_VEC_CRYPTO +#define PPC_FEATURE2_VEC_CRYPTO 0x02000000 +#endif + +struct ppc_features +{ + int have_crypto_ext; +}; + +static void +get_ppc_features (struct ppc_features *features) +{ + unsigned long hwcap2 = 0; +#if defined(__FreeBSD__) +#if __FreeBSD__ < 12 + size_t len = sizeof(hwcap2); + sysctlbyname("hw.cpu_features2", &hwcap2, &len, NULL, 0); +#else + elf_aux_info(AT_HWCAP2, &hwcap2, sizeof(hwcap2)); +#endif +#else + hwcap2 = getauxval(AT_HWCAP2); +#endif + features->have_crypto_ext = + (hwcap2 & PPC_FEATURE2_VEC_CRYPTO) == PPC_FEATURE2_VEC_CRYPTO ? 1 : 0; +} + +DECLARE_FAT_FUNC(_nettle_aes_encrypt, aes_crypt_internal_func) +DECLARE_FAT_FUNC_VAR(aes_encrypt, aes_crypt_internal_func, c) +DECLARE_FAT_FUNC_VAR(aes_encrypt, aes_crypt_internal_func, ppc64) + +DECLARE_FAT_FUNC(_nettle_aes_decrypt, aes_crypt_internal_func) +DECLARE_FAT_FUNC_VAR(aes_decrypt, aes_crypt_internal_func, c) +DECLARE_FAT_FUNC_VAR(aes_decrypt, aes_crypt_internal_func, ppc64) + +#if GCM_TABLE_BITS == 8 +DECLARE_FAT_FUNC(_nettle_gcm_init_key, gcm_init_key_func) +DECLARE_FAT_FUNC_VAR(gcm_init_key, gcm_init_key_func, c) +DECLARE_FAT_FUNC_VAR(gcm_init_key, gcm_init_key_func, ppc64) + +DECLARE_FAT_FUNC(_nettle_gcm_hash, gcm_hash_func) +DECLARE_FAT_FUNC_VAR(gcm_hash, gcm_hash_func, c) +DECLARE_FAT_FUNC_VAR(gcm_hash, gcm_hash_func, ppc64) +#endif /* GCM_TABLE_BITS == 8 */ + +DECLARE_FAT_FUNC(_nettle_gcm_fill, gcm_fill_func) +DECLARE_FAT_FUNC_VAR(gcm_fill, gcm_fill_func, c) +DECLARE_FAT_FUNC_VAR(gcm_fill, gcm_fill_func, ppc64) + +static void CONSTRUCTOR +fat_init (void) +{ + struct ppc_features features; + int verbose; + + get_ppc_features (&features); + + verbose = getenv (ENV_VERBOSE) != NULL; + if (verbose) + fprintf (stderr, "libnettle: cpu features: %s\n", + features.have_crypto_ext ? "crypto extensions" : ""); + + if (features.have_crypto_ext) + { + if (verbose) + fprintf (stderr, "libnettle: enabling arch 2.07 code.\n"); + _nettle_aes_encrypt_vec = _nettle_aes_encrypt_ppc64; + _nettle_aes_decrypt_vec = _nettle_aes_decrypt_ppc64; +#if GCM_TABLE_BITS == 8 + /* Make sure _nettle_gcm_init_key_vec function is compatible + with _nettle_gcm_hash_vec function e.g. _nettle_gcm_init_key_c() + fills gcm_key table with values that are incompatible with + _nettle_gcm_hash_ppc64() */ + _nettle_gcm_init_key_vec = _nettle_gcm_init_key_ppc64; + _nettle_gcm_hash_vec = _nettle_gcm_hash_ppc64; +#endif /* GCM_TABLE_BITS == 8 */ + _nettle_gcm_fill_vec = _nettle_gcm_fill_ppc64; + } + else + { + _nettle_aes_encrypt_vec = _nettle_aes_encrypt_c; + _nettle_aes_decrypt_vec = _nettle_aes_decrypt_c; +#if GCM_TABLE_BITS == 8 + _nettle_gcm_init_key_vec = _nettle_gcm_init_key_c; + _nettle_gcm_hash_vec = _nettle_gcm_hash_c; +#endif /* GCM_TABLE_BITS == 8 */ + _nettle_gcm_fill_vec = _nettle_gcm_fill_c; + } +} + +DEFINE_FAT_FUNC(_nettle_aes_encrypt, void, + (unsigned rounds, const uint32_t *keys, + const struct aes_table *T, + size_t length, uint8_t *dst, + const uint8_t *src), + (rounds, keys, T, length, dst, src)) + +DEFINE_FAT_FUNC(_nettle_aes_decrypt, void, + (unsigned rounds, const uint32_t *keys, + const struct aes_table *T, + size_t length, uint8_t *dst, + const uint8_t *src), + (rounds, keys, T, length, dst, src)) + +#if GCM_TABLE_BITS == 8 +DEFINE_FAT_FUNC(_nettle_gcm_init_key, void, + (union nettle_block16 *table), + (table)) + +DEFINE_FAT_FUNC(_nettle_gcm_hash, void, + (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data), + (key, x, length, data)) +#endif /* GCM_TABLE_BITS == 8 */ + +DEFINE_FAT_FUNC(_nettle_gcm_fill, void, + (uint8_t *ctr, size_t blocks, + union nettle_block16 *buffer), + (ctr, blocks, buffer)) diff --git a/fat-setup.h b/fat-setup.h index 58b687fd..9793eebb 100644 --- a/fat-setup.h +++ b/fat-setup.h @@ -161,6 +161,15 @@ typedef void aes_crypt_internal_func (unsigned rounds, const uint32_t *keys, size_t length, uint8_t *dst, const uint8_t *src); +#if GCM_TABLE_BITS == 8 +typedef void gcm_init_key_func (union nettle_block16 *table); + +typedef void gcm_hash_func (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data); + +typedef void gcm_fill_func (uint8_t *ctr, size_t blocks, union nettle_block16 *buffer); +#endif /* GCM_TABLE_BITS == 8 */ + typedef void *(memxor_func)(void *dst, const void *src, size_t n); typedef void salsa20_core_func (uint32_t *dst, const uint32_t *src, unsigned rounds); diff --git a/powerpc64/fat/aes-decrypt-internal-2.asm b/powerpc64/fat/aes-decrypt-internal-2.asm new file mode 100644 index 00000000..593bf6cf --- /dev/null +++ b/powerpc64/fat/aes-decrypt-internal-2.asm @@ -0,0 +1,37 @@ +C powerpc64/fat/aes-decrypt-internal-2.asm + + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +dnl PROLOGUE(_nettle_aes_decrypt) picked up by configure + +define(<fat_transform>, <$1_ppc64>) +include_src(<powerpc64/P8/aes-decrypt-internal.asm>) diff --git a/powerpc64/fat/aes-encrypt-internal-2.asm b/powerpc64/fat/aes-encrypt-internal-2.asm new file mode 100644 index 00000000..73a4d543 --- /dev/null +++ b/powerpc64/fat/aes-encrypt-internal-2.asm @@ -0,0 +1,37 @@ +C powerpc64/fat/aes-encrypt-internal-2.asm + + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +dnl PROLOGUE(_nettle_aes_encrypt) picked up by configure + +define(<fat_transform>, <$1_ppc64>) +include_src(<powerpc64/P8/aes-encrypt-internal.asm>) diff --git a/powerpc64/fat/gcm-hash.asm b/powerpc64/fat/gcm-hash.asm new file mode 100644 index 00000000..dbb827db --- /dev/null +++ b/powerpc64/fat/gcm-hash.asm @@ -0,0 +1,40 @@ +C powerpc64/fat/gcm-hash.asm + + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +dnl picked up by configure +dnl PROLOGUE(_nettle_gcm_init_key) +dnl PROLOGUE(_nettle_gcm_hash) +dnl PROLOGUE(_nettle_gcm_fill) + +define(<fat_transform>, <$1_ppc64>) +include_src(<powerpc64/P8/gcm-hash.asm>) -- 2.17.1

3 5

[PATCH] "PowerPC64" Detect and check ABI
by Maamoun TK 20 Aug '20

20 Aug '20

--- configure.ac | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/configure.ac b/configure.ac index 49db7af1..6db3fc42 100644 --- a/configure.ac +++ b/configure.ac @@ -322,6 +322,17 @@ case "$host_cpu" in AC_TRY_COMPILE([ #if defined(__sgi) && defined(__LP64__) #error 64-bit mips +#endif + ], [], [ + ABI=32 + ], [ + ABI=64 + ]) + ;; + powerpc*) + AC_TRY_COMPILE([ +#if defined(__PPC64__) +#error 64-bit powerpc #endif ], [], [ ABI=32 @@ -438,13 +449,15 @@ if test "x$enable_assembler" = xyes ; then fi fi ;; - *powerpc64*) - if test "x$enable_fat" = xyes ; then - asm_path="powerpc64/fat powerpc64" - OPT_NETTLE_SOURCES="fat-ppc.c $OPT_NETTLE_SOURCES" - else - if test "x$enable_power_crypto_ext" = xyes ; then - asm_path="powerpc64/P8 powerpc64" + powerpc*) + if test "$ABI" = 64 ; then + if test "x$enable_fat" = xyes ; then + asm_path="powerpc64/fat powerpc64" + OPT_NETTLE_SOURCES="fat-ppc.c $OPT_NETTLE_SOURCES" + else + if test "x$enable_power_crypto_ext" = xyes ; then + asm_path="powerpc64/p8 powerpc64" + fi fi fi ;; -- 2.17.1

1 0

[PATCH] Check for ENV_OVERRIDE in get_ppc_features()
by Maamoun TK 02 Aug '20

02 Aug '20

This patch doesn't add FAT_TEST_LIST to ppc in configure.ac because the check-fat patch hasn't merged to the power-asm-wip branch. It can be enabled easily by adding FAT_TEST_LIST="crypto_ext" to the ppc condition in configure.ac once both patches are merged together. Regards, Mamone --- fat-ppc.c | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/fat-ppc.c b/fat-ppc.c index eca689fe..e0b50199 100644 --- a/fat-ppc.c +++ b/fat-ppc.c @@ -68,26 +68,48 @@ struct ppc_features int have_crypto_ext; }; +#define MATCH(s, slen, literal, llen) \ + ((slen) == (llen) && memcmp ((s), (literal), llen) == 0) + static void get_ppc_features (struct ppc_features *features) { + const char *s; + features->have_crypto_ext = 0; + + s = secure_getenv (ENV_OVERRIDE); + if (s) + for (;;) + { + const char *sep = strchr (s, ','); + size_t length = sep ? (size_t) (sep - s) : strlen(s); + + if (MATCH (s, length, "crypto_ext", 10)) + features->have_crypto_ext = 1; + if (!sep) + break; + s = sep + 1; + } + else + { #if defined(_AIX) && defined(__power_8_andup) - features->have_crypto_ext = __power_8_andup() != 0 ? 1 : 0; + features->have_crypto_ext = __power_8_andup() != 0 ? 1 : 0; #else - unsigned long hwcap2 = 0; + unsigned long hwcap2 = 0; #if defined(__linux__) - hwcap2 = getauxval(AT_HWCAP2); + hwcap2 = getauxval(AT_HWCAP2); #elif defined(__FreeBSD__) #if __FreeBSD__ >= 12 - elf_aux_info(AT_HWCAP2, &hwcap2, sizeof(hwcap2)); + elf_aux_info(AT_HWCAP2, &hwcap2, sizeof(hwcap2)); #else - size_t len = sizeof(hwcap2); - sysctlbyname("hw.cpu_features2", &hwcap2, &len, NULL, 0); + size_t len = sizeof(hwcap2); + sysctlbyname("hw.cpu_features2", &hwcap2, &len, NULL, 0); #endif #endif - features->have_crypto_ext = - (hwcap2 & PPC_FEATURE2_VEC_CRYPTO) == PPC_FEATURE2_VEC_CRYPTO ? 1 : 0; + features->have_crypto_ext = + (hwcap2 & PPC_FEATURE2_VEC_CRYPTO) == PPC_FEATURE2_VEC_CRYPTO ? 1 : 0; #endif + } } DECLARE_FAT_FUNC(_nettle_aes_encrypt, aes_crypt_internal_func) -- 2.17.1

1 0

[PATCH 2/6] "PowerPC64" Add optimized AES [Enc|Dec]
by Maamoun TK 01 Aug '20

01 Aug '20

I measured the latency and throughput of vcipher/vncipher/vxor instructions for POWER8 vcipher/vncipher throughput 6 instructions per cycle latency 0.91 clock cycles vxor throughput 6 instructions per cycle latency 0.32 clock cycles So the ideal option for POWER8 is processing 8 blocks, it has +12% performance over processing 4 blocks. --- powerpc64/P8/aes-decrypt-internal.asm | 367 ++++++++++++++++++++++++++++++++++ powerpc64/P8/aes-encrypt-internal.asm | 344 +++++++++++++++++++++++++++++++ 2 files changed, 711 insertions(+) create mode 100644 powerpc64/P8/aes-decrypt-internal.asm create mode 100644 powerpc64/P8/aes-encrypt-internal.asm diff --git a/powerpc64/P8/aes-decrypt-internal.asm b/powerpc64/P8/aes-decrypt-internal.asm new file mode 100644 index 00000000..f5d64548 --- /dev/null +++ b/powerpc64/P8/aes-decrypt-internal.asm @@ -0,0 +1,367 @@ +C powerpc64/P8/aes-decrypt-internal.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<ROUNDS>, <3>) +define(<KEYS>, <4>) +define(<LENGTH>, <6>) +define(<DST>, <7>) +define(<SRC>, <8>) + +define(<swap_mask>, <0>) + +define(<K>, <1>) +define(<S0>, <2>) +define(<S1>, <3>) +define(<S2>, <4>) +define(<S3>, <5>) +define(<S4>, <6>) +define(<S5>, <7>) +define(<S6>, <8>) +define(<S7>, <9>) + +define(<KX>, <33>) +define(<S0X>, <34>) +define(<S1X>, <35>) +define(<S2X>, <36>) +define(<S3X>, <37>) +define(<S4X>, <38>) +define(<S5X>, <39>) +define(<S6X>, <40>) +define(<S7X>, <41>) + +C ZERO vector register is used in place of RoundKey +C for vncipher instruction because the order of InvMixColumns +C and Xor processes are flipped in that instruction. +C The Xor process with RoundKey is executed afterward. +define(<ZERO>, <10>) + +.file "aes-decrypt-internal.asm" + +IF_LE(<.abiversion 2>) +.text + + C _aes_decrypt(unsigned rounds, const uint32_t *keys, + C const struct aes_table *T, + C size_t length, uint8_t *dst, + C uint8_t *src) + +define(<FUNC_ALIGN>, <5>) +PROLOGUE(_nettle_aes_decrypt) + vxor ZERO,ZERO,ZERO + + DATA_LOAD_VEC(swap_mask,.swap_mask,5) + + subi ROUNDS,ROUNDS,1 + srdi LENGTH,LENGTH,4 + + srdi 5,LENGTH,3 #8x loop count + cmpldi 5,0 + beq L4x + + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 25,0x10 + li 26,0x20 + li 27,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + +.align 5 +Lx8_loop: + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + lxvd2x S1X,25,SRC + lxvd2x S2X,26,SRC + lxvd2x S3X,27,SRC + lxvd2x S4X,28,SRC + lxvd2x S5X,29,SRC + lxvd2x S6X,30,SRC + lxvd2x S7X,31,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L8x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vncipher S4,S4,ZERO + vncipher S5,S5,ZERO + vncipher S6,S6,ZERO + vncipher S7,S7,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + addi 10,10,0x10 + bdnz L8x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + vncipherlast S4,S4,K + vncipherlast S5,S5,K + vncipherlast S6,S6,K + vncipherlast S7,S7,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask>) + + stxvd2x S0X,0,DST + stxvd2x S1X,25,DST + stxvd2x S2X,26,DST + stxvd2x S3X,27,DST + stxvd2x S4X,28,DST + stxvd2x S5X,29,DST + stxvd2x S6X,30,DST + stxvd2x S7X,31,DST + + addi SRC,SRC,0x80 + addi DST,DST,0x80 + subic. 5,5,1 + bne Lx8_loop + + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi LENGTH,LENGTH,61 + +L4x: + srdi 5,LENGTH,2 + cmpldi 5,0 + beq L2x + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L4x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + addi 10,10,0x10 + bdnz L4x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask>) + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + + addi SRC,SRC,0x40 + addi DST,DST,0x40 + + clrldi LENGTH,LENGTH,62 + +L2x: + srdi 5,LENGTH,1 + cmpldi 5,0 + beq L1x + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L2x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vxor S0,S0,K + vxor S1,S1,K + addi 10,10,0x10 + bdnz L2x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask>) + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + + addi SRC,SRC,0x20 + addi DST,DST,0x20 + + clrldi LENGTH,LENGTH,63 + +L1x: + cmpldi LENGTH,0 + beq Ldone + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask>) + + vxor S0,S0,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L1x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vxor S0,S0,K + addi 10,10,0x10 + bdnz L1x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + +IF_LE(<vperm S0,S0,S0,swap_mask>) + + stxvd2x S0X,0,DST + +Ldone: + blr +EPILOGUE(_nettle_aes_decrypt) + + .data + .align 4 +.swap_mask: +IF_LE(<.byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7>) +IF_BE(<.byte 3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12>) diff --git a/powerpc64/P8/aes-encrypt-internal.asm b/powerpc64/P8/aes-encrypt-internal.asm new file mode 100644 index 00000000..3e0fa6f0 --- /dev/null +++ b/powerpc64/P8/aes-encrypt-internal.asm @@ -0,0 +1,344 @@ +C powerpc64/P8/aes-encrypt-internal.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<ROUNDS>, <3>) +define(<KEYS>, <4>) +define(<LENGTH>, <6>) +define(<DST>, <7>) +define(<SRC>, <8>) + +define(<swap_mask>, <0>) + +define(<K>, <1>) +define(<S0>, <2>) +define(<S1>, <3>) +define(<S2>, <4>) +define(<S3>, <5>) +define(<S4>, <6>) +define(<S5>, <7>) +define(<S6>, <8>) +define(<S7>, <9>) + +define(<KX>, <33>) +define(<S0X>, <34>) +define(<S1X>, <35>) +define(<S2X>, <36>) +define(<S3X>, <37>) +define(<S4X>, <38>) +define(<S5X>, <39>) +define(<S6X>, <40>) +define(<S7X>, <41>) + +.file "aes-encrypt-internal.asm" + +IF_LE(<.abiversion 2>) +.text + + C _aes_encrypt(unsigned rounds, const uint32_t *keys, + C const struct aes_table *T, + C size_t length, uint8_t *dst, + C uint8_t *src) + +define(<FUNC_ALIGN>, <5>) +PROLOGUE(_nettle_aes_encrypt) + DATA_LOAD_VEC(swap_mask,.swap_mask,5) + + subi ROUNDS,ROUNDS,1 + srdi LENGTH,LENGTH,4 + + srdi 5,LENGTH,3 #8x loop count + cmpldi 5,0 + beq L4x + + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 25,0x10 + li 26,0x20 + li 27,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + +.align 5 +Lx8_loop: + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + lxvd2x S1X,25,SRC + lxvd2x S2X,26,SRC + lxvd2x S3X,27,SRC + lxvd2x S4X,28,SRC + lxvd2x S5X,29,SRC + lxvd2x S6X,30,SRC + lxvd2x S7X,31,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L8x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + vcipher S4,S4,K + vcipher S5,S5,K + vcipher S6,S6,K + vcipher S7,S7,K + addi 10,10,0x10 + bdnz L8x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + vcipherlast S4,S4,K + vcipherlast S5,S5,K + vcipherlast S6,S6,K + vcipherlast S7,S7,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask>) + + stxvd2x S0X,0,DST + stxvd2x S1X,25,DST + stxvd2x S2X,26,DST + stxvd2x S3X,27,DST + stxvd2x S4X,28,DST + stxvd2x S5X,29,DST + stxvd2x S6X,30,DST + stxvd2x S7X,31,DST + + addi SRC,SRC,0x80 + addi DST,DST,0x80 + subic. 5,5,1 + bne Lx8_loop + + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi LENGTH,LENGTH,61 + +L4x: + srdi 5,LENGTH,2 + cmpldi 5,0 + beq L2x + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L4x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + addi 10,10,0x10 + bdnz L4x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask>) + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + + addi SRC,SRC,0x40 + addi DST,DST,0x40 + + clrldi LENGTH,LENGTH,62 + +L2x: + srdi 5,LENGTH,1 + cmpldi 5,0 + beq L1x + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L2x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + addi 10,10,0x10 + bdnz L2x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask>) + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + + addi SRC,SRC,0x20 + addi DST,DST,0x20 + + clrldi LENGTH,LENGTH,63 + +L1x: + cmpldi LENGTH,0 + beq Ldone + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask>) + + vxor S0,S0,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L1x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + addi 10,10,0x10 + bdnz L1x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + +IF_LE(<vperm S0,S0,S0,swap_mask>) + + stxvd2x S0X,0,DST + +Ldone: + blr +EPILOGUE(_nettle_aes_encrypt) + + .data + .align 4 +.swap_mask: +IF_LE(<.byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7>) +IF_BE(<.byte 3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12>) -- 2.17.1

2 10

[PATCH] "PowerPC64" Add README
by Maamoun TK 01 Aug '20

01 Aug '20

--- powerpc64/README | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 powerpc64/README diff --git a/powerpc64/README b/powerpc64/README new file mode 100644 index 00000000..6d6f3fbb --- /dev/null +++ b/powerpc64/README @@ -0,0 +1,86 @@ +General-Purpose Register Conventions + +Register Status Use + +GPR0 volatile In function prologs. +GPR1 dedicated Stack pointer. +GPR2 dedicated Table of Contents (TOC) pointer. +GPR3 volatile First word of a function's argument list; + first word of a scalar function return. +GPR4 volatile Second word of a function's argument list; + second word of a scalar function return. +GPR5 volatile Third word of a function's argument list. +GPR6 volatile Fourth word of a function's argument list. +GPR7 volatile Fifth word of a function's argument list. +GPR8 volatile Sixth word of a function's argument list. +GPR9 volatile Seventh word of a function's argument list. +GPR10 volatile Eighth word of a function's argument list. +GPR11 volatile In calls by pointer and as an environment pointer for languages + that require it (for example, PASCAL). +GPR12 volatile For special exception handling required by certain languages and in + glink code. +GPR13 reserved Reserved under 64-bit environment; not restored across system calls. +GPR14:GPR31 nonvolatile These registers must be preserved across a function call. + +Vector Register Conventions + +Register Status + +VR0 Volatile +VR1 Volatile +VR2 Volatile +VR3 Volatile +VR4 Volatile +VR5 Volatile +VR6 Volatile +VR7 Volatile +VR8 Volatile +VR9 Volatile +VR10 Volatile +VR11 Volatile +VR12 Volatile +VR13 Volatile +VR14 Volatile +VR15 Volatile +VR16 Volatile +VR17 Volatile +VR18 Volatile +VR19 Volatile +VR20:31 Nonvolatile (extended ABI mode) their values are preserved across function calls + +Addressing memory + +There are many ways to reference data, in the sake of writing position-independent code +the current implementations uses GOT-indirect addressing (Accessing data through the global +offset table): +1. Define data in .data section +2. Load the address of data into register from the global offset table e.g. ld 7, my_var@got(2) +3. Use the address to load the value of data into register e.g. ld 3, 0(7) +Refer to [2] for more information + +VSX instructions "lxvd2x/stxvd2x" are used to load and store data to memory +instead of VR instructions "lvx/stvx" as it produces a fewer instructions +"lvx/stvx" can be used to load/store data into storage operands +but additional instructions are needed to access unaligned storage operands, refer +to "6.4.1 Accessing Unaligned Storage Operands" in [3] +to see an example of accessing unaligned storage operands. "lxvd2x/stxvd2x" can +be used to load/store data into unaligned storage operands but permuting is needed +for loading and storing data in little-endian mode +VSX registers are defined with "X" suffix +TODO: use architecture 3.0 instructions "lxv/stxv" instead for POWER9 and newer + +Function Prologue + +Big-endian systems only support ELFv1 ABI which requires the following steps in the +function prologue: +1. Write the "official procedure descriptor" in ".opd","aw" section +2. Write procedure description for .my_func in my_func label +3. Switch back to ".text" section for program code +4. Label the beginning of the code .my_func +Refer to [1] for more information +Little-endian systems are compatible with ELFv2 ABI, an example of function prologue +for ELFv2 ABI can be seen in [2] + +[1] http://www.ibm.com/developerworks/linux/library/l-powasm1.html +[2] https://openpowerfoundation.org/?resource_lib=64-bit-elf-v2-abi-specificati… +[3] https://openpowerfoundation.org/?resource_lib=ibm-power-isa-version-2-07-b -- 2.17.1

2 2

Optimizing salsa20
by nisse＠lysator.liu.se 22 Jul '20

22 Jul '20

I've written some new ARM Neon assembly for salsa20. See https://gitlab.com/gnutls/nettle/-/commit/2ac58a1ce729a6cfe1d3703f4deb6da88…, when configured with --enable-arm-neon. It interleaves the processing of two blocks, which gives a speedup of 50% -- 100% on the ARM cores where I've tested it. Before merging, I need to fix fat builds to use the new code on processors that support it. To make it work also on big-endian ARM, I'd need some help. (I think the qemu-user package supports big-endian ARM, at least, it includes a program named qemu-armeb. But I'm missing a cross compiler and cross debugger). I'd like to do the same for x86_64. And for chacha, it might give even greater speedup to interleave processing of three blocks, which may be possible since I think chacha needs fewer registers for temporaries. For both x86_64 and ARM neon, the current code uses 128-bit wide registers. Processors with 256-bit wide simd registers (at least 16 of them) could do twice as many blocks at a time. Regards, /Niels -- Niels Möller. PGP-encrypted email is preferred. Keyid 368C6677. Internet email is subject to wholesale government surveillance.

3 5

Add ppc64le arch to Gitlab CI
by Maamoun TK 21 Jul '20

21 Jul '20

I investigated this issue. The Debian image used for Gitlab CI only supports the following archs amd64 mips armhf arm64. To add a new arch, this arch should be added to the sources list of apt and install the required packages to build and check nettle library.

3 4

[PowerPC64] Add AIX to cpu detection
by Maamoun TK 20 Jul '20

20 Jul '20

--- fat-ppc.c | 29 ++++++++++++++++++++--------- 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/fat-ppc.c b/fat-ppc.c index e09b2097..eca689fe 100644 --- a/fat-ppc.c +++ b/fat-ppc.c @@ -39,10 +39,17 @@ #include <stdio.h> #include <stdlib.h> #include <string.h> -#if defined(__FreeBSD__) && __FreeBSD__ < 12 -#include <sys/sysctl.h> -#else + +#if defined(_AIX) +#include <sys/systemcfg.h> +#elif defined(__linux__) +#include <sys/auxv.h> +#elif defined(__FreeBSD__) +#if __FreeBSD__ >= 12 #include <sys/auxv.h> +#else +#include <sys/sysctl.h> +#endif #endif #include "nettle-types.h" @@ -64,19 +71,23 @@ struct ppc_features static void get_ppc_features (struct ppc_features *features) { +#if defined(_AIX) && defined(__power_8_andup) + features->have_crypto_ext = __power_8_andup() != 0 ? 1 : 0; +#else unsigned long hwcap2 = 0; -#if defined(__FreeBSD__) -#if __FreeBSD__ < 12 +#if defined(__linux__) + hwcap2 = getauxval(AT_HWCAP2); +#elif defined(__FreeBSD__) +#if __FreeBSD__ >= 12 + elf_aux_info(AT_HWCAP2, &hwcap2, sizeof(hwcap2)); +#else size_t len = sizeof(hwcap2); sysctlbyname("hw.cpu_features2", &hwcap2, &len, NULL, 0); -#else - elf_aux_info(AT_HWCAP2, &hwcap2, sizeof(hwcap2)); #endif -#else - hwcap2 = getauxval(AT_HWCAP2); #endif features->have_crypto_ext = (hwcap2 & PPC_FEATURE2_VEC_CRYPTO) == PPC_FEATURE2_VEC_CRYPTO ? 1 : 0; +#endif } DECLARE_FAT_FUNC(_nettle_aes_encrypt, aes_crypt_internal_func) -- 2.17.1

2 2

checking ECC point at infinity
by Daiki Ueno 20 Jul '20

20 Jul '20

Hello, SP800-56A (revision 3) section 5.6.2.3.3 now mandates a check that the generated public key (Q) multiplied by the curve order (n) results in an identity element (= an infinity point). It seem that it is not possible to implement this check with the Nettle's public API. The attached patch naively multiplies Q by n but it causes the valgrind errors below. As it works with the curve order minus 1, I added the following check instead in my library, though I'm not sure if this satisfies the original requirement: P = (n - 1) * Q where P's x-coordinate is the same as Q's, and P also lies on the same curve so that indicates P + Q (= n * Q) is an infinity point, according to the group law. Is there a better (direct) way to implement the check? Suggestions appreciated. Conditional jump or move depends on uninitialised value(s) at 0x4880DFB: _nettle_ecc_mul_a (ecc-mul-a.c:145) by 0x48815F8: nettle_ecc_point_mul (ecc-point-mul.c:55) by 0x4012EB: main (ecc-test.c:36) Conditional jump or move depends on uninitialised value(s) at 0x487D1D9: _nettle_sec_tabselect (sec-tabselect.c:54) by 0x4880E1B: _nettle_ecc_mul_a (ecc-mul-a.c:147) by 0x48815F8: nettle_ecc_point_mul (ecc-point-mul.c:55) by 0x4012EB: main (ecc-test.c:36) Conditional jump or move depends on uninitialised value(s) at 0x487E0F3: _nettle_ecc_mod_add (ecc-mod-arith.c:53) by 0x487F51B: _nettle_ecc_dup_jj (ecc-dup-jj.c:81) by 0x4880E66: _nettle_ecc_mul_a (ecc-mul-a.c:171) by 0x48815F8: nettle_ecc_point_mul (ecc-point-mul.c:55) by 0x4012EB: main (ecc-test.c:36) Regards, -- Daiki Ueno #include <nettle/ecdsa.h> #include <nettle/ecc-curve.h> #include <assert.h> #include <string.h> static void myrandom (void *ctx, size_t length, uint8_t *dst) { memset (dst, 0, length); *dst = 0xff; } int main (void) { const struct ecc_curve *curve = nettle_get_secp_256r1 (); struct ecc_point pub; struct ecc_scalar key; mpz_t nz, x, y; struct ecc_scalar n; struct ecc_point r; ecc_point_init (&pub, curve); ecc_scalar_init (&key, curve); ecc_scalar_init (&n, curve); ecc_point_init (&r, curve); ecdsa_generate_keypair (&pub, &key, NULL, myrandom); /* Calculate P = nQ and check if it is an infinity point. */ mpz_init_set_str (nz, "ffffffff00000000ffffffffffffffff" "bce6faada7179e84f3b9cac2fc632551", 16); ecc_scalar_set (&n, nz); mpz_clear (nz); ecc_point_mul (&r, &n, &pub); mpz_init (x); mpz_init (y); ecc_point_get (&r, x, y); assert (mpz_cmp_ui (x, 0) == 0); assert (mpz_cmp_ui (y, 0) == 0); mpz_clear (x); mpz_clear (y); ecc_point_clear (&pub); ecc_scalar_clear (&key); ecc_scalar_clear (&n); ecc_point_clear (&r); return 0; } /** * Local variables: * compile-command: "gcc -o ecc-test ecc-test.c `pkg-config hogweed --cflags --libs` -lgmp" * End: */

2 2

Handling of ORIGIN-based rpaths and runpaths
by Jeffrey Walton 20 Jul '20

20 Jul '20

Hi Everyone, I build OpenSSH for downlevel machines, like OS X and Solaris. I install into /opt/ssh, and I use a runpath of $ORIGIN/../lib. The LDFLAGS are: -Wl,-runpath,'$ORIGIN/../lib' -Wl,-runpath,$(prefix)/lib -Wl,--enable-new-dtags I noticed Nettle does not handle the ORIGIN-based runpath properly: /opt/ssh/lib/libhogweed.so.6: RUNPATH: RIGIN/../lib:/opt/ssh/lib RPATH: RIGIN/../lib:/opt/ssh/lib And: /opt/ssh/lib/libnettle.so.8.0: RUNPATH: RIGIN/../lib:/opt/ssh/lib RPATH: RIGIN/../lib:/opt/ssh/lib Besides $ORIGIN, Nettle may encounter $LIB and $PLATFORM. Also see ld.so man page (https://man7.org/linux/man-pages/man8/ld.so.8.html). I believe the fix is to escape the dollar sign in the makefile. That is, when Nettle creates its makefiles, it must use: -Wl,-runpath,'$$ORIGIN/../lib' ... Jeff

2 4

Assembly recipe missing ASFLAGS
by Jeffrey Walton 17 Jul '20

17 Jul '20

Hi Everyone, I noticed *.s recipes do not use ASFLAGS. ASFLAGS includes -Wa,--noexecstack:

1 1

relocation error: R_AMD64_64
by Jeffrey Walton 17 Jul '20

17 Jul '20

Hi Everyone, I'm seeing a fair amount of "ld: warning: relocation error: R_AMD64_64" when linking from nettle-3.6 release tarball: gcc -g2 -O2 -m64 -march=native -fPIC -pthread -ggdb3 -Wall -W -Wno-sign-compare -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wpointer-arith -Wbad-function-cast -Wnested-externs -L.. -L/opt/ssh/lib -m64 -Wl,-R,'RIGIN/../lib' -Wl,-R,/opt/ssh/lib -Wl,-z,now rsa-sign.o io.o read_rsa_key.o \ -lhogweed -lnettle -lgmp -ldl -lpthread -o rsa-sign ld: warning: relocation error: R_AMD64_64: file ../getopt.o: symbol optarg: external symbolic relocation against non-allocatable section .debug_info; cannot be processed at runtime: relocation ignored It looks like getopt was not built correctly. Jeff

2 1

[PATCH 1/4] Check for PowerPC64 assembly if crypto extensions are available
by Maamoun TK 14 Jul '20

14 Jul '20

--- Makefile.in | 2 +- aclocal.m4 | 52 ++++++++++++++++++++++++++++++++++++++++ configure.ac | 11 +++++++++ powerpc64/README | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++++ powerpc64/machine.m4 | 24 +++++++++++++++++++ 5 files changed, 155 insertions(+), 1 deletion(-) create mode 100644 powerpc64/README create mode 100644 powerpc64/machine.m4 diff --git a/Makefile.in b/Makefile.in index 77efb5c9..1508e8f4 100644 --- a/Makefile.in +++ b/Makefile.in @@ -603,7 +603,7 @@ distdir: $(DISTFILES) done set -e; for d in sparc32 sparc64 x86 \ x86_64 x86_64/aesni x86_64/sha_ni x86_64/fat \ - arm arm/neon arm/v6 arm/fat ; do \ + arm arm/neon arm/v6 arm/fat powerpc64 ; do \ mkdir "$(distdir)/$$d" ; \ find "$(srcdir)/$$d" -maxdepth 1 '(' -name '*.asm' -o -name '*.m4' ')' \ -exec cp '{}' "$(distdir)/$$d" ';' ; \ diff --git a/aclocal.m4 b/aclocal.m4 index 513b2df4..e38181ad 100644 --- a/aclocal.m4 +++ b/aclocal.m4 @@ -462,6 +462,58 @@ foo: fi ]) +dnl NETTLE_CHECK_POWER_CRYPTO_EXT +dnl --------------------- +dnl Check if POWER crypto extensions should be used. +dnl Obeys enable_power_crypto_ext, which should be set earlier. +AC_DEFUN([NETTLE_CHECK_POWER_CRYPTO_EXT], +[if test "$enable_power_crypto_ext" = auto ; then + if test "$cross_compiling" = yes ; then + dnl Check if compiler/assembler accepts it. + AC_CACHE_CHECK([if assembler accepts crypto instructions], + nettle_cv_asm_power_vcrypto, + [GMP_TRY_ASSEMBLE([ +.text +foo: + vcipher 0, 0, 1 +], + [nettle_cv_asm_power_vcrypto=yes], + [nettle_cv_asm_power_vcrypto=no])]) + enable_power_crypto_ext="$nettle_cv_asm_power_vcrypto" + else + AC_CACHE_CHECK([if crypto extensions supported], + nettle_cv_asm_power_vcrypto, + [AC_RUN_IFELSE([AC_LANG_PROGRAM([[ +#if defined(__FreeBSD__) && __FreeBSD__ < 12 +#include <sys/sysctl.h> +#else +#include <sys/auxv.h> +#endif +// Define from arch/powerpc/include/uapi/asm/cputable.h in Linux kernel +#ifndef PPC_FEATURE2_VEC_CRYPTO +#define PPC_FEATURE2_VEC_CRYPTO 0x02000000 +#endif + unsigned long hwcap2 = 0; + ]], [[ +#if defined(__FreeBSD__) +#if __FreeBSD__ < 12 + size_t len = sizeof(hwcap2); + sysctlbyname("hw.cpu_features2", &hwcap2, &len, NULL, 0); +#else + elf_aux_info(AT_HWCAP2, &hwcap2, sizeof(hwcap2)); +#endif +#else + hwcap2 = getauxval(AT_HWCAP2); +#endif + return (hwcap2 & PPC_FEATURE2_VEC_CRYPTO) == PPC_FEATURE2_VEC_CRYPTO ? 0 : 1; + ]])], + [nettle_cv_asm_power_vcrypto=yes], + [nettle_cv_asm_power_vcrypto=no])]) + enable_power_crypto_ext="$nettle_cv_asm_power_vcrypto" + fi +fi +]) + dnl NETTLE_CHECK_IFUNC dnl ------------------ dnl Check if __attribute__ ((ifunc(...))) works diff --git a/configure.ac b/configure.ac index 1c0b7393..cc0d67ec 100644 --- a/configure.ac +++ b/configure.ac @@ -89,6 +89,10 @@ AC_ARG_ENABLE(x86-sha-ni, AC_HELP_STRING([--enable-x86-sha-ni], [Enable x86_64 sha_ni instructions. (default=no)]),, [enable_x86_sha_ni=no]) +AC_ARG_ENABLE(power-crypto-ext, + AC_HELP_STRING([--enable-power-crypto-ext], [Enable POWER crypto extentions. (default=auto)]),, + [enable_power_crypto_ext=auto]) + AC_ARG_ENABLE(mini-gmp, AC_HELP_STRING([--enable-mini-gmp], [Enable mini-gmp, used instead of libgmp.]),, [enable_mini_gmp=no]) @@ -434,6 +438,13 @@ if test "x$enable_assembler" = xyes ; then fi fi ;; + *powerpc64*) + NETTLE_CHECK_POWER_CRYPTO_EXT + + if test "x$enable_power_crypto_ext" = xyes ; then + asm_path=powerpc64 + fi + ;; *) enable_assembler=no ;; diff --git a/powerpc64/README b/powerpc64/README new file mode 100644 index 00000000..1890ee9a --- /dev/null +++ b/powerpc64/README @@ -0,0 +1,67 @@ +General-Purpose Register Conventions + +Register Status Use + +GPR0 volatile In function prologs. +GPR1 dedicated Stack pointer. +GPR2 dedicated Table of Contents (TOC) pointer. +GPR3 volatile First word of a function's argument list; + first word of a scalar function return. +GPR4 volatile Second word of a function's argument list; + second word of a scalar function return. +GPR5 volatile Third word of a function's argument list. +GPR6 volatile Fourth word of a function's argument list. +GPR7 volatile Fifth word of a function's argument list. +GPR8 volatile Sixth word of a function's argument list. +GPR9 volatile Seventh word of a function's argument list. +GPR10 volatile Eighth word of a function's argument list. +GPR11 volatile In calls by pointer and as an environment pointer for languages + that require it (for example, PASCAL). +GPR12 volatile For special exception handling required by certain languages and in + glink code. +GPR13 reserved Reserved under 64-bit environment; not restored across system calls. +GPR14:GPR31 nonvolatile These registers must be preserved across a function call. + +Vector Register Conventions + +Register Status + +VR0 Volatile +VR1 Volatile +VR2 Volatile +VR3 Volatile +VR4 Volatile +VR5 Volatile +VR6 Volatile +VR7 Volatile +VR8 Volatile +VR9 Volatile +VR10 Volatile +VR11 Volatile +VR12 Volatile +VR13 Volatile +VR14 Volatile +VR15 Volatile +VR16 Volatile +VR17 Volatile +VR18 Volatile +VR19 Volatile +VR20:31 Nonvolatile (extended ABI mode) their values are preserved across function calls + +Addressing memory + +There are many ways to reference data, the current implementations uses GOT-indirect addressing +(Accessing data through the global offset table): +1. Define data in .data section +2. Load the address of data into register from the global offset table e.g. ld 7, my_var@got(2) +3. Use the address to load the value of data into register e.g. ld 3, 0(7) + +VSX instructions (lxvd2x and stxvd2x) are used to load and store data to memory +instead of VR instructions (lvx and stvx) as it produces a fewer instructions +(lvx and stvx) can be used to load and store data into storage operands +but additional instructions are needed to access unaligned storage operands, please +refer to "6.4.1 Accessing Unaligned Storage Operands" in "POWER ISA Version 2.07 B" +to see an example of accessing unaligned storage operands (lxvd2x and stxvd2x) can +be used to load and store data into unaligned storage operands but permuting is needed +for loading and storing data in little-endian mode +VSX registers are defined with "X" suffix diff --git a/powerpc64/machine.m4 b/powerpc64/machine.m4 new file mode 100644 index 00000000..c8005cd8 --- /dev/null +++ b/powerpc64/machine.m4 @@ -0,0 +1,24 @@ +define(<PROLOGUE>, +<ifelse(WORDS_BIGENDIAN,no, +<.align 5 +.globl C_NAME($1) +DECLARE_FUNC(C_NAME($1)) +C_NAME($1): +addis 2,12,(.TOC.-C_NAME($1))@ha +addi 2,2,(.TOC.-C_NAME($1))@l +.localentry C_NAME($1), .-C_NAME($1)>, +<.globl C_NAME($1) +DECLARE_FUNC(C_NAME($1)) +.section ".opd","aw" +.align 3 +C_NAME($1): +.quad .C_NAME($1),.TOC.@tocbase,0 +.previous +.align 5 +.C_NAME($1):>)>) + +define(<EPILOGUE>, +<ifelse(WORDS_BIGENDIAN,no, +<.size C_NAME($1), . - C_NAME($1)>, +<.size .C_NAME($1), . - .C_NAME($1) +.size C_NAME($1), . - .C_NAME($1)>)>) -- 2.17.1

2 2

[PATCH 4/4] Add AES [Enc|Dec] optimized implementations for PowerPC64
by Maamoun TK 14 Jul '20

14 Jul '20

--- powerpc64/aes-decrypt-internal.asm | 579 +++++++++++++++++++++++++++++++++++++ powerpc64/aes-encrypt-internal.asm | 540 ++++++++++++++++++++++++++++++++++ 2 files changed, 1119 insertions(+) create mode 100644 powerpc64/aes-decrypt-internal.asm create mode 100644 powerpc64/aes-encrypt-internal.asm diff --git a/powerpc64/aes-decrypt-internal.asm b/powerpc64/aes-decrypt-internal.asm new file mode 100644 index 00000000..b194d65f --- /dev/null +++ b/powerpc64/aes-decrypt-internal.asm @@ -0,0 +1,579 @@ +C powerpc64/aes-decrypt-internal.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<ROUNDS>, <3>) +define(<KEYS>, <4>) +define(<LENGTH>, <6>) +define(<DST>, <7>) +define(<SRC>, <8>) + +define(<swap_mask>, <0>) + +define(<K>, <1>) +define(<S0>, <2>) +define(<S1>, <3>) +define(<S2>, <4>) +define(<S3>, <5>) +define(<S4>, <6>) +define(<S5>, <7>) +define(<S6>, <8>) +define(<S7>, <9>) +define(<S8>, <10>) +define(<S9>, <11>) +define(<S10>, <12>) +define(<S11>, <13>) +define(<S12>, <14>) +define(<S13>, <15>) +define(<S14>, <16>) +define(<S15>, <17>) + +define(<KX>, <33>) +define(<S0X>, <34>) +define(<S1X>, <35>) +define(<S2X>, <36>) +define(<S3X>, <37>) +define(<S4X>, <38>) +define(<S5X>, <39>) +define(<S6X>, <40>) +define(<S7X>, <41>) +define(<S8X>, <42>) +define(<S9X>, <43>) +define(<S10X>, <44>) +define(<S11X>, <45>) +define(<S12X>, <46>) +define(<S13X>, <47>) +define(<S14X>, <48>) +define(<S15X>, <49>) + +C ZERO vector register is used in place of RoundKey +C for vncipher instruction because the order of InvMixColumns +C and Xor processes are flipped in that instruction. +C The Xor process with RoundKey is executed afterward. +define(<ZERO>, <18>) + +.file "aes-decrypt-internal.asm" + +IF_LE(<.abiversion 2>) +.text + + C _aes_decrypt(unsigned rounds, const uint32_t *keys, + C const struct aes_table *T, + C size_t length, uint8_t *dst, + C uint8_t *src) + +PROLOGUE(_nettle_aes_decrypt) + vxor ZERO,ZERO,ZERO + + ld 5,.swap_mask@got(TOCP) + lvx swap_mask,0,5 + + subi ROUNDS,ROUNDS,1 + srdi LENGTH,LENGTH,4 + + srdi 5,LENGTH,4 #16x loop count + cmpldi 5,0 + beq L8x + + std 17,-120(SP); + std 18,-112(SP); + std 19,-104(SP); + std 20,-96(SP); + std 21,-88(SP); + std 22,-80(SP); + std 23,-72(SP); + std 24,-64(SP); + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 17,0x10 + li 18,0x20 + li 19,0x30 + li 20,0x40 + li 21,0x50 + li 22,0x60 + li 23,0x70 + li 24,0x80 + li 25,0x90 + li 26,0xA0 + li 27,0xB0 + li 28,0xC0 + li 29,0xD0 + li 30,0xE0 + li 31,0xF0 + +.align 5 +Lx16_loop: + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + lxvd2x S1X,17,SRC + lxvd2x S2X,18,SRC + lxvd2x S3X,19,SRC + lxvd2x S4X,20,SRC + lxvd2x S5X,21,SRC + lxvd2x S6X,22,SRC + lxvd2x S7X,23,SRC + lxvd2x S8X,24,SRC + lxvd2x S9X,25,SRC + lxvd2x S10X,26,SRC + lxvd2x S11X,27,SRC + lxvd2x S12X,28,SRC + lxvd2x S13X,29,SRC + lxvd2x S14X,30,SRC + lxvd2x S15X,31,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L16x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vncipher S4,S4,ZERO + vncipher S5,S5,ZERO + vncipher S6,S6,ZERO + vncipher S7,S7,ZERO + vncipher S8,S8,ZERO + vncipher S9,S9,ZERO + vncipher S10,S10,ZERO + vncipher S11,S11,ZERO + vncipher S12,S12,ZERO + vncipher S13,S13,ZERO + vncipher S14,S14,ZERO + vncipher S15,S15,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + addi 10,10,0x10 + bdnz L16x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + vncipherlast S4,S4,K + vncipherlast S5,S5,K + vncipherlast S6,S6,K + vncipherlast S7,S7,K + vncipherlast S8,S8,K + vncipherlast S9,S9,K + vncipherlast S10,S10,K + vncipherlast S11,S11,K + vncipherlast S12,S12,K + vncipherlast S13,S13,K + vncipherlast S14,S14,K + vncipherlast S15,S15,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask>) + + stxvd2x S0X,0,DST + stxvd2x S1X,17,DST + stxvd2x S2X,18,DST + stxvd2x S3X,19,DST + stxvd2x S4X,20,DST + stxvd2x S5X,21,DST + stxvd2x S6X,22,DST + stxvd2x S7X,23,DST + stxvd2x S8X,24,DST + stxvd2x S9X,25,DST + stxvd2x S10X,26,DST + stxvd2x S11X,27,DST + stxvd2x S12X,28,DST + stxvd2x S13X,29,DST + stxvd2x S14X,30,DST + stxvd2x S15X,31,DST + + addi SRC,SRC,0x100 + addi DST,DST,0x100 + subic. 5,5,1 + bne Lx16_loop + + ld 17,-120(SP); + ld 18,-112(SP); + ld 19,-104(SP); + ld 20,-96(SP); + ld 21,-88(SP); + ld 22,-80(SP); + ld 23,-72(SP); + ld 24,-64(SP); + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi LENGTH,LENGTH,60 + +L8x: + srdi 5,LENGTH,3 + cmpldi 5,0 + beq L4x + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + addi 9,9,0x10 + lxvd2x S4X,9,SRC + addi 9,9,0x10 + lxvd2x S5X,9,SRC + addi 9,9,0x10 + lxvd2x S6X,9,SRC + addi 9,9,0x10 + lxvd2x S7X,9,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L8x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vncipher S4,S4,ZERO + vncipher S5,S5,ZERO + vncipher S6,S6,ZERO + vncipher S7,S7,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + addi 10,10,0x10 + bdnz L8x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + vncipherlast S4,S4,K + vncipherlast S5,S5,K + vncipherlast S6,S6,K + vncipherlast S7,S7,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask>) + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + addi 9,9,0x10 + stxvd2x S4X,9,DST + addi 9,9,0x10 + stxvd2x S5X,9,DST + addi 9,9,0x10 + stxvd2x S6X,9,DST + addi 9,9,0x10 + stxvd2x S7X,9,DST + + addi SRC,SRC,0x80 + addi DST,DST,0x80 + + clrldi LENGTH,LENGTH,61 + +L4x: + srdi 5,LENGTH,2 + cmpldi 5,0 + beq L2x + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L4x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vncipher S2,S2,ZERO + vncipher S3,S3,ZERO + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + addi 10,10,0x10 + bdnz L4x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + vncipherlast S2,S2,K + vncipherlast S3,S3,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask>) + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + + addi SRC,SRC,0x40 + addi DST,DST,0x40 + + clrldi LENGTH,LENGTH,62 + +L2x: + srdi 5,LENGTH,1 + cmpldi 5,0 + beq L1x + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L2x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vncipher S1,S1,ZERO + vxor S0,S0,K + vxor S1,S1,K + addi 10,10,0x10 + bdnz L2x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + vncipherlast S1,S1,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask>) + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + + addi SRC,SRC,0x20 + addi DST,DST,0x20 + + clrldi LENGTH,LENGTH,63 + +L1x: + cmpldi LENGTH,0 + beq Ldone + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask>) + + vxor S0,S0,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L1x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipher S0,S0,ZERO + vxor S0,S0,K + addi 10,10,0x10 + bdnz L1x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vncipherlast S0,S0,K + +IF_LE(<vperm S0,S0,S0,swap_mask>) + + stxvd2x S0X,0,DST + +Ldone: + blr +EPILOGUE(_nettle_aes_decrypt) + + .data + .align 4 +.swap_mask: +IF_LE(<.byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7>) +IF_BE(<.byte 3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12>) diff --git a/powerpc64/aes-encrypt-internal.asm b/powerpc64/aes-encrypt-internal.asm new file mode 100644 index 00000000..76630222 --- /dev/null +++ b/powerpc64/aes-encrypt-internal.asm @@ -0,0 +1,540 @@ +C powerpc64/aes-encrypt-internal.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<ROUNDS>, <3>) +define(<KEYS>, <4>) +define(<LENGTH>, <6>) +define(<DST>, <7>) +define(<SRC>, <8>) + +define(<swap_mask>, <0>) + +define(<K>, <1>) +define(<S0>, <2>) +define(<S1>, <3>) +define(<S2>, <4>) +define(<S3>, <5>) +define(<S4>, <6>) +define(<S5>, <7>) +define(<S6>, <8>) +define(<S7>, <9>) +define(<S8>, <10>) +define(<S9>, <11>) +define(<S10>, <12>) +define(<S11>, <13>) +define(<S12>, <14>) +define(<S13>, <15>) +define(<S14>, <16>) +define(<S15>, <17>) + +define(<KX>, <33>) +define(<S0X>, <34>) +define(<S1X>, <35>) +define(<S2X>, <36>) +define(<S3X>, <37>) +define(<S4X>, <38>) +define(<S5X>, <39>) +define(<S6X>, <40>) +define(<S7X>, <41>) +define(<S8X>, <42>) +define(<S9X>, <43>) +define(<S10X>, <44>) +define(<S11X>, <45>) +define(<S12X>, <46>) +define(<S13X>, <47>) +define(<S14X>, <48>) +define(<S15X>, <49>) + +.file "aes-encrypt-internal.asm" + +IF_LE(<.abiversion 2>) +.text + + C _aes_encrypt(unsigned rounds, const uint32_t *keys, + C const struct aes_table *T, + C size_t length, uint8_t *dst, + C uint8_t *src) + +PROLOGUE(_nettle_aes_encrypt) + ld 5,.swap_mask@got(TOCP) + lvx swap_mask,0,5 + + subi ROUNDS,ROUNDS,1 + srdi LENGTH,LENGTH,4 + + srdi 5,LENGTH,4 #16x loop count + cmpldi 5,0 + beq L8x + + std 17,-120(SP); + std 18,-112(SP); + std 19,-104(SP); + std 20,-96(SP); + std 21,-88(SP); + std 22,-80(SP); + std 23,-72(SP); + std 24,-64(SP); + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 17,0x10 + li 18,0x20 + li 19,0x30 + li 20,0x40 + li 21,0x50 + li 22,0x60 + li 23,0x70 + li 24,0x80 + li 25,0x90 + li 26,0xA0 + li 27,0xB0 + li 28,0xC0 + li 29,0xD0 + li 30,0xE0 + li 31,0xF0 + +.align 5 +Lx16_loop: + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + lxvd2x S1X,17,SRC + lxvd2x S2X,18,SRC + lxvd2x S3X,19,SRC + lxvd2x S4X,20,SRC + lxvd2x S5X,21,SRC + lxvd2x S6X,22,SRC + lxvd2x S7X,23,SRC + lxvd2x S8X,24,SRC + lxvd2x S9X,25,SRC + lxvd2x S10X,26,SRC + lxvd2x S11X,27,SRC + lxvd2x S12X,28,SRC + lxvd2x S13X,29,SRC + lxvd2x S14X,30,SRC + lxvd2x S15X,31,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + vxor S8,S8,K + vxor S9,S9,K + vxor S10,S10,K + vxor S11,S11,K + vxor S12,S12,K + vxor S13,S13,K + vxor S14,S14,K + vxor S15,S15,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L16x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + vcipher S4,S4,K + vcipher S5,S5,K + vcipher S6,S6,K + vcipher S7,S7,K + vcipher S8,S8,K + vcipher S9,S9,K + vcipher S10,S10,K + vcipher S11,S11,K + vcipher S12,S12,K + vcipher S13,S13,K + vcipher S14,S14,K + vcipher S15,S15,K + addi 10,10,0x10 + bdnz L16x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + vcipherlast S4,S4,K + vcipherlast S5,S5,K + vcipherlast S6,S6,K + vcipherlast S7,S7,K + vcipherlast S8,S8,K + vcipherlast S9,S9,K + vcipherlast S10,S10,K + vcipherlast S11,S11,K + vcipherlast S12,S12,K + vcipherlast S13,S13,K + vcipherlast S14,S14,K + vcipherlast S15,S15,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask + vperm S8,S8,S8,swap_mask + vperm S9,S9,S9,swap_mask + vperm S10,S10,S10,swap_mask + vperm S11,S11,S11,swap_mask + vperm S12,S12,S12,swap_mask + vperm S13,S13,S13,swap_mask + vperm S14,S14,S14,swap_mask + vperm S15,S15,S15,swap_mask>) + + stxvd2x S0X,0,DST + stxvd2x S1X,17,DST + stxvd2x S2X,18,DST + stxvd2x S3X,19,DST + stxvd2x S4X,20,DST + stxvd2x S5X,21,DST + stxvd2x S6X,22,DST + stxvd2x S7X,23,DST + stxvd2x S8X,24,DST + stxvd2x S9X,25,DST + stxvd2x S10X,26,DST + stxvd2x S11X,27,DST + stxvd2x S12X,28,DST + stxvd2x S13X,29,DST + stxvd2x S14X,30,DST + stxvd2x S15X,31,DST + + addi SRC,SRC,0x100 + addi DST,DST,0x100 + subic. 5,5,1 + bne Lx16_loop + + ld 17,-120(SP); + ld 18,-112(SP); + ld 19,-104(SP); + ld 20,-96(SP); + ld 21,-88(SP); + ld 22,-80(SP); + ld 23,-72(SP); + ld 24,-64(SP); + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi LENGTH,LENGTH,60 + +L8x: + srdi 5,LENGTH,3 + cmpldi 5,0 + beq L4x + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + addi 9,9,0x10 + lxvd2x S4X,9,SRC + addi 9,9,0x10 + lxvd2x S5X,9,SRC + addi 9,9,0x10 + lxvd2x S6X,9,SRC + addi 9,9,0x10 + lxvd2x S7X,9,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + vxor S4,S4,K + vxor S5,S5,K + vxor S6,S6,K + vxor S7,S7,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L8x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + vcipher S4,S4,K + vcipher S5,S5,K + vcipher S6,S6,K + vcipher S7,S7,K + addi 10,10,0x10 + bdnz L8x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + vcipherlast S4,S4,K + vcipherlast S5,S5,K + vcipherlast S6,S6,K + vcipherlast S7,S7,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask + vperm S4,S4,S4,swap_mask + vperm S5,S5,S5,swap_mask + vperm S6,S6,S6,swap_mask + vperm S7,S7,S7,swap_mask>) + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + addi 9,9,0x10 + stxvd2x S4X,9,DST + addi 9,9,0x10 + stxvd2x S5X,9,DST + addi 9,9,0x10 + stxvd2x S6X,9,DST + addi 9,9,0x10 + stxvd2x S7X,9,DST + + addi SRC,SRC,0x80 + addi DST,DST,0x80 + + clrldi LENGTH,LENGTH,61 + +L4x: + srdi 5,LENGTH,2 + cmpldi 5,0 + beq L2x + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + addi 9,9,0x10 + lxvd2x S2X,9,SRC + addi 9,9,0x10 + lxvd2x S3X,9,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + vxor S2,S2,K + vxor S3,S3,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L4x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + vcipher S2,S2,K + vcipher S3,S3,K + addi 10,10,0x10 + bdnz L4x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + vcipherlast S2,S2,K + vcipherlast S3,S3,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask + vperm S2,S2,S2,swap_mask + vperm S3,S3,S3,swap_mask>) + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + addi 9,9,0x10 + stxvd2x S2X,9,DST + addi 9,9,0x10 + stxvd2x S3X,9,DST + + addi SRC,SRC,0x40 + addi DST,DST,0x40 + + clrldi LENGTH,LENGTH,62 + +L2x: + srdi 5,LENGTH,1 + cmpldi 5,0 + beq L1x + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + li 9,0x10 + lxvd2x S1X,9,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask>) + + vxor S0,S0,K + vxor S1,S1,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L2x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + vcipher S1,S1,K + addi 10,10,0x10 + bdnz L2x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + vcipherlast S1,S1,K + +IF_LE(<vperm S0,S0,S0,swap_mask + vperm S1,S1,S1,swap_mask>) + + stxvd2x S0X,0,DST + li 9,0x10 + stxvd2x S1X,9,DST + + addi SRC,SRC,0x20 + addi DST,DST,0x20 + + clrldi LENGTH,LENGTH,63 + +L1x: + cmpldi LENGTH,0 + beq Ldone + + lxvd2x KX,0,KEYS + vperm K,K,K,swap_mask + + lxvd2x S0X,0,SRC + +IF_LE(<vperm S0,S0,S0,swap_mask>) + + vxor S0,S0,K + + mtctr ROUNDS + li 10,0x10 +.align 5 +L1x_round_loop: + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipher S0,S0,K + addi 10,10,0x10 + bdnz L1x_round_loop + + lxvd2x KX,10,KEYS + vperm K,K,K,swap_mask + vcipherlast S0,S0,K + +IF_LE(<vperm S0,S0,S0,swap_mask>) + + stxvd2x S0X,0,DST + +Ldone: + blr +EPILOGUE(_nettle_aes_encrypt) + + .data + .align 4 +.swap_mask: +IF_LE(<.byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7>) +IF_BE(<.byte 3,2,1,0,7,6,5,4,11,10,9,8,15,14,13,12>) -- 2.17.1

2 2

[PATCH 6/6] "PowerPC64" Update configure.ac and Makefile.in
by Maamoun TK 14 Jul '20

14 Jul '20

--- Makefile.in | 2 +- configure.ac | 21 +++++++++++++++++++-- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/Makefile.in b/Makefile.in index 042ebe5f..92216d55 100644 --- a/Makefile.in +++ b/Makefile.in @@ -603,7 +603,7 @@ distdir: $(DISTFILES) done set -e; for d in sparc32 sparc64 x86 \ x86_64 x86_64/aesni x86_64/sha_ni x86_64/fat \ - arm arm/neon arm/v6 arm/fat ; do \ + arm arm/neon arm/v6 arm/fat powerpc64 powerpc64/P8 powerpc64/fat ; do \ mkdir "$(distdir)/$$d" ; \ find "$(srcdir)/$$d" -maxdepth 1 '(' -name '*.asm' -o -name '*.m4' ')' \ -exec cp '{}' "$(distdir)/$$d" ';' ; \ diff --git a/configure.ac b/configure.ac index a01eb7d3..49db7af1 100644 --- a/configure.ac +++ b/configure.ac @@ -89,6 +89,10 @@ AC_ARG_ENABLE(x86-sha-ni, AC_HELP_STRING([--enable-x86-sha-ni], [Enable x86_64 sha_ni instructions. (default=no)]),, [enable_x86_sha_ni=no]) +AC_ARG_ENABLE(power-crypto-ext, + AC_HELP_STRING([--enable-power-crypto-ext], [Enable POWER crypto extentions. (default=no)]),, + [enable_power_crypto_ext=no]) + AC_ARG_ENABLE(mini-gmp, AC_HELP_STRING([--enable-mini-gmp], [Enable mini-gmp, used instead of libgmp.]),, [enable_mini_gmp=no]) @@ -434,6 +438,16 @@ if test "x$enable_assembler" = xyes ; then fi fi ;; + *powerpc64*) + if test "x$enable_fat" = xyes ; then + asm_path="powerpc64/fat powerpc64" + OPT_NETTLE_SOURCES="fat-ppc.c $OPT_NETTLE_SOURCES" + else + if test "x$enable_power_crypto_ext" = xyes ; then + asm_path="powerpc64/P8 powerpc64" + fi + fi + ;; *) enable_assembler=no ;; @@ -453,7 +467,7 @@ asm_replace_list="aes-encrypt-internal.asm aes-decrypt-internal.asm \ sha3-permute.asm umac-nh.asm umac-nh-n.asm machine.m4" # Assembler files which generate additional object files if they are used. -asm_nettle_optional_list="gcm-hash8.asm cpuid.asm \ +asm_nettle_optional_list="gcm-hash.asm gcm-hash8.asm cpuid.asm \ aes-encrypt-internal-2.asm aes-decrypt-internal-2.asm memxor-2.asm \ chacha-core-internal-2.asm salsa20-2core.asm \ salsa20-core-internal-2.asm sha1-compress-2.asm sha256-compress-2.asm \ @@ -571,7 +585,10 @@ AH_VERBATIM([HAVE_NATIVE], #undef HAVE_NATIVE_ecc_secp384r1_redc #undef HAVE_NATIVE_ecc_secp521r1_modp #undef HAVE_NATIVE_ecc_secp521r1_redc -#undef HAVE_NATIVE_gcm_init_key8 +#undef HAVE_NATIVE_aes_decrypt +#undef HAVE_NATIVE_aes_encrypt +#undef HAVE_NATIVE_gcm_init_key +#undef HAVE_NATIVE_gcm_hash #undef HAVE_NATIVE_gcm_hash8 #undef HAVE_NATIVE_gcm_fill #undef HAVE_NATIVE_salsa20_core -- 2.17.1

1 0

[PATCH 5/6] "PowerPC64" Add README
by Maamoun TK 14 Jul '20

14 Jul '20

--- powerpc64/README | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 powerpc64/README diff --git a/powerpc64/README b/powerpc64/README new file mode 100644 index 00000000..f78357ab --- /dev/null +++ b/powerpc64/README @@ -0,0 +1,86 @@ +General-Purpose Register Conventions + +Register Status Use + +GPR0 volatile In function prologs. +GPR1 dedicated Stack pointer. +GPR2 dedicated Table of Contents (TOC) pointer. +GPR3 volatile First word of a function's argument list; + first word of a scalar function return. +GPR4 volatile Second word of a function's argument list; + second word of a scalar function return. +GPR5 volatile Third word of a function's argument list. +GPR6 volatile Fourth word of a function's argument list. +GPR7 volatile Fifth word of a function's argument list. +GPR8 volatile Sixth word of a function's argument list. +GPR9 volatile Seventh word of a function's argument list. +GPR10 volatile Eighth word of a function's argument list. +GPR11 volatile In calls by pointer and as an environment pointer for languages + that require it (for example, PASCAL). +GPR12 volatile For special exception handling required by certain languages and in + glink code. +GPR13 reserved Reserved under 64-bit environment; not restored across system calls. +GPR14:GPR31 nonvolatile These registers must be preserved across a function call. + +Vector Register Conventions + +Register Status + +VR0 Volatile +VR1 Volatile +VR2 Volatile +VR3 Volatile +VR4 Volatile +VR5 Volatile +VR6 Volatile +VR7 Volatile +VR8 Volatile +VR9 Volatile +VR10 Volatile +VR11 Volatile +VR12 Volatile +VR13 Volatile +VR14 Volatile +VR15 Volatile +VR16 Volatile +VR17 Volatile +VR18 Volatile +VR19 Volatile +VR20:31 Nonvolatile (extended ABI mode) their values are preserved across function calls + +Addressing memory + +There are many ways to reference data, in the sake of writing position-independent code +the current implementations uses GOT-indirect addressing (Accessing data through the global +offset table): +1. Define data in .data section +2. Load the address of data into register from the global offset table e.g. ld 7, my_var@got(2) +3. Use the address to load the value of data into register e.g. ld 3, 0(7) +Refer to [2] for more information + +VSX instructions "lxvd2x/stxvd2x" are used to load and store data to memory +instead of VR instructions "lvx/stvx" as it produces a fewer instructions +"lvx/stvx" can be used to load/store data into storage operands +but additional instructions are needed to access unaligned storage operands, refer +to "6.4.1 Accessing Unaligned Storage Operands" in [3] +to see an example of accessing unaligned storage operands. "lxvd2x/stxvd2x" can +be used to load/store data into unaligned storage operands but permuting is needed +for loading and storing data in little-endian mode +VSX registers are defined with "X" suffix +TODO: use architecture 3.0 instructions "lxv/stxv" instead for POWER9 and newer + +Function Prologue + +Big-endian systems only support ELFv1 ABI which requires the following steps in the +function prologue: +1. Write the "official procedure descriptor" in ".opd","aw" section +2. Write procedure description for .my_func in my_func label +3. Switch back to ".text" section for program code +4. Label the beginning of the code .my_func +Refer to [1] for more information +Little-endian systems are compatible with ELFv2 ABI, an example of function prologue +for ELFv2 ABI can be seen in [2] + +[1] http://www.ibm.com/developerworks/linux/library/l-powasm1.html +[2] https://openpowerfoundation.org/?resource_lib=64-bit-elf-v2-abi-specificati… +[3] https://openpowerfoundation.org/?resource_lib=ibm-power-isa-version-2-07-b -- 2.17.1

1 0

[PATCH 3/6] "PowerPC64" Add optimized GHASH
by Maamoun TK 14 Jul '20

14 Jul '20

--- gcm.c | 82 +++- powerpc64/P8/gcm-hash.asm | 998 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1066 insertions(+), 14 deletions(-) create mode 100644 powerpc64/P8/gcm-hash.asm diff --git a/gcm.c b/gcm.c index cf615daf..935d4420 100644 --- a/gcm.c +++ b/gcm.c @@ -140,6 +140,26 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) memcpy (x->b, Z.b, sizeof(Z)); } # elif GCM_TABLE_BITS == 8 +# if HAVE_NATIVE_gcm_init_key + +#define gcm_init_key _nettle_gcm_init_key +void +_nettle_gcm_init_key (union nettle_block16 *table); +/* For fat builds */ +void +_nettle_gcm_init_key_c (union nettle_block16 *table); +# endif /* HAVE_NATIVE_gcm_init_key */ +# if HAVE_NATIVE_gcm_hash + +#define gcm_hash _nettle_gcm_hash +void +_nettle_gcm_hash (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data); +/* For fat builds */ +void +_nettle_gcm_hash_c (const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data); +# endif /* HAVE_NATIVE_gcm_hash */ # if HAVE_NATIVE_gcm_hash8 #define gcm_hash _nettle_gcm_hash8 @@ -225,9 +245,45 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) #endif /* GCM_TABLE_BITS */ +#if HAVE_NATIVE_gcm_fill + +#define gcm_fill _nettle_gcm_fill +void +_nettle_gcm_fill (uint8_t *ctr, size_t blocks, union nettle_block16 *buffer); +/* For fat builds */ +void +_nettle_gcm_fill_c (uint8_t *ctr, size_t blocks, union nettle_block16 *buffer); +#endif /* HAVE_NATIVE_gcm_fill */ + /* Increment the rightmost 32 bits. */ #define INC32(block) INCREMENT(4, (block.b) + GCM_BLOCK_SIZE - 4) +#ifdef gcm_init_key +void +_nettle_gcm_init_key_c(union nettle_block16 *table) +#else +static void +gcm_init_key(union nettle_block16 *table) +#endif /* !gcm_init_key */ +{ +#if GCM_TABLE_BITS + /* Middle element if GCM_TABLE_BITS > 0, otherwise the first + element */ + unsigned i = (1<<GCM_TABLE_BITS)/2; + + /* Algorithm 3 from the gcm paper. First do powers of two, then do + the rest by adding. */ + while (i /= 2) + block16_mulx_ghash(&table[i], &table[2*i]); + for (i = 2; i < 1<<GCM_TABLE_BITS; i *= 2) + { + unsigned j; + for (j = 1; j < i; j++) + block16_xor3(&table[i+j], &table[i], &table[j]); + } +#endif +} + /* Initialization of GCM. * @ctx: The context of GCM * @cipher: The context of the underlying block cipher @@ -245,24 +301,18 @@ gcm_set_key(struct gcm_key *key, memset(key->h[0].b, 0, GCM_BLOCK_SIZE); f (cipher, GCM_BLOCK_SIZE, key->h[i].b, key->h[0].b); -#if GCM_TABLE_BITS - /* Algorithm 3 from the gcm paper. First do powers of two, then do - the rest by adding. */ - while (i /= 2) - block16_mulx_ghash(&key->h[i], &key->h[2*i]); - for (i = 2; i < 1<<GCM_TABLE_BITS; i *= 2) - { - unsigned j; - for (j = 1; j < i; j++) - block16_xor3(&key->h[i+j], &key->h[i],&key->h[j]); - } -#endif + gcm_init_key(key->h); } -#ifndef gcm_hash +#ifdef gcm_hash +void +_nettle_gcm_hash_c(const struct gcm_key *key, union nettle_block16 *x, + size_t length, const uint8_t *data) +#else static void gcm_hash(const struct gcm_key *key, union nettle_block16 *x, size_t length, const uint8_t *data) +#endif /* !gcm_hash */ { for (; length >= GCM_BLOCK_SIZE; length -= GCM_BLOCK_SIZE, data += GCM_BLOCK_SIZE) @@ -276,7 +326,6 @@ gcm_hash(const struct gcm_key *key, union nettle_block16 *x, gcm_gf_mul (x, key->h); } } -#endif /* !gcm_hash */ static void gcm_hash_sizes(const struct gcm_key *key, union nettle_block16 *x, @@ -333,9 +382,14 @@ gcm_update(struct gcm_ctx *ctx, const struct gcm_key *key, ctx->auth_size += length; } +#ifdef gcm_fill +void +_nettle_gcm_fill_c(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) +#else static nettle_fill16_func gcm_fill; static void gcm_fill(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) +#endif /* !gcm_fill */ { uint32_t c; diff --git a/powerpc64/P8/gcm-hash.asm b/powerpc64/P8/gcm-hash.asm new file mode 100644 index 00000000..b8f2178e --- /dev/null +++ b/powerpc64/P8/gcm-hash.asm @@ -0,0 +1,998 @@ +C powerpc64/P8/gcm-hash.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Alignment of gcm_key table elements, which is declared in gcm.h +define(<TableElemAlign>, <0x100>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<TABLE>, <3>) +define(<X>, <4>) +define(<LENGTH>, <5>) +define(<DATA>, <6>) + +define(<zero>, <0>) +define(<swap_mask>, <1>) +define(<hidw_mask>, <2>) +define(<lodw_mask>, <3>) +define(<poly>, <4>) +define(<poly_h>, <4>) +define(<poly_l>, <5>) +define(<RP>, <6>) +define(<Mh>, <7>) +define(<Ml>, <8>) +define(<H>, <9>) +define(<Hh>, <10>) +define(<Hl>, <11>) +define(<RP2>, <9>) +define(<M2h>, <10>) +define(<M2l>, <11>) + +define(<HX>, <41>) +define(<HhX>, <42>) +define(<HlX>, <43>) +define(<H_HhX>, <44>) +define(<H_HX>, <45>) +define(<H_HlX>, <46>) + +define(<sl1>, <1>) +define(<msb>, <5>) +define(<H2>, <6>) +define(<H2h>, <7>) +define(<H2l>, <8>) +define(<H_h>, <12>) +define(<H_m>, <13>) +define(<H_l>, <14>) +define(<H_Hh>, <12>) +define(<H_H>, <13>) +define(<H_Hl>, <14>) +define(<H_t>, <15>) +define(<H2_h>, <16>) +define(<H2_m>, <17>) +define(<H2_l>, <18>) +define(<H2_t>, <19>) + +define(<C0X>, <38>) +define(<C1X>, <39>) +define(<C2X>, <40>) +define(<C3X>, <44>) +define(<C4X>, <38>) +define(<C5X>, <39>) +define(<C6X>, <40>) +define(<C7X>, <44>) + +define(<CX>, <45>) + +define(<C0>, <6>) +define(<C1>, <7>) +define(<C2>, <8>) +define(<C3>, <12>) +define(<C4>, <6>) +define(<C5>, <7>) +define(<C6>, <8>) +define(<C7>, <12>) + +define(<C>, <13>) + +define(<Ch>, <14>) +define(<Cl>, <15>) +define(<Cm>, <16>) + +define(<C01h>, <14>) +define(<C01l>, <15>) +define(<C01>, <16>) +define(<C23h>, <17>) +define(<C23l>, <18>) +define(<C23>, <19>) +define(<C45h>, <20>) +define(<C45l>, <21>) +define(<C45>, <22>) +define(<C67h>, <6>) +define(<C67l>, <7>) +define(<C67>, <8>) + +define(<H21>, <9>) +define(<H21h>, <10>) +define(<H21l>, <11>) +define(<H43>, <23>) +define(<H43h>, <24>) +define(<H43l>, <25>) +define(<H65>, <26>) +define(<H65h>, <27>) +define(<H65l>, <28>) +define(<H87>, <29>) +define(<H87h>, <30>) +define(<H87l>, <31>) + +define(<H21X>, <41>) +define(<H21hX>, <42>) +define(<H21lX>, <43>) +define(<H43X>, <55>) +define(<H43hX>, <56>) +define(<H43lX>, <57>) +define(<H65X>, <58>) +define(<H65hX>, <59>) +define(<H65lX>, <60>) +define(<H87X>, <61>) +define(<H87hX>, <62>) +define(<H87lX>, <63>) + +# gcm_fill registers: + +define(<CTR>, <3>) +define(<BLOCKS>, <4>) +define(<BUFFER>, <5>) + +define(<CTR0>, <2>) +define(<CTR0S>, <3>) +define(<CTR1>, <4>) +define(<CTR2>, <5>) +define(<CTR3>, <6>) +define(<CTR4>, <7>) +define(<CTR5>, <8>) +define(<CTR6>, <9>) +define(<CTR7>, <10>) + +define(<CTR0X>, <34>) +define(<CTR0SX>, <35>) +define(<CTR1X>, <36>) +define(<CTR2X>, <37>) +define(<CTR3X>, <38>) +define(<CTR4X>, <39>) +define(<CTR5X>, <40>) +define(<CTR6X>, <41>) +define(<CTR7X>, <42>) + +define(<I1>, <11>) +define(<I2>, <12>) +define(<I3>, <13>) +define(<I4>, <14>) +define(<I5>, <15>) +define(<I6>, <16>) +define(<I7>, <17>) +define(<I8>, <18>) + +.file "gcm-hash.asm" + +IF_LE(<.abiversion 2>) +.text + + # void gcm_init_key (union gcm_block *table) + +define(<FUNC_ALIGN>, <5>) +PROLOGUE(_nettle_gcm_init_key) + DATA_LOAD_VEC(poly,.polynomial,7) +IF_LE(<DATA_LOAD_VEC(swap_mask,.swap_mask,7)>) + DATA_LOAD_VEC(hidw_mask,.hidw_mask,7) + DATA_LOAD_VEC(lodw_mask,.lodw_mask,7) + + li 10,8*TableElemAlign + lxvd2x HX,10,TABLE # load H +IF_LE(<vperm H,H,H,swap_mask>) + + # --- calculate H = H shift left 1 modulo polynomial --- + + vupkhsw msb,H # most significant bit word-extend + vspltisb sl1,1 # splat 1 for shift left + vspltw msb,msb,0 # most significant bit extend + vsl H,H,sl1 # H shift left 1 + vand msb,msb,poly + vxor zero,zero,zero + vxor H_t,H,msb + + vsldoi H,H_t,H_t,8 # doubleword swap + vsldoi Hh,H,zero,8 + vsldoi Hl,zero,H,8 + + # --- calculate H^2 = H*H --- + + # reduction pre-processing + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + # polynomial multiplication "classical" + vpmsumd H_h,H_t,Hh # H^1h*H^1h + vpmsumd H_l,H_t,Hl # H^1l*H^1l + vpmsumd H_m,H_t,H # H^1h*H^1l⊕H^1l*H^1h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] + vsldoi Ml,H_m,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor H_h,H_h,Mh # [2] + vxor H_l,H_l,Ml # [2] + vxor H_l,H_l,RP # [1] + + # reduction second phase + vpmsumd RP,H_l,poly_l + vxor H_h,H_l,H_h + vxor H2_t,H_h,RP + + vsldoi H2,H2_t,H2_t,8 + vsldoi H2h,H2,zero,8 + vsldoi H2l,zero,H2,8 + + # --- calculate [H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store H,[H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + li 8,0*TableElemAlign + li 9,1*TableElemAlign + li 10,2*TableElemAlign + stxvd2x HlX,8,TABLE + stxvd2x HX,9,TABLE + stxvd2x HhX,10,TABLE + + li 8,3*TableElemAlign + li 9,4*TableElemAlign + li 10,5*TableElemAlign + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^3,H^4 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^2l + vpmsumd H_m,H_t,H2 # H^1h*H^2l⊕H^1l*H^2h + vpmsumd H_h,H_t,H2h # H^1h*H^2h + vpmsumd H2_l,H2_t,H2l # H^2l*H^2l + vpmsumd H2_m,H2_t,H2 # H^2h*H^2l⊕H^2l*H^2h + vpmsumd H2_h,H2_t,H2h # H^2h*H^2h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^3 + vpmsumd RP2,H2_l,poly_h # [1] H^4 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^3 + vsldoi M2h,zero,H2_m,8 # [2] H^4 + vsldoi Ml,H_m,zero,8 # [2] H^3 + vsldoi M2l,H2_m,zero,8 # [2] H^4 + vsldoi RP,RP,RP,8 # [1] H^3 + vsldoi RP2,RP2,RP2,8 # [1] H^4 + vxor H_h,H_h,Mh # [2] H^3 + vxor H2_h,H2_h,M2h # [2] H^4 + vxor H_l,H_l,Ml # [2] H^3 + vxor H2_l,H2_l,M2l # [2] H^4 + vxor H_l,H_l,RP # [1] H^3 + vxor H2_l,H2_l,RP2 # [1] H^4 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^3 + vpmsumd RP2,H2_l,poly_l # H^4 + vxor H_h,H_l,H_h # H^3 + vxor H2_h,H2_l,H2_h # H^4 + vxor H_h,H_h,RP # H^3 + vxor H2_h,H2_h,RP2 # H^4 + + vsldoi H2,H2_h,H2_h,8 # H^4 + vsldoi H,H_h,H_h,8 # H^3 + vsldoi H2l,zero,H2,8 # H^4 + vsldoi H2h,H2,zero,8 # H^4 + + # --- calculate [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + li 8,6*TableElemAlign + li 9,7*TableElemAlign + li 10,8*TableElemAlign + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^5,H^6 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^4l + vpmsumd H_m,H_t,H2 # H^1h*H^4l⊕H^1l*H^4h + vpmsumd H_h,H_t,H2h # H^1h*H^4h + vpmsumd H2_l,H2_t,H2l # H^2l*H^4l + vpmsumd H2_m,H2_t,H2 # H^2h*H^4l⊕H^2l*H^4h + vpmsumd H2_h,H2_t,H2h # H^2h*H^4h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^5 + vpmsumd RP2,H2_l,poly_h # [1] H^6 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^5 + vsldoi M2h,zero,H2_m,8 # [2] H^6 + vsldoi Ml,H_m,zero,8 # [2] H^5 + vsldoi M2l,H2_m,zero,8 # [2] H^6 + vsldoi RP,RP,RP,8 # [1] H^5 + vsldoi RP2,RP2,RP2,8 # [1] H^6 + vxor H_h,H_h,Mh # [2] H^5 + vxor H2_h,H2_h,M2h # [2] H^6 + vxor H_l,H_l,Ml # [2] H^5 + vxor H2_l,H2_l,M2l # [2] H^6 + vxor H_l,H_l,RP # [1] H^5 + vxor H2_l,H2_l,RP2 # [1] H^6 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^5 + vpmsumd RP2,H2_l,poly_l # H^6 + vxor H_h,H_l,H_h # H^5 + vxor H2_h,H2_l,H2_h # H^6 + vxor H_h,H_h,RP # H^5 + vxor H2_h,H2_h,RP2 # H^6 + + vsldoi H2,H2_h,H2_h,8 # H^6 + vsldoi H,H_h,H_h,8 # H^5 + vsldoi H2l,zero,H2,8 # H^6 + vsldoi H2h,H2,zero,8 # H^6 + + # --- calculate [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + li 8,9*TableElemAlign + li 9,10*TableElemAlign + li 10,11*TableElemAlign + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^7,H^8 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^6l + vpmsumd H_m,H_t,H2 # H^1h*H^6l⊕H^1l*H^6h + vpmsumd H_h,H_t,H2h # H^1h*H^6h + vpmsumd H2_l,H2_t,H2l # H^2l*H^6l + vpmsumd H2_m,H2_t,H2 # H^2h*H^6l⊕H^2l*H^6h + vpmsumd H2_h,H2_t,H2h # H^2h*H^6h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^7 + vpmsumd RP2,H2_l,poly_h # [1] H^8 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^7 + vsldoi M2h,zero,H2_m,8 # [2] H^8 + vsldoi Ml,H_m,zero,8 # [2] H^7 + vsldoi M2l,H2_m,zero,8 # [2] H^8 + vsldoi RP,RP,RP,8 # [1] H^7 + vsldoi RP2,RP2,RP2,8 # [1] H^8 + vxor H_h,H_h,Mh # [2] H^7 + vxor H2_h,H2_h,M2h # [2] H^8 + vxor H_l,H_l,Ml # [2] H^7 + vxor H2_l,H2_l,M2l # [2] H^8 + vxor H_l,H_l,RP # [1] H^7 + vxor H2_l,H2_l,RP2 # [1] H^8 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^7 + vpmsumd RP2,H2_l,poly_l # H^8 + vxor H_h,H_l,H_h # H^7 + vxor H2_h,H2_l,H2_h # H^8 + vxor H_h,H_h,RP # H^7 + vxor H2_h,H2_h,RP2 # H^8 + + vsldoi H,H_h,H_h,8 # H^7 + vsldoi H2,H2_h,H2_h,8 # H^8 + + # --- calculate [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + li 8,12*TableElemAlign + li 9,13*TableElemAlign + li 10,14*TableElemAlign + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + blr +EPILOGUE(_nettle_gcm_init_key) + + # void gcm_hash (const struct gcm_key *key, union gcm_block *x, + # size_t length, const uint8_t *data) + +define(<FUNC_ALIGN>, <5>) +PROLOGUE(_nettle_gcm_hash) + vxor zero,zero,zero + + DATA_LOAD_VEC(poly,.polynomial,7) +IF_LE(<DATA_LOAD_VEC(swap_mask,.swap_mask,7)>) + DATA_LOAD_VEC(hidw_mask,.hidw_mask,7) + DATA_LOAD_VEC(lodw_mask,.lodw_mask,7) + + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + lxvd2x CX,0,X # load X +IF_LE(<vperm C,C,C,swap_mask>) + + srdi 7,LENGTH,7 # 8x loop count + cmpldi 7,0 + beq L2x + + # backup registers + stdu SP,-224(SP) + std 28,216(SP) + std 29,208(SP) + std 30,200(SP) + std 31,192(SP) + li 8,176 + stvx 20,8,SP + subi 8,8,16 + stvx 21,8,SP + subi 8,8,16 + stvx 22,8,SP + subi 8,8,16 + stvx 23,8,SP + subi 8,8,16 + stvx 24,8,SP + subi 8,8,16 + stvx 25,8,SP + subi 8,8,16 + stvx 26,8,SP + subi 8,8,16 + stvx 27,8,SP + subi 8,8,16 + stvx 28,8,SP + subi 8,8,16 + stvx 29,8,SP + subi 8,8,16 + stvx 30,8,SP + subi 8,8,16 + stvx 31,8,SP + + # table loading + li 8,3*TableElemAlign + li 9,4*TableElemAlign + li 10,5*TableElemAlign + lxvd2x H21hX,8,TABLE + lxvd2x H21X,9,TABLE + lxvd2x H21lX,10,TABLE + li 8,6*TableElemAlign + li 9,7*TableElemAlign + li 10,8*TableElemAlign + lxvd2x H43hX,8,TABLE + lxvd2x H43X,9,TABLE + lxvd2x H43lX,10,TABLE + li 8,9*TableElemAlign + li 9,10*TableElemAlign + li 10,11*TableElemAlign + lxvd2x H65hX,8,TABLE + lxvd2x H65X,9,TABLE + lxvd2x H65lX,10,TABLE + li 8,12*TableElemAlign + li 9,13*TableElemAlign + li 10,14*TableElemAlign + lxvd2x H87hX,8,TABLE + lxvd2x H87X,9,TABLE + lxvd2x H87lX,10,TABLE + + li 8,0x10 + li 9,0x20 + li 10,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + + mtctr 7 +.align 5 +L8x_loop: + # input loading + lxvd2x C0X,0,DATA # load C0 + lxvd2x C1X,8,DATA # load C1 + lxvd2x C2X,9,DATA # load C2 + lxvd2x C3X,10,DATA # load C3 + + # swap permuting +IF_LE(<vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask + vperm C2,C2,C2,swap_mask + vperm C3,C3,C3,swap_mask>) + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C23h,C2,C3,hidw_mask + vperm C23l,C2,C3,lodw_mask + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + + # input loading + lxvd2x C4X,28,DATA # load C4 + lxvd2x C5X,29,DATA # load C5 + lxvd2x C6X,30,DATA # load C6 + lxvd2x C7X,31,DATA # load C7 + + # swap permuting +IF_LE(<vperm C4,C4,C4,swap_mask + vperm C5,C5,C5,swap_mask + vperm C6,C6,C6,swap_mask + vperm C7,C7,C7,swap_mask>) + + # polynomial multiplication "karatsuba" pre-processing + vperm C45h,C4,C5,hidw_mask + vperm C45l,C4,C5,lodw_mask + vperm C67h,C6,C7,hidw_mask + vperm C67l,C6,C7,lodw_mask + vxor C23,C23h,C23l + vxor C01,C01h,C01l + vxor C45,C45h,C45l + vxor C67,C67h,C67l + + # polynomial multiplication "karatsuba" + vpmsumd C23h,C23h,H65h # H23 = H^6h*C2h⊕H^5h*C3h + vpmsumd C23l,C23l,H65l # L23 = H^6l*C2l⊕H^5l*C3l + vpmsumd C01h,C01h,H87h # H01 = H^8h*C0h⊕H^7h*C1h + vpmsumd C01l,C01l,H87l # L01 = H^8l*C0l⊕H^7l*C1l + vpmsumd C67h,C67h,H21h # H67 = H^2h*C6h⊕H^1h*C7h + vpmsumd C67l,C67l,H21l # L67 = H^2l*C6l⊕H^1l*C7l + vpmsumd C45h,C45h,H43h # H45 = H^4h*C4h⊕H^3h*C5h + vpmsumd C45l,C45l,H43l # L45 = H^4l*C4l⊕H^3l*C5l + vpmsumd C23,C23,H65 # M23 = (H^6h⊕H^5h)*(C2h⊕C3h)⊕(H^6l⊕H^5l)*(C2l⊕C3l) + vpmsumd C01,C01,H87 # M01 = (H^8h⊕H^7h)*(C0h⊕C1h)⊕(H^8l⊕H^7l)*(C0l⊕C1l) + vpmsumd C45,C45,H43 # M45 = (H^4h⊕H^3h)*(C4h⊕C5h)⊕(H^4l⊕H^3l)*(C4l⊕C5l) + vpmsumd C67,C67,H21 # M67 = (H^2h⊕H^1h)*(C6h⊕C7h)⊕(H^2l⊕H^1l)*(C6l⊕C7l) + + # polynomial multiplication "karatsuba" post-processing + vxor C23,C23,C23h + vxor C01,C01,C01h + vxor C45,C45,C45h + vxor C67,C67,C67h + vxor C23,C23,C23l + vxor C01,C01,C01l + vxor C45,C45,C45l + vxor C67,C67,C67l + + # deferred recombination of partial products + vxor C01h,C01h,C23h # H0 = H01⊕H23 + vxor C45h,C45h,C67h # H1 = H45⊕H67 + vxor C01l,C01l,C23l # L0 = L01⊕L23 + vxor C45l,C45l,C67l # L1 = L45⊕L45 + vxor C01,C01,C23 # M0 = M01⊕M23 + vxor C45,C45,C67 # M1 = M45⊕M45 + vxor C01h,C01h,C45h # H = H0⊕H1 + vxor C01l,C01l,C45l # L = L0⊕L1 + vxor C01,C01,C45 # M = M0⊕M1 + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x80 + bdnz L8x_loop + + # restore registers + li 8,0 + lvx 31,8,SP + addi 8,8,16 + lvx 30,8,SP + addi 8,8,16 + lvx 29,8,SP + addi 8,8,16 + lvx 28,8,SP + addi 8,8,16 + lvx 27,8,SP + addi 8,8,16 + lvx 26,8,SP + addi 8,8,16 + lvx 25,8,SP + addi 8,8,16 + lvx 24,8,SP + addi 8,8,16 + lvx 23,8,SP + addi 8,8,16 + lvx 22,8,SP + addi 8,8,16 + lvx 21,8,SP + addi 8,8,16 + lvx 20,8,SP + ld 31,192(SP) + ld 30,200(SP) + ld 29,208(SP) + ld 28,216(SP) + addi SP,SP,224 + + clrldi LENGTH,LENGTH,57 +L2x: + srdi 7,LENGTH,5 + cmpldi 7,0 + beq L1x + + # table loading + li 8,3*TableElemAlign + li 9,4*TableElemAlign + li 10,5*TableElemAlign + lxvd2x H21hX,8,TABLE + lxvd2x H21X,9,TABLE + lxvd2x H21lX,10,TABLE + + li 10,0x10 + + mtctr 7 +.align 5 +L2x_loop: + # input loading + lxvd2x C0X,0,DATA # load C0 + lxvd2x C1X,10,DATA # load C1 + + # swap permuting +IF_LE(<vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask>) + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + vxor C01,C01h,C01l + + # polynomial multiplication "karatsuba" + vpmsumd C01h,C01h,H21h # H01 = H^2h*C0h⊕H^1h*C1h + vpmsumd C01l,C01l,H21l # L01 = H^2l*C0l⊕H^1l*C1l + vpmsumd C01,C01,H21 # M01 = (H^2h⊕H^1h)*(C0h⊕C1h)⊕(H^2l⊕H^1l)*(C0l⊕C1l) + + # polynomial multiplication "karatsuba" post-processing + vxor C01,C01,C01h + vxor C01,C01,C01l + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x20 + bdnz L2x_loop + + clrldi LENGTH,LENGTH,59 +L1x: + srdi 7,LENGTH,4 + cmpldi 7,0 + beq Lrem + + # table loading + li 9,1*TableElemAlign + li 10,2*TableElemAlign + lxvd2x HlX,0,TABLE + lxvd2x HX, 9,TABLE + lxvd2x HhX,10,TABLE + + # input loading + lxvd2x C0X,0,DATA # load C0 + + # swap permuting +IF_LE(<vperm C0,C0,C0,swap_mask>) + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + + addi DATA,DATA,0x10 + clrldi LENGTH,LENGTH,60 +Lrem: + cmpldi LENGTH,0 + beq Ldone + + # table loading + li 9,1*TableElemAlign + li 10,2*TableElemAlign + lxvd2x HlX,0,TABLE + lxvd2x HX, 9,TABLE + lxvd2x HhX,10,TABLE + + # input loading + stdu SP,-16(SP) + stvx zero,0,SP +Lst_loop: + subic. LENGTH,LENGTH,1 + lbzx 7,LENGTH,DATA + stbx 7,LENGTH,SP + bne Lst_loop + lxvd2x C0X,0,SP + addi SP,SP,16 + + # swap permuting +IF_LE(<vperm C0,C0,C0,swap_mask>) + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + +Ldone: +IF_LE(<vperm C,C,C,swap_mask>) + stxvd2x CX,0,X # store C + blr +EPILOGUE(_nettle_gcm_hash) + + # gcm_fill (uint8_t *ctr, size_t blocks, union gcm_block *buffer) + +define(<FUNC_ALIGN>, <5>) +PROLOGUE(_nettle_gcm_fill) +IF_LE(<DATA_LOAD_VEC(swap_mask,.swap_mask,6)>) + + vxor zero,zero,zero + vspltisb I1,1 + vspltisb I2,2 + vspltisb I3,3 + vspltisb I4,4 + vspltisb I5,5 + vspltisb I6,6 + vspltisb I7,7 + vspltisb I8,8 + vsldoi I1,zero,I1,1 + vsldoi I2,zero,I2,1 + vsldoi I3,zero,I3,1 + vsldoi I4,zero,I4,1 + vsldoi I5,zero,I5,1 + vsldoi I6,zero,I6,1 + vsldoi I7,zero,I7,1 + vsldoi I8,zero,I8,1 + + lxvd2x CTR0X,0,CTR + IF_LE(<vperm CTR0,CTR0,CTR0,swap_mask>) + + srdi 6,BLOCKS,3 # 8x loop count + cmpldi 6,0 + beq Lfill_4x + + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 25,0x10 + li 26,0x20 + li 27,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + + mtctr 6 + L8x_fill_loop: + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + vadduwm CTR4,CTR0,I4 + vadduwm CTR5,CTR0,I5 + vadduwm CTR6,CTR0,I6 + vadduwm CTR7,CTR0,I7 + + IF_LE(<vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask + vperm CTR4,CTR4,CTR4,swap_mask + vperm CTR5,CTR5,CTR5,swap_mask + vperm CTR6,CTR6,CTR6,swap_mask + vperm CTR7,CTR7,CTR7,swap_mask>) + + IF_LE(<stxvd2x CTR0SX,0,BUFFER>) + IF_BE(<stxvd2x CTR0X,0,BUFFER>) + stxvd2x CTR1X,25,BUFFER + stxvd2x CTR2X,26,BUFFER + stxvd2x CTR3X,27,BUFFER + stxvd2x CTR4X,28,BUFFER + stxvd2x CTR5X,29,BUFFER + stxvd2x CTR6X,30,BUFFER + stxvd2x CTR7X,31,BUFFER + + vadduwm CTR0,CTR0,I8 + addi BUFFER,BUFFER,0x80 + bdnz L8x_fill_loop + + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi BLOCKS,BLOCKS,61 + + Lfill_4x: + srdi 6,BLOCKS,2 + cmpldi 6,0 + beq Lfill_2x + + li 8,0x10 + li 9,0x20 + li 10,0x30 + + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + + IF_LE(<vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask>) + + IF_LE(<stxvd2x CTR0SX,0,BUFFER>) + IF_BE(<stxvd2x CTR0X,0,BUFFER>) + stxvd2x CTR1X,8,BUFFER + stxvd2x CTR2X,9,BUFFER + stxvd2x CTR3X,10,BUFFER + + vadduwm CTR0,CTR0,I4 + addi BUFFER,BUFFER,0x40 + + clrldi BLOCKS,BLOCKS,62 + + Lfill_2x: + srdi 6,BLOCKS,1 + cmpldi 6,0 + beq Lfill_1x + + li 10,0x10 + + vadduwm CTR1,CTR0,I1 + + IF_LE(<vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask>) + + IF_LE(<stxvd2x CTR0SX,0,BUFFER>) + IF_BE(<stxvd2x CTR0X,0,BUFFER>) + stxvd2x CTR1X,10,BUFFER + + vadduwm CTR0,CTR0,I2 + addi BUFFER,BUFFER,0x20 + + clrldi BLOCKS,BLOCKS,63 + + Lfill_1x: + cmpldi BLOCKS,0 + beq Lfill_done + + IF_LE(<vperm CTR0S,CTR0,CTR0,swap_mask>) + + IF_LE(<stxvd2x CTR0SX,0,BUFFER>) + IF_BE(<stxvd2x CTR0X,0,BUFFER>) + + vadduwm CTR0,CTR0,I1 + + Lfill_done: + IF_LE(<vperm CTR0,CTR0,CTR0,swap_mask>) + stxvd2x CTR0X,0,CTR + + blr +EPILOGUE(_nettle_gcm_fill) + + .data +IF_LE(<.align 4 +.polynomial: + .byte 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0xc2 + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 + .align 4 +.hidw_mask: + .byte 23,22,21,20,19,18,17,16,7,6,5,4,3,2,1,0 + .align 4 +.lodw_mask: + .byte 31,30,29,28,27,26,25,24,15,14,13,12,11,10,9,8>) +IF_BE(<.align 4 +.polynomial: + .byte 0xc2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1 + .align 4 +.hidw_mask: + .byte 0,1,2,3,4,5,6,7,16,17,18,19,20,21,22,23 + .align 4 +.lodw_mask: + .byte 8,9,10,11,12,13,14,15,24,25,26,27,28,29,30,31>) -- 2.17.1

1 0

[PATCH 1/6] "PowerPC64" Add machine.m4
by Maamoun TK 14 Jul '20

14 Jul '20

--- powerpc64/machine.m4 | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 powerpc64/machine.m4 diff --git a/powerpc64/machine.m4 b/powerpc64/machine.m4 new file mode 100644 index 00000000..3a121260 --- /dev/null +++ b/powerpc64/machine.m4 @@ -0,0 +1,32 @@ +define(<PROLOGUE>, +<.globl C_NAME($1) +DECLARE_FUNC(C_NAME($1)) +ifelse(WORDS_BIGENDIAN,no, +<ifdef(<FUNC_ALIGN>,<.align FUNC_ALIGN>) +C_NAME($1): +addis 2,12,(.TOC.-C_NAME($1))@ha +addi 2,2,(.TOC.-C_NAME($1))@l +.localentry C_NAME($1), .-C_NAME($1)>, +<.section ".opd","aw" +.align 3 +C_NAME($1): +.quad .C_NAME($1),.TOC.@tocbase,0 +.previous +ifdef(<FUNC_ALIGN>,<.align FUNC_ALIGN>) +.C_NAME($1):>) +undefine(<FUNC_ALIGN>)>) + +define(<EPILOGUE>, +<ifelse(WORDS_BIGENDIAN,no, +<.size C_NAME($1), . - C_NAME($1)>, +<.size .C_NAME($1), . - .C_NAME($1) +.size C_NAME($1), . - .C_NAME($1)>)>) + +C Load the quadword in DATA_SRC storage into +C VEC_DST. GPR is general-purpose register +C used to obtain the effective address of +C DATA_SRC storage. +C DATA_LOAD_VEC(VEC_DST, DATA_SRC, GPR) +define(<DATA_LOAD_VEC>, +<ld $3,$2@got(2) +lvx $1,0,$3>) -- 2.17.1

1 0

[PATCH 3/4] Add test 128 bytes to gcm-test
by Maamoun TK 09 Jul '20

09 Jul '20

--- testsuite/gcm-test.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/testsuite/gcm-test.c b/testsuite/gcm-test.c index c8174019..df1fc94a 100644 --- a/testsuite/gcm-test.c +++ b/testsuite/gcm-test.c @@ -170,6 +170,29 @@ test_main(void) "16aedbf5a0de6a57a637b39b"), SHEX("619cc5aefffe0bfa462af43c1699d050")); + /* Test 128 bytes */ + test_aead(&nettle_gcm_aes128, NULL, + SHEX("feffe9928665731c6d6a8f9467308308"), + SHEX(""), + SHEX("d9313225f88406e5a55909c5aff5269a" + "86a7a9531534f7da2e4c303d8a318a72" + "1c3c0c95956809532fcf0e2449a6b525" + "b16aedf5aa0de657ba637b391aafd255" + "5ae376bc5e9f6a1b08e34db7a6ee0736" + "9ba662ea12f6f197e6bc3ed69d2480f3" + "ea5691347f2ba69113eb37910ebc18c8" + "0f697234582016fa956ca8f63ae6b473"), + SHEX("42831ec2217774244b7221b784d0d49c" + "e3aa212f2c02a4e035c17e2329aca12e" + "21d514b25466931c7d8f6a5aac84aa05" + "1ba30b396a0aac973d58e091473f5985" + "874b1178906ddbeab04ab2fe6cce8c57" + "8d7e961bd13fd6a8c56b66ca5e576492" + "1a48cd8bda04e66343e73055118b69b9" + "ced486813846958a11e602c03cfc232b"), + SHEX("cafebabefacedbaddecaf888"), + SHEX("796836f1246c9d735c5e1be0a715ccc3")); + /* Test case 7 */ test_aead(&nettle_gcm_aes192, NULL, SHEX("00000000000000000000000000000000" -- 2.17.1

2 1

[PATCH] Add missing undef directives in configure.ac"
by Maamoun TK 09 Jul '20

09 Jul '20

--- configure.ac | 2 ++ 1 file changed, 2 insertions(+) diff --git a/configure.ac b/configure.ac index cc0d67ec..d7020fd6 100644 --- a/configure.ac +++ b/configure.ac @@ -582,7 +582,9 @@ AH_VERBATIM([HAVE_NATIVE], #undef HAVE_NATIVE_ecc_secp384r1_redc #undef HAVE_NATIVE_ecc_secp521r1_modp #undef HAVE_NATIVE_ecc_secp521r1_redc +#undef HAVE_NATIVE_gcm_init_key8 #undef HAVE_NATIVE_gcm_hash8 +#undef HAVE_NATIVE_gcm_fill #undef HAVE_NATIVE_salsa20_core #undef HAVE_NATIVE_sha1_compress #undef HAVE_NATIVE_sha256_compress -- 2.17.1

2 1

[PATCH 2/4] Add GHASH optimized implementation for PowerPC64
by Maamoun TK 09 Jul '20

09 Jul '20

--- gcm.c | 19 +- powerpc64/gcm-hash8.asm | 1004 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1022 insertions(+), 1 deletion(-) create mode 100644 powerpc64/gcm-hash8.asm diff --git a/gcm.c b/gcm.c index cf615daf..809c03bc 100644 --- a/gcm.c +++ b/gcm.c @@ -140,6 +140,12 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) memcpy (x->b, Z.b, sizeof(Z)); } # elif GCM_TABLE_BITS == 8 +# if HAVE_NATIVE_gcm_init_key8 + +#define gcm_init_key _nettle_gcm_init_key8 +void +_nettle_gcm_init_key8 (union nettle_block16 *table); +# endif /* HAVE_NATIVE_gcm_init_key8 */ # if HAVE_NATIVE_gcm_hash8 #define gcm_hash _nettle_gcm_hash8 @@ -225,6 +231,13 @@ gcm_gf_mul (union nettle_block16 *x, const union nettle_block16 *table) #endif /* GCM_TABLE_BITS */ +#if HAVE_NATIVE_gcm_fill + +#define gcm_fill _nettle_gcm_fill +void +_nettle_gcm_fill (uint8_t *ctr, size_t blocks, union nettle_block16 *buffer); +#endif /* HAVE_NATIVE_gcm_fill */ + /* Increment the rightmost 32 bits. */ #define INC32(block) INCREMENT(4, (block.b) + GCM_BLOCK_SIZE - 4) @@ -245,7 +258,9 @@ gcm_set_key(struct gcm_key *key, memset(key->h[0].b, 0, GCM_BLOCK_SIZE); f (cipher, GCM_BLOCK_SIZE, key->h[i].b, key->h[0].b); -#if GCM_TABLE_BITS +#ifdef gcm_init_key + gcm_init_key(key->h); +#elif GCM_TABLE_BITS /* Algorithm 3 from the gcm paper. First do powers of two, then do the rest by adding. */ while (i /= 2) @@ -333,6 +348,7 @@ gcm_update(struct gcm_ctx *ctx, const struct gcm_key *key, ctx->auth_size += length; } +#ifndef gcm_fill static nettle_fill16_func gcm_fill; static void gcm_fill(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) @@ -349,6 +365,7 @@ gcm_fill(uint8_t *ctr, size_t blocks, union nettle_block16 *buffer) WRITE_UINT32(ctr + GCM_BLOCK_SIZE - 4, c); } +#endif /* !gcm_fill */ void gcm_encrypt (struct gcm_ctx *ctx, const struct gcm_key *key, diff --git a/powerpc64/gcm-hash8.asm b/powerpc64/gcm-hash8.asm new file mode 100644 index 00000000..7d5d3ec9 --- /dev/null +++ b/powerpc64/gcm-hash8.asm @@ -0,0 +1,1004 @@ +C powerpc64/gcm-hash8.asm + +ifelse(< + Copyright (C) 2020 Mamone Tarsha + This file is part of GNU Nettle. + + GNU Nettle is free software: you can redistribute it and/or + modify it under the terms of either: + + * the GNU Lesser General Public License as published by the Free + Software Foundation; either version 3 of the License, or (at your + option) any later version. + + or + + * the GNU General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your + option) any later version. + + or both in parallel, as here. + + GNU Nettle is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received copies of the GNU General Public License and + the GNU Lesser General Public License along with this program. If + not, see http://www.gnu.org/licenses/. +>) + +C Alignment of gcm_key table elements, which is declared in gcm.h +define(<TableElemAlign>, <0x100>) + +C Register usage: + +define(<SP>, <1>) +define(<TOCP>, <2>) + +define(<TABLE>, <3>) +define(<X>, <4>) +define(<LENGTH>, <5>) +define(<DATA>, <6>) + +define(<zero>, <0>) +define(<swap_mask>, <1>) +define(<hidw_mask>, <2>) +define(<lodw_mask>, <3>) +define(<poly>, <4>) +define(<poly_h>, <4>) +define(<poly_l>, <5>) +define(<RP>, <6>) +define(<Mh>, <7>) +define(<Ml>, <8>) +define(<H>, <9>) +define(<Hh>, <10>) +define(<Hl>, <11>) +define(<RP2>, <9>) +define(<M2h>, <10>) +define(<M2l>, <11>) + +define(<HX>, <41>) +define(<HhX>, <42>) +define(<HlX>, <43>) +define(<H_HhX>, <44>) +define(<H_HX>, <45>) +define(<H_HlX>, <46>) + +define(<sl1>, <1>) +define(<msb>, <5>) +define(<H2>, <6>) +define(<H2h>, <7>) +define(<H2l>, <8>) +define(<H_h>, <12>) +define(<H_m>, <13>) +define(<H_l>, <14>) +define(<H_Hh>, <12>) +define(<H_H>, <13>) +define(<H_Hl>, <14>) +define(<H_t>, <15>) +define(<H2_h>, <16>) +define(<H2_m>, <17>) +define(<H2_l>, <18>) +define(<H2_t>, <19>) + +define(<C0X>, <38>) +define(<C1X>, <39>) +define(<C2X>, <40>) +define(<C3X>, <44>) +define(<C4X>, <38>) +define(<C5X>, <39>) +define(<C6X>, <40>) +define(<C7X>, <44>) + +define(<CX>, <45>) + +define(<C0>, <6>) +define(<C1>, <7>) +define(<C2>, <8>) +define(<C3>, <12>) +define(<C4>, <6>) +define(<C5>, <7>) +define(<C6>, <8>) +define(<C7>, <12>) + +define(<C>, <13>) + +define(<Ch>, <14>) +define(<Cl>, <15>) +define(<Cm>, <16>) + +define(<C01h>, <14>) +define(<C01l>, <15>) +define(<C01>, <16>) +define(<C23h>, <17>) +define(<C23l>, <18>) +define(<C23>, <19>) +define(<C45h>, <20>) +define(<C45l>, <21>) +define(<C45>, <22>) +define(<C67h>, <6>) +define(<C67l>, <7>) +define(<C67>, <8>) + +define(<H21>, <9>) +define(<H21h>, <10>) +define(<H21l>, <11>) +define(<H43>, <23>) +define(<H43h>, <24>) +define(<H43l>, <25>) +define(<H65>, <26>) +define(<H65h>, <27>) +define(<H65l>, <28>) +define(<H87>, <29>) +define(<H87h>, <30>) +define(<H87l>, <31>) + +define(<H21X>, <41>) +define(<H21hX>, <42>) +define(<H21lX>, <43>) +define(<H43X>, <55>) +define(<H43hX>, <56>) +define(<H43lX>, <57>) +define(<H65X>, <58>) +define(<H65hX>, <59>) +define(<H65lX>, <60>) +define(<H87X>, <61>) +define(<H87hX>, <62>) +define(<H87lX>, <63>) + +# gcm_fill registers: + +define(<CTR>, <3>) +define(<BLOCKS>, <4>) +define(<BUFFER>, <5>) + +define(<CTR0>, <2>) +define(<CTR0S>, <3>) +define(<CTR1>, <4>) +define(<CTR2>, <5>) +define(<CTR3>, <6>) +define(<CTR4>, <7>) +define(<CTR5>, <8>) +define(<CTR6>, <9>) +define(<CTR7>, <10>) + +define(<CTR0X>, <34>) +define(<CTR0SX>, <35>) +define(<CTR1X>, <36>) +define(<CTR2X>, <37>) +define(<CTR3X>, <38>) +define(<CTR4X>, <39>) +define(<CTR5X>, <40>) +define(<CTR6X>, <41>) +define(<CTR7X>, <42>) + +define(<I1>, <11>) +define(<I2>, <12>) +define(<I3>, <13>) +define(<I4>, <14>) +define(<I5>, <15>) +define(<I6>, <16>) +define(<I7>, <17>) +define(<I8>, <18>) + +.file "gcm-hash8.asm" + +IF_LE(<.abiversion 2>) +.text + + # void gcm_init_key (union gcm_block *table) + +PROLOGUE(_nettle_gcm_init_key8) + ld 7,.polynomial@got(TOCP) + lvx poly,0,7 +IF_LE(<ld 7,.swap_mask@got(TOCP) + lvx swap_mask,0,7>) + ld 7,.hidw_mask@got(TOCP) + lvx hidw_mask,0,7 + ld 7,.lodw_mask@got(TOCP) + lvx lodw_mask,0,7 + + li 10,8*TableElemAlign + lxvd2x HX,10,TABLE # load H +IF_LE(<vperm H,H,H,swap_mask>) + + # --- calculate H = H shift left 1 modulo polynomial --- + + vupkhsw msb,H # most significant bit word-extend + vspltisb sl1,1 # splat 1 for shift left + vspltw msb,msb,0 # most significant bit extend + vsl H,H,sl1 # H shift left 1 + vand msb,msb,poly + vxor zero,zero,zero + vxor H_t,H,msb + + vsldoi H,H_t,H_t,8 # doubleword swap + vsldoi Hh,H,zero,8 + vsldoi Hl,zero,H,8 + + # --- calculate H^2 = H*H --- + + # reduction pre-processing + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + # polynomial multiplication "classical" + vpmsumd H_h,H_t,Hh # H^1h*H^1h + vpmsumd H_l,H_t,Hl # H^1l*H^1l + vpmsumd H_m,H_t,H # H^1h*H^1l⊕H^1l*H^1h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] + vsldoi Ml,H_m,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor H_h,H_h,Mh # [2] + vxor H_l,H_l,Ml # [2] + vxor H_l,H_l,RP # [1] + + # reduction second phase + vpmsumd RP,H_l,poly_l + vxor H_h,H_l,H_h + vxor H2_t,H_h,RP + + vsldoi H2,H2_t,H2_t,8 + vsldoi H2h,H2,zero,8 + vsldoi H2l,zero,H2,8 + + # --- calculate [H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store H,[H^2.Hi⊕H^2.Lo:H^1.Hi⊕H^1.Lo] --- + + li 8,0*TableElemAlign + li 9,1*TableElemAlign + li 10,2*TableElemAlign + stxvd2x HlX,8,TABLE + stxvd2x HX,9,TABLE + stxvd2x HhX,10,TABLE + + li 8,3*TableElemAlign + li 9,4*TableElemAlign + li 10,5*TableElemAlign + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^3,H^4 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^2l + vpmsumd H_m,H_t,H2 # H^1h*H^2l⊕H^1l*H^2h + vpmsumd H_h,H_t,H2h # H^1h*H^2h + vpmsumd H2_l,H2_t,H2l # H^2l*H^2l + vpmsumd H2_m,H2_t,H2 # H^2h*H^2l⊕H^2l*H^2h + vpmsumd H2_h,H2_t,H2h # H^2h*H^2h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^3 + vpmsumd RP2,H2_l,poly_h # [1] H^4 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^3 + vsldoi M2h,zero,H2_m,8 # [2] H^4 + vsldoi Ml,H_m,zero,8 # [2] H^3 + vsldoi M2l,H2_m,zero,8 # [2] H^4 + vsldoi RP,RP,RP,8 # [1] H^3 + vsldoi RP2,RP2,RP2,8 # [1] H^4 + vxor H_h,H_h,Mh # [2] H^3 + vxor H2_h,H2_h,M2h # [2] H^4 + vxor H_l,H_l,Ml # [2] H^3 + vxor H2_l,H2_l,M2l # [2] H^4 + vxor H_l,H_l,RP # [1] H^3 + vxor H2_l,H2_l,RP2 # [1] H^4 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^3 + vpmsumd RP2,H2_l,poly_l # H^4 + vxor H_h,H_l,H_h # H^3 + vxor H2_h,H2_l,H2_h # H^4 + vxor H_h,H_h,RP # H^3 + vxor H2_h,H2_h,RP2 # H^4 + + vsldoi H2,H2_h,H2_h,8 # H^4 + vsldoi H,H_h,H_h,8 # H^3 + vsldoi H2l,zero,H2,8 # H^4 + vsldoi H2h,H2,zero,8 # H^4 + + # --- calculate [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^4.Hi⊕H^4.Lo:H^3.Hi⊕H^3.Lo] --- + + li 8,6*TableElemAlign + li 9,7*TableElemAlign + li 10,8*TableElemAlign + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^5,H^6 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^4l + vpmsumd H_m,H_t,H2 # H^1h*H^4l⊕H^1l*H^4h + vpmsumd H_h,H_t,H2h # H^1h*H^4h + vpmsumd H2_l,H2_t,H2l # H^2l*H^4l + vpmsumd H2_m,H2_t,H2 # H^2h*H^4l⊕H^2l*H^4h + vpmsumd H2_h,H2_t,H2h # H^2h*H^4h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^5 + vpmsumd RP2,H2_l,poly_h # [1] H^6 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^5 + vsldoi M2h,zero,H2_m,8 # [2] H^6 + vsldoi Ml,H_m,zero,8 # [2] H^5 + vsldoi M2l,H2_m,zero,8 # [2] H^6 + vsldoi RP,RP,RP,8 # [1] H^5 + vsldoi RP2,RP2,RP2,8 # [1] H^6 + vxor H_h,H_h,Mh # [2] H^5 + vxor H2_h,H2_h,M2h # [2] H^6 + vxor H_l,H_l,Ml # [2] H^5 + vxor H2_l,H2_l,M2l # [2] H^6 + vxor H_l,H_l,RP # [1] H^5 + vxor H2_l,H2_l,RP2 # [1] H^6 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^5 + vpmsumd RP2,H2_l,poly_l # H^6 + vxor H_h,H_l,H_h # H^5 + vxor H2_h,H2_l,H2_h # H^6 + vxor H_h,H_h,RP # H^5 + vxor H2_h,H2_h,RP2 # H^6 + + vsldoi H2,H2_h,H2_h,8 # H^6 + vsldoi H,H_h,H_h,8 # H^5 + vsldoi H2l,zero,H2,8 # H^6 + vsldoi H2h,H2,zero,8 # H^6 + + # --- calculate [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^6.Hi⊕H^6.Lo:H^5.Hi⊕H^5.Lo] --- + + li 8,9*TableElemAlign + li 9,10*TableElemAlign + li 10,11*TableElemAlign + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + # --- calculate H^7,H^8 --- + + # polynomial multiplication "classical" + vpmsumd H_l,H_t,H2l # H^1l*H^6l + vpmsumd H_m,H_t,H2 # H^1h*H^6l⊕H^1l*H^6h + vpmsumd H_h,H_t,H2h # H^1h*H^6h + vpmsumd H2_l,H2_t,H2l # H^2l*H^6l + vpmsumd H2_m,H2_t,H2 # H^2h*H^6l⊕H^2l*H^6h + vpmsumd H2_h,H2_t,H2h # H^2h*H^6h + + # reduction first phase # [1] + vpmsumd RP,H_l,poly_h # [1] H^7 + vpmsumd RP2,H2_l,poly_h # [1] H^8 + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,H_m,8 # [2] H^7 + vsldoi M2h,zero,H2_m,8 # [2] H^8 + vsldoi Ml,H_m,zero,8 # [2] H^7 + vsldoi M2l,H2_m,zero,8 # [2] H^8 + vsldoi RP,RP,RP,8 # [1] H^7 + vsldoi RP2,RP2,RP2,8 # [1] H^8 + vxor H_h,H_h,Mh # [2] H^7 + vxor H2_h,H2_h,M2h # [2] H^8 + vxor H_l,H_l,Ml # [2] H^7 + vxor H2_l,H2_l,M2l # [2] H^8 + vxor H_l,H_l,RP # [1] H^7 + vxor H2_l,H2_l,RP2 # [1] H^8 + + # reduction second phase + vpmsumd RP,H_l,poly_l # H^7 + vpmsumd RP2,H2_l,poly_l # H^8 + vxor H_h,H_l,H_h # H^7 + vxor H2_h,H2_l,H2_h # H^8 + vxor H_h,H_h,RP # H^7 + vxor H2_h,H2_h,RP2 # H^8 + + vsldoi H,H_h,H_h,8 # H^7 + vsldoi H2,H2_h,H2_h,8 # H^8 + + # --- calculate [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + vperm H_Hh,H2,H,lodw_mask + vperm H_Hl,H2,H,hidw_mask + vxor H_H,H_Hh,H_Hl + + # --- store [H^8.Hi⊕H^8.Lo:H^7.Hi⊕H^7.Lo] --- + + li 8,12*TableElemAlign + li 9,13*TableElemAlign + li 10,14*TableElemAlign + stxvd2x H_HhX,8,TABLE + stxvd2x H_HX,9,TABLE + stxvd2x H_HlX,10,TABLE + + blr +EPILOGUE(_nettle_gcm_init_key8) + + # void gcm_hash (const struct gcm_key *key, union gcm_block *x, + # size_t length, const uint8_t *data) + +PROLOGUE(_nettle_gcm_hash8) + vxor zero,zero,zero + + ld 7,.polynomial@got(TOCP) + lvx poly,0,7 +IF_LE(<ld 7,.swap_mask@got(TOCP) + lvx swap_mask,0,7>) + ld 7,.hidw_mask@got(TOCP) + lvx hidw_mask,0,7 + ld 7,.lodw_mask@got(TOCP) + lvx lodw_mask,0,7 + + vsldoi poly_h,zero,poly,8 + vsldoi poly_l,poly_h,poly_h,8 + + lxvd2x CX,0,X # load X +IF_LE(<vperm C,C,C,swap_mask>) + + srdi 7,LENGTH,7 # 8x loop count + cmpldi 7,0 + beq L2x + + # backup registers + stdu SP,-224(SP) + std 28,216(SP) + std 29,208(SP) + std 30,200(SP) + std 31,192(SP) + li 8,176 + stvx 20,8,SP + subi 8,8,16 + stvx 21,8,SP + subi 8,8,16 + stvx 22,8,SP + subi 8,8,16 + stvx 23,8,SP + subi 8,8,16 + stvx 24,8,SP + subi 8,8,16 + stvx 25,8,SP + subi 8,8,16 + stvx 26,8,SP + subi 8,8,16 + stvx 27,8,SP + subi 8,8,16 + stvx 28,8,SP + subi 8,8,16 + stvx 29,8,SP + subi 8,8,16 + stvx 30,8,SP + subi 8,8,16 + stvx 31,8,SP + + # table loading + li 8,3*TableElemAlign + li 9,4*TableElemAlign + li 10,5*TableElemAlign + lxvd2x H21hX,8,TABLE + lxvd2x H21X,9,TABLE + lxvd2x H21lX,10,TABLE + li 8,6*TableElemAlign + li 9,7*TableElemAlign + li 10,8*TableElemAlign + lxvd2x H43hX,8,TABLE + lxvd2x H43X,9,TABLE + lxvd2x H43lX,10,TABLE + li 8,9*TableElemAlign + li 9,10*TableElemAlign + li 10,11*TableElemAlign + lxvd2x H65hX,8,TABLE + lxvd2x H65X,9,TABLE + lxvd2x H65lX,10,TABLE + li 8,12*TableElemAlign + li 9,13*TableElemAlign + li 10,14*TableElemAlign + lxvd2x H87hX,8,TABLE + lxvd2x H87X,9,TABLE + lxvd2x H87lX,10,TABLE + + li 8,0x10 + li 9,0x20 + li 10,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + + mtctr 7 +.align 5 +L8x_loop: + # input loading + lxvd2x C0X,0,DATA # load C0 + lxvd2x C1X,8,DATA # load C1 + lxvd2x C2X,9,DATA # load C2 + lxvd2x C3X,10,DATA # load C3 + + # swap permuting +IF_LE(<vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask + vperm C2,C2,C2,swap_mask + vperm C3,C3,C3,swap_mask>) + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C23h,C2,C3,hidw_mask + vperm C23l,C2,C3,lodw_mask + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + + # input loading + lxvd2x C4X,28,DATA # load C4 + lxvd2x C5X,29,DATA # load C5 + lxvd2x C6X,30,DATA # load C6 + lxvd2x C7X,31,DATA # load C7 + + # swap permuting +IF_LE(<vperm C4,C4,C4,swap_mask + vperm C5,C5,C5,swap_mask + vperm C6,C6,C6,swap_mask + vperm C7,C7,C7,swap_mask>) + + # polynomial multiplication "karatsuba" pre-processing + vperm C45h,C4,C5,hidw_mask + vperm C45l,C4,C5,lodw_mask + vperm C67h,C6,C7,hidw_mask + vperm C67l,C6,C7,lodw_mask + vxor C23,C23h,C23l + vxor C01,C01h,C01l + vxor C45,C45h,C45l + vxor C67,C67h,C67l + + # polynomial multiplication "karatsuba" + vpmsumd C23h,C23h,H65h # H23 = H^6h*C2h⊕H^5h*C3h + vpmsumd C23l,C23l,H65l # L23 = H^6l*C2l⊕H^5l*C3l + vpmsumd C01h,C01h,H87h # H01 = H^8h*C0h⊕H^7h*C1h + vpmsumd C01l,C01l,H87l # L01 = H^8l*C0l⊕H^7l*C1l + vpmsumd C67h,C67h,H21h # H67 = H^2h*C6h⊕H^1h*C7h + vpmsumd C67l,C67l,H21l # L67 = H^2l*C6l⊕H^1l*C7l + vpmsumd C45h,C45h,H43h # H45 = H^4h*C4h⊕H^3h*C5h + vpmsumd C45l,C45l,H43l # L45 = H^4l*C4l⊕H^3l*C5l + vpmsumd C23,C23,H65 # M23 = (H^6h⊕H^5h)*(C2h⊕C3h)⊕(H^6l⊕H^5l)*(C2l⊕C3l) + vpmsumd C01,C01,H87 # M01 = (H^8h⊕H^7h)*(C0h⊕C1h)⊕(H^8l⊕H^7l)*(C0l⊕C1l) + vpmsumd C45,C45,H43 # M45 = (H^4h⊕H^3h)*(C4h⊕C5h)⊕(H^4l⊕H^3l)*(C4l⊕C5l) + vpmsumd C67,C67,H21 # M67 = (H^2h⊕H^1h)*(C6h⊕C7h)⊕(H^2l⊕H^1l)*(C6l⊕C7l) + + # polynomial multiplication "karatsuba" post-processing + vxor C23,C23,C23h + vxor C01,C01,C01h + vxor C45,C45,C45h + vxor C67,C67,C67h + vxor C23,C23,C23l + vxor C01,C01,C01l + vxor C45,C45,C45l + vxor C67,C67,C67l + + # deferred recombination of partial products + vxor C01h,C01h,C23h # H0 = H01⊕H23 + vxor C45h,C45h,C67h # H1 = H45⊕H67 + vxor C01l,C01l,C23l # L0 = L01⊕L23 + vxor C45l,C45l,C67l # L1 = L45⊕L45 + vxor C01,C01,C23 # M0 = M01⊕M23 + vxor C45,C45,C67 # M1 = M45⊕M45 + vxor C01h,C01h,C45h # H = H0⊕H1 + vxor C01l,C01l,C45l # L = L0⊕L1 + vxor C01,C01,C45 # M = M0⊕M1 + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x80 + bdnz L8x_loop + + # restore registers + li 8,0 + lvx 31,8,SP + addi 8,8,16 + lvx 30,8,SP + addi 8,8,16 + lvx 29,8,SP + addi 8,8,16 + lvx 28,8,SP + addi 8,8,16 + lvx 27,8,SP + addi 8,8,16 + lvx 26,8,SP + addi 8,8,16 + lvx 25,8,SP + addi 8,8,16 + lvx 24,8,SP + addi 8,8,16 + lvx 23,8,SP + addi 8,8,16 + lvx 22,8,SP + addi 8,8,16 + lvx 21,8,SP + addi 8,8,16 + lvx 20,8,SP + ld 31,192(SP) + ld 30,200(SP) + ld 29,208(SP) + ld 28,216(SP) + addi SP,SP,224 + + clrldi LENGTH,LENGTH,57 +L2x: + srdi 7,LENGTH,5 + cmpldi 7,0 + beq L1x + + # table loading + li 8,3*TableElemAlign + li 9,4*TableElemAlign + li 10,5*TableElemAlign + lxvd2x H21hX,8,TABLE + lxvd2x H21X,9,TABLE + lxvd2x H21lX,10,TABLE + + li 10,0x10 + + mtctr 7 +.align 5 +L2x_loop: + # input loading + lxvd2x C0X,0,DATA # load C0 + lxvd2x C1X,10,DATA # load C1 + + # swap permuting +IF_LE(<vperm C0,C0,C0,swap_mask + vperm C1,C1,C1,swap_mask>) + + # previous digest combining + vxor C0,C0,C + + # polynomial multiplication "karatsuba" pre-processing + vperm C01h,C0,C1,hidw_mask + vperm C01l,C0,C1,lodw_mask + vxor C01,C01h,C01l + + # polynomial multiplication "karatsuba" + vpmsumd C01h,C01h,H21h # H01 = H^2h*C0h⊕H^1h*C1h + vpmsumd C01l,C01l,H21l # L01 = H^2l*C0l⊕H^1l*C1l + vpmsumd C01,C01,H21 # M01 = (H^2h⊕H^1h)*(C0h⊕C1h)⊕(H^2l⊕H^1l)*(C0l⊕C1l) + + # polynomial multiplication "karatsuba" post-processing + vxor C01,C01,C01h + vxor C01,C01,C01l + + # reduction first phase # [1] + vpmsumd RP,C01l,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,C01,8 # [2] + vsldoi Ml,C01,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor C01h,C01h,Mh # [2] + vxor C01l,C01l,Ml # [2] + vxor C01l,C01l,RP # [1] + + # reduction second phase + vpmsumd RP,C01l,poly_l + vxor C01h,C01l,C01h + vxor C,C01h,RP + + addi DATA,DATA,0x20 + bdnz L2x_loop + + clrldi LENGTH,LENGTH,59 +L1x: + srdi 7,LENGTH,4 + cmpldi 7,0 + beq Lrem + + # table loading + li 9,1*TableElemAlign + li 10,2*TableElemAlign + lxvd2x HlX,0,TABLE + lxvd2x HX, 9,TABLE + lxvd2x HhX,10,TABLE + + # input loading + lxvd2x C0X,0,DATA # load C0 + + # swap permuting +IF_LE(<vperm C0,C0,C0,swap_mask>) + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + + addi DATA,DATA,0x10 + clrldi LENGTH,LENGTH,60 +Lrem: + cmpldi LENGTH,0 + beq Ldone + + # table loading + li 9,1*TableElemAlign + li 10,2*TableElemAlign + lxvd2x HlX,0,TABLE + lxvd2x HX, 9,TABLE + lxvd2x HhX,10,TABLE + + # input loading + stdu SP,-16(SP) + stvx zero,0,SP +Lst_loop: + subic. LENGTH,LENGTH,1 + lbzx 7,LENGTH,DATA + stbx 7,LENGTH,SP + bne Lst_loop + lxvd2x C0X,0,SP + addi SP,SP,16 + + # swap permuting +IF_LE(<vperm C0,C0,C0,swap_mask>) + + # previous digest combining + vxor C0,C0,C + + vpmsumd Cl,C0,Hl # L = Hl*Cl + vpmsumd Cm,C0,H # M = Hh*Cl⊕Hl*Ch + vpmsumd Ch,C0,Hh # H = Hh*Ch + + # reduction first phase # [1] + vpmsumd RP,Cl,poly_h # [1] + + # polynomial multiplication post-processing # [2] + vsldoi Mh,zero,Cm,8 # [2] + vsldoi Ml,Cm,zero,8 # [2] + vsldoi RP,RP,RP,8 # [1] + vxor Ch,Ch,Mh # [2] + vxor Cl,Cl,Ml # [2] + vxor Cl,Cl,RP # [1] + + # reduction second phase + vpmsumd RP,Cl,poly_l + vxor Ch,Cl,Ch + vxor C,Ch,RP + +Ldone: +IF_LE(<vperm C,C,C,swap_mask>) + stxvd2x CX,0,X # store C + blr +EPILOGUE(_nettle_gcm_hash8) + + # gcm_fill (uint8_t *ctr, size_t blocks, union gcm_block *buffer) + +PROLOGUE(_nettle_gcm_fill) +IF_LE(<ld 6,.swap_mask@got(TOCP) + lvx swap_mask,0,6>) + + vxor zero,zero,zero + vspltisb I1,1 + vspltisb I2,2 + vspltisb I3,3 + vspltisb I4,4 + vspltisb I5,5 + vspltisb I6,6 + vspltisb I7,7 + vspltisb I8,8 + vsldoi I1,zero,I1,1 + vsldoi I2,zero,I2,1 + vsldoi I3,zero,I3,1 + vsldoi I4,zero,I4,1 + vsldoi I5,zero,I5,1 + vsldoi I6,zero,I6,1 + vsldoi I7,zero,I7,1 + vsldoi I8,zero,I8,1 + + lxvd2x CTR0X,0,CTR + IF_LE(<vperm CTR0,CTR0,CTR0,swap_mask>) + + srdi 6,BLOCKS,3 # 8x loop count + cmpldi 6,0 + beq Lfill_4x + + std 25,-56(SP); + std 26,-48(SP); + std 27,-40(SP); + std 28,-32(SP); + std 29,-24(SP); + std 30,-16(SP); + std 31,-8(SP); + + li 25,0x10 + li 26,0x20 + li 27,0x30 + li 28,0x40 + li 29,0x50 + li 30,0x60 + li 31,0x70 + + mtctr 6 + L8x_fill_loop: + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + vadduwm CTR4,CTR0,I4 + vadduwm CTR5,CTR0,I5 + vadduwm CTR6,CTR0,I6 + vadduwm CTR7,CTR0,I7 + + IF_LE(<vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask + vperm CTR4,CTR4,CTR4,swap_mask + vperm CTR5,CTR5,CTR5,swap_mask + vperm CTR6,CTR6,CTR6,swap_mask + vperm CTR7,CTR7,CTR7,swap_mask>) + + IF_LE(<stxvd2x CTR0SX,0,BUFFER>) + IF_BE(<stxvd2x CTR0X,0,BUFFER>) + stxvd2x CTR1X,25,BUFFER + stxvd2x CTR2X,26,BUFFER + stxvd2x CTR3X,27,BUFFER + stxvd2x CTR4X,28,BUFFER + stxvd2x CTR5X,29,BUFFER + stxvd2x CTR6X,30,BUFFER + stxvd2x CTR7X,31,BUFFER + + vadduwm CTR0,CTR0,I8 + addi BUFFER,BUFFER,0x80 + bdnz L8x_fill_loop + + ld 25,-56(SP); + ld 26,-48(SP); + ld 27,-40(SP); + ld 28,-32(SP); + ld 29,-24(SP); + ld 30,-16(SP); + ld 31,-8(SP); + + clrldi BLOCKS,BLOCKS,61 + + Lfill_4x: + srdi 6,BLOCKS,2 + cmpldi 6,0 + beq Lfill_2x + + li 8,0x10 + li 9,0x20 + li 10,0x30 + + vadduwm CTR1,CTR0,I1 + vadduwm CTR2,CTR0,I2 + vadduwm CTR3,CTR0,I3 + + IF_LE(<vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask + vperm CTR2,CTR2,CTR2,swap_mask + vperm CTR3,CTR3,CTR3,swap_mask>) + + IF_LE(<stxvd2x CTR0SX,0,BUFFER>) + IF_BE(<stxvd2x CTR0X,0,BUFFER>) + stxvd2x CTR1X,8,BUFFER + stxvd2x CTR2X,9,BUFFER + stxvd2x CTR3X,10,BUFFER + + vadduwm CTR0,CTR0,I4 + addi BUFFER,BUFFER,0x40 + + clrldi BLOCKS,BLOCKS,62 + + Lfill_2x: + srdi 6,BLOCKS,1 + cmpldi 6,0 + beq Lfill_1x + + li 10,0x10 + + vadduwm CTR1,CTR0,I1 + + IF_LE(<vperm CTR0S,CTR0,CTR0,swap_mask + vperm CTR1,CTR1,CTR1,swap_mask>) + + IF_LE(<stxvd2x CTR0SX,0,BUFFER>) + IF_BE(<stxvd2x CTR0X,0,BUFFER>) + stxvd2x CTR1X,10,BUFFER + + vadduwm CTR0,CTR0,I2 + addi BUFFER,BUFFER,0x20 + + clrldi BLOCKS,BLOCKS,63 + + Lfill_1x: + cmpldi BLOCKS,0 + beq Lfill_done + + IF_LE(<vperm CTR0S,CTR0,CTR0,swap_mask>) + + IF_LE(<stxvd2x CTR0SX,0,BUFFER>) + IF_BE(<stxvd2x CTR0X,0,BUFFER>) + + vadduwm CTR0,CTR0,I1 + + Lfill_done: + IF_LE(<vperm CTR0,CTR0,CTR0,swap_mask>) + stxvd2x CTR0X,0,CTR + + blr +EPILOGUE(_nettle_gcm_fill) + + .data +IF_LE(<.align 4 +.polynomial: + .byte 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0xc2 + .align 4 +.swap_mask: + .byte 8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7 + .align 4 +.hidw_mask: + .byte 23,22,21,20,19,18,17,16,7,6,5,4,3,2,1,0 + .align 4 +.lodw_mask: + .byte 31,30,29,28,27,26,25,24,15,14,13,12,11,10,9,8>) +IF_BE(<.align 4 +.polynomial: + .byte 0xc2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1 + .align 4 +.hidw_mask: + .byte 0,1,2,3,4,5,6,7,16,17,18,19,20,21,22,23 + .align 4 +.lodw_mask: + .byte 8,9,10,11,12,13,14,15,24,25,26,27,28,29,30,31>) -- 2.17.1

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

nettle-bugs