Hi,
I committed a change to update nettle version numbers, which implies a new symbol version for internal symbols.
That seems to break the gnutls ci build, https://gitlab.com/gnutls/nettle/-/jobs/487360242
The error is
1217 ./bootstrap: getting translations into po/.reference for gnutls... 1218 wget: /lib64/libhogweed.so.5: version `HOGWEED_INTERNAL_5_0' not found (required by /lib64/libgnutls.so.30) 1219 wget: /lib64/libnettle.so.7: version `NETTLE_INTERNAL_7_0' not found (required by /lib64/libgnutls.so.30)
I don't quite understand all details. This job buils and installs nettle, as
./configure --disable-documentation --prefix=/usr --libdir=/usr/lib64 && make -j4 && make install
replacing any previously instaled version with the same soname. So that's a somewhat broader test than I had expected. That explains why linking of wget is affected.
The more worring part is that the installed gnutls library seems to depend on internal nettle symbols (it would have been nice if the error message named the offending symbols). Is that intended?
It's not necessarily wrong, but any packaged version of gnutls built that way *must* be marked to depend on exactly the same versions of nettle with which it was built. And the ci rule would need to use a different prefix and some other way to tell the gnutls build to use the newly built nettle library instead of the system version.
On my own debian machine I have libgnutls.so.30, but apparently an older version. According to
objdump -T /usr/lib/x86_64-linux-gnu/libgnutls.so.30
the version I have refers only to symbols with version NETTLE_6 and HOGWEED_4, i.e., nettle version older than 3.5. And no mention of NETTLE_INTERNAL or HOGWEED_INTERNAL.
Regards, /Niels
On Mon, Mar 30, 2020 at 1:23 PM Niels Möller nisse@lysator.liu.se wrote:
Hi,
I committed a change to update nettle version numbers, which implies a new symbol version for internal symbols.
That seems to break the gnutls ci build, https://gitlab.com/gnutls/nettle/-/jobs/487360242
The error is
1217 ./bootstrap: getting translations into po/.reference for gnutls... 1218 wget: /lib64/libhogweed.so.5: version `HOGWEED_INTERNAL_5_0' not found (required by /lib64/libgnutls.so.30) 1219 wget: /lib64/libnettle.so.7: version `NETTLE_INTERNAL_7_0' not found (required by /lib64/libgnutls.so.30)
I don't quite understand all details. This job buils and installs nettle, as
./configure --disable-documentation --prefix=/usr --libdir=/usr/lib64 && make -j4 && make install
That works well as long as binary compatibility is kept. As nettle breaks it, applications in the system will be unable to run. We may want to install nettle somewhere separately and instruct gnutls to use it from there rather than the default locations.
regards, Nikos
Nikos Mavrogiannopoulos n.mavrogiannopoulos@gmail.com writes:
On Mon, Mar 30, 2020 at 1:23 PM Niels Möller nisse@lysator.liu.se wrote:
The error is
1217 ./bootstrap: getting translations into po/.reference for gnutls... 1218 wget: /lib64/libhogweed.so.5: version `HOGWEED_INTERNAL_5_0' not found (required by /lib64/libgnutls.so.30) 1219 wget: /lib64/libnettle.so.7: version `NETTLE_INTERNAL_7_0' not found (required by /lib64/libgnutls.so.30)
I don't quite understand all details. This job buils and installs nettle, as
./configure --disable-documentation --prefix=/usr --libdir=/usr/lib64 && make -j4 && make install
That works well as long as binary compatibility is kept. As nettle breaks it, applications in the system will be unable to run.
But advertised binary compatibility between Nettle releases considers only the symbols related to the public api, not the symbols with *INTERNAL* as version. When gnutls refers to internal symbols, there's no binary compatibility at all between nettle versions.
I would strongly recommend that the default configuration of gnutls don't refer to internal nettle symbols. And any packaging of a version with a non-default configuration enabling use of nettle internals (e.g, to support more gost curves) needs to depend on a particular version of nettle, rather than on the soname, which only captures binary compatibility of the public ABI.
We may want to install nettle somewhere separately and instruct gnutls to use it from there rather than the default locations.
That seems to be the right solution to the immediate problem.
Regards, /Niels
On Tue, Mar 31, 2020 at 9:41 AM Niels Möller nisse@lysator.liu.se wrote:
Nikos Mavrogiannopoulos n.mavrogiannopoulos@gmail.com writes:
On Mon, Mar 30, 2020 at 1:23 PM Niels Möller nisse@lysator.liu.se wrote:
The error is
1217 ./bootstrap: getting translations into po/.reference for gnutls... 1218 wget: /lib64/libhogweed.so.5: version `HOGWEED_INTERNAL_5_0' not found (required by /lib64/libgnutls.so.30) 1219 wget: /lib64/libnettle.so.7: version `NETTLE_INTERNAL_7_0' not found (required by /lib64/libgnutls.so.30)
I don't quite understand all details. This job buils and installs nettle, as
./configure --disable-documentation --prefix=/usr --libdir=/usr/lib64 && make -j4 && make install
That works well as long as binary compatibility is kept. As nettle breaks it, applications in the system will be unable to run.
But advertised binary compatibility between Nettle releases considers only the symbols related to the public api, not the symbols with *INTERNAL* as version. When gnutls refers to internal symbols, there's no binary compatibility at all between nettle versions.
I would strongly recommend that the default configuration of gnutls don't refer to internal nettle symbols. And any packaging of a version with a non-default configuration enabling use of nettle internals (e.g, to support more gost curves) needs to depend on a particular version of nettle, rather than on the soname, which only captures binary compatibility of the public ABI.
That is not possible as we are introducing features into gnutls in a faster pace than they are introduced in nettle (e.g., x448 or gost). Without a predictable cadence of nettle releases we have to duplicate and bundle a lot of code into gnutls because it is unknown when the next nettle release will be. The options we have considered were bundling nettle into gnutls or backporting selected features even when depending on nettle's ABI. The latter is the approach we currently take as it still keeps separation between the projects. The main problem lies however with the unpredictability and long delay of nettle's releases. If that could be addressed it would solve the dependency issue as well.
regards, Nikos
nisse@lysator.liu.se (Niels Möller) writes:
Nikos Mavrogiannopoulos n.mavrogiannopoulos@gmail.com writes:
We may want to install nettle somewhere separately and instruct gnutls to use it from there rather than the default locations.
That seems to be the right solution to the immediate problem.
Below patch seems to work. See https://gitlab.com/gnutls/nettle/-/jobs/495327357. Does it look right? Some questions:
1. Gnutls' configure uses pkg-config to check for existence and version of nettle, but it doesn't use flags provided by pkg-config. We therefore need to set *both* PKG_CONFIG_PATH and CPPFLAGS/LDFLAGS to use a Nettle library in a custom location. Is it supposed to work that way? I would expect most things to either rely exclusively on pkg-config, or use other means of configuration. Except that I don't know if pkg-config can or should be used to supply options like -Wl,-rpath,...
2. What's the right way to add line breaks for the very long configure command in .gitlab-ci.yml?
Regards, /Niels
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 663f98f5..c215f850 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -96,11 +96,11 @@ build/gnutls: image: $CI_REGISTRY/$BUILD_IMAGES_PROJECT:$FEDORA_BUILD script: - ./.bootstrap && - - ./configure --disable-documentation --prefix=/usr --libdir=/usr/lib64 && + ./configure --disable-documentation --prefix="$(pwd)/local" --libdir="$(pwd)/local/lib" && make -j4 && make install - git clone --depth 1 --branch master https://gitlab.com/gnutls/gnutls.git gnutls-git - cd gnutls-git && git submodule update --init && ./bootstrap && - ./configure --disable-gost --disable-cxx --disable-guile --disable-doc && make -j$(nproc) && + ./configure PKG_CONFIG_PATH="$(pwd)/../local/lib/pkgconfig" CPPFLAGS="-I$(pwd)/../local/include" LDFLAGS="-L$(pwd)/../local/lib -Wl,-rpath,$(pwd)/../local/lib" --disable-gost --disable-cxx --disable-guile --disable-doc && make -j$(nproc) && make -j $(nproc) check tags: - shared
On 4/2/20 8:03 AM, Niels Möller wrote:
nisse@lysator.liu.se (Niels Möller) writes:
Nikos Mavrogiannopoulos n.mavrogiannopoulos@gmail.com writes:
We may want to install nettle somewhere separately and instruct gnutls to use it from there rather than the default locations.
That seems to be the right solution to the immediate problem.
Below patch seems to work. See https://gitlab.com/gnutls/nettle/-/jobs/495327357. Does it look right? Some questions:
- Gnutls' configure uses pkg-config to check for existence and version of nettle, but it doesn't use flags provided by pkg-config. We therefore need to set *both* PKG_CONFIG_PATH and CPPFLAGS/LDFLAGS to use a Nettle library in a custom location. Is it supposed to work that way? I would expect most things to either rely exclusively on pkg-config, or use other means of configuration. Except that I don't know if pkg-config can or should be used to supply options like -Wl,-rpath,...
When installed to a custom location, you have to set PKG_CONFIG_PATH correctly. And the configure.ac code for nettle should be something like (taken from wget2): PKG_CHECK_MODULES([NETTLE], nettle, [ with_libnettle=yes LIBS="$NETTLE_LIBS $LIBS" CFLAGS="$NETTLE_CFLAGS $CFLAGS" AC_DEFINE([WITH_LIBNETTLE], [1], [Use libnettle]) ], [ with_libnettle=no AC_MSG_WARN(*** LIBNETTLE was not found. Fallback to libgcrypt for hashing and checksumming.) ])
*BUT* I had some issues with just setting PKG_CONFIG_PATH alone on Debian unstable a while ago. I had to adjust to something different.
This is from my current MinGW recipe: unset CC unset CXX
PREFIX=x86_64-w64-mingw32
export PATH="/usr/$PREFIX/bin:$PATH" export INSTALLDIR="$PWD/$PREFIX" export PKG_CONFIG_PATH=$INSTALLDIR/lib/pkgconfig:/usr/$PREFIX/lib/pkgconfig export PKG_CONFIG_LIBDIR="$INSTALLDIR/lib/pkgconfig" export PKG_CONFIG="/usr/bin/${PREFIX}-pkg-config" export CPPFLAGS="-I$INSTALLDIR/include" export LDFLAGS="-L$INSTALLDIR/lib"
cd nettle bash .bootstrap ./configure --build=x86_64-pc-linux-gnu --host=$PREFIX --enable-mini-gmp --enable-shared --disable-documentation --prefix=$INSTALLDIR make clean make -j$(nproc) make install
Hope that helps a bit.
- What's the right way to add line breaks for the very long configure command in .gitlab-ci.yml?
Just breaking the line (linefeed) is enough, as long as the continuation line doesn't start with a dash '-'. What is also ok and used in many places is adding a '' plus linefeed.
Regards, Tim
Tim Rühsen tim.ruehsen@gmx.de writes:
When installed to a custom location, you have to set PKG_CONFIG_PATH correctly. And the configure.ac code for nettle should be something like (taken from wget2): PKG_CHECK_MODULES([NETTLE], nettle, [ with_libnettle=yes LIBS="$NETTLE_LIBS $LIBS" CFLAGS="$NETTLE_CFLAGS $CFLAGS" AC_DEFINE([WITH_LIBNETTLE], [1], [Use libnettle]) ], [ with_libnettle=no AC_MSG_WARN(*** LIBNETTLE was not found. Fallback to libgcrypt for hashing and checksumming.) ])
*BUT* I had some issues with just setting PKG_CONFIG_PATH alone on Debian unstable a while ago. I had to adjust to something different.
This is the failure I got when setting PKG_CONFIG_PATH but not CPPFLAGS: https://gitlab.com/gnutls/nettle/-/jobs/494919451
In file included from iov.c:26: ../lib/gnutls_int.h:56:10: fatal error: nettle/memxor.h: No such file or directory 56 | #include <nettle/memxor.h> | ^~~~~~~~~~~~~~~~~ compilation terminated.
So either gnutls doesn't use pkg-config include flags, or there's something wrong with nettle's .pc files. I can't investigate that right now.
- What's the right way to add line breaks for the very long configure command in .gitlab-ci.yml?
Just breaking the line (linefeed) is enough, as long as the continuation line doesn't start with a dash '-'.
I see, thanks.
Regards, /Niels
On Mon, Mar 30, 2020 at 7:23 AM Niels Möller nisse@lysator.liu.se wrote:
I committed a change to update nettle version numbers, which implies a new symbol version for internal symbols.
That seems to break the gnutls ci build, https://gitlab.com/gnutls/nettle/-/jobs/487360242
The error is
1217 ./bootstrap: getting translations into po/.reference for gnutls... 1218 wget: /lib64/libhogweed.so.5: version `HOGWEED_INTERNAL_5_0' not found (required by /lib64/libgnutls.so.30) 1219 wget: /lib64/libnettle.so.7: version `NETTLE_INTERNAL_7_0' not found (required by /lib64/libgnutls.so.30)
I don't quite understand all details. This job buils and installs nettle, as
./configure --disable-documentation --prefix=/usr --libdir=/usr/lib64 && make -j4 && make install
replacing any previously instaled version with the same soname. So that's a somewhat broader test than I had expected. That explains why linking of wget is affected.
The more worring part is that the installed gnutls library seems to depend on internal nettle symbols (it would have been nice if the error message named the offending symbols). Is that intended?
It's not necessarily wrong, but any packaged version of gnutls built that way *must* be marked to depend on exactly the same versions of nettle with which it was built. And the ci rule would need to use a different prefix and some other way to tell the gnutls build to use the newly built nettle library instead of the system version.
On my own debian machine I have libgnutls.so.30, but apparently an older version. According to
objdump -T /usr/lib/x86_64-linux-gnu/libgnutls.so.30
the version I have refers only to symbols with version NETTLE_6 and HOGWEED_4, i.e., nettle version older than 3.5. And no mention of NETTLE_INTERNAL or HOGWEED_INTERNAL.
It is not limited to Nettle.
Last week, /usr/lib/x86_64-linux-gnu/libgnutls.so.30 could not find a symbol in libidn2.so I had installed into /usr/local or /opt/local. The problem cascaded to the point I could not open a Terminal to fix things.
I had to reboot into safe mode, and delete both /usr/local or /opt/local. Then that promiscuous-ass loader fixed its cache.
Would the genius who thought it was a good idea to compile and link against one library, and then runtime link against the wrong library, please step forward to claim their Darwin award...
Jeff
On Tue, Mar 31, 2020 at 7:42 AM Jeffrey Walton noloader@gmail.com wrote:
On Mon, Mar 30, 2020 at 7:23 AM Niels Möller nisse@lysator.liu.se wrote:
I committed a change to update nettle version numbers, which implies a new symbol version for internal symbols.
That seems to break the gnutls ci build, https://gitlab.com/gnutls/nettle/-/jobs/487360242
The error is
1217 ./bootstrap: getting translations into po/.reference for gnutls... 1218 wget: /lib64/libhogweed.so.5: version `HOGWEED_INTERNAL_5_0' not found (required by /lib64/libgnutls.so.30) 1219 wget: /lib64/libnettle.so.7: version `NETTLE_INTERNAL_7_0' not found (required by /lib64/libgnutls.so.30)
I don't quite understand all details. This job buils and installs nettle, as
./configure --disable-documentation --prefix=/usr --libdir=/usr/lib64 && make -j4 && make install
replacing any previously instaled version with the same soname. So that's a somewhat broader test than I had expected. That explains why linking of wget is affected.
The more worring part is that the installed gnutls library seems to depend on internal nettle symbols (it would have been nice if the error message named the offending symbols). Is that intended?
It's not necessarily wrong, but any packaged version of gnutls built that way *must* be marked to depend on exactly the same versions of nettle with which it was built. And the ci rule would need to use a different prefix and some other way to tell the gnutls build to use the newly built nettle library instead of the system version.
On my own debian machine I have libgnutls.so.30, but apparently an older version. According to
objdump -T /usr/lib/x86_64-linux-gnu/libgnutls.so.30
the version I have refers only to symbols with version NETTLE_6 and HOGWEED_4, i.e., nettle version older than 3.5. And no mention of NETTLE_INTERNAL or HOGWEED_INTERNAL.
It is not limited to Nettle.
Last week, /usr/lib/x86_64-linux-gnu/libgnutls.so.30 could not find a symbol in libidn2.so I had installed into /usr/local or /opt/local. The problem cascaded to the point I could not open a Terminal to fix things.
My problem appeared again. I'm going to have to boot into recovery mode, delete /usr/local, and reboot to fix it.
System libraries have no business going into my directories. I don't mess with /bin, /usr/bin, /lib and /usr/lib. The system should stay out of our directories.
$ sudo apt-get update ... /usr/lib/apt/methods/http: relocation error: /usr/lib/x86_64-linux-gnu/libgnutls.so.30: symbol _idn2_punycode_decode version IDN2_0.0.0 not defined in file libidn2.so.0 with link time reference E: Method http has died unexpectedly! E: Sub-process http returned an error code (127) E: Method /usr/lib/apt/methods/http did not start correctly
On 4/3/20 10:29 AM, Jeffrey Walton wrote:
On Tue, Mar 31, 2020 at 7:42 AM Jeffrey Walton noloader@gmail.com wrote:
On Mon, Mar 30, 2020 at 7:23 AM Niels Möller nisse@lysator.liu.se wrote:
I committed a change to update nettle version numbers, which implies a new symbol version for internal symbols.
That seems to break the gnutls ci build, https://gitlab.com/gnutls/nettle/-/jobs/487360242
The error is
1217 ./bootstrap: getting translations into po/.reference for gnutls... 1218 wget: /lib64/libhogweed.so.5: version `HOGWEED_INTERNAL_5_0' not found (required by /lib64/libgnutls.so.30) 1219 wget: /lib64/libnettle.so.7: version `NETTLE_INTERNAL_7_0' not found (required by /lib64/libgnutls.so.30)
I don't quite understand all details. This job buils and installs nettle, as
./configure --disable-documentation --prefix=/usr --libdir=/usr/lib64 && make -j4 && make install
replacing any previously instaled version with the same soname. So that's a somewhat broader test than I had expected. That explains why linking of wget is affected.
The more worring part is that the installed gnutls library seems to depend on internal nettle symbols (it would have been nice if the error message named the offending symbols). Is that intended?
It's not necessarily wrong, but any packaged version of gnutls built that way *must* be marked to depend on exactly the same versions of nettle with which it was built. And the ci rule would need to use a different prefix and some other way to tell the gnutls build to use the newly built nettle library instead of the system version.
On my own debian machine I have libgnutls.so.30, but apparently an older version. According to
objdump -T /usr/lib/x86_64-linux-gnu/libgnutls.so.30
the version I have refers only to symbols with version NETTLE_6 and HOGWEED_4, i.e., nettle version older than 3.5. And no mention of NETTLE_INTERNAL or HOGWEED_INTERNAL.
It is not limited to Nettle.
Last week, /usr/lib/x86_64-linux-gnu/libgnutls.so.30 could not find a symbol in libidn2.so I had installed into /usr/local or /opt/local. The problem cascaded to the point I could not open a Terminal to fix things.
My problem appeared again. I'm going to have to boot into recovery mode, delete /usr/local, and reboot to fix it.
System libraries have no business going into my directories. I don't mess with /bin, /usr/bin, /lib and /usr/lib. The system should stay out of our directories.
IMO, you have that in your hand. Check /etc/ld.so.conf.d/* for /usr/local/lib. Prefix that file to be included as last one (or even remove it). Execute 'ldconfig' when done.
That *should* change your library search path as wanted.
Regards, Tim
nettle-bugs@lists.lysator.liu.se
-
Jeffrey Walton -
Nikos Mavrogiannopoulos -
Nikos Mavrogiannopoulos -
nisse＠lysator.liu.se -
Tim Rühsen