Hi, When we initially introduced symbol versioning in nettle we bundled all symbols from the library in a single version. That means that new symbols added to a release like nettle_get_hashes() may cause issues like this: https://bugzilla.redhat.com/show_bug.cgi?id=1549190
The underlying issue is in rpm-based systems which detect which symbols exist in which library version based on the version. That way, when a dependency is tracked, the version with the right symbol will be retrieved. Debian-based systems work differently so that shouldn't affect it.
To address these problems in gnutls, I follow these additional rules in to having a map file: https://gitlab.com/gnutls/gnutls/blob/master/CONTRIBUTING.md#symbol-and-libr...
which in turn comes from libvirt: https://www.berrange.com/posts/2011/01/13/versioning-in-the-libvirt-library/
That of course adds overhead on releases.
nettle doesn't necessarily need to follow that process, I send that issue in order to spark discussion on whether that's desirable to have.
regards, Nikos
On Tue, Feb 27, 2018 at 02:01:06PM +0100, Nikos Mavrogiannopoulos wrote:
Hi, When we initially introduced symbol versioning in nettle we bundled all symbols from the library in a single version. That means that new symbols added to a release like nettle_get_hashes() may cause issues like this: https://bugzilla.redhat.com/show_bug.cgi?id=1549190
The underlying issue is in rpm-based systems which detect which symbols exist in which library version based on the version. That way, when a dependency is tracked, the version with the right symbol will be retrieved. Debian-based systems work differently so that shouldn't affect it.
To address these problems in gnutls, I follow these additional rules in to having a map file: https://gitlab.com/gnutls/gnutls/blob/master/CONTRIBUTING.md#symbol-and-libr...
You mention here that symbol versioning lets you expose multiple symbols with the same name, but different versions, as is commonly seen with glibc.
That is correct, but beware that this trick only works on platforms using certain linkers. In particular Windows linker scripts will not support this. In fact windows doesn't support versioning at all - you just have a single flat list of exported symbols. For libvirt we just take the list of versioned symbols & merge them into a list suitable for Wndows DLL linking. So while using versioned symbols is fine if you only ever export one symbol for a given name, you should wary of relying on the ability to export the same name twice with different versions as that's non-portable.
which in turn comes from libvirt: https://www.berrange.com/posts/2011/01/13/versioning-in-the-libvirt-library/
That of course adds overhead on releases.
IME, this per-release versioning has not actually caused any notable overhead on libvirt's upstream development. At worst, you get contributors sending patches with old versions and have to point out to update the patch to use the correct next version number.
The only overhead we experianced falls on downstream vendors, if they try to fork the library ABI/API, by backporting new symbols to older releases. You can't make the backported symbol use the old version number, as that will cause an ABI incompatibilty if you later rebase to the new version. Using the new version number on the backported symbol, gives a false record of the ABI, as it makes the old version appear to support the new version's ABI whereas in fact it only supports 1 out of potentially many APIs from the new version.
For libvirt we simply refuse to do backporting of APIs in distros where I am involved of maintenence. If someone needs a newer API, libvirt just has to be rebased to the suitable version.
Regards, Daniel
Nikos Mavrogiannopoulos nmav@redhat.com writes:
When we initially introduced symbol versioning in nettle we bundled all symbols from the library in a single version. That means that new symbols added to a release like nettle_get_hashes() may cause issues like this: https://bugzilla.redhat.com/show_bug.cgi?id=1549190
I aim to ensure that a program compiled with nettle version x should work fine when linked with nettle version x+1, with exceptions told to the linker by changing the soname. When I talk about backwards compatibility, it's always this case I have in mind.
In general, I don't try to support compiling a program using libnettle version x, and then linking it with libnettle version x-1, and I think that should be referred to as forward compatibility rather backward comatibility. (I might try if I do releases with only bug fixes, typically with unchanged minor version).
Maybe that's the old-fashioned way, but if you want to be sure the program can link successfully with nettle version x-1, you need to compile it using nettle x-1 development files.
So if a distro builds stuff using nettle version x on the build machine, the dependency on the resulting binary packages should express that it needs shared nettle libraries which
(i) correspond to a package version >= x, and (ii) uses the same soname.
The underlying issue is in rpm-based systems which detect which symbols exist in which library version based on the version. That way, when a dependency is tracked, the version with the right symbol will be retrieved.
That's clever, and should work fine for packages that make the extra effort to have fine-grained symbol versions, but it can't be the *only* way to handle libraries with rpm?
Regards, /Niels
On Tue, Mar 27, 2018 at 11:04:28PM +0200, Niels Möller wrote:
Nikos Mavrogiannopoulos nmav@redhat.com writes:
When we initially introduced symbol versioning in nettle we bundled all symbols from the library in a single version. That means that new symbols added to a release like nettle_get_hashes() may cause issues like this: https://bugzilla.redhat.com/show_bug.cgi?id=1549190
I aim to ensure that a program compiled with nettle version x should work fine when linked with nettle version x+1, with exceptions told to the linker by changing the soname. When I talk about backwards compatibility, it's always this case I have in mind.
In general, I don't try to support compiling a program using libnettle version x, and then linking it with libnettle version x-1, and I think that should be referred to as forward compatibility rather backward comatibility. (I might try if I do releases with only bug fixes, typically with unchanged minor version).
Maybe that's the old-fashioned way, but if you want to be sure the program can link successfully with nettle version x-1, you need to compile it using nettle x-1 development files.
So if a distro builds stuff using nettle version x on the build machine, the dependency on the resulting binary packages should express that it needs shared nettle libraries which
(i) correspond to a package version >= x, and (ii) uses the same soname.
The underlying issue is in rpm-based systems which detect which symbols exist in which library version based on the version. That way, when a dependency is tracked, the version with the right symbol will be retrieved.
That's clever, and should work fine for packages that make the extra effort to have fine-grained symbol versions, but it can't be the *only* way to handle libraries with rpm?
The traditional way is for developers to update the dependancies to have an explicit version against the library they require. eg if libvirt requires some symbol introduced in nettle x.y, the maintainer would add
Requires: nettle >= x.y
the problem is that humans are fallible and so these versioned dependancies frequently get missed, because package maintainers don't often have accurate info on which min version is needed by a build. By adding fine grained symbol versioning to the nettle library, humans are taken out of the loop, in favour of automation in the toolchain. The linker embeds elf versions in the downstream app linking to nettle, so the linker will refuse to load against an outdated nettle. RPM automatically extracts these versions and turns then into correct versioned dependancies ensuring old versions can't be installed in the first place. The only scope for errors here is when the nettle developers are creating the symbol version file, and this is pretty minor IME maintaining such version files for libvirt.
Regards, Daniel
Daniel P. Berrangé berrange@redhat.com writes:
The traditional way is for developers to update the dependancies to have an explicit version against the library they require. eg if libvirt requires some symbol introduced in nettle x.y, the maintainer would add
Requires: nettle >= x.y
the problem is that humans are fallible and so these versioned dependancies frequently get missed, because package maintainers don't often have accurate info on which min version is needed by a build.
I agree it's going to be a bit brittle to do that manually. E.g., in the example at hand, the package source code only used the old nettle_hashes symbol, but the new version of nettle headers turned that into a macro calling a new function.
What I would suggest is filling out the >= x.y automatically based on the version of nettle header files actually used when compiling the package. As far as I see, that should be both reliable and fairly easy (get the version either from package metadata, or from nettle/version.h, whatever fits the package building tools best). It might sometimes produce an unnecessarily strict dependency (specifying a version higher than really necessary), but I think that's a lot better then erring in the other direction.
The only scope for errors here is when the nettle developers are creating the symbol version file,
That's me... And I imagine we'll get into the territory of subtle breakages the first time I mess up an update to that file.
and this is pretty minor IME maintaining such version files for libvirt.
How do you write testcases to verify that the version list is correct?
Regards, /Niels
On Fri, Mar 30, 2018 at 09:29:51AM +0200, Niels Möller wrote:
Daniel P. Berrangé berrange@redhat.com writes:
The only scope for errors here is when the nettle developers are creating the symbol version file,
That's me... And I imagine we'll get into the territory of subtle breakages the first time I mess up an update to that file.
and this is pretty minor IME maintaining such version files for libvirt.
How do you write testcases to verify that the version list is correct?
We've not added any tests for symbol versioning correctness - we just rely on whoever adds the API to pay attention to the file comments.
eg the tail of the libvirt_public.syms file current looks like this...
... LIBVIRT_3.9.0 { global: virDomainSetLifecycleAction; } LIBVIRT_3.7.0;
LIBVIRT_4.1.0 { global: virStoragePoolLookupByTargetPath; } LIBVIRT_3.9.0;
# .... define new API here using predicted next version number ....
and we rely on our code reviewers to watch for people making mistakes. We've not had any mistakes committed before but then we are lucky to have a large pool of active reviewers to notice this.
Regards, Daniel
On Fri, 2018-03-30 at 09:29 +0200, Niels Möller wrote:
Daniel P. Berrangé berrange@redhat.com writes:
The traditional way is for developers to update the dependancies to have an explicit version against the library they require. eg if libvirt requires some symbol introduced in nettle x.y, the maintainer would add
Requires: nettle >= x.y
the problem is that humans are fallible and so these versioned dependancies frequently get missed, because package maintainers don't often have accurate info on which min version is needed by a build.
I agree it's going to be a bit brittle to do that manually. E.g., in the example at hand, the package source code only used the old nettle_hashes symbol, but the new version of nettle headers turned that into a macro calling a new function.
A colleague is working on a script to automate that version script creation. That tool is on its infancy, but if you find it useful please send feedback at: https://github.com/ansasaki/symbol_version
The idea is that if the exported symbols per release can be provided to the script, it will create the right symbol map file.
regards, Nikos
nettle-bugs@lists.lysator.liu.se
-
Daniel P. Berrangé -
Nikos Mavrogiannopoulos -
nisse＠lysator.liu.se