[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <bE-0gPJy86FuZCqdfKSDz--xvumVkJH-395hH-0cjzrHnMHF0YVDs7OgosvBpmDAi7bGaHWrB4rpfvwQrkS0kGIdmkXxesnQhaNGpFjJP2M=@harmenstoppels.nl>
Date: Wed, 12 Jan 2022 14:52:39 +0000
From: Harmen Stoppels <me@...menstoppels.nl>
To: Rich Felker <dalias@...c.org>
Cc: musl@...ts.openwall.com
Subject: Re: Cannot dlopen() an already loaded shared library by its SONAME name
> >
> > 1. Julia [1] splits binary dependencies into separate packages, so when
> > liba.so depends on libb.so, they live in a different dir, where
> > the absolute and relative paths are only known when the julia
> > interpreter has started, so neither rpaths or LD_LIBRARY_PATH can
> > be used.
> > So they dlopen libb.so, and then dlopen liba.so in that
> > order, and then assume liba.so does not have to locate libb.so
> > again, because its soname is already seens before.
> > The proposed workaround was: don't list libb.so in the
> > DT_NEEDED of liba.so (that is, if you're already doing the work of
> > the linker, you might as well not use the linker at all for locating
> > libs). However, being able to run executables shipped with julia
> > packages would still be nice (e.g. a subprocess with LD_LIBRARY_PATH
> > set properly)
> >
> > 2. The Nix / Guix / Spack people are trying to reduce startup time of
> > executables with many shared libraries (as well as fixing library
> > paths once and for all to keep executables run deterministically).
> > In Guix there's a blog post where they call this the "stat storm" [2],
> > and they solve it in a glibc patch: using context dependent ld.so.cache,
> > that is, a reverse mapping soname => library path.
> > In Nix the proposal to fix the "stat storm" is to replace DT_NEEDED
> > in executables with absolute paths of all required libs (also
> > transient ones). This works fine, except on musl, where a dlopen by
> > soname will still do a search.
>
> This could be solved much better by making an application-specific
> directory full of symlinks to the libraries it uses and putting that
> directory as the first thing in the program binary's rpath.
So the proposal is basically to replicate an application-specific
ld.so.cache in the filesystem? Create a dir per ELF file, put its
path in the RPATH, fill the dir with symlinks from soname => library.
One problem with this is that $ORIGIN starts behaving differently
compared to ld.so.cache. $ORIGIN is now relative to the directory
of the symlink, not to the realpath of library. So if a library
of a dependent package dlopen's a library in its own prefix by soname,
relying on an rpath of say $ORIGIN/plugins, this will fail if the
symlink dir is just a flat file list.
So it means you'd effectively need to merge the prefixes, and this
won't fly in Nix / Spack.
Also it won't be a manageable solution for Julia, cause they require immutability
of each prefix (and if they knew the relative path ahead of time, they
wouldn't need this at all).
> but having this happen on libraries without any SONAME is really an anti-feature.
What do you mean? I think we're talking only about libraries that have a SONAME,
do you mean dlopen-by-soname?
So, do I understand correctly that loading a lib by path & putting its soname in
a dict, so that future libs opened by soname can early exit is fine?
But loading a lib by soname first, and then by path, and upon opening it
happens to have a soname seen before, but is a different file (st_dev/st_ino),
then it should continue with this lib, not early exit?
Download attachment "publickey - me@...menstoppels.nl - 0xFD537C88.asc" of type "application/pgp-keys" (1815 bytes)
Download attachment "signature.asc" of type "application/pgp-signature" (510 bytes)
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
