[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAMbhsRQQkmxbyxYhsjJoCs2Opv970vpvA5qpGQ9CmGdAPwgOtQ@mail.gmail.com>
Date: Mon, 22 Aug 2022 17:22:06 -0700
From: Colin Cross <ccross@...roid.com>
To: Colin Cross <ccross@...roid.com>, musl@...ts.openwall.com,
Ryan Prichard <rprichard@...gle.com>
Subject: Re: Running musl executables without a preinstalled dynamic linker
On Sat, Aug 20, 2022 at 2:43 AM Szabolcs Nagy <nsz@...t70.net> wrote:
>
> * Colin Cross <ccross@...roid.com> [2022-08-15 14:35:33 -0700]:
> > I would like to distribute dynamic binaries built against musl to
> > systems that do not have the musl dynamic linker installed in any
> > known location (e.g. /lib/ld-musl-$ARCH.so.1). I have two prototypes
> > that enable this, and I’d like to gauge whether either is something
> > that would be of interest to check in to musl, or whether it would be
> > something we should keep in our project.
> >
> > The first solution is based on the embedded linker we use to test
> > bionic libc on non-Android systems. The dynamic linker is compiled as
> > usual, then the resulting elf file is embedded as raw data into
> > Scrt1.o and the PT_INTERP section removed. The entry point is changed
> > to point to a trampoline that modifies AT_BASE, AT_ENTRY and AT_PHDR
> > to simulate how the kernel would initialize them if the dynamic linker
> > was mapped separately by the kernel instead of as part of the main
> > executable, and then jumps to the dynamic linker.
> >
> > This embedded linker solution works relatively well, except that the
> > dynamic linker’s elf sections are inside the main executable’s elf
> > sections, which can break reasonable assumptions. For example, musl’s
> > dladdr fails to find symbols in the embedded linker, and gdb has
> > trouble finding debug information from the linker. Musl’s reuse of
> > libc.so as the linker means that these problems apply to everything in
> > libc.so, and also increases the size of every binary by including all
> > of libc.so.
> >
> > These problems with the embedded linker could be somewhat mitigated by
> > splitting the dynamic linker out of libc.so when using the embedded
> > linker. That requires compiling the ldso sources against a statically
> > linked libc.a, tweaking some of the initialization, and forwarding the
> > dl* calls from libc.so to the separate linker. The changes are
> > relatively small, but result in a pretty big difference in musl’s
> > internals with and without the embedded linker that may be hard to
> > maintain.
> >
>
> that breaks atomic update of the libc and introduces libc internal abi.
> (i.e. bad for long term security and maintainability)
The intent was to distribute the binary with the embedded linker
alongside a matching copy of libc, but yes, this would increase the
chances of an unintended version skew.
> > The second solution we call “relinterp”. It was originally designed
> > by Ryan Prichard as a standalone trampoline that could be used with
> > musl, glibc or bionic, but I’ve more tightly integrated it with musl
> > in order to reuse CRTJMP for architecture portability and some of
> > musl’s string functions to reduce the size of the code. It uses a
> > similar trampoline in Scrt1.o, but with a much larger implementation
> > that reads DT_RUNPATH to construct a path to the dynamic linker that
> > is relative to the executable. It then maps the dynamic linker as the
> > kernel would, modifies AT_BASE, AT_ENTRY and AT_PHDR, and jumps to the
> > dynamic linker.
> >
>
> i think this is a better approach.
Agreed, I generally prefer this approach.
> i would not use Scrt1.o though, the same toolchain should be
> usable for normal linking and relinterp linking, just use a
> different name like Xcrt1.o.
Is there some way to get gcc/clang to use Xcrt1.o without using
-nostdlib and passing all the crtbegin/end objects manually?
> > The current prototype of relinterp is tricky to compile, as it
> > requires using -fvisibility=hidden and ld -r partial linking to build
> > a Scrt1.o file that uses some of the src/string/*.c sources without
> > any relocations, and then objcopy –keep-global-symbol to hide the
> > string symbols. It’s only useful if DT_RUNPATH contains $ORIGIN so
> > that the dynamic linker can be distributed alongside the executable,
> > so it is probably never going to be suitable for setuid binaries.
> >
> > If relinterp were going to be included with musl I’d refactor it to
> > reuse the __dls* bootstrapping from dynlink.c so that it can link
> > against libc.a and not worry about avoiding any relocations.
> >
>
> i would make Xcrt1.o self-contained and size optimized: it only
> runs at start up, this is a different requirement from the -O3
> build of normal string functions. and then there is no dependency
> on libc internals (which may have various instrumentations that
> does not work in Xcrt1.o).
Doesn't this same logic apply to most of the code in dynlink.c? My
main worry with a self contained implementation is that it requires
reimplementations of various string functions that are easy to get
wrong. The current prototype reuses the C versions of musl's string
functions, but implements its own syscall wrappers to avoid
interactions with musl internals like errno.
> > An alternative solution to these two would be to distribute statically
> > linked binaries, which precludes the use of dlopen, or to wrap every
> > executable in a shell script that runs the dynamic linker directly.
> >
>
> i think it is possible to support static linking such that if dlopen
> is linked then the entire libc gets linked into the main exe with
> libc apis exported. then dlopen can work from an otherwise static exe.
> (may not be easy to implement in practice though)
>
> > Do either of these prototypes seem interesting enough to clean up and
> > post as upstream patches, or should I keep them as a side project that
> > I can bolt on to musl with minimal invasive changes?
> >
> > Colin
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
