[<prev] [day] [month] [year] [list]
Message-ID: <CADpJO7wMD+YutqS8Qv8Hw=3-xQuJ-J+ESsib93=CtCTQWRTnAg@mail.gmail.com>
Date: Mon, 19 Oct 2026 17:38:53 -0400
From: Aliaksey Kandratsenka <alkondratenko@...il.com>
To: libc-coord@...ts.openwall.com
Cc: fweimer@...hat.com,
Adhemerval Zanella Netto <adhemerval.zanella@...aro.org>, dalias@...ifal.cx, i@...kray.me,
riastradh@...bsd.org
Subject: RFC: dl_events_subscribe for ELF dynamic loaders
Hi libc-coord members,
In my aw-backtrace project
(https://github.com/gperftools/aw-backtrace) I currently use glibc's
_dl_find_object API, which works great. However, I also need a
reliable cache-invalidation signal on module unloads, as well as a way
to support _dl_find_object semantics on non-glibc systems without
requiring every libc to maintain RCU/seqlock data structures
internally (or resorting to fragile dlopen/dlclose interposition).
I would like to propose a small dynamic linker extension,
dl_events_subscribe, to solve this. As far as I can see, it also
cleanly addresses several long-standing dynamic linker pain points in
other projects (such as libbacktrace and BDWGC).
I have working prototype patches for glibc, musl, and NetBSD, along
with a prototype user-space _dl_find_object and async-signal-safe
snapshot dl_iterate_phdr implementation built on top of this API:
https://github.com/alk/dl-find-object
Below is the design document (also available as a Google Doc at
https://tinyurl.com/dl-events-subscribe-design; some people may prefer
to comment there). Feedback on the concept and API shape is very
welcome. I am also reaching out to the libbacktrace and BDWGC
maintainers about the use cases described below.
I am posting here so that we can focus on the API rather than on my
specific patches, but the code is available for anyone interested.
Musl prototype patch can be checked here:
https://github.com/alk/dl-find-object/blob/main/doc/0001-rfc-prototype-of-dl_events_subscribe-for-musl.patch
NetBSD patch is at:
https://github.com/alk/dl-find-object/blob/main/doc/0001-rfc-ld.elf_so-prototype-of-dl_events_subscribe-for-n.patch
Glibc patch is here:
https://github.com/alk/dl-find-object/blob/main/doc/glibc/0004-rfc-elf-prototype-of-dl_events_subscribe-for-glibc.patch
(depends on 0003 in the series that moves some code). Patches
0001-0003 in that directory are independent bug fixes that I am
posting to libc-alpha separately; only 0004 is the RFC.
---
dl_events_subscribe for ELF dynamic loaders
===========================================
Aliaksei Kandratsenka, alkondratenko@...il.com
Oct 13 2026
Self link: https://tinyurl.com/dl-events-subscribe-design
**TL;DR:** I am proposing a new dynamic linker API endpoint to
register callbacks invoked on module load and unload events. It can be
used to implement features like _dl_find_object with arbitrary
extensions (e.g., cache invalidation or additional metadata fields)
and resolves several long-standing issues with dl_iterate_phdr.
_dl_find_object is great, but as of this writing it remains
glibc-only. Not every libc is eager to maintain the RCU-like
synchronization required to make it async-signal-safe—which is the
primary reason to have it in the first place.
I propose adding dl_events_subscribe(callback, callback_data), where
callback is a function pointer of type void (*)(struct dl_phdr_info
*info, size_t info_size, void *callback_data, int is_unload). This
matches the dl_iterate_phdr callback signature, except with an
additional is_unload parameter and a void return type.
One way to think of this is as a “streaming dl_iterate_phdr”. The
callback is invoked once for each currently loaded module upon
registration, and subsequently from inside dlopen and dlclose as
modules are loaded and unloaded. For newly loaded modules, callbacks
are invoked before module constructors run, under the same loader
locks that are held (or dropped) around constructors (for example,
musl and NetBSD drop loader locks around constructors, whereas glibc
holds its load lock). For unloaded modules, callbacks are invoked
after module destructors finish. The core invariant is that a module’s
existence is announced before any of its code can run, and its removal
is announced immediately after its destructors complete and before its
address range can be reused.
The intended use of this API is to allow user code to maintain its own
snapshots of loaded module state. It can be used both to implement a
portable equivalent of _dl_find_object and to maintain an
async-signal-safe snapshot of loaded modules (e.g., for conservative
garbage collectors scanning GC roots). Unload events can also drive
cache invalidation (a primary use case for aw-backtrace, and equally
applicable to libbacktrace,
https://github.com/ianlancetaylor/libbacktrace).
Subscriptions allow moving non-trivial tracking and per-module setup
to module load/unload time, rather than performing that work at much
trickier execution points (such as during stop-the-world GC, inside
SIGPROF handlers, or when capturing and symbolizing backtraces from
fatal signal handlers).
Module load and unload events are rare in most programs, whereas
queries against the currently loaded module set are often frequent
(for instance, capturing a backtrace on every malloc call). Moving
precomputation and cache invalidation work to load and unload time is
therefore a clear performance win.
As is standard for low-level runtime APIs, the high-level design
provides the weakest guarantees that still get the job done. Proposed
specification guarantees (individual implementations may be stricter,
but this defines the broad contract):
- dl_events_subscribe invokes the newly registered callback for all
currently loaded modules across all dynamic linker namespaces.
- Callbacks may observe “spurious” unload events (an unload event
without a prior matching load event).
- Load events are announced before the corresponding module is
initialized (before its constructors run).
- Unload events are announced after the module’s destructors have
completed and before the module is unmapped.
- Unload events are always announced after their matching load events
(aside from spurious unloads as noted above).
- Load events may be duplicated (e.g., if a concurrent dlopen races
with the initial enumeration in dl_events_subscribe).
- Registered callbacks for a single event are invoked sequentially in an
unspecified order.
- Unrelated events may invoke callbacks concurrently on different
threads without internal serialization.
- Whether the module currently being loaded or unloaded is visible to
_dl_find_object or dl_iterate_phdr during the callback is
unspecified.
- Callbacks must not exit non-locally (e.g., via longjmp or by
throwing C++ exceptions across the callback boundary).
Invoking user callbacks from inside the dynamic loader naturally
raises synchronization concerns, such as deadlocks. The current
prototype implementations enforce a “mutation ban” on the thread
executing callbacks: a callback (and anything it invokes) is forbidden
from calling dynamic linker mutation endpoints (dlopen, dlmopen,
dlclose, dl_events_subscribe). This prevents the most common
re-entrancy hazards and limits Hyrum’s Law dependencies.
Common plugin architectures that load arbitrary code (such as PAM or
NSS modules) already place very few restrictions on what locks may be
held around calls to dlopen or dlclose—and, by extension, around
subscription callbacks—because module constructors themselves are
largely unrestricted.
Occasionally, however, code loads specially constructed modules under
stricter assumptions (for example, assuming a data-only module has no
initializers, or that its initializers only run tightly constrained
code). Running arbitrary subscriber callbacks in those contexts could
theoretically interact poorly with caller locks. In practice,
subscription callbacks only need to invoke primitives already used
internally by the dynamic loader (such as malloc) or acquire “leaf
locks” (locks under which no other locks are acquired), which is more
than sufficient for the intended use cases.
Lack of unsubscribe
~~~~~~~~~~~~~~~~~~~
The current prototypes do not provide an unsubscribe API. Instead,
subscriptions are tied to the module containing the callback
function’s address. When that module is unloaded via dlclose, its
subscriptions are automatically removed before its destructors
run. This is similar to how __cxa_atexit and pthread_atfork are
managed, though the prototype implementation is simpler. Keeping the
API surface small reduces both implementation complexity and the risk
of misuse.
Common use-cases
----------------
_dl_find_object implementation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In glibc, _dl_find_object hooks into essentially the same internal
add/remove points. Re-implementing it on top of dl_events_subscribe
using a two-version seqlock and async-signal-safe revalidating lookups
is straightforward. At https://github.com/alk/dl-find-object, I have
an alternative implementation that stores module metadata in a B-tree;
lookups similarly validate that the data version remains unchanged
after each access.
libbacktrace: reacting to dlopen and dlclose
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Currently, libbacktrace’s elf.c gathers DSOs only once during state
initialization. There is a long-standing FIXME in dwarf.c (used by the
ELF backend) noting the lack of updates for dlopen-loaded modules (and
a corresponding issue when modules are unloaded). On Windows,
libbacktrace’s pecoff.c already solves this cleanly using a very
similar OS facility (LdrRegisterDllNotification) to update debug
info. Updating elf.c to use dl_events_subscribe would be a small,
natural change, and synchronizing via a lock with signals masked is
straightforward.
BDWGC: deadlock-free stop-the-world module iteration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To scan static data roots safely, BDWGC
(https://github.com/bdwgc/bdwgc) needs to stop all threads and then
enumerate all loaded modules. dl_events_subscribe solves this cleanly
without calling dl_iterate_phdr while threads are stopped:
- In the subscription callback, acquire a dedicated BDWGC snapshot lock
and update an internal snapshot of loaded modules.
- When stopping the world:
- First, acquire the BDWGC snapshot lock.
- Next, stop all threads.
- Finally, scan the snapshot of loaded modules for GC roots (and
release the lock after resuming threads).
Because the GC uses its own independent lock and never touches
internal dynamic linker locks during collection, this works safely
even if a suspended thread happens to be stopped while holding
internal dynamic linker locks. It eliminates the deadlocks and fragile
dlopen interposition workarounds currently required with
dl_iterate_phdr.
Alternatives
------------
- **Async-signal-safe generation counters (dlpi_adds / dlpi_subs):**
Purely for aw-backtrace’s cache-invalidation needs, one alternative
is an async-signal-safe API to read the counters currently exposed
via dlpi_adds and dlpi_subs across all loader namespaces (since
dl_iterate_phdr only reports the caller’s namespace). If fast
enough, this would cover cache invalidation for aw-backtrace and
potentially libbacktrace (if paired with _dl_find_object and lazy
DWARF loading), but it is a much less flexible primitive.
- **_r_debug:** Not viable for libraries. _r_debug is designed for
external debuggers that can place breakpoints and halt all
threads. An in-process library like aw-backtrace cannot safely use
it.
- **LD_AUDIT:** Not viable for general-purpose libraries. It requires
wrapper scripts or users manually launching binaries with
environment variables set (LD_AUDIT=...), which works for standalone
profiling runs but not for libraries linked or loaded into normal
applications.
- **No libc changes (interposing dlopen/dlclose + /proc/self/maps):**
A rough equivalent of _dl_find_object can be built by wrapping
dlopen/dlclose for cache invalidation and parsing /proc/pid/maps (or
getdents + readlink on /proc/pid/map_files) to locate mapped
modules. This will likely serve as aw-backtrace’s fallback on older
or non-Linux systems, but it has major drawbacks: (a) reading
/proc/pid/maps scales poorly and can self-DDoS under heavy
contention; (b) cleanly interposing dlopen/dlclose across all
linking models (including static linking) is fragile; and (c)
/proc/pid/maps is non-portable and may not be mounted in chroots or
containers.
- **Async-signal-safe, lockless dl_iterate_phdr:** Libcs could expose
a snapshot-based or lockless variant of dl_iterate_phdr (for
example, by delaying module unmapping). This could be a reasonable
fallback for musl, which does not unload modules on dlclose:
dlpi_adds already provides a cache-invalidation signal, so safe
lockless iteration only requires an atomic tail pointer to fully
loaded DSOs. Note that musl already drops its loader lock around
invocations of the dl_iterate_phdr callback, which breaks tools that
assume dl_iterate_phdr serializes against dynamic linker activity
(see, for example, LSan’s assumptions in LLVM review D67738,
https://reviews.llvm.org/D67738).
Prior Art: Windows and macOS
----------------------------
Both Windows and macOS provide similar notification APIs:
- **macOS** provides _dyld_register_func_for_add_image and
_dyld_register_func_for_remove_image
(https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dyld.3.html).
Its
documentation notes that callbacks are invoked with the dynamic
linker lock held, and registration immediately invokes the callback
for already-loaded images.
- **Windows** provides LdrRegisterDllNotification and
LdrUnregisterDllNotification
(https://learn.microsoft.com/en-us/windows/win32/devnotes/ldrdllnotification).
While
not explicitly documented by Microsoft, internet gossip indicates
that its callbacks are likewise invoked under the loader lock.
Potential questions
-------------------
- **Should there be an explicit unsubscribe API?** And if so, should
it be restricted (e.g., to module destructors)? Currently
implemented automatic cleanup on module unload avoids race
conditions between concurrent callback execution and
unregistration. A smaller API surface means fewer risks of abusing
it. While JIT-compiled or libffi-trampoline callbacks might not map
to a standard DSO, those cases can simply remain registered for the
lifetime of the process.
- **Should subscription callbacks be able to “veto” (fail) a load or
unload?** In glibc, _dl_find_object_update can cause dlopen to fail
if it runs out of memory. One could argue a third-party
_dl_find_object-style subscriber might want to do the same.
However, supporting veto-capable callbacks—especially on the unload
path, where user implementations may also allocate
memory—significantly complicates loader rollback logic. Some
prototypes invoke load callbacks after the loader’s “point of no
return”. Even in glibc, while _dl_find_object_dlclose avoids
allocation by atomically zeroing the address range, scope updates
during dlclose still call malloc with imperfect OOM handling.
Keeping callbacks void and non-failing keeps loader integration much
simpler.
- **Aborted dlopen and error-recovery paths:** Allowing spurious
unload events (as well as load-then-immediate-unload sequences)
simplifies error handling inside the dynamic loader. If a dlopen
fails partway through (e.g., due to a relocation failure or an OOM
during internal bookkeeping like _dl_find_object_update) and runs
its internal dlclose cleanup path on partially loaded objects, the
loader does not need to track whether each object already fired a
load callback. It can unconditionally fire unload callbacks during
cleanup: modules that already reported a load event are cleanly
rolled back, and modules that failed prior to the load event simply
emit a benign spurious unload event.
- **Should dl_events_subscribe replay currently loaded modules, or
only report deltas?** Windows reports only deltas, requiring callers
to enumerate existing modules separately. We could do the same, but
because dl_iterate_phdr in glibc only enumerates the caller’s
namespace, a delta-only API would also require introducing a new
dl_iterate_phdr_all function. Replaying currently loaded modules
upon subscription keeps the API surface down to a single function.
- **Dynamic linker namespaces (dlmopen):** Currently a glibc-specific
feature (see man dlmopen). Whereas dl_iterate_phdr only visits DSOs
in the caller’s namespace, _dl_find_object operates globally across
all namespaces. dl_events_subscribe is likewise global across all
namespaces to match _dl_find_object.
- **What arguments should be passed to subscription callbacks?** The
current prototypes mirror dl_iterate_phdr (passing struct
dl_phdr_info with dlpi_adds and dlpi_subs set to 0) plus
is_unload. Reusing struct dl_phdr_info is familiar and sufficient to
match loads and unloads, though extensions (such as exposing the
namespace ID or struct link_map) are possible.
- **Process exit behavior:** In the glibc and NetBSD prototypes,
starting process shutdown (_dl_fini) disables new subscriptions and
cancels all existing subscriptions before running global ELF
destructors. This matches dlclose semantics and avoids firing
redundant unload events during process termination. Note that if
dl_events_subscribe is called after process shutdown has begun—or
with a callback belonging to a module currently being unloaded by
dlclose—the implementation still invokes the callback for all
currently loaded modules to honor the base contract, and then
silently drops the subscription.
- **Reporting errors from dl_events_subscribe:** The current patches
have dl_events_subscribe return 0 on success and a positive errno
value (ENOMEM, EDEADLK) on failure, matching modern pthread APIs
rather than using the awkward dlerror facility.
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
