Linux kernel remote logging: approaches, challenges, implementation
These are the slides of Solar Designer's talk at
BSidesZagreb 2024.
A video of the talk is available on YouTube.
This talk is based on research conducted for our Linux Kernel Runtime Guard (LKRG) project,
which is a Linux kernel module that performs runtime integrity checking of the kernel and detection of security
vulnerability exploits against the kernel.
Delivery, storage, and processing of LKRG security events to/on a remote system is a natural extension of LKRG's functionality.
Remote logging is also valuable on its own, including for troubleshooting and post-mortem analyses of (non-)security incidents,
where the system's local logs might be unavailable, incomplete, or tampered with.
In this talk, I start by briefly examining pre-existing remote logging solutions and their suitability.
Then I proceed to our own considerations and choices for transport and security protocols and software design,
including many of the challenges and trade-offs encountered.
Finally, I introduce and demonstrate the initial implementation in LKRG, released just in time for the talk, as well as
its integration in Rocky Linux via the Security SIG package.
For the live demo (not seen on the slides), I used Valentina Palmiotti's (@chompie1337)
exploit of an old vulnerability in the eBPF subsystem
running current LKRG on a deliberately out-of-date Ubuntu VPS in New York,
delivering logs to a VPS in Amsterdam running AlmaLinux 8.9 with Rocky Linux 8.9's SIG/Security package of lkrg-logger installed.
The attack was detected and blocked (process killed before it could spawn a root shell),
and LKRG messages promptly delivered to and logged on the other continent.
And yes, we encourage and provide instructions for reuse of Rocky Linux SIG/Security
packages on other Enterprise Linux distros.
This research and initial implementation have been sponsored by
Binarly software supply chain security platform,
whereas the public release, Rocky Linux integration, and this talk are due to my work at CIQ,
the primary corporate sponsor of Rocky Linux.
Please click on the slides for higher-resolution versions.
You can also
download a PDF file with all of the slides (6 MB) or
view them at Speaker Deck.
2248
