[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <20240701083838.GA12787@localhost.localdomain>
Date: Mon, 1 Jul 2024 08:40:06 +0000
From: Qualys Security Advisory <qsa@...lys.com>
To: "oss-security@...ts.openwall.com" <oss-security@...ts.openwall.com>
Subject: CVE-2024-6387: RCE in OpenSSH's server, on glibc-based Linux systems
Qualys Security Advisory
regreSSHion: RCE in OpenSSH's server, on glibc-based Linux systems
(CVE-2024-6387)
========================================================================
Contents
========================================================================
Summary
SSH-2.0-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3 (Debian 3.0r6, from 2005)
- Theory
- Practice
- Timing
SSH-2.0-OpenSSH_4.2p1 Debian-7ubuntu3 (Ubuntu 6.06.1, from 2006)
- Theory, take one
- Theory, take two
- Practice
- Timing
SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2 (Debian 12.5.0, from 2024)
- Theory
- Practice
- Timing
Towards an amd64 exploit
Patches and mitigation
Acknowledgments
Timeline
========================================================================
Summary
========================================================================
All it takes is a leap of faith
-- The Interrupters, "Leap of Faith"
Preliminary note: OpenSSH is one of the most secure software in the
world; this vulnerability is one slip-up in an otherwise near-flawless
implementation. Its defense-in-depth design and code are a model and an
inspiration, and we thank OpenSSH's developers for their exemplary work.
We discovered a vulnerability (a signal handler race condition) in
OpenSSH's server (sshd): if a client does not authenticate within
LoginGraceTime seconds (120 by default, 600 in old OpenSSH versions),
then sshd's SIGALRM handler is called asynchronously, but this signal
handler calls various functions that are not async-signal-safe (for
example, syslog()). This race condition affects sshd in its default
configuration.
On investigation, we realized that this vulnerability is in fact a
regression of CVE-2006-5051 ("Signal handler race condition in OpenSSH
before 4.4 allows remote attackers to cause a denial of service (crash),
and possibly execute arbitrary code"), which was reported in 2006 by
Mark Dowd.
This regression was introduced in October 2020 (OpenSSH 8.5p1) by commit
752250c ("revised log infrastructure for OpenSSH"), which accidentally
removed an "#ifdef DO_LOG_SAFE_IN_SIGHAND" from sigdie(), a function
that is directly called by sshd's SIGALRM handler. In other words:
- OpenSSH < 4.4p1 is vulnerable to this signal handler race condition,
if not backport-patched against CVE-2006-5051, or not patched against
CVE-2008-4109, which was an incorrect fix for CVE-2006-5051;
- 4.4p1 <= OpenSSH < 8.5p1 is not vulnerable to this signal handler race
condition (because the "#ifdef DO_LOG_SAFE_IN_SIGHAND" that was added
to sigdie() by the patch for CVE-2006-5051 transformed this unsafe
function into a safe _exit(1) call);
- 8.5p1 <= OpenSSH < 9.8p1 is vulnerable again to this signal handler
race condition (because the "#ifdef DO_LOG_SAFE_IN_SIGHAND" was
accidentally removed from sigdie()).
This vulnerability is exploitable remotely on glibc-based Linux systems,
where syslog() itself calls async-signal-unsafe functions (for example,
malloc() and free()): an unauthenticated remote code execution as root,
because it affects sshd's privileged code, which is not sandboxed and
runs with full privileges. We have not investigated any other libc or
operating system; but OpenBSD is notably not vulnerable, because its
SIGALRM handler calls syslog_r(), an async-signal-safer version of
syslog() that was invented by OpenBSD in 2001.
To exploit this vulnerability remotely (to the best of our knowledge,
CVE-2006-5051 has never been successfully exploited before), we drew
inspiration from a visionary paper, "Delivering Signals for Fun and
Profit", which was published in 2001 by Michal Zalewski:
https://lcamtuf.coredump.cx/signals.txt
Nevertheless, we immediately faced three major problems:
- From a theoretical point of view, we must find a useful code path
that, if interrupted at the right time by SIGALRM, leaves sshd in an
inconsistent state, and we must then exploit this inconsistent state
inside the SIGALRM handler.
- From a practical point of view, we must find a way to reach this
useful code path in sshd, and maximize our chances of interrupting it
at the right time.
- From a timing point of view, we must find a way to further increase
our chances of interrupting this useful code path at the right time,
remotely.
To focus on these three problems without having to immediately fight
against all the modern operating system protections (in particular, ASLR
and NX), we decided to exploit old OpenSSH versions first, on i386, and
then, based on this experience, recent versions:
- First, "SSH-2.0-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3", from
"debian-30r6-dvd-i386-binary-1_NONUS.iso": this is the first Debian
version that has privilege separation enabled by default and that is
patched against all the critical vulnerabilities of that era (in
particular, CVE-2003-0693 and CVE-2002-0640).
To remotely exploit this version, we interrupt a call to free() with
SIGALRM (inside sshd's public-key parsing code), leave the heap in an
inconsistent state, and exploit this inconsistent state during another
call to free(), inside the SIGALRM handler.
In our experiments, it takes ~10,000 tries on average to win this race
condition; i.e., with 10 connections (MaxStartups) accepted per 600
seconds (LoginGraceTime), it takes ~1 week on average to obtain a
remote root shell.
- Second, "SSH-2.0-OpenSSH_4.2p1 Debian-7ubuntu3", from
"ubuntu-6.06.1-server-i386.iso": this is the last Ubuntu version that
is still vulnerable to CVE-2006-5051 ("Signal handler race condition
in OpenSSH before 4.4").
To remotely exploit this version, we interrupt a call to pam_start()
with SIGALRM, leave one of PAM's structures in an inconsistent state,
and exploit this inconsistent state during a call to pam_end(), inside
the SIGALRM handler.
In our experiments, it takes ~10,000 tries on average to win this race
condition; i.e., with 10 connections (MaxStartups) accepted per 120
seconds (LoginGraceTime), it takes ~1-2 days on average to obtain a
remote root shell.
- Finally, "SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2", from
"debian-12.5.0-i386-DVD-1.iso": this is the current Debian stable
version, and it is vulnerable to the regression of CVE-2006-5051.
To remotely exploit this version, we interrupt a call to malloc() with
SIGALRM (inside sshd's public-key parsing code), leave the heap in an
inconsistent state, and exploit this inconsistent state during another
call to malloc(), inside the SIGALRM handler (more precisely, inside
syslog()).
In our experiments, it takes ~10,000 tries on average to win this race
condition, so ~3-4 hours with 100 connections (MaxStartups) accepted
per 120 seconds (LoginGraceTime). Ultimately, it takes ~6-8 hours on
average to obtain a remote root shell, because we can only guess the
glibc's address correctly half of the time (because of ASLR).
This research is still a work in progress:
- we have targeted virtual machines only, not bare-metal servers, on a
mostly stable network link (~10ms packet jitter);
- we are convinced that various aspects of our exploits can be greatly
improved;
- we have started to work on an amd64 exploit, which is much harder
because of the stronger ASLR.
A few days after we started our work on amd64, we noticed the following
bug report (in OpenSSH's public Bugzilla), about a deadlock in sshd's
SIGALRM handler:
https://bugzilla.mindrot.org/show_bug.cgi?id=3690
We therefore decided to contact OpenSSH's developers immediately (to let
them know that this deadlock is caused by an exploitable vulnerability),
we put our amd64 work on hold, and we started to write this advisory.
========================================================================
SSH-2.0-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3 (Debian 3.0r6, from 2005)
========================================================================
------------------------------------------------------------------------
Theory
------------------------------------------------------------------------
But that's not like me, I'm breaking free
-- The Interrupters, "Haven't Seen the Last of Me"
The SIGALRM handler of this OpenSSH version calls packet_close(), which
calls buffer_free(), which calls xfree() and hence free(), which is not
async-signal-safe:
------------------------------------------------------------------------
302 grace_alarm_handler(int sig)
303 {
...
307 packet_close();
------------------------------------------------------------------------
329 packet_close(void)
330 {
...
341 buffer_free(&input);
342 buffer_free(&output);
343 buffer_free(&outgoing_packet);
344 buffer_free(&incoming_packet);
------------------------------------------------------------------------
35 buffer_free(Buffer *buffer)
36 {
37 memset(buffer->buf, 0, buffer->alloc);
38 xfree(buffer->buf);
39 }
------------------------------------------------------------------------
51 xfree(void *ptr)
52 {
53 if (ptr == NULL)
54 fatal("xfree: NULL pointer given as argument");
55 free(ptr);
56 }
------------------------------------------------------------------------
Consequently, we started to read the malloc code of this Debian's glibc
(2.2.5), to see if a first call to free() can be interrupted by SIGALRM
and exploited during a second call to free() inside the SIGALRM handler
(at lines 341-344, above). Because this glibc's malloc is not hardened
against the unlink() technique pioneered by Solar Designer in 2000, we
quickly spotted an interesting code path in chunk_free() (which is
called internally by free()):
------------------------------------------------------------------------
1028 struct malloc_chunk
1029 {
1030 INTERNAL_SIZE_T prev_size; /* Size of previous chunk (if free). */
1031 INTERNAL_SIZE_T size; /* Size in bytes, including overhead. */
1032 struct malloc_chunk* fd; /* double links -- used only if free. */
1033 struct malloc_chunk* bk;
1034 };
------------------------------------------------------------------------
2516 #define unlink(P, BK, FD) \
2517 { \
2518 BK = P->bk; \
2519 FD = P->fd; \
2520 FD->bk = BK; \
2521 BK->fd = FD; \
2522 } \
------------------------------------------------------------------------
3160 chunk_free(arena *ar_ptr, mchunkptr p)
....
3164 {
3165 INTERNAL_SIZE_T hd = p->size; /* its head field */
....
3177 sz = hd & ~PREV_INUSE;
3178 next = chunk_at_offset(p, sz);
3179 nextsz = chunksize(next);
....
3230 if (!(inuse_bit_at_offset(next, nextsz))) /* consolidate forward */
3231 {
....
3241 unlink(next, bck, fwd);
....
3244 }
3245 else
3246 set_head(next, nextsz); /* clear inuse bit */
....
3251 frontlink(ar_ptr, p, sz, idx, bck, fwd);
------------------------------------------------------------------------
To exploit this code path, we arrange for sshd's heap to have the
following layout (chunk_X, chunk_Y, and chunk_Z are malloc()ated chunks
of memory, and p, s, f, b are their prev_size, size, fd, and bk fields):
-----|---+---------------|---+---------------|---+---------------|-----
... |p|s|f|b| chunk_X |p|s|f|b| chunk_Y |p|s|f|b| chunk_Z | ...
-----|---+---------------|---+---------------|---+---------------|-----
|<------------->|
user data
- First, if a call to free(chunk_Y) is interrupted by SIGALRM *after*
line 3246 but *before* line 3251, then chunk_Y is already marked as
free (because chunk_Z's PREV_INUSE bit is cleared at line 3246) but it
is not yet linked into its doubly-linked list (at line 3251): in other
words, chunk_Y's fd and bk pointers still contain user data (attacker-
controlled data).
- Second, if (inside the SIGALRM handler) packet_close() calls
free(chunk_X), then the code block at lines 3230-3244 is entered
(because chunk_Y is marked as free) and chunk_Y is unlink()ed (at line
3241): a so-called aa4bmo primitive (almost arbitrary 4 bytes mirrored
overwrite), because chunk_Y's fd and bk pointers are still attacker-
controlled. For more information on the unlink() technique and the
aa4bmo primitive:
https://www.openwall.com/articles/JPEG-COM-Marker-Vulnerability#exploit
http://phrack.org/issues/61/6.html#article
- Last, with this aa4bmo primitive we overwrite the glibc's __free_hook
function pointer (this old Debian version does not have ASLR, nor NX)
with the address of our shellcode in the heap, thus achieving remote
code execution during the next call to free() in packet_close().
------------------------------------------------------------------------
Practice
------------------------------------------------------------------------
Now they're taking over and they got complete control
-- The Interrupters, "Liberty"
To mount this attack against sshd, we interrupt a call to free() inside
sshd's parsing code of a DSA public key (i.e., line 144 below is our
free(chunk_Y)) and exploit it during one of the free() calls in
packet_close() (i.e., one of the lines 341-344 above is our
free(chunk_X)):
------------------------------------------------------------------------
136 buffer_get_bignum2(Buffer *buffer, BIGNUM *value)
137 {
138 u_int len;
139 u_char *bin = buffer_get_string(buffer, &len);
...
143 BN_bin2bn(bin, len, value);
144 xfree(bin);
145 }
------------------------------------------------------------------------
Initially, however, we were never able to win this race condition (i.e.,
interrupt the free() call at line 144 at the right time). Eventually, we
realized that we could greatly improve our chances of winning this race:
the DSA public-key parsing code allows us to call free() four times (at
lines 704-707 below), and furthermore sshd allows us to attempt six user
authentications (AUTH_FAIL_MAX); if any one of these 24 free() calls is
interrupted at the right time, then we later achieve remote code
execution inside the SIGALRM handler.
------------------------------------------------------------------------
678 key_from_blob(u_char *blob, int blen)
679 {
...
693 switch (type) {
...
702 case KEY_DSA:
703 key = key_new(type);
704 buffer_get_bignum2(&b, key->dsa->p);
705 buffer_get_bignum2(&b, key->dsa->q);
706 buffer_get_bignum2(&b, key->dsa->g);
707 buffer_get_bignum2(&b, key->dsa->pub_key);
------------------------------------------------------------------------
With this improvement, we finally won the race condition after ~1 month:
we were happy (and did a root-shell dance), but we also felt that there
was still room for improvement.
------------------------------------------------------------------------
Timing
------------------------------------------------------------------------
Don't worry, just wait and see
-- The Interrupters, "Haven't Seen the Last of Me"
We therefore implemented the following threefold timing strategy:
- We do not wait until the last moment to send our (rather large) DSA
public-key packet to sshd: instead, we send the entire packet minus
one byte (the last byte) long before the LoginGraceTime, and send the
very last byte at the very last moment, to minimize the effects of
network delays. (And we disable the Nagle algorithm.)
- We keep track of the median round-trip time (by regularly sending
packets that produce a response from sshd), and keep track of the
difference between the moment we are expecting our connection to be
closed by sshd (essentially the moment we receive the first byte of
sshd's banner, plus LoginGraceTime) and the moment our connection is
really closed by sshd, and accordingly adjust our timing (i.e., the
moment when we send the last byte of our DSA packet).
These time differences allow us to track clock skews and network
delays, which show predictable patterns over time: we experimented
with linear and spline regressions, but in the end, nothing worked
better than simply re-using the most recent measurement. Possibly,
deep learning might yield even better results; this is left as an
exercise for the interested reader.
- More importantly, we further increase our chances of winning this race
condition by slowly adjusting our timing through involuntary feedback
from sshd:
- if we receive a response (SSH2_MSG_USERAUTH_FAILURE) to our DSA
public-key packet, then we sent it too early (sshd had the time to
receive our packet in the unprivileged child, parse it, send it to
the privileged child, parse it there, and send a response all the
way back to us);
- if we cannot even send the last byte of our DSA packet, then we
waited too long (sshd already received the SIGALRM and closed our
connection);
- if we can send the last byte of our DSA packet, and receive no
response before sshd closes our connection, then our timing was
reasonably accurate.
This feedback allows us to target what we call the "large" race
window: hitting it does not guarantee that we win the race condition,
but inside this large window are the 24 "small" race windows (inside
the 24 free() calls) that, if hit, guarantee that we do win the race
condition.
With these improvements, it takes ~10,000 tries on average to win this
race condition; i.e., with 10 connections (MaxStartups) accepted per 600
seconds (LoginGraceTime), it takes ~1 week on average to obtain a remote
root shell.
========================================================================
SSH-2.0-OpenSSH_4.2p1 Debian-7ubuntu3 (Ubuntu 6.06.1, from 2006)
========================================================================
------------------------------------------------------------------------
Theory, take one
------------------------------------------------------------------------
I sleep when the sun starts to rise
-- The Interrupters, "Alien"
The SIGALRM handler of this OpenSSH version does not call packet_close()
anymore; moreover, this Ubuntu's glibc (2.3.6) always takes a mandatory
lock when entering the functions of the malloc family (even if single-
threaded like sshd), which prevents us from interrupting a call to one
of the malloc functions and later exploiting it during another call to
these functions (they would always deadlock). We must find another
solution.
CVE-2006-5051 mentions a double-free in GSSAPI, but GSSAPI (or Kerberos)
is not enabled by default, so this does not sound very appealing. On the
other hand, PAM is enabled by default, and pam_end() is called by sshd's
SIGALRM handler (and is, of course, not async-signal-safe). We therefore
searched for a PAM function that, if interrupted by SIGALRM at the right
time, would leave PAM's internal structures in an inconsistent state,
exploitable during pam_end() in the SIGALRM handler. We found
pam_set_data():
------------------------------------------------------------------------
33 int pam_set_data(
34 pam_handle_t *pamh,
..
37 void (*cleanup)(pam_handle_t *pamh, void *data, int error_status))
38 {
39 struct pam_data *data_entry;
..
57 } else if ((data_entry = malloc(sizeof(*data_entry)))) {
..
65 data_entry->next = pamh->data;
66 pamh->data = data_entry;
..
74 data_entry->cleanup = cleanup;
------------------------------------------------------------------------
If this function is interrupted by SIGALRM *after* line 66 but *before*
line 74, then data_entry is already linked into PAM's structures (pamh),
but its cleanup field (a function pointer) is not yet initialized (since
the malloc() at line 57 does not initialize its memory). If we are able
to control cleanup (through leftovers from previous heap allocations),
then we can execute arbitrary code when pam_end() (inside the SIGALRM
handler) calls _pam_free_data() (at line 118):
------------------------------------------------------------------------
104 void _pam_free_data(pam_handle_t *pamh, int status)
105 {
106 struct pam_data *last;
107 struct pam_data *data;
...
112 data = pamh->data;
113
114 while (data) {
115 last = data;
116 data = data->next;
117 if (last->cleanup) {
118 last->cleanup(pamh, last->data, status);
------------------------------------------------------------------------
This would have been an extremely simple exploit; unfortunately, we
completely overlooked that pam_set_data() can only be called from PAM
modules: if we interrupt it with SIGALRM, then pamh->caller_is is still
_PAM_CALLED_FROM_MODULE, in which case pam_end() returns immediately,
without ever calling _pam_free_data(). Back to the drawing board.
------------------------------------------------------------------------
Theory, take two
------------------------------------------------------------------------
Not giving up, it's not what we do
-- The Interrupters, "Title Holder"
We noticed that, at line 601 below, sshd passes a pointer to its global
sshpam_handle pointer directly to pam_start() (which is called once per
connection):
------------------------------------------------------------------------
202 static pam_handle_t *sshpam_handle = NULL;
------------------------------------------------------------------------
584 sshpam_init(Authctxt *authctxt)
585 {
...
600 sshpam_err =
601 pam_start(SSHD_PAM_SERVICE, user, &store_conv, &sshpam_handle);
------------------------------------------------------------------------
We therefore decided to look into pam_start() itself: if interrupted by
SIGALRM, it might leave the structure pointed to by sshpam_handle in an
inconsistent state, which could then be exploited inside the SIGALRM
handler, when "pam_end(sshpam_handle, sshpam_err)" is called.
------------------------------------------------------------------------
18 int pam_start (
..
22 pam_handle_t **pamh)
23 {
..
32 if ((*pamh = calloc(1, sizeof(**pamh))) == NULL) {
...
110 if ( _pam_init_handlers(*pamh) != PAM_SUCCESS ) {
------------------------------------------------------------------------
319 int _pam_init_handlers(pam_handle_t *pamh)
320 {
...
398 retval = _pam_parse_conf_file(pamh, f, pamh->service_name, PAM_T_ANY
------------------------------------------------------------------------
66 static int _pam_parse_conf_file(pam_handle_t *pamh, FILE *f
..
73 {
...
252 res = _pam_add_handler(pamh, must_fail, other
------------------------------------------------------------------------
581 int _pam_add_handler(pam_handle_t *pamh
...
585 {
...
755 the_handlers = (other) ? &pamh->handlers.other : &pamh->handlers.conf;
...
767 handler_p = &the_handlers->authenticate;
...
874 if ((*handler_p = malloc(sizeof(struct handler))) == NULL) {
...
886 (*handler_p)->next = NULL;
------------------------------------------------------------------------
At line 32, pam_start() immediately sets sshd's sshpam_handle to a
calloc()ated chunk of memory; this is safe, because calloc() initializes
this memory to zero. On the other hand, if _pam_add_handler() (which is
called multiple times by pam_start()) is interrupted by SIGALRM *after*
line 874 but *before* line 886, then a malloc()ated structure is linked
into pamh, but its next field is not yet initialized. If we are able to
control next (through leftovers from previous heap allocations), then we
can pass an arbitrary pointer to free() during the call to pam_end()
(inside the SIGALRM handler), at line 1020 (and line 1017) below:
------------------------------------------------------------------------
11 int pam_end(pam_handle_t *pamh, int pam_status)
12 {
..
31 if ((ret = _pam_free_handlers(pamh)) != PAM_SUCCESS) {
------------------------------------------------------------------------
925 int _pam_free_handlers(pam_handle_t *pamh)
926 {
...
954 _pam_free_handlers_aux(&(pamh->handlers.conf.authenticate));
------------------------------------------------------------------------
1009 void _pam_free_handlers_aux(struct handler **hp)
1010 {
1011 struct handler *h = *hp;
1012 struct handler *last;
....
1015 while (h) {
1016 last = h;
1017 _pam_drop(h->argv); /* This is all alocated in a single chunk */
1018 h = h->next;
1019 memset(last, 0, sizeof(*last));
1020 free(last);
1021 }
------------------------------------------------------------------------
Because the malloc of this Ubuntu's glibc is already hardened against
the old unlink() technique, we decided to transform our arbitrary free()
into the Malloc Maleficarum's House of Mind (fastbin version): we free()
our own NON_MAIN_ARENA chunk, point our fake arena to sshd's .got.plt
(this Ubuntu's sshd has ASLR but not PIE), and overwrite _exit()'s entry
with the address of our shellcode in the heap (this Ubuntu's heap is
still executable by default). For more information on the Malloc
Maleficarum:
https://seclists.org/bugtraq/2005/Oct/118
------------------------------------------------------------------------
Practice
------------------------------------------------------------------------
I learned everything the hard way
-- The Interrupters, "The Hard Way"
To mount this attack against sshd, we initially faced three problems:
- The House of Mind requires us to store the pointer to our fake arena
at address 0x08100000 in the heap; but are we able to store attacker-
controlled data at such a high address? Because sshd calls pam_start()
at the very beginning of the user authentication, we do not control
anything except the user name itself; luckily, a user name of length
~128KB (shorter than DEFAULT_MMAP_THRESHOLD) allows us to store our
own data at address 0x08100000.
- The size field of our fake NON_MAIN_ARENA chunk must not be too large
(to pass free()'s security checks); i.e., it must contain null bytes.
But our long user name is a null-terminated string that cannot contain
null bytes; luckily we remembered that _pam_free_handlers_aux() zeroes
the structures that it free()s (line 1019 above): we therefore "patch"
the size field of our fake chunk with such a memset(0), and only then
free() it.
- We must survive several calls to free() (at lines 1017 and 1020 above)
before the free() of our fake NON_MAIN_ARENA chunk. We transform these
free()s into no-ops by pointing them to fake IS_MMAPPED chunks: free()
calls munmap_chunk(), which calls munmap(), which fails because these
fake IS_MMAPPED chunks are misaligned; effectively a no-op, because
assert()ion failures are not enforced in this Ubuntu's glibc.
Finally, our long user name also allows us to control the potentially
uninitialized next field of 20 different structures (through leftovers
from temporary copies of our long user name), because pam_start() calls
_pam_add_handler() multiple times; i.e., our large race window contains
20 small race windows.
------------------------------------------------------------------------
Timing
------------------------------------------------------------------------
Same tricks they used before
-- The Interrupters, "Divide Us"
For this attack against Ubuntu 6.06.1, we simply re-used the timing
strategy that we used against Debian 3.0r6: it takes ~10,000 tries on
average to win the race condition, and with 10 connections (MaxStartups)
accepted per 120 seconds (LoginGraceTime), it takes ~1-2 days on average
to obtain a remote root shell.
Note: because this Ubuntu's glibc always takes a mandatory lock when
entering the functions of the malloc family, an unlucky attacker might
deadlock all 10 MaxStartups connections before obtaining a root shell;
we have not tried to work around this problem because our ultimate goal
was to exploit a modern OpenSSH version anyway.
========================================================================
SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2 (Debian 12.5.0, from 2024)
========================================================================
------------------------------------------------------------------------
Theory
------------------------------------------------------------------------
Now you're ready, take the demons head on
-- The Interrupters, "Be Gone"
The SIGALRM handler of this OpenSSH version does not call packet_close()
nor pam_end(); in fact it calls only one interesting function, syslog():
------------------------------------------------------------------------
358 grace_alarm_handler(int sig)
359 {
...
370 sigdie("Timeout before authentication for %s port %d",
371 ssh_remote_ipaddr(the_active_state),
372 ssh_remote_port(the_active_state));
------------------------------------------------------------------------
96 #define sigdie(...) sshsigdie(__FILE__, __func__, __LINE__, 0, SYSLOG_LEVEL_ERROR, NULL, __VA_ARGS__)
------------------------------------------------------------------------
451 sshsigdie(const char *file, const char *func, int line, int showfunc,
452 LogLevel level, const char *suffix, const char *fmt, ...)
453 {
...
457 sshlogv(file, func, line, showfunc, SYSLOG_LEVEL_FATAL,
458 suffix, fmt, args);
------------------------------------------------------------------------
464 sshlogv(const char *file, const char *func, int line, int showfunc,
465 LogLevel level, const char *suffix, const char *fmt, va_list args)
466 {
...
489 do_log(level, forced, suffix, fmt2, args);
------------------------------------------------------------------------
337 do_log(LogLevel level, int force, const char *suffix, const char *fmt,
338 va_list args)
339 {
...
419 syslog(pri, "%.500s", fmtbuf);
------------------------------------------------------------------------
Our two key questions, then, are: Does the syslog() of this Debian's
glibc (2.36) call async-signal-unsafe functions such as malloc() and
free()? And if yes, does this glibc still take a mandatory lock when
entering the functions of the malloc family?
- Luckily for us attackers, the answer to our first question is yes; if,
and only if, the syslog() inside the SIGALRM handler is the very first
call to syslog(), then __localtime64_r() (which is called by syslog())
calls malloc(304) to allocate a FILE structure (at line 166) and calls
malloc(4096) to allocate an internal read buffer (at line 186):
------------------------------------------------------------------------
28 __localtime64_r (const __time64_t *t, struct tm *tp)
29 {
30 return __tz_convert (*t, 1, tp);
------------------------------------------------------------------------
567 __tz_convert (__time64_t timer, int use_localtime, struct tm *tp)
568 {
...
577 tzset_internal (tp == &_tmbuf && use_localtime);
------------------------------------------------------------------------
367 tzset_internal (int always)
368 {
...
405 __tzfile_read (tz, 0, NULL);
------------------------------------------------------------------------
105 __tzfile_read (const char *file, size_t extra, char **extrap)
106 {
...
109 FILE *f;
...
166 f = fopen (file, "rce");
...
186 if (__builtin_expect (__fread_unlocked ((void *) &tzhead, sizeof (tzhead),
187 1, f) != 1, 0)
------------------------------------------------------------------------
Note: because we do not control anything about these malloc()ations
(not their order, not their sizes, not their contents), we took the
"rce" at line 166 as a much-needed good omen.
- And luckily for us, the answer to our second question is no; since
October 2017, the glibc's malloc functions do not take any lock
anymore, when single-threaded (like sshd):
https://sourceware.org/git?p=glibc.git;a=commit;h=a15d53e2de4c7d83bda251469d92a3c7b49a90db
https://sourceware.org/git?p=glibc.git;a=commit;h=3f6bb8a32e5f5efd78ac08c41e623651cc242a89
https://sourceware.org/git?p=glibc.git;a=commit;h=905a7725e9157ea522d8ab97b4c8b96aeb23df54
Moreover, this Debian version suffers from the ASLR weakness described
in the following great blog posts (by Justin Miller and Mathias Krause,
respectively):
https://zolutal.github.io/aslrnt/
https://grsecurity.net/toolchain_necromancy_past_mistakes_haunting_aslr
Concretely, in the case of sshd on i386, every memory mapping is
randomized normally (sshd's PIE, the heap, most libraries, the stack),
but the glibc itself is always mapped either at address 0xb7200000 or at
address 0xb7400000; in other words, we can correctly guess the glibc's
address half of the time (a small price to pay for defeating ASLR). In
our exploit we assume that the glibc is mapped at address 0xb7400000,
because it is slightly more common than 0xb7200000.
Our next question is: which code paths inside the glibc's malloc
functions, if interrupted by SIGALRM at the right time, leave the heap
in an inconsistent state, exploitable during one of the malloc() calls
inside the SIGALRM handler?
We found several interesting (and surprising!) code paths, but the one
we chose involves only relative sizes, not absolute addresses (unlike
various code paths inside unlink_chunk(), for example); this difference
might prove crucial for a future amd64 exploit. This code path, inside
malloc(), splits a large free chunk (victim) into two smaller chunks;
the first chunk is returned to malloc()'s caller (at line 4345) and the
second chunk (remainder) is linked into an unsorted list of free chunks
(at lines 4324-4327):
------------------------------------------------------------------------
1449 #define set_head(p, s) ((p)->mchunk_size = (s))
------------------------------------------------------------------------
3765 _int_malloc (mstate av, size_t bytes)
3766 {
....
3798 nb = checked_request2size (bytes);
....
4295 size = chunksize (victim);
....
4300 remainder_size = size - nb;
....
4316 remainder = chunk_at_offset (victim, nb);
....
4320 bck = unsorted_chunks (av);
4321 fwd = bck->fd;
....
4324 remainder->bk = bck;
4325 remainder->fd = fwd;
4326 bck->fd = remainder;
4327 fwd->bk = remainder;
....
4337 set_head (victim, nb | PREV_INUSE |
4338 (av != &main_arena ? NON_MAIN_ARENA : 0));
4339 set_head (remainder, remainder_size | PREV_INUSE);
....
4343 void *p = chunk2mem (victim);
....
4345 return p;
------------------------------------------------------------------------
- If this code path is interrupted by SIGALRM *after* line 4327 but
*before* line 4339, then the remainder chunk of this split is already
linked into the unsorted list of free chunks (lines 4324-4327), but
its size field (mchunk_size) is not yet initialized (line 4339).
- If we are able to control its size field (through leftovers from
previous heap allocations), then we can make this remainder chunk
larger and overlap with other heap chunks, and therefore corrupt heap
memory when this enlarged, overlapping remainder chunk is eventually
malloc()ated and written to (inside the SIGALRM handler).
Our last question, then, is: given that we do not control anything about
the malloc() calls inside the SIGALRM handler, what can we overwrite in
the heap to achieve arbitrary code execution before sshd calls _exit()
(in sshsigdie())?
Because __tzfile_read() (inside the SIGALRM handler) malloc()ates a FILE
structure in the heap (at line 166 above), and because FILE structures
have a long history of abuse for arbitrary code execution, we decided to
aim our heap corruption at this FILE structure. This is, however, easier
said than done: our heap corruption is very limited, and FILE structures
have been significantly hardened over the years (by IO_validate_vtable()
and PTR_DEMANGLE(), for example).
Eventually, we devised the following technique (which seems to be
specific to the i386 glibc -- the amd64 glibc does not seem to use
_vtable_offset at all):
- with our limited heap corruption, we overwrite the _vtable_offset
field (a single signed char) of __tzfile_read()'s FILE structure;
- the glibc's libio functions will therefore look for this FILE
structure's vtable pointer (a pointer to an array of function
pointers) at a non-zero offset (our overwritten _vtable_offset),
instead of the default zero offset;
- we (attackers) can easily control this fake vtable pointer (through
leftovers from previous heap allocations), because the FILE structure
around this offset is not explicitly initialized by fopen();
- to pass the glibc's security checks, our fake vtable pointer must
point somewhere into the __libc_IO_vtables section: we decided to
point it to the vtable for wide-character streams, _IO_wfile_jumps
(i.e., to 0xb761b740, since we assume that the glibc is mapped at
address 0xb7400000);
- as a result, __fread_unlocked() (at line 186 above) calls
_IO_wfile_underflow() (instead of _IO_file_underflow()), which calls a
function pointer (__fct) that basically comes from a structure whose
pointer (_codecvt) is yet another field of the FILE structure;
- we (attackers) can easily control this _codecvt pointer (through
leftovers from previous heap allocations, because this field of the
FILE structure is not explicitly initialized by fopen()), which also
allows us to control the __fct function pointer.
In summary, by overwriting a single byte (_vtable_offset) of the FILE
structure malloc()ated by fopen(), we can call our own __fct function
pointer and execute arbitrary code during __fread_unlocked().
------------------------------------------------------------------------
Practice
------------------------------------------------------------------------
I wanted it perfect, no wrinkles in it
-- The Interrupters, "In the Mirror"
To mount this attack against sshd's privileged child, let us first
imagine the following heap layout (the "XXX"s are "barrier" chunks that
allow us to make holes in the heap; for example, small memory-leaked
chunks):
---|----------------------------------------------|---|------------|---
XXX| large hole |XXX| small hole |XXX
---|----------------------------------------------|---|------------|---
| ~8KB | | 320B |
- shortly before sshd receives the SIGALRM, we malloc()ate a ~4KB chunk
that splits the large ~8KB hole into two smaller chunks:
---|-----------------------|----------------------|---|------------|---
XXX| large allocated chunk | free remainder chunk |XXX| small hole |XXX
---|-----------------------|----------------------|---|------------|---
| ~4KB | ~4KB | | 320B |
- but if this malloc() is interrupted by SIGALRM *after* line 4327 but
*before* line 4339, then the remainder chunk of this split is already
linked into the unsorted list of free chunks, but its size field is
under our control (through leftovers from previous heap allocations),
and this artificially enlarged remainder chunk overlaps with the
following small hole:
---|-----------------------|----------------------|---|------------|---
XXX| large allocated chunk | real remainder chunk |XXX| small hole |XXX
---|-----------------------|----------------------|---|------------|---
| ~4KB |<------------------------------------->|
artificially enlarged remainder chunk
- when the SIGALRM handler calls syslog() and hence __tzfile_read(),
fopen() malloc()ates the small hole for its FILE structure, and
__fread_unlocked() malloc()ates a 4KB read buffer, thereby splitting
the enlarged remainder chunk in two (the 4KB read buffer and a small
remainder chunk):
---|-----------------------|----------------------|---|------------|---
XXX| large allocated chunk | |XXX| FILE |XXX
---|-----------------------|----------------------|---|--|---------|---
| ~4KB |<--------------------------->|<------->|
4KB read buffer remainder
- we therefore overwrite parts of the FILE structure with the internal
header of this small remainder chunk: more precisely, we overwrite the
FILE's _vtable_offset with the third byte of this header's bk field,
which is a pointer to the unsorted list of free chunks, 0xb761d7f8
(i.e., we overwrite _vtable_offset with 0x61);
- then, as explained in the "Theory" subsection, __fread_unlocked()
calls _IO_wfile_underflow() (instead of _IO_file_underflow()), which
calls our own __fct function pointer (through our own _codecvt
pointer) and executes our arbitrary code.
Note: we have not yet explained how to reliably go from a controlled
_codecvt pointer to a controlled __fct function pointer; we will do
so, but we must first solve a more pressing problem.
Indeed, we learned from our work on older OpenSSH versions that we will
never win this signal handler race condition if our large race window
contains only one small race window. Consequently, we implemented the
following strategy, based on the following heap layout:
---|------------|---|------------|---|------------|---|------------|---
XXX|large hole 1|XXX|small hole 1|XXX|large hole 2|XXX|small hole 2|...
---|------------|---|------------|---|------------|---|------------|---
| ~8KB | | 320B | | ~8KB | | 320B |
The last packet that we send to sshd (shortly before the delivery of
SIGALRM) forces sshd to perform the following sequence of malloc()
calls: malloc(~4KB), malloc(304), malloc(~4KB), malloc(304), etc.
1/ Our first malloc(~4KB) splits the large hole 1 in two:
- if this first split is interrupted by SIGALRM at the right time, then
the fopen() inside the SIGALRM handler malloc()ates the small hole 1
for its FILE structure, and we achieve arbitrary code execution as
explained above;
- if not, then we malloc()ate the small hole 1 ourselves with our first
malloc(304), and:
2/ Our second malloc(~4KB) splits the large hole 2 in two:
- if this second split is interrupted by SIGALRM at the right time, then
the fopen() inside the SIGALRM handler malloc()ates the small hole 2
for its FILE structure, and we achieve arbitrary code execution as
explained above;
- if not, then we malloc()ate the small hole 2 ourselves with our second
malloc(304), etc.
We were able to make 27 pairs of such large and small holes in sshd's
heap (28 would exceed PACKET_MAX_SIZE, 256KB): our large race window now
contains 27 small race windows! Achieving this complex heap layout was
extremely painful and time-consuming, but the two highlights are:
- We abuse sshd's public-key parsing code to perform arbitrary sequences
of malloc() and free() calls (at lines 1805 and 573):
------------------------------------------------------------------------
1754 cert_parse(struct sshbuf *b, struct sshkey *key, struct sshbuf *certbuf)
1755 {
....
1797 while (sshbuf_len(principals) > 0) {
....
1805 if ((ret = sshbuf_get_cstring(principals, &principal,
....
1820 key->cert->principals[key->cert->nprincipals++] = principal;
1821 }
------------------------------------------------------------------------
562 cert_free(struct sshkey_cert *cert)
563 {
...
572 for (i = 0; i < cert->nprincipals; i++)
573 free(cert->principals[i]);
------------------------------------------------------------------------
- We were unable to find a memory leak for our small "barrier" chunks;
instead, we use tcache chunks (which are never really freed, because
their inuse bit is never cleared) as makeshift "barrier" chunks.
To reliably achieve this heap layout, we send five different public-key
packets to sshd (packets a/ to d/ can be sent long before SIGALRM; most
of packet e/ can also be sent long before SIGALRM, but its very last
byte must be sent at the very last moment):
a/ We malloc()ate and free() a variety of tcache chunks, to ensure that
the heap allocations that we do not control end up in these tcache
chunks and do not interfere with our careful heap layout.
b/ We malloc()ate and free() chunks of various sizes, to make our 27
pairs of large and small holes (and the corresponding "barrier" chunks).
c/ We malloc()ate and free() ~4KB chunks and 320B chunks, to:
- write the fake header (the large size field) of our potentially
enlarged remainder chunk, into the middle of our large holes;
- write the fake footer of our potentially enlarged remainder chunk, to
the end of our small holes (to pass the glibc's security checks);
- write our fake vtable and _codecvt pointers, into our small holes
(which are potential FILE structures).
d/ We malloc()ate and free() one very large string (nearly 256KB), to
ensure that our large and small holes are removed from the unsorted list
of free chunks and placed into their respective malloc bins.
e/ We force sshd to perform our final sequence of malloc() calls
(malloc(~4KB), malloc(304), malloc(~4KB), malloc(304), etc), to open our
27 small race windows.
Attentive readers may have noticed that we have still not addressed
(literally and figuratively) the problem of _codecvt. In fact, _codecvt
is a pointer to a structure (_IO_codecvt) that contains a pointer to a
structure (__gconv_step) that contains the __fct function pointer that
allows us to execute arbitrary code. To reliably control __fct through
_codecvt, we simply point _codecvt to one of the glibc's malloc bins,
which conveniently contains a pointer to one of our free chunks in the
heap, which contains our own __fct function pointer to arbitrary glibc
code (all of these glibc addresses are known to us, because we assume
that the glibc is mapped at address 0xb7400000).
------------------------------------------------------------------------
Timing
------------------------------------------------------------------------
We're running out of time
-- The Interrupters, "As We Live"
As we implemented this third exploit, it became clear that we could not
simply re-use the timing strategy that we had used against the two older
OpenSSH versions: we were never winning this new race condition.
Eventually, we understood why:
- It takes a long time (~10ms) for sshd to parse our fifth and last
public key (packet e/ above); in other words, our large race window is
too large (our 27 small race windows are like needles in a haystack).
- The user_specific_delay() that was introduced recently (OpenSSH 7.8p1)
delays sshd's response to our last public-key packet by up to ~9ms and
therefore destroys our feedback-based timing strategy.
As a result, we developed a completely different timing strategy:
- from time to time, we send our last public-key packet with a little
mistake that produces an error response (lines 138-142 below), right
before the call to sshkey_from_blob() that parses our public key;
- from time to time, we send our last public-key packet with another
little mistake that produces an error response (lines 151-155 below),
right after the call to sshkey_from_blob() that parses our public key;
- the difference between these two response times is the time that it
takes for sshd to parse our last public key, and this allows us to
precisely time the transmission of our last packets (to ensure that
sshd has the time to parse our public key in the unprivileged child,
send it to the privileged child, and start to parse it there, before
the delivery of SIGALRM).
------------------------------------------------------------------------
88 userauth_pubkey(struct ssh *ssh, const char *method)
89 {
...
138 if (pktype == KEY_UNSPEC) {
139 /* this is perfectly legal */
140 verbose_f("unsupported public key algorithm: %s", pkalg);
141 goto done;
142 }
143 if ((r = sshkey_from_blob(pkblob, blen, &key)) != 0) {
144 error_fr(r, "parse key");
145 goto done;
146 }
...
151 if (key->type != pktype) {
152 error_f("type mismatch for decoded key "
153 "(received %d, expected %d)", key->type, pktype);
154 goto done;
155 }
------------------------------------------------------------------------
With this change in strategy, it takes ~10,000 tries on average to win
the race condition; i.e., with 100 connections (MaxStartups) accepted
per 120 seconds (LoginGraceTime), it takes ~3-4 hours on average to win
the race condition, and ~6-8 hours to obtain a remote root shell
(because of ASLR).
========================================================================
Towards an amd64 exploit
========================================================================
What's your plan for tomorrow?
-- The Interrupters, "Take Back the Power"
We decided to target Rocky Linux 9 (a Red Hat Enterprise Linux 9
derivative), from "Rocky-9.4-x86_64-minimal.iso", for two reasons:
- its OpenSSH version (8.7p1) is vulnerable to this signal handler race
condition and its glibc is always mapped at a multiple of 2MB (because
of the ASLR weakness discussed in the previous "Theory" subsection),
which makes partial pointer overwrites much more powerful;
- the syslog() function (which is async-signal-unsafe but is called by
sshd's SIGALRM handler) of this glibc version (2.34) internally calls
__open_memstream(), which malloc()ates a FILE structure in the heap,
and also calls calloc(), realloc(), and free() (which gives us some
much-needed freedom).
With a heap corruption as a primitive, two FILE structures malloc()ated
in the heap, and 21 fixed bits in the glibc's addresses, we believe that
this signal handler race condition is exploitable on amd64 (probably not
in ~6-8 hours, but hopefully in less than a week). Only time will tell.
Side note: we discovered that Ubuntu 24.04 does not re-randomize the
ASLR of its sshd children (it is randomized only once, at boot time); we
tracked this down to the patch below, which turns off sshd's rexec_flag.
This is generally a bad idea, but in the particular case of this signal
handler race condition, it prevents sshd from being exploitable: the
syslog() inside the SIGALRM handler does not call any of the malloc
functions, because it is never the very first call to syslog().
https://git.launchpad.net/ubuntu/+source/openssh/tree/debian/patches/systemd-socket-activation.patch
========================================================================
Patches and mitigation
========================================================================
The storm has come and gone
-- The Interrupters, "Good Things"
On June 6, 2024, this signal handler race condition was fixed by commit
81c1099 ("Add a facility to sshd(8) to penalise particular problematic
client behaviours"), which moved the async-signal-unsafe code from
sshd's SIGALRM handler to sshd's listener process, where it can be
handled synchronously:
https://github.com/openssh/openssh-portable/commit/81c1099d22b81ebfd20a334ce986c4f753b0db29
Because this fix is part of a large commit (81c1099), on top of an even
larger defense-in-depth commit (03e3de4, "Start the process of splitting
sshd into separate binaries"), it might prove difficult to backport. In
that case, the signal handler race condition itself can be fixed by
removing or commenting out the async-signal-unsafe code from the
sshsigdie() function; for example:
------------------------------------------------------------------------
sshsigdie(const char *file, const char *func, int line, int showfunc,
LogLevel level, const char *suffix, const char *fmt, ...)
{
#if 0
va_list args;
va_start(args, fmt);
sshlogv(file, func, line, showfunc, SYSLOG_LEVEL_FATAL,
suffix, fmt, args);
va_end(args);
#endif
_exit(1);
}
------------------------------------------------------------------------
Finally, if sshd cannot be updated or recompiled, this signal handler
race condition can be fixed by simply setting LoginGraceTime to 0 in the
configuration file. This makes sshd vulnerable to a denial of service
(the exhaustion of all MaxStartups connections), but it makes it safe
from the remote code execution presented in this advisory.
========================================================================
Acknowledgments
========================================================================
We thank OpenSSH's developers for their outstanding work and close
collaboration on this release. We also thank the distros@...nwall.
Finally, we dedicate this advisory to Sophia d'Antoine.
========================================================================
Timeline
========================================================================
2024-05-19: We contacted OpenSSH's developers. Successive iterations of
patches and patch reviews followed.
2024-06-20: We contacted the distros@...nwall.
2024-07-01: Coordinated Release Date.
Powered by blists - more mailing lists
Please check out the
Open Source Software Security Wiki, which is counterpart to this
mailing list.
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
