[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <LfF7Bgg--3-1@tuta.io>
Date: Sun, 19 May 2019 14:49:32 +0200 (CEST)
From: Wire Snark <wsnark@...a.io>
To: <oss-security@...ts.openwall.com>
Subject: Potential DoS vulnerability in CGit
Hello, oss-security list
I need your advice on the following bug in CGit disclosured by me recently in the cgit 'at' lists.zx2c4.com [1].
CGit is a hyperfast web frontend for git repositories written in C [2].
There is no formal security contact at [3], contacting CGit author and maintainer Jason Donenfeld directly didn't work either (probably my mail ends up in spam or whatever). My posting to CGit mailing list hasn't received a reply yet (since May 15) so I'm not sure whether someone has even read it so far.
My question: is this a valid security issue (DoS) that's worth applying for CVE? This is my first bug in public software actually, so your advice on this will be very helpful. Should I do more thorough performance measures, or this qualitative analysis below is enough?
[1] https://lists.zx2c4.com/pipermail/cgit/2019-May/004364.html <https://lists.zx2c4.com/pipermail/cgit/2019-May/004364.html>
[2] https://git.zx2c4.com/cgit/ <https://git.zx2c4.com/cgit/>
[3] https://git.zx2c4.com/cgit/about/ <https://git.zx2c4.com/cgit/about/>
### Bug description
A specially crafted URL in the request is processed by cgit with a sort of non-linear(quadratic) function, excessively using CPU and network resources. That is, given input with len(input) = n, output produced by cgit becomes len(output) ~ C * n^2.
Severity: Low (?)
### Reproducers
Hand-crafted reproducers I have come with so far look like:
curl http://localhost:8080/mycgit/tree/0/0/ <http://localhost:8080/mycgit/tree/0/0/><...>/0/0/0
Where "localhost:8080" is where my web server is, "mycgit" is a valid repository name, and the number of /0/ blocks that can be filled is determined by the maximum URL length configured at the particular web server.
Reproducer for my local server setup: local_repr.txt (attached)
Input len = 8056, Output is ~15.5Mb.
For kernel.org, that is using cgit: kernel.org_repr.txt (attached)
There are 2 reproducers:
- URL with len=1305 bytes. Output is html with len=424 kbytes.
- URL with len=1875 (maximum that is accepted at kernel.org at the moment). Output is html with len=863 kbytes.
The dependency seems to be quadratic with C ~ 0.25.
Original hang reproducer generated by AFL: afl_repr.bin (attached)
Input: 34kb, Output: ~600Mb
Note: afl_repr.bin file format is tab-separated values for all used env variables by cgit; the reproducer is different and contains some non-printable chars (in my fuzzing setup cgit reads env variables from stdin, allowing arbitrary input)
### Analysis
Backtrace from interrupting cgit during processing of this input (with output in terminal, i.e. slow):
Program received signal SIGINT, Interrupt.
0x00007ffff7d741c5 in write () from /usr/lib/libpthread.so.0
(gdb) bt
#0 0x00007ffff7d741c5 in write () from /usr/lib/libpthread.so.0
#1 0x00005555555647d4 in html_raw (data=<optimized out>, size=6948) at ../html.c:83
#2 0x0000555555564ec1 in html (txt=<optimized out>) at ../html.c:211
#3 html_url_path (txt=<optimized out>, txt@...ry=0x55555574ff30 "Oe", '/' <repeats 198 times>...)
at ../html.c:211
#4 0x000055555556e70d in repolink (title=title@...ry=0x0, class=class@...ry=0x0,
page=page@...ry=0x5555556b175b "tree", head=head@...ry=0x555555759450 "fuzzing",
path=path@...ry=0x55555574ff30 "Oe", '/' <repeats 198 times>...) at ../ui-shared.c:288
#5 0x000055555556e853 in reporevlink (page=page@...ry=0x5555556b175b "tree",
name=name@...ry=0x555555751a54 "", title=title@...ry=0x0, class=class@...ry=0x0,
head=head@...ry=0x555555759450 "fuzzing", rev=0x0,
path=0x55555574ff30 "Oe", '/' <repeats 198 times>...) at ../ui-shared.c:319
#6 0x000055555556fbc9 in cgit_tree_link (path=0x55555574ff30 "Oe", '/' <repeats 198 times>...,
rev=<optimized out>, head=<optimized out>, class=0x0, title=0x0, name=0x555555751a54 "")
at ../ui-shared.c:345
#7 cgit_self_link (name=name@...ry=0x555555751a54 "", class=0x0, title=0x0) at ../ui-shared.c:527
#8 0x0000555555571347 in cgit_print_path_crumbs (path=<optimized out>) at ../ui-shared.c:957
#9 cgit_print_pageheader () at ../ui-shared.c:1103
#10 0x00005555555718cf in cgit_print_layout_start () at ../ui-shared.c:863
#11 cgit_print_error_page (code=code@...ry=404, msg=msg@...ry=0x55555568d951 "Not found",
fmt=fmt@...ry=0x555555690f7c "Path not found") at ../ui-shared.c:852
#12 0x0000555555575d5b in cgit_print_tree (rev=0x555555759450 "fuzzing",
path=0x55555574ff30 "Oe", '/' <repeats 198 times>...) at ../ui-tree.c:382
#13 0x0000555555561993 in process_request () at ../cgit.c:800
#14 0x0000555555563579 in cache_process (size=<optimized out>, path=<optimized out>,
key=<optimized out>, ttl=<optimized out>, fn=fn@...ry=0x555555561890 <process_request>)
at ../cache.c:370
#15 0x000055555556236f in cmd_main (argc=<optimized out>, argv=<optimized out>) at ../cgit.c:1161
#16 0x000055555555ed4f in main (argc=2, argv=0x7fffffffe7e8) at common-main.c:45
As I understand, the issue is in ui-shared.c, cgit_print_path_crumbs():
ctx.qry.path = p = path;
while (p < end) {
if (!(q = strchr(p, '/')))
q = end;
*q = '\0';
html_txt("/");
cgit_self_link(p, NULL, NULL);
if (q < end)
*q = '/';
p = q + 1;
}
It attempts to print a cgit_self_link() on each subpath in the URL, resulting in O(n^2) for n as number of subpaths in the url.
### How to fix
I don't really know cgit internals, so I can propose only very simple fix - limit the depth of path crumbs handling, e.g.
diff --git a/ui-shared.c b/ui-shared.c
index d27a5fd..279862f 100644
--- a/ui-shared.c
+++ b/ui-shared.c
@@ -949,7 +949,9 @@ static void cgit_print_path_crumbs(char *path)
ctx.qry.path = NULL;
cgit_self_link("root", NULL, NULL);
ctx.qry.path = p = path;
- while (p < end) {
+ int maxdepth = 10;
+ while (p < end && maxdepth > 0) {
+ maxdepth--;
if (!(q = strchr(p, '/')))
q = end;
*q = '\0';
Probably this magic 10 should be defined somewhere (do not think it should be configurable though). Also I don't know what is valid path depth expected here.
With this fix I confirm the output size is reduced to normal (8056-len URL gives 50K, 34K AFL-generated one gives 204K html output).
### Security implications
I think this issue can be leveraged to cause Denial of Service condition on the cgit server. I have tried following experiment: at the localhost start 100 curl instances with reproducer (8056 one) and "--limit-rate 10K -sS >/dev/null" options so they do not consume output html too fast. This results in a few seconds of high CPU usage at the target server (I used 4 vCPU VM from some old Core i7 mobile CPU). 100 curls cause load ~1, adding more can push to 2 and so on; curls do not seem to consume much CPU themselves. I use nginx, fcgiwrap and cgit.cgi, so nginx worker process and fcgiwrap were using the most of CPU, probably until all cgit output has been saved in nginx buffer (not really sure here, but seems like nginx memory usage grows). After initial CPU usage burst cgit finishes rendering and terminates, so CPU usage goes down. Some clients may receive one of the two errors:
curl: (18) transfer closed with outstanding read data remaining
curl: (56) Recv failure: Connection reset by peer
Most of the clients continued to work. Sometimes (especially if increasing a number of clients) there are fcgiwrap failures like:
[crit] 21295#21295: *1182 pwritev() "/var/lib/nginx/fastcgi/2/57/0000000572" has written only 936 of 8184 while reading upstream, client: 127.0.0.1, server: localhost, request: "GET /mycgit/tree/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/0/<...cut...>/0/
After applying the fix, none of the issues have been observed with 100-400 curl readers at the same setup (even if throttled to 1K instead of 10K as before, to keep connections open). CPU bursts only for a very short amount of time (mostly when 100s of curls are forked/exec'd, though slight fcgiwrap CPU usage has been observed in htop too).
### About the author
My name is Fyodor [Wire Snark], I'm an amateur security researcher at DC7831 (http://defcon-nn.ru <http://defcon-nn.ru>), our local DEF CON group in Nizhniy Novgorod, Russia.
This report has been prepared as a result of my self-studying fuzzing with AFL and LibFuzzer from LLVM and applying them to cgit. I haven't seen any such fuzzing reported for cgit, so if you know any previous work on this, I'd be glad to know (I plan to publish a blog post about my fuzzing setup and these results; no crashes have been observed so far).
Best regards,
Wire Snark
Powered by blists - more mailing lists
Please check out the
Open Source Software Security Wiki, which is counterpart to this
mailing list.
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
