[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20160914112400.GS1280@port70.net>
Date: Wed, 14 Sep 2016 13:24:00 +0200
From: Szabolcs Nagy <nsz@...t70.net>
To: musl@...ts.openwall.com
Subject: Re: incompatibility between libtheora/mmx and musl ?
* u-uy74@...ey.se <u-uy74@...ey.se> [2016-09-14 12:32:53 +0200]:
> On Tue, Sep 13, 2016 at 04:43:04PM -0400, Rich Felker wrote:
> > > Libtheora contains an assembler part for ia32 and x86_64 which has been
> > > in use for many years, with at least both glibc and uclibc.
> > >
> > > Now, musl-based builds of libtheora for ia32 with this code enabled
> > > lead to the encoder segfaulting. No problem when using the C version.
>
> > > https://trac.xiph.org/ticket/2287
> > >
> > > There is also a corresponding
> > >
> > > https://bugs.alpinelinux.org/issues/6132
> >
> > The most likely explanation is that they're overflowing a heap buffer.
> > Perhaps it would be possible to pad all their malloc/realloc calls
> > with +64 or so to see if that makes the problem go away. If so that
> > gives a good starting point for tracking down the bug.
>
> Thanks for the suggestion, indeed libtheora seems to consistently use
> ogg_malloc() which is redefinable.
>
> Building with
>
> #define _ogg_malloc(x) malloc((x)+256)
> #define _ogg_calloc(x,y) calloc((x)+256,(y))
> #define _ogg_realloc(y,x) realloc((y),(x)+256)
> #define _ogg_free free
>
> instead of the default
>
> #define _ogg_malloc malloc
> #define _ogg_calloc calloc
> #define _ogg_realloc realloc
> #define _ogg_free free
>
> did not make any difference. The crash on a test file occurs in the same
> way and the resulting partial output file is as long as otherwise.
>
> This may mean that this is not a simple overflowing but rather
> overwriting or reading distant "random" places (?) (register corruption?)
>
can be underflow (or the way they align the pointer returned by malloc)
you can increase/decrease alignment of musl's alloc by
changing SIZE_ALIGN in src/malloc/malloc.c
(or you can try some hack in _ogg_malloc/free if you are
sure that's what they are using)
there can be some call abi issue (register clobbering,
stack alignment,..) because of the asm, but that's hard
to check.
you may try tracing malloc calls (i don't know an easy
way other than instrumenting musl, you can try python
scripting gdb, the default gdb command language is not
enough for reporting malloc args and return values).
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
