Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20160914144145.GT1280@port70.net>
Date: Wed, 14 Sep 2016 16:41:45 +0200
From: Szabolcs Nagy <nsz@...t70.net>
To: musl@...ts.openwall.com
Subject: Re: incompatibility between libtheora/mmx and musl ?

* Rich Felker <dalias@...c.org> [2016-09-14 10:28:42 -0400]:
> On Wed, Sep 14, 2016 at 04:04:50PM +0200, u-uy74@...ey.se wrote:
> > On Wed, Sep 14, 2016 at 01:24:00PM +0200, Szabolcs Nagy wrote:
> > > there can be some call abi issue (register clobbering,
> > > stack alignment,..) because of the asm, but that's hard
> > > to check.
> > 
> > Is musl in any way special compared to glibc/uclibc in its register usage?
> 
> Not in principle; this is mandated by the ABI. But it's possible that
> their violation of ABI contracts is visible with some implementations
> but not others. For example if they're calling malloc from code that's
> using asm it's possible that they assume the floating point registers
> (or mmx state) are call-saved rather than call-clobbered. This is an
> invalid assumption that might happen to actively break on musl but not
> glibc. IIRC you need some special instructions to switch between x87
> and (original) mmx usage; perhaps they're missing this somewhere.
> 

this might be an issue:
musl uses float instructions in malloc,
if mmx needs different fpu state and
they don't change it back for a malloc
call that can corrupt the heap.

to test this, try to use the 'non-float bin index'
code in musl from here:
http://port70.net/~nsz/musl/malloc.c

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.