musl - "non-float" malloc (was: incompatibility between libtheora/mmx and musl)

Follow @Openwall on Twitter for new release announcements and other news

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20161002105950.GM28325@example.net>
Date: Sun, 2 Oct 2016 12:59:50 +0200
From: u-uy74@...ey.se
To: musl@...ts.openwall.com
Subject: "non-float" malloc (was: incompatibility between libtheora/mmx and
 musl)

Hello Szabolcs,

On Wed, Sep 14, 2016 at 04:41:45PM +0200, Szabolcs Nagy wrote:
> this might be an issue:
> musl uses float instructions in malloc,
> if mmx needs different fpu state and
> they don't change it back for a malloc
> call that can corrupt the heap.
> 
> to test this, try to use the 'non-float bin index'
> code in musl from here:
> http://port70.net/~nsz/musl/malloc.c

Thanks again for the solution to the libtheora mystery.

Now I have reported the same issue affecting ffmpeg, it contains quite
a bit of simd assembler code and assumed until now among others that
malloc() never uses floating point.

Hopefully a fix is on the way there.

For the time being, to have a working ffmpeg build,
I tried to use your "non-float" malloc.

Unfortunately this leads to a segfault when used with the explicit runtime
loader, because brk() happily grows over the dynamic loader's data.
The modified variant of the malloc code seems to not apply
the traverses_stack_p() check.

I guess as long as you want to keep the special malloc.c
you would like it to be robust.

Would you mind adding the brk()/stack overlap checking to this variant
of the code?

Regards,
Rune

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.