Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20181030150039.GT5150@brightrain.aerifal.cx>
Date: Tue, 30 Oct 2018 11:00:39 -0400
From: Rich Felker <dalias@...c.org>
To: musl@...ts.openwall.com
Subject: Re: musl: about malloc 'expand heap' issue

On Tue, Oct 30, 2018 at 11:11:07AM +0000, zhangwentao (M) wrote:
> 
> Hi all,
>   I am using musl in my project and I found an issue about the malloc function in musl:
> 
> Issue Description:
> *             When in muti-threads environment, malloc/free are called in high concurrency<http://dict.cn/high%20concurrency>.
> 
> Malloc:
>   Will find 'struct bin' from bitmap(without lock), and allocate memory from the bin (with lock).
> 
> Free:
>  Will merge the chunk together if the free memory is 'connected' to the existing chunk.
> 
> ? It will remove the old chunk first then combine the chunk to a larger one..
> 
> ? After merge operation done, insert the chunk to the bin list.
> 
> ? Each of the chunk operation is locked while merging, but the whole steps aren't within a lock.
> 
> So here is the issue:
> 
> 1.      There is only one chunk in largest bin list, and Free is on process, just remove the largest bins chunk from bin, the bitmap(mal.binmap) on that bit will be zero.
> 
> 2.      A malloc comes, the bitmap is zero, and goes to expand heap. (Actually there is enough memories in process)
> 
> 3.      Free operation goes on, and put the merged big chunk to bins.
> 
> But in operation 2, the process has expand heap.
> 
> If we have a loop on step 1-3, the process will expand heap frequently.
> So it will cost more Virtual Memory  (of course, physical memory would be freed by calling '__madvise' if the chunk is big enough)
> 
> In my environment , we do not have that much virtual memory. I think stop expand heap would a better choice.
> 
> Do you have plan to fix it ??

This is a known issue, and intended to be fixed in the complete
redesign of malloc. Fixing it right in the current design seems to
impose significant performance costs that I thought were equivalent
to, or worse than, just using one global lock. However if it's causing
major problems I may be able to make a quick fix that's not too
expensive -- I'll take a look again today or tomorrow.

Rich

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.