[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <9ee0b4efeb68ac9c81d1e78321f7594f@smtp.hushmail.com>
Date: Mon, 21 Apr 2014 23:06:07 +0200
From: magnum <john.magnum@...hmail.com>
To: john-dev@...ts.openwall.com
Subject: Re: mmap(2) in wordlist.c
On 2014-04-21 19:05, magnum wrote:
> I started looking at this. The current implementation alters the buffer
> (turns CR/LF into NULLs) and we can't do this with a shared read-only
> buffer. It also creates a char array (which is 114 MB alone, for
> Rockyou) so you can address any given word with an index and use it just
> like that. This is not compatible with (so silently disabled for)
> external filters.
>
> One solution would be to scrap that array and make something like
> fgetl() for the mapped buffer. This would sometimes be a tad slower than
> current solution but for some situations it would be a lot better. Think
> 32 separate processes using the Rockyou list - all using the same 140 MB
> buffer instead of a total of 4 GB of duplicated data.
Actually it's even worse: Including the mentioned array (it's 109 MB
rather than 114), rockyou in 32 processes would eat 7.7 GB with current
code compared to a shared map, no array, totalling a mere 133 MB (the
actual file size x1).
magnum
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
