[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <d24fb7.7e2.1398f43a522.Webtop.0@cox.net>
Date: Mon, 3 Sep 2012 23:12:18 -0400 (EDT)
From: jfoug@....net
To: john-dev@...ts.openwall.com
Subject: RE: memory usage (was: [john-users] JTR against 135
millions MD5 hashes)
Thank you for the long and insightful write up.
On Mon, Sep 3, 2012 at 8:49 PM, Solar Designer wrote:
> Jim -
>
> Thank you for sharing your thoughts on this.
>
> On Mon, Sep 03, 2012 at 11:16:44AM -0400, jfoug@....net wrote:
>> The memory reduction due to the source() function, and also due to
>> removing some wasteful allocations within the prepare() and valid()
>> within dynamic. I believe the wasteful allocations in dynamic have
>> been removed, all the way back to the J7-RC. The memory
>> requirements were greatly reduced early on, but have slowly started
>> to be lost some (the bitmaps). However, total mem usage is still
>> much less than it was, when we were keeping the full source buffer
>> for each hash.
>
> I did not realize that there were wasteful allocations in prepare()
> and
> valid(). Weren't they temporary?
They were done using alloc_memory_tiny. Now, I simply use static
buffers. When raw hashes were being used, and a few other cases, there
could be one or more of these type allocations, which are very similar
to memory leaks, when done on temp transient data.
> Another potential source of memory usage reduction are the alignments.
> For raw MD5 hashes, a 4-byte alignment should suffice (they're 4x4
> bytes), yet we were using 8-byte alignment on 64-bit builds.
Very good point. I had not even thought of things like these new
alignment requirements. I really think for most formats, 4 is the
correct alignment, due to dereference of an arch_word_32 in some of the
lookup functions.
I do not have time to fully address all of the issues brought up in your
reply, but will try to look more in depth at this tomorrow afternoon.
Jim.
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
