Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Wed, 16 Sep 2015 05:28:17 +0300
From: Solar Designer <>
Subject: larger bitmaps and hash tables

Fred, magnum, all -

I am trying to stop the mess in the "Judy array" thread by starting
separate threads for the various related topics that come up.  Please
help me with this by also using per-topic threads.

Attached are two patches - john-huge-largehash.diff is preparing us for
larger bitmaps and possibly hash tables and cleaning up our use of
integer types a little bit in general (I think we should commit this
one), and john-huge-largehash-plus1.diff increases the size of the
largest bitmap for raw-md5 by a factor of 2 (and breaks other formats).

Previously, my best speed for Fred's testcase on 2x E5420 was:

real    0m55.630s
user    4m7.385s
sys     0m16.687s

With these two patches, I am getting:

real    0m53.729s
user    3m36.475s
sys     0m16.151s

That's still with PASSWORD_HASH_SHR at 0, as I set it in previous
patches (in the "Judy array" thread).  With SHR=1:

real    0m52.976s
user    3m39.218s
sys     0m16.227s

With SHR=2:

real    0m54.221s
user    3m44.508s
sys     0m19.503s

So maybe we should in fact increase our largest bitmap size, and at the
same time use larger SHR (to keep the largest hash table from increasing).

To beat Fred's reported speed, we need to get below:

47*233/250 = 43.8 seconds

adjusted for CPU clock rate difference.

Of course, we'd still be using far more RAM than MDXfind did in that test.


View attachment "john-huge-largehash.diff" of type "text/plain" (2399 bytes)

View attachment "john-huge-largehash-plus1.diff" of type "text/plain" (2477 bytes)

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.