Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Fri, 3 Feb 2017 13:53:46 +0100
From: Patrick Proniewski <p+password@...atpro.net>
To: john-users@...ts.openwall.com
Subject: Re: to Single or not to Single

Alexander,

On 03 févr. 2017, at 13:14, Solar Designer wrote:

> What version of JtR are you using?  Is it almost the latest
> bleeding-jumbo (like this month's off GitHub) or something older?

I'm running a not-too-old Bleeding Jumbo, from the 6th of november 2016. (on FreeBSD 10.x). I've fully upgraded my OS few days ago, I should probably build a newer Bleeding jumbo on top of that.


> On Fri, Feb 03, 2017 at 07:26:29AM +0100, Patrick Proniewski wrote:
>> Oops. forgot to write a title on the graph. It's the number of "Cracked" per minute, extracted from the log file.
> 
> Well, it's in fact puzzling to me why this would be decreasing given
> what you have stated before.  This would be non-surprising if you had
> rules enabled, or had "SingleRetestGuessed = Y".
> 
>> 2799705g 1:08:45:08 50.00% (ETA: 2017-02-04 15:31) 23.74g/s 392.6p/s 392.6c/s 392.6C/s erikita222
> 
> Ouch.  That's extremely low speed for dynamic_25 (you said this is what
> you had?), which is sha1($s.$p).  It should be giving speeds of about
> 10M c/s per CPU core.

This is indeed very low, but I can't remember having achieved millions c/s on this hash type, even with other attack mode. I'll make some tests to check that.


> One obvious reason for a lower speed is that you're presumably testing
> only 1 password per salt, whereas we're optimizing for many passwords to
> be tested at once.  On a non-OpenMP build with AVX:
> 
> $ ./john --list=format-all-details --format=dynamic_25 | fgrep keys
> Min. keys per crypt                  64
> Max. keys per crypt                  1680
> 
> So you possibly get a 64x performance hit by testing only 1 password,
> and maybe worse.

If I understand well, I should be able to test 64 candidates in the same time I currently test 1. Meaning that if I write a few simple rules (1st char upper case, all char upper case, suffix 123, ...) that create less than 64 candidates, it could be a huge win.


> You could want to ensure you're using a non-OpenMP build (although
> recently OpenMP is disabled by default for "fast hash" formats,
> including this one, so simply using recent bleeding-jumbo should suffice).

OpenMP is not enabled on my install (./configure CC=gcc5 LDFLAGS=-L/usr/lib -L/usr/local/lib/gcc5 CFLAGS=-g -O2 -I/usr/include/openssl --disable-pkg-config --disable-openmp)


>>> along with their preceding Loaded and Remaining lines.
>> 
>> How do I get those numbers?
> 
> John prints them at startup.

Oh! Of course. 


>>> Your use of single mode sounds appropriate for what you're doing.
>>> To reduce its memory needs, try "--save-memory=2".
>> 
>> I'll give it a try. For now, I've split the huge file and I reclaim memory by stoping/restoring john every hour (yielding to this result <https://www.patpro.net/~patpro/john-memory-day.png>).
> 
> That graph clearly shows your hourly restarts, but it doesn't show a
> reason for you to be doing them - there's no obvious growth in memory
> usage during those hours.

The reason I restart is not good: I just want to give RAM back to the OS (other things running). I'll probably split my files again to make them smaller, and stop stoping/restarting.


> "Single crack" is not restore-friendly.  It's "--restore" is based on
> rules, which you don't use.  You're probably having it do a lot of
> duplicate work each time you restart.

Damn. I'm going to have to do something about that.

> Anyway, at ridiculously low speeds like what you're getting (need to
> figure out exactly why, but that's a separate issue), you could as well
> go for "--save-memory=3" and probably not notice a further slowdown -
> but you'd save lots of memory, and should be able to load all of your
> hashes at once.

loading the whole file would be impressive, but I would really prefer a huge speed increase. I'll update my bleeding jumbo, and make few benches.

thanks a lot!
patpro

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.