john-dev - Re: revised incremental mode and charset files

Follow @Openwall on Twitter for new release announcements and other news

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20130430220652.GA21333@openwall.com>
Date: Wed, 1 May 2013 02:06:52 +0400
From: Solar Designer <solar@...nwall.com>
To: john-dev@...ts.openwall.com
Subject: Re: revised incremental mode and charset files

magnum, all -

Here's a really nasty problem that shows up with the new incremental
mode, and which I haven't figured out yet.  With the following in john.pot:

G02RZCpx7eF9E:2222
S2xnJvMcW5yeA:3141
b3mCnNjDdlN7k:1975
o8gCCzassVUWo:2252
sAC8cBLwaGPow:1332
XBcFMUNRUUKJQ:5252
5F8Ix0/FQ1yQo:1911
1HHxNDNOaD5zs:1022
YKZwA4qPukCEI:1214
sOoLCUP6RVKFE:5121
oQz1vttf53NAs:1234
vWWAhctESXefY:2112
nXdKWmH5NBtew:1991
Xc3VyFlRO0nPU:1956
8daJx7olZGvfs:4055
ZfiANbLUCiUyI:1213
Eh9MXVguwk.r2:6969
nj6MWnr5Ydqjs:3112
doTSXVW04Kq/2:1316
Ep0MHpguMaXF2:1313
sp9wRHIBvQYsQ:1948
DrPokBObKA2PU:1955
cr/zatcrrV7wc:1978
1sVdlYoBvx6yQ:1996
TuDrIAi/ML3Y.:1952
puFzFyeDYN06w:1973
gLlTSBjH9OEno:1818
BjXfSWu98nx5s:3533
JnSlZ4MQ9liYM:6301
sA4qnJKaMSPE.:4788
efwmSD4As54iM:0007

generate digits.chr and define:

[Incremental:Digits]
File = $JOHN/digits.chr
MinLen = 4
MaxLen = 4

Then try cracking the same hashes with an OpenMP-enabled build of John.
It will crack the 31 hashes.  Now do the same, but interrupt/restore it
a few times.  It will sometimes or often mysteriously crack fewer.
Sometimes much fewer.

With two threads, it had max_keys_per_crypt at 4096 (per the log file).
Apparently, an entire block like this fails to be checked correctly upon
restore in some cases.  The real mystery is that even when I added
printing of the passwords being tested right into the format's
set_key(), I got the 10000 different 4-digit numbers printed (total
across several interrupts/restores), yet not all passwords got cracked
by that very same run.

Now this could suggest a problem with my DES code.  However, I managed
to reproduce the problem also with c3_fmt.c and sha512crypt in Ubuntu
12.04's glibc.  (And the DES problem I reproduced on two machines, one
32-bit Owl, the other the 64-bit Ubuntu system.  Also with different
thread counts.  Even with 1 thread.  I think the large
max_keys_per_crypt helps trigger the problem, and OpenMP per se is
irrelevant.)  With sha512crypt, the problem is triggerable both with
OpenMP and without.  To test with sha512crypt, obviously hash the same
passwords above with that.

I'd appreciate it if you play with this too, and maybe figure it out
before I do.  One thing to try is AddressSanitizer.

Thanks,

Alexander

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.