[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <BLU0-SMTP38950DD319BF98C75B40BF7FD270@phx.gbl>
Date: Sat, 5 Jan 2013 02:34:19 +0100
From: Frank Dittrich <frank_dittrich@...mail.com>
To: john-dev@...ts.openwall.com
Subject: Re: New plugin load order magic
On 01/05/2013 01:10 AM, magnum wrote:
> On 1 Jan, 2013, at 23:30 , Frank Dittrich <frank_dittrich@...mail.com> wrote:
>> On 01/01/2013 10:48 PM, magnum wrote:
>>> What we could do if you insist, is force raw-sha512 to load before blake.
>> I do not insist. I am just unsure what the best solution is.
>> Knowing what formats *could* possibly crack a given hash also is an
>> important feature.
>> May be you are right, and it is more important to let the user know that
>> there are possibly more formats which could crack a given hash than
>> allowing him not to specify --format=, and still expect a raw hash to be
>> treated as raw-sha512.
>>
>>> Either by renaming one of the source files so the plugin globbing will put them in a different order, or by making raw-sha512 a non-plugin.
>>
>> No, spending/causing effort for such crude workarounds is not what I
>> wanted, although, in this particular case it might be a bit unfortunate
>> that BLAKE2 would grab raw-sha512 hastes, even if BLAKE2 probably isn't
>> really used as a password hash function.
>> There might be users who will complain on john-users when they are not
>> longer able to crack raw-sha512 hashes if they forget the --format switch.
>>
>> May be this really should have been discussed on john-users as well, so
>> tomorrow I'll try to get this discussed on john-users.
>
> After Lukas' struct name fixes, I opted to try the struct-name sort order, committed now. I took the chance to rename fmt_rawSHA512 to fmt_raw0_SHA512 which ensures it loads before the Blake2 format. BTW I also renamed (talking format *labels* now) blake2-512 to raw-blake2-512, I hope anyone doesn't disagree with that?
>
> Some review of fmt_registers.h did not reveal any other cases I should fix. The MSSQL formats are in correct order, as is the NETLM ones, raw-sha1, NT and .
>
> Here is the old load order:
> (des, bsdi, md5, bf, afs, lm, dynamic_n)
> bfegg, dmd5, dominosec, epi, fortigate, hdaa, ipb2, krb4, krb5, mschapv2,
> netlm, netlmv2, netntlm, netntlmv2, nethalflm, md5ns, nt, phps, po,
> sybasease, xsha512, xsha, chap, clipperz, crc32, sha256crypt, sha512crypt,
> dmg, dragonfly3-32, dragonfly3-64, dragonfly4-32, dragonfly4-64, drupal7,
> encfs, episerver, formspring, gost, gpg, hmac-sha224, hmac-sha256,
> hmac-sha384, hmac-sha512, hmailserver, ike, keepass, keychain, keyring,
> krb5pa-md5, krb5pa-sha1, lp, lastpass, lotus5, md4-gen, mediawiki, mongodb,
> mscash, mscash2, mssql, mssql05, mssql12, mysql-sha1, mysql, mysqlna, npdf,
> nsldap, nt2, nk, o5logon, odf, office, oldoffice, oracle11, oracle, osc,
> pbkdf2-hmac-sha512, phpass, pix-md5, pkzip, postgre, pst, putty, pwsafe,
> racf, radmin, blake2-512, raw-md4, raw-md5, raw-sha1, raw-sha1-linkedin,
> raw-sha224, raw-sha256, raw-sha384, raw-sha512, raw-md5u, salted-sha1, sapb,
> sapg, sha1-gen, sip, ssh-ng, strip, sunmd5, sxc, openvms, vnc, wbb3, wowsrp
> (hmac-md5, hmac-sha1, raw-sha, django, tc_ripemd160, tc_sha512, tc_whirlpool,
> raw-sha1-ng, crypt, trip, ssh, pfx, pdf, wpapsk, rar, zip, dummy)
>
> Here is the new one:
> (des, bsdi, md5, bf, afs, lm, dynamic_n, bfegg)
> dmd5, dominosec, epi, fortigate, formspring, hdaa, ipb2, krb4, krb5,
> keepass, mschapv2, mysql, nethalflm, netlm, netlmv2, netntlm, netntlmv2,
> md5ns, nt, nt2, osc, phps, po, sybasease, openvms, xsha, xsha512, wowsrp,
> chap, clipperz, crc32, sha256crypt, sha512crypt, dmg, dragonfly3-32,
> dragonfly3-64, dragonfly4-32, dragonfly4-64, drupal7, encfs, episerver,
> gost, gpg, hmac-sha224, hmac-sha256, hmac-sha384, hmac-sha512, hmailserver,
> ike, keychain, keyring, krb5pa-sha1, lp, lotus5, md4-gen, mediawiki,
> mongodb, mscash, mscash2, krb5pa-md5, mssql, mssql05, mssql12, mysql-sha1,
> mysqlna, nk, npdf, nsldap, o5logon, odf, office, oldoffice, oracle,
> oracle11, pbkdf2-hmac-sha512, phpass, pix-md5, pkzip, postgre, pst, putty,
> pwsafe, racf, radmin, raw-sha512, raw-blake2-512, raw-md4, raw-md5,
> raw-sha1, raw-sha1-linkedin, raw-sha224, raw-sha256, raw-sha384, raw-md5u,
> salted-sha1, sapb, sapg, sha1-gen, sip, lastpass, ssh-ng, strip, sunmd5,
> sxc, vnc, wbb3, hmac-md5
> (hmac-sha1, raw-sha, django, tc_ripemd160, tc_sha512, tc_whirlpool,
> raw-sha1-ng, crypt, trip, ssh, pfx, pdf, wpapsk, rar, zip, dummy)
>
> Please test/review/comment.
I began testing this on my 32bit Linux system even without your commit
(i.e., without the renaming of fmt_raw_SHA512 to fmt_raw0_SHA512, and
with manually changing Makefile for the sort order).
First, I built an OMP version with plugin formats sorted by LC_ALL=C
sort. Then, another one using LC_ALL=C sort -r, for the reverse sort order.
Afterwards, I started test runs for each of these two versions, and
checked the output with relbench.
(unstable-jumbo)run $ ./relbench test-john-sort.txt test-john-sort-r.txt
More than one benchmark for NT MD4:Raw in file 1
More than one benchmark for NT MD4:Raw in file 2
Number of benchmarks: 293
Minimum: 0.76404 real, 0.76404 virtual
Maximum: 1.73783 real, 1.35359 virtual
Median: 0.99895 real, 0.99926 virtual
Median absolute deviation: 0.00562 real, 0.00576 virtual
Geometric mean: 0.99826 real, 0.99783 virtual
Geometric standard deviation: 1.04313 real, 1.03040 virtual
$ ./relbench -v test-john-sort.txt test-john-sort-r.txt |grep
"^Ratio:"|cut -f 2-
0.76404 real, 0.76404 virtual MS SQL 2005 SHA-1:Only one salt
0.83397 real, 0.98742 virtual OpenPGP / GnuPG Secret Key:Only one salt
0.84306 real, 0.84306 virtual Oracle 11g SHA-1:Many salts
0.90196 real, 0.90196 virtual BLAKE2 512:Raw
0.92598 real, 0.95994 virtual CRC-32:Many salts
0.92773 real, 0.93744 virtual pfx:Raw
0.94533 real, 0.95016 virtual generic crypt(3) DES:Many salts
0.94980 real, 0.94980 virtual Raw SHA-1 LinkedIn:Raw
0.95147 real, 0.94198 virtual generic crypt(3) DES:Only one salt
0.96503 real, 0.97477 virtual dynamic_82: sha512($p.$s):Only one salt
...
1.03409 real, 1.01804 virtual EPiServer salted SHA-1/SHA-256:Only one salt
1.04082 real, 1.00388 virtual WPA-PSK PBKDF2-HMAC-SHA-1:Raw
1.04139 real, 1.04121 virtual EPiServer salted SHA-1/SHA-256:Many salts
1.04252 real, 1.02131 virtual PKZIP:Many salts
1.04392 real, 1.02818 virtual SIP MD5:Raw
1.05087 real, 1.02437 virtual PKZIP:Only one salt
1.05487 real, 1.05487 virtual MS SQL SHA-1:Only one salt
1.05829 real, 1.05829 virtual Raw SHA-1:Raw
1.11241 real, 1.11241 virtual MySQL 4.1 double-SHA-1:Raw
1.73783 real, 1.35359 virtual PostgreSQL MD5 challenge-response:Raw
When I repeated the experiment, I noticed that even with the same john
binary, I had such variations:
$ ./relbench -v test-john-sort.txt test-john-sort2.txt |grep
"^Ratio:"|cut -f 2-|sort -n
0.74209 real, 1.00234 virtual SXC SHA-1 Blowfish:Raw
0.95602 real, 0.95602 virtual dynamic_25: sha1($s.$p):Many salts
0.95703 real, 0.95573 virtual pfx:Raw
0.96532 real, 0.98522 virtual HalfLM C/R DES:Only one salt
0.97442 real, 1.00415 virtual dynamic_82: sha512($p.$s):Many salts
0.97468 real, 0.98952 virtual PDF MD5 SHA-2 RC4 / AES:Many salts
0.97595 real, 0.98092 virtual MySQL Network Authentication SHA1:Raw
0.97690 real, 0.99166 virtual Mac OS X 10.7+ salted SHA-512:Many salts
0.97710 real, 0.99660 virtual HalfLM C/R DES:Many salts
0.97721 real, 0.98221 virtual Fortigate FortiOS:Many salts
...
1.01613 real, 1.00592 virtual DragonFly BSD $3$ SHA-256 w/ bug,
64-bit:Many salts
1.01698 real, 1.01187 virtual DragonFly BSD $3$ SHA-256 w/ bug,
32-bit:Only one salt
1.01837 real, 1.01320 virtual SIP MD5:Raw
1.02181 real, 1.00658 virtual NTLMv1 C/R MD4 DES (ESS MD5):Only one salt
1.03106 real, 1.01046 virtual PDF MD5 RC4:Only one salt
1.03986 real, 1.03986 virtual generic crypt(3) DES:Only one salt
1.04056 real, 1.04586 virtual generic crypt(3) DES:Many salts
1.04082 real, 1.00971 virtual WPA-PSK PBKDF2-HMAC-SHA-1:Raw
1.04978 real, 1.01811 virtual PKZIP:Only one salt
1.69725 real, 1.34221 virtual PostgreSQL MD5 challenge-response:Raw
I tried to keep the system idle besides the john --test, but who knows.
May be I shouldn't have used OMP.
(BTW: I think we really need another script to support relbench:
a script which checks several benchmark files created with the same
binary, just picks the best c/s rates for each hash format, and writes a
new "best-of" benchmark result to stdout.)
So, I started another test on my 64bit Linux system after fetching the
latest git (latest commit: 018ddb99).
Then, I uncommented HAVE_NSS, HAVE_GMP and HAVE_KRB5, used
$ make clean; make linux-x86-64-native
renamed ../run/john to ../run/john-sort
Then, I changed "$(SORT)" to "$(SORT) -r" (both occurrences in Makefile)
Then, to save some time, I just removed john.o, fmt_externs.h and
mt_registers (this shouldn't be necessary because I changed Makefile).
Then, I just used
$ make linux-x86-64-native
(without a make clean.
Then, I renamed ../run/john to ../run/john-sort-r, and repeated the
experiment I did on my 32bit Linux system.
This time, the differences weren't that large (Ratio from 0.922 to 1.08,
with one exception:
The dummy format was 1.24 times faster with the john-sort-r binary!
This was easily reproducable, even if I tested --format=dummy separately.
58949K vs. 73213K.
For some reason I don't understand, the 2 binaries differed in size!
john-sort: 2146752 bytes.
john-sort-r: 2146816 bytes.
When I do a
$ make clean
before I build the john binary with the reverted sort order, both
binaries have the same size, but performance for --format=dummy drops to
the performance of the john-sort version.
When I change the Makefile again (use $(SORT) without -r), remove
john.o, and build a new john version without a make clean, I get a
binary of size of 2146816 bytes.
The dummy format test gives between 62825K and 64214K now, which is
somewhere between the c/s rates of the other binaries.
Admittedly, dummy isn't the most important format.
But I don't understand these fairly reproducible c/s rate differences.
And I really don't know why the binaries differ in size.
May be I should get some sleep and hope someone else can come up with an
explanation.
Frank
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
