[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <20110603020509.GA28869@openwall.com>
Date: Fri, 3 Jun 2011 06:05:09 +0400
From: Solar Designer <solar@...nwall.com>
To: john-users@...ts.openwall.com
Subject: JtR 1.7.7-jumbo-5
Hi,
After several development-only revisions of the jumbo patch, we've
finally released one intended for actual use. Yes, it's an upgrade from
1.7.7-jumbo-1 straight to -jumbo-5 for the adventurous users. Quite
possibly, this is the largest single jumbo patch update we made so far.
The full 1.7.7-jumbo-5 tarball and just the patch may be downloaded here:
http://www.openwall.com/john/
http://www.openwall.com/john/#contrib
http://download.openwall.net/pub/projects/john/1.7.7/
Most of the work towards this update was by JimF and magnum (or vice
versa), although it also integrates major contributions by bartavelle
and smaller ones by others in the community (thanks to all of you!)
Here's a summary of the major changes:
* MD5 and SHA-1 based hashes have been sped up with SSE2 intrinsics.
* md5_gen has been expanded with more hash types.
* UTF-8 support has been added ("--utf8").
* MPI parallelization support for all cracking modes has been integrated.
* OpenMP parallelization support has been added to more hash types.
* New formats have been added: mskrb5 (offline attack on MS Kerberos 5
pre-authentication data), rawMD5unicode (MD5 of UCS-2 encoded
plaintext), and salted_sha1 (faster handling of some LDAP {SSHA} hashes).
* The "unique" program, Markov mode, ETA display, and programming interfaces
have been enhanced.
Below is a more complete list of changes, compiled by JimF and posted to
john-dev (thanks!), with only very minor edits by me (such as
incorporating corrections from followup postings on john-dev). I don't
feel the strength to fully "process" this, sorry, nor would I manage to
come up with a similar list myself (the changes since -jumbo-1 are just
too extensive, and I'm not even familiar with many of them).
---
Even though this update is heavily loaded with new functionality/enhancements,
it should also hopefully prove over time to be a stable release as well.
Intrinsic SSE code incorporated into the jumbo. Numerous formats patched
(listed below), and sse-intrinsics.c/.h files added (bartavelle)
--utf8 (UTF-8) support, in several formats, and with rules support and other
infrastructure support.
Support code is now all in unicode.c/h;
removed file smbencrypt.c (older support code) (magnum)
MPI/OpenMP code and support added (magnum)
Structural changes to fmt_main, including new prepare() function (and changes
to init and valid). prepare() allows formats to control line morphing, which
before was hacked into loader.c. These changes touched the
source for every format file, and they will not be listed separately. (JimF)
Changes to md5_gen so it appears as multiple individual formats, and not a
format that morphs, required the fmt_main changes. Also all of the md5_gen
specific code stripped out of formats.c and bench.c as that code is not
needed anymore. (JimF)
md5_gen back and forth from X86 to MMX code (allows formats which used to
not work on SSE builds to be used on them now) (JimF)
md5_gen: Added limited support for SHA-1 hashing, uppercase, lowercase,
ISO-8859-1 to UTF-8 support, 'username', and support for any multiple salts,
or any other GECOS field(s) (JimF)
md5_gen: Added new built-in formats (md5_gen(18) to md5_gen(28)). These
include PO, PIX, salted PIX, HDAA, some SHA stuff, FreeBSD/Apache MD5 (JimF)
md5_gen changed 'thin' format linking, thus had to update PHPS, phpass, and
raw-md5 to use the new methods (JimF)
timer.c/.h added for portable high res timing 'class' used in some testing
code for md5_gen. This code can be reused in other parts of john. (JimF)
Speed improvement for md5_gen for non-MMX/SSE builds. Now using MD5 and X2
MD5 from MD5_std.c (JimF)
New formats: mskrb5 (magnum), rawMD5unicode (magnum),
salted_sha1 (bartavelle) - this one is nsldaps, but using SSE2i.
Enhancements to Markov engine, adding minlevel and minlen (bartavelle)
ETA now working for -incremental runs. (magnum)
format 'listing' in options.c now done dynamically (magnum)
-mem-file-size=0 will load any sized wordfile into memory;
also -mem-file-size=1 or =2 for special MPI usage (magnum).
New options (-inp=fname, -ex_file=fname, -ex_file_only=fname and -mem=num)
added to unique program, for usability and performance (JimF)
Removed DOS line endings from DMD5 source file (JimF)
Increased max password length of HDAA from 16 to 32, and changed some
#defines for STDC (JimF/magnum)
Increased max password length of Netscreen from 15 to 25. (JimF)
Increased max password length of mssql05/mssql from 20 to 25. (JimF)
Fixed bug in mssql/mssql05 MMX/SSE2 builds crypt_all() function, which
caused them to not work for more than 1 salt, even though they passed self
test (JimF)
In several formats, switch from malloc, to mem_alloc_tiny (JimF)
prepare() in LM_fmt.c to handle pwdump format (was in loader.c) (JimF)
SSE intrinsics added to md5_fmt.c, and got intrinsic working for both $1$ and
$apr1$ (bartavelle / JimF)
SSE intrinsics also added to raw-sha1 and raw-md5 (through thin format to
md5_gen(0)). Intrinsics also used in other formats. (bartavelle)
Code moved from loader.c into prepare() for: LM, NT, oracle, md5_gen, mscash2,
mssql05, MSCHAPv2, NETLM, NETLMv2, NETNTLM, NETNTLMv2, NETHALFLM (JimF)
UTF-8 support added to: MSCHAPv2, NETLMv2m NETNTLM, NETNTLMv2, NT, mscash,
mscashv2, mskrb5. mssql, mssql05, oracle, sapG. (magnum)
OpenMP support added to: MSCHAPv2, NETHALFLM, NETLM, NETLMv2, NETNTLM,
NETNTLMv2, bench.c (magnum)
Fixed REQ_ALIGN crash problems in: HDAA, MSCHAPv2, NETLM, NETLMv2, NETNTLM,
NETNTLMv2, NSLDAPS, NSLDAP, OPENLDAPS, mscash, mscashv2, mssql, mssql05,
oracle, raw-md4. Note, some crash problems existed in Jumbo-1, and some
were needed due to J1->J5 performance changes, and UTF-8 support. (JimF)
Changed NT's asm code (x86_sse.S) to properly work with UTF-8. (Alain Espinosa)
Fixed compile problem in bench.c for Solaris. (Solar)
Additional information added to self-tests (helps in debugging) (magnum)
Markov code: added minlevel and minlen;
bumped level=0 from 350 to 400. (bartavelle)
hmac_md5 code sped up (but this turned up several REQ_ALIGNMENT issues that
had to be addressed). (magnum)
"make check" is now working again (was broken in john.c in -jumbo-1);
also works for Win32 builds. (JimF)
-nolog option added (keeps data from being written to john.log file). (JimF)
Made changes to md5_mmx.S/md5.h sha1_mmx.S/sha1.h so that the object files
built by MinGW can be used by M$ Visual C. (JimF)
Changes to md5_go.c, added a MD5_Go2() function where the output buffer
space is passed in. (JimF)
Added 'cleanup' code to mem_alloc_tiny() calls, allowing better checking for
leaked memory at program exit. (JimF)
Added memory image dumping functions for flat and for MMX buffers. (bartavelle)
Added shammx_nofinalbyteswap function, which leaves result in LE format. (JimF)
mysqlSHA1, reduced memset(), and improved speed 10-15%; use the new
shammx_nofinalbyteswap function, and skip swap and unswap (2-3% speed
increase). (JimF)
raw-md5 converted to a 'thin' md5_gen format (bartavelle)
Changes made to x86-64.h (and some 32-bit arch.h files), so that different
compilers would get different SSE_PARA values. Better tuning ability. May
still need a little work, as values chosen were done so based upon
assumptions from prior email discussions and not physical testing. (JimF)
Created a new x86-sse2i.h to allow intrinsics to be used with 32-bit
compilers, while still leaving the 32-bit .S SSE2 code working. This is
more for testing, since most 32-bit builds are much slower with intrinsic vs
.S builds. (JimF)
---
Whew. Did you manage to read and understand this? I guess not...
Maybe we shouldn't have included this many changes at once, although we
did have our reasons to do so.
Anyway, please give this new revision a try and provide some feedback. :-)
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
