[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <20130530081728.GA19142@openwall.com>
Date: Thu, 30 May 2013 12:17:28 +0400
From: Solar Designer <solar@...nwall.com>
To: announce@...ts.openwall.com, john-users@...ts.openwall.com
Subject: John the Ripper 1.8.0
Hi,
I've just released John the Ripper 1.8.0, available from the usual place:
http://www.openwall.com/john/
Available are a source code tarball (compressed with gzip or xz) and an
extra charset files tarball (xz only). Besides the source code,
included in the main tarball are 3 .chr files that are arguably most
essential, whereas the extras tarball contains 7 additional .chr files.
The default john.conf provides pre-defined incremental modes for all 10
of these files.
Although only JtR 1.8.0 source code and .chr files are available at this
time (to be used mostly on Unix-like systems, where building from source
is customary and convenient), we do intend to release Windows build(s)
of JtR 1.8.x, a jumbo based on 1.8.x, and a Pro based on 1.8.x at a
later time. Meanwhile, magnum is regularly sync'ing the bleeding-jumbo
branch on GitHub to be based on the latest core tree (which means 1.8.0
at the moment).
Much of the work towards this release was sponsored under Rapid7's
Magnificent7 program as previously announced here:
http://www.openwall.com/lists/announce/2012/10/17/1
Here are the Magnificent7 sponsored changes, with their short
descriptions as extracted from doc/CHANGES:
* Revised the incremental mode to let the current character counts grow
for each character position independently, with the aim to improve
efficiency in terms of successful guesses per candidate passwords tested.
Here are some test results of the old (1.7.9's) vs. new incremental
mode, using the same training and test sets for both versions:
http://www.openwall.com/lists/john-dev/2013/05/18/26
http://www.openwall.com/lists/john-dev/2013/05/19/1
http://www.openwall.com/lists/john-dev/2013/05/21/7
These show an improvement in the number of passwords cracked (for the
same number of candidate passwords tested) ranging from 1% to 91% (that
is, up to almost twice more passwords cracked) for different points in
time. Greater improvements are seen at 10 thousand to 100 million
candidates tested. Then the old version slowly catches up and by 1e12
candidates tested the new version's advantage decreases to only 1%,
which is nevertheless significant. Tests on other samples show somewhat
different results, but the new version generally does stay ahead.
I'd like to thank magnum and bartavelle for running additional tests,
which first helped spot a problem and then confirmed that, after my fix,
the new incremental mode consistently outperformed the old one in their
tests as well.
* Revised the pre-defined incremental modes, as well as external mode
filters that are used to generate .chr files.
* Added makechr, a script to (re-)generate .chr files.
As of version 1.8.0, pre-defined incremental modes are "ASCII" (all 95
printable ASCII characters), "LM_ASCII" (for use on LM hashes), "Alnum"
(all 62 alphanumeric characters), "Alpha" (all 52 letters), "LowerNum"
(lowercase letters plus digits, for 36 total), "UpperNum" (uppercase
letters plus digits, for 36 total), "LowerSpace" (lowercase letters plus
space, for 27 total), "Lower" (lowercase letters), "Upper" (uppercase
letters), and "Digits" (digits only). The supplied .chr files include
data for lengths up to 13 for all of these modes except for "LM_ASCII"
(where password portions input to the LM hash halves are assumed to be
truncated at length 7) and "Digits" (where the supplied .chr file and
pre-defined incremental mode work for lengths up to 20).
The charset files for "ASCII", "LM_ASCII", and "Digits" are included in
the main tarball. The remaining 7 are available in the extras tarball.
The new compile-time defaults for JtR permit for incremental mode
lengths of up to 24, and for the entire 8-bit character set (excluding
only the NUL character). In other words, you may generate additional
.chr files with data for longer passwords and/or non-ASCII characters
while using the normal JtR build that is compatible with the supplied
.chr files. (The suggested limit of 13, except for digits, is based on
our testing, though. It usually does not make sense to go for higher
lengths with incremental mode.) You may also use "Extra = ..." in the
configuration file to add more characters manually, including non-ASCII
characters.
* Enhanced the status reporting to include four distinct speed metrics
(g/s, p/s, c/s, and C/s).
The new status reporting is illustrated in this posting:
http://www.openwall.com/lists/john-dev/2013/04/28/1
The new status line may include: successful guess count ("g"), session
duration (in the D:HH:MM:SS format for days, hours, minutes, and
seconds), progress indicator (percent done and optionally pass number
out of the total number of passes), up to four speed metrics ("g/s",
"p/s", "c/s", and "C/s"), and the current (range of) candidate
password(s) being tested (John is often able to test multiple candidate
passwords in parallel for better performance, hence a range). The four
speed metrics are as follows: g/s is successful guesses per second (so
it'll stay at 0 until at least one password is cracked), p/s is
candidate passwords tested per second, c/s is "crypts" (password hash or
cipher computations) per second, and C/s is combinations of candidate
password and target hash per second. Versions of John prior to 1.8.0
displayed only the C/s rate (calling it c/s). When you restore a
pre-1.8.0 session with version 1.8.0 or newer, only the g/s and C/s
rates will be displayed, because the older .rec file format lacked
information needed to compute p/s and c/s.
* Added the "--fork=N" and "--node=MIN[-MAX]/TOTAL" options for trivial
parallel and distributed processing.
Naturally, these are documented in doc/OPTIONS.
"--fork=N" is only available on Unix-like systems. It is an easy way
to make use of multiple CPUs or CPU cores - you simply specify the
number of John processes that you'd like to run. You may use "--fork"
as an alternative to OpenMP, for formats currently lacking OpenMP
support, or/and along with OpenMP (e.g., on a machine with 64 logical
CPUs you might choose to run with "--fork=8" for 8 processes and use
OpenMP to run 8 threads per process).
You may use "--fork" along with "--node" to use multiple machines while
also running multiple John processes per machine. For example, to use
two similar 8-core machines you may run "--fork=8 --node=1-8/16" on one
of the machines and "--fork=8 --node=9-16/16" on the other.
You may also use "--node" separately from "--fork".
Please refer to doc/OPTIONS for a lot more detail on these new options:
http://www.openwall.com/john/doc/OPTIONS.shtml
* In the external mode compiler, treat character literals as unsigned.
This is to make 8-bit character literals specified in external modes
always behave as expected. Previously, they were problematic if naively
put e.g. into a revision of the Keyboard external mode:
http://www.openwall.com/lists/john-users/2012/02/26/2
Now this problem has been addressed both in John itself and in the
revised Keyboard external mode (in case it is used on an older version
of John, and 8-bit characters are introduced into it).
* Renamed many of the formats.
John the Ripper 1.8.0's valid format names are descrypt, bsdicrypt,
md5crypt, bcrypt, LM, AFS, tripcode, dummy, and crypt (and many more are
added in jumbo). We no longer misuse just "des" or just "md5" to refer
to the crypt(3) flavors; this made some sense in 1990s, before raw MD5
started to be misused for password hashing in web apps. However, with
today's (still) widespread use of raw MD5 in web apps (and especially in
leaked password hash dumps) our use of "md5" to refer to md5crypt was
causing confusion.
* Updated the documentation.
Naturally. :-)
Other recent changes unrelated to Magnificent7 are:
* Relaxed the license for many source files to cut-down BSD.
* Relaxed the license for John the Ripper as a whole from GPLv2 (exact
version) to GPLv2 or newer with optional OpenSSL and unRAR exceptions.
80% of John the Ripper 1.8.0 source files (in the src/ directory) are
now available under a cut-down BSD license (which is obviously
compatible with the GNU GPL, letting me release the entire thing under
GNU GPL v2 or newer).
Other/older changes (also unrelated to Magnificent7) that I made since
version 1.7.9 are as follows:
* Enhanced the support for DES-based tripcodes by making use of the
bitslice DES implementation and supporting OpenMP parallelization.
An equivalent change, providing for much faster tripcode cracking than
JtR's earlier proof-of-concept implementation did, has been in jumbo
for a while, although it originated from the core tree - and is now
properly released "officially".
* Implemented bitmaps for fast initial comparison of computed hashes
against those loaded for cracking. This is applied before hash table
lookups, and it allows for the use of smaller hash tables (thereby
saving memory) while achieving the same or greater speed that larger
hash tables previously did. The speed increase is due to improved
locality of reference (where only the smaller bitmap is accessed all the
time, whereas the larger hash table behind it is only accessed for a
percentage of comparisons and additionally it is smaller than it would
otherwise need to be).
* Tuned the bitmap and hash table sizes and thresholds based on testing
on saltless hashes on a Core 2'ish CPU.
This provides a substantial performance improvement when cracking large
numbers of fast hashes. This change is a reason why in the incremental
mode tests referenced above the new version achieved the same number of
candidate passwords tested much sooner than the old version did. (To
remove this effect on our incremental modes' comparison, we compared
their successful guess counts at the same numbers of candidates tested,
not at the same wall clock time.)
* When cracking LM hashes, don't store the ASCII encodings of the hashes
in memory, but instead reconstruct them from the binary hashes for
writing into john.pot when a password gets cracked.
This saves some memory, and the same functionality is also usable for
other saltless hashes in jumbo.
* With 32-bit x86 builds and at least MMX enabled, the "two hashes at a
time" code for bcrypt is now enabled for GCC 4.2 and newer. This change
is made based on benchmark results for different builds made with
different versions of GCC on CPUs ranging from Pentium 3 to Core i7.
Unfortunately, there's a known performance regression with this change
on Atom. Previously, this code was only enabled for x86-64 and/or
OpenMP-enabled builds.
This is faster bcrypt cracking on some old and new computers running
32-bit operating systems or VMs for whatever reason.
* The formats interface has been enhanced to better support GPU
implementations (in jumbo), as well as fast hashes on multi-CPU systems
(not yet made use of).
1.8.0 was a good time for revising the formats interface, which we'll
make use of in jumbo shortly.
* Assorted minor corrections to Cygwin builds were made.
These are essentially the same corrections that were included in Windows
builds of 1.7.9 (but were not yet included in 1.7.9 proper).
* Fixed a bug in the Keyboard external mode (uninitialized variables on
"--restore" or when minlength is greater than 1).
* Enhanced the generic crypt(3) format to handle possible NULL returns
from crypt() and crypt_r().
Oops. These are fixed now.
* Assorted other changes have been made.
OK, that was enough for an announcement. Feedback is welcome on the
john-users list, as usual.
I'd like to thank Rapid7 and their Magnificent7 program for enabling me
to dedicate more time to implementing many of these changes and bringing
them to release quality.
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
