[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20240815133229.GA13222@openwall.com>
Date: Thu, 15 Aug 2024 15:32:29 +0200
From: Solar Designer <solar@...nwall.com>
To: john-users@...ts.openwall.com
Subject: Re: problem running out of memory - john 20240608.1
Hi Jeff,
On Wed, Aug 14, 2024 at 08:44:31PM -0700, jeff wrote:
> I have quite a lot of NTLM hashes. I have broken them up into files of
> about 1 gigabyte each.
> Please help me find a path forward. I have a total of about 30 gigabytes
> of NTLM hashes I wish to process.
It sounds like you're playing with HIBP / PwnedPasswords hashes.
One way to approach this problem is to not split that file. Try loading
them all at once, however don't use --fork at first. Maybe this will
fit in memory for you. As most hashes will get cracked after a while,
you'll be able to slowly increase the --fork count for further attacks
(but those should be different attacks).
I've just tested on Linux, and 100M of unique NTLM hashes take 11 GB RAM
with default settings, taking about 1 minute to load. So HIBPv8 847M
will probably fit in 128 GB RAM.
Since these hashes do not use salts, it is wasteful to crack them in
batches. Sure this enables you to use more CPU cores, but then you do
duplicate hash computations across the batches. OTOH, it is non-obvious
how this affects the overall progress, because at these counts hash
comparisons and not hash computation take most time, and these are
slightly faster at lower loaded hash counts. So you can try with and
without splitting.
> K:\password-cracking\john_bleeding_2024_0608.1\run>john.exe --format=LM
> --fork=60 \password-cracking\pwnedpasswords_ntlm.rawest.xaa
One issue here is you confused NTLM and LM. The correct option for NTLM
hashes is --format=NT.
> 3: Disabling duplicate candidate password suppressor
These hashes - LM and NTLM - are so fast that it's pointless to use the
duplicate candidate password suppressor (it is quicker to compute the
hashes than suppress occasional duplicates). This is getting detected
and the suppressor gets auto-disabled. You can save a little bit of
memory by disabling it explicitly. With specific cracking modes, you'd
do this on the command line with --dupe=0. With batch mode (which is
what you ran here), you'd have to edit this in john.conf:
# Opportunistic duplicate candidate password suppressor for wordlist+rules
[Options:Suppressor]
# Memory allocation size in MiB per process, 0 to disable unless forced.
# The default is 256 MiB, the maximum is just below 256 GiB (on 64-bit).
Size = 256
You can set it to 0, but you'll need to remember to revert this edit
when you attack slower hashes.
Batch mode isn't necessarily a good idea for these hashes. Its first
pass tries to derive passwords from per-user information such as login
names, but you probably don't have those. Looks like it had completed
almost instantly for you (as your status lines show you were already at
pass 2), yet it may have wasted some memory. So you can instead jump
right to wordlist with rules, e.g.:
john --format=nt --verbosity=1 --wordlist --rules --dupe=0 \password-cracking\pwnedpasswords_ntlm.rawest
Once this completes, you could similarly run incremental mode (would
have been batch mode's pass 3):
john --format=nt --verbosity=1 --incremental \password-cracking\pwnedpasswords_ntlm.rawest
... and maybe by this point you could already afford --fork=2, then
eventually more.
The decreased verbosity is so that your terminal window doesn't get
flooded with guesses, and so that this doesn't slow down progress.
Another thing you could try next is --format=nt-opencl --mask (then
adjust the mask, and use it along with other modes, see doc/MASK). This
will let you use all cores at once via OpenCL, so without --fork, but
you'd need to install an OpenCL "driver" (such as Intel's) and you'd be
limited to mask mode (including in combination with other modes) for
efficiency (other modes without mask are inefficient with nt-opencl).
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
