[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20160908233612.GA2253@openwall.com>
Date: Fri, 9 Sep 2016 01:36:12 +0200
From: Solar Designer <solar@...nwall.com>
To: john-users@...ts.openwall.com
Subject: Re: possible memory leak on FreeBSD?
On Thu, Sep 08, 2016 at 08:42:38AM +0200, patpro@...pro.net wrote:
> I'm running JtR (JohnTheRipper-bleeding-jumbo 20160728) on FreeBSD 10.1-RELEASE, and I'm experiencing some nasty memory problem with some settings.
> I'm cracking huge passwords dump (10s of millions records), and my current pot file is about 4.3 GB. The server has 16 GB ram (but also runs other softwares).
>
> For example, --incremental will apparently very slowly consume memory on this server. I can't make really sure about this, but I can see the consumed swap size slowly increase overnight. Under normal usage, this server never swap a single bit.
> It becomes blatant when I use --fork=4 with --incremental: the memory is exhausted in about 10-30 minutes and swap piles up. If I don't kill john, the box ends up crashing (swap exhaustion on ZFS is not good). Oddly, top output does not show a real increase in john's memory usage while free memory on host is depleting.
> Same goes with --loopback --fork=4, even with a smaller pot file.
>
> Other attack modes like --wordlist are OK.
The way "--fork" works, there's initially a lot of data sharing between
the 4 processes, but the more passwords they crack, the less sharing
there remains. Thus, their combined memory usage will in fact increase
when John is running and is successfully cracking passwords. With
password hash counts like yours, such increases can easily be in the
gigabytes. My guess is that incremental mode was somehow more effective
at getting you more cracks (that were not already in john.pot) than
wordlist mode, or maybe you didn't use "--fork" with wordlist mode.
Given that you're close to bumping into your total RAM size, I recommend
that you get most passwords cracked when running without "--fork" (e.g.,
for a few hours or a day) and then re-add the "--fork=4" when the passwords
are no longer getting cracked this frequently. Unfortunately, there's
no easy way to continue a non-forked session with "--fork" added, so
some processing time will be lost, but at least you'll hopefully bypass
the issue you're running into now. (And it does sound like you need more
RAM to efficiently crack all of your passwords at once.)
You should also use "--save-memory=1" (but not higher), which might help
a little bit (but likely not enough, hence the above primary suggestion).
Incremental mode may also need more memory as it runs, but not much more.
Specifically, it defers allocation of per character position tables for
the maximum length until that length is actually reached. This is not a
leak, but just a deferred allocation, so that some runs can benefit from
lower memory usage. However, this allocation is on the order of 100 MB
(or four times that, for "--fork=4") and not gigabytes, so is probably
unrelated to what you're seeing.
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
