Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 12 Sep 2011 10:35:09 +0200
From: Pablo Fernandez <>
Subject: Re: External filter question


Thanks a lot for your answers!

> This sounds even worse than the [List.External:Parallel] sample, which
> is pretty bad on its own.  I recommend that you consider other approaches:

Actually, I have been making myself some numbers, and it turns out to be 
better than Parallel, given the same conditions: jobs will have a limited 
The reason for being better is because each job only "skips" until it's its 
turn, and then finishes. On the Parallel all jobs "skip" all the time. For 
example, with blocks, given you want to test the first 10k passwords with 10 
- First jobs doesn't skip. It computes 1k passwords and exits
- Second job skips 1k, computes 1k, exits
- Last job skips 9k, computes 1k, exits.
All in all, time spent skipping: 0+1+2+3+..+9 = 45k skips.
The same, with Parallel, 10k passwords, 10 jobs:
- All jobs compute 1k, and skip 9k.
All in all, time spent skipping: 10*9 = 90k skips.

Does this make sense?

> Is your reason to go for "blocks" instead of having each node try every
> Nth password (like [List.External:Parallel] does) to be able to retry
> skipped/failed blocks from a master node, and to adjust the number of
> nodes on the fly?  If so, this makes sense.  But having each node skip
> to its starting password number each time it's invoked on a relatively
> small block is going to kill performance unless you're running against
> salted and extremely slow hashes.

Indeed, yes. I wanted to make it variable-size blocks, depending on the 
"expected" free time to use in the cluster. And also make it variable number 
of compute nodes, you never know... so, as flexible as possible. No 
MPI/OpenMP, by design. And Markov is a bit too strict.

Anyway, Parallel or Block are not too bad, if you have slow hashes (with Linux 
- my target system - you barely see DES) and/or you have many salts.

> Worse, it will also not even find some passwords, because there's not
> only I/O buffering, but also crypto algorithm related buffering in JtR.

Is there a "safe" limit I can use? I could make the block to perform, let's 
say, 200 more hashes than it should (if would be less than a second) even if 
it overlaps with the next block. What I can't admit is to perform *less* than 
it should.

> We might introduce a way for filter() to ask for process termination in
> a later version.  In fact, maybe external mode functions should also be
> able to ask for the status line to be displayed (simulate keypress).

Those two would be awesome to have! Any idea on how to do it? Maybe I could do 
my work on a "patched" version, pretending to work also in a future release.

> > - Or maybe, is there anything I can do to force John to flush the IO
> > buffer before doing the word=1/0 operation?
> No, not from external mode, and it's not only about the I/O anyway.

I have one question here... what happens if you do a sigkill? Does it exit 
nicely (IO/crypt buffers)? Is there a way to send a message to an external 
application, so that it can send the sigkill back to john?

Thanks again,

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.