Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Tue, 5 Jul 2011 12:03:57 -0500
From: "JimF" <jfoug@....net>
To: <john-users@...ts.openwall.com>
Subject: Re: Charset options

Hi Maximilian,

First off, you should become aware, that all the charset mode does, is 
perform a full permutation of the characters contained within the set.  That 
is all it does.  It is similar to how multiple for loops would build a 
string of 'words' by looping through all of the valid characters.

Now, the charset mode is much 'smarter' than most.  It will generate 
candidates based upon statistical analysis of the data which was used to 
build the charset.  Thus, for the rockyou charset, the character patterns 
which make up the most frequent seen patterns in that wordlist will appear 
much earlier in the run.  However, 'all' of the character permutations would 
eventually be output by the incremental engine.

Now, you mentioned having passwords which are of an 'additional' pasword 
policy.  The way which this can be done, usually is with an external rule 
set.  The incremental engine will generate a word, and then that word is 
presented to the external rule filter.  If the rule is satisfied, then this 
word is 'used', and the hash is computed.  If the rule does not match, the 
word is not tested.     The assumption here, is that the external rule 
processor is faster than the hash being tested for (since it DOES take time 
to run the filter).  Now, if a word does not match the policy, it can not be 
a proper candidate.  But if the external filter takes longer to process than 
processing the word, then it may be better to simply blindly run the 
'incremental' code.  For instance, if this were a simple MD5, and you were 
running this in GPU mode (cracking 100's of millions per second), then you 
absolutely would not want to run the words through the external filter. 
However, if the passwords were for salted DES, then you certainly would want 
to filter out words which could NEVER be a proper candidate.

You mention a 'huge' performance hit.  Well, is it really huge?  If you are 
only processing 1/2 of the candidate words (or only 5%), then a 20% overall 
speed reduction is actually almost a 2x improvement (18x improvement if only 
5% of the words pass the filter).  These are the type of things you need to 
keep in mind.  Often, it is good to run a 6 char min / 6 char max test (or 
5), and see how long it takes both types to complete (with and without a 
filter).  If it completes faster with the filter, even though the speed 
'lists' to be slower, then your actual throughput is faster.

As for the warning about case, it sounds like you might be doing LM testing 
(or some other force upcase or force low case).  There is no 'way' to tell 
the charset (that I am aware of) to not use the a or b or c ... characters. 
Each of the characters is it's own beast, and has it's own rules about how 
likely it is, when following other characters.    If you want a data set 
that is only of specfic characters, then you have to create it.

If you are doing the rockyou and want to make an LM cracker, then I would 
recomend:

1. take rockyou, and chop it up into 7 character items. These are the first 
7 characters of the line (or as many as possible), and then the 2nd 7 
characters of the line.  NOTE, the newer unique will do this for you :) 
using the -cut=LM switch
2. take this new file (with the chopped lines of 7 chars), and perform 
uppercase conversion of all text (unique's -cut=LM has already done this for 
you).
3. uniq this file (again unique has done this)
4. Now build a charset file using this data.  Call it rockyou_lm.chr.  It 
should only be built for 7 char max, and the up case vs low case issue has 
been resolved.

That rockyou_lm.chr is the proper file to use  (again, I am 'assuming' that 
you are doing LM).

If this was some other format that is case insensitive, then you would need 
to perform the steps 1 to 4, but would have to change the 'logic' some. 
Possibly just upcase the file and strip any words with symbols (if the 
format only handles up case and numbers).

Jim.

----- Original Message ----- 
From: "Maximilian Melcher" <melcher.maximilian@...glemail.com>


> I have a small question: Is it possible to modify a charset like 
> rockyou.chr
> to stick to a password policy? E.g. password must be 8 characters long and
> must contain at least 1 letter (all upper or lower), 1 digit and 1 special
> sign ?
>
> I've searched the FAQ/wiki and tried it with the external filters but:
>
> 1. I experience a huge performance inpact (~20%). As far as I understand 
> the
> filtering is worth it because not that many words are being hashed but
> limiting the charset before the generation of the words would be better,
> right? But on the other hand I do not want to loose the presumption of the
> charset... Any solutions for that?
>
> 2. John still tells me "Warning: mixed-case charset, but the current hash
> type is case-insensitive; some candidate passwords may be unnecessarily
> tried more than once." - how to tell the word generator? :)

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.