[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <20181216180412.GA6704@openwall.com>
Date: Sun, 16 Dec 2018 19:04:12 +0100
From: Solar Designer <solar@...nwall.com>
To: john-dev@...ts.openwall.com
Cc: Claudio Andr?? <claudioandre.br@...il.com>
Subject: Re: Anyone knows what does this fuzzer message mean?
On Sun, Dec 16, 2018 at 06:50:05PM +0100, Solar Designer wrote:
> On Sat, Dec 15, 2018 at 10:08:25PM -0200, Claudio Andr?? wrote:
> > Warning: excessive partial hash collisions detected
> The message comes from the loader when it encouters too many (by
> default, more than 1000) same binary_hash()es, which slows down its dupe
> hash check and is likely to slow down actual cracking as well (if the
> same or a similar binary_hash*() is used within a salt, and the
> collisions are seen within the salt as well). During fuzzing, this is
> normal, because the fuzzer alters our test vector hashes in many simple
> ways, resulting in a set of fake hashes that are not random-looking.
> I suggest we suppress these warnings with the following patch:
>
> diff --git a/src/john.c b/src/john.c
> index c7181c8..fd15f9e 100644
> --- a/src/john.c
> +++ b/src/john.c
> @@ -1694,6 +1694,13 @@ static void john_run(void)
> #ifdef HAVE_FUZZ
> else
> if (options.flags & FLG_FUZZ_CHK || options.flags & FLG_FUZZ_DUMP_CHK) {
> +/*
> + * Suppress dupe hash check because fuzzed ones often result in too many
> + * partial hash collisions.
> + */
> + options.loader.flags |= DB_WORDS;
> + list_init(&single_seed); /* Required for DB_WORDS */
> +
> ldr_init_database(&database, &options.loader);
> exit_status = fuzz(&database);
> }
>
> Alternatively, we could export the skip_dupe_checking flag (currently
> local to a function in loader.c) and set it from fuzz_test().
>
> Or we could set a flag in the fuzzer and check it from the loader.
Thinking of this some more, it would actually be helpful if the fuzzer
let us reliably discover formats that have non-working binary_hash*().
So as a separate task we might want to add a function to the fuzzer
where it'd generate random-looking hashes (also based on the test
vectors, but altering them in ways that are unlikely to result in same
binary_hash*() values?), and we wouldn't suppress the loader's dupe
check for those - in fact, we could lower the threshold for that test.
The tricky part is how to alter arbitrary hash strings in ways that are
unlikely to result in same binary_hash*() values, yet keep many of those
strings valid for the loader. If we e.g. replace all chars in base64
charsets with different ones (so that all parts of the hash would
change), we'd also be likely to end up replacing part of the hash id
prefix string, so the loader wouldn't process those hashes. Maybe do it
only to the portion on the right of the last '$' or '*'?
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
