[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20150820213138.GA29462@openwall.com>
Date: Fri, 21 Aug 2015 00:31:38 +0300
From: Solar Designer <solar@...nwall.com>
To: john-dev@...ts.openwall.com
Subject: Re: BENCHMARK_LENGTH bugs
On Thu, Aug 20, 2015 at 07:30:17PM +0200, magnum wrote:
> On 2015-08-20 19:13, Kai Zhao wrote:
> >I create a patch for benchmark_length:
> >
> >https://github.com/loverszhaokai/JohnTheRipper/commit/810a62c938d86a724f5398d312bbb1c5736a1147
>
> >Testing: 7z, 7-Zip (512K iterations) [SHA256 AES 32/64]... (8xOMP)
> >FAILED (This format is very slow, but it will report separate "Many
> >salts" vs. "Only one salt" speeds)
>
> Your patch has a hard-coded list of "slow hashes". That's not good from
> any point of view.
I agree that this kind of a hard-coded list isn't helpful. If these are
the only formats for which warnings may be printed, then we could as
well just review them manually and correct whatever wrong settings there
might be.
> As seen in the above figures, you should do calculate
> a figure many/one like this:
>
> >$ ../john --test --format=7z
> >
> >Will run 8 OpenMP threads
> >Benchmarking: 7z, 7-Zip (512K iterations) [SHA256 AES 32/64]... (8xOMP)
> >DONE
> >Speed for cost 1 (iteration count) of 524288
> >Many salts: 3056 c/s real, 3034 c/s virtual
> >Only one salt: 18.0 c/s real, 18.1 c/s virtual
>
> 3056/18 = 169.78x faster. Definitely nothing to complain about.
>
> >$ ../john --test --format=dominosec8
> >
> >Will run 8 OpenMP threads
> >Benchmarking: dominosec8, Lotus Notes/Domino 8 [8/64]... (8xOMP) DONE
> >Warning: "Many salts" test limited: 1/256
> >Many salts: 3471 c/s real, 435 c/s virtual
> >Only one salt: 3303 c/s real, 416 c/s virtual
>
> 3471/3303 = 1.05x so a 5% speedup. I'm not sure we should complain even
> here.
>
> I think you should complain if speedup is less than 1-2%. Also, I'm not
> sure you should FAIL but maybe just "Warning: ..." like with alignment
> issues. Solar?
The threshold should be way below 1%. We do want to see separate
results if they are expected to differ by 1% or more.
Unfortunately, this prevents reliable automatic detection. It is
possible that on a given run results just happen to differ by 2% even if
there's no good reason for them to differ (a slight change in system
load may cause this), or on the contrary are almost the same (difference
below 1%) even if they are expected to differ by a few percent.
Yet we could try detecting and warning about Many / One salt results
that are too similar (difference less than 1%).
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
