[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20130604234824.GB8374@openwall.com>
Date: Wed, 5 Jun 2013 03:48:24 +0400
From: Solar Designer <solar@...nwall.com>
To: john-users@...ts.openwall.com
Subject: Re: Phoronix Test Suite vs JtR
Lukas,
On Wed, Jun 05, 2013 at 12:28:24AM +0200, Lukas Odzioba wrote:
> when I saw this article:
> http://www.phoronix.com/scan.php?page=article&item=intel_4770k_linux&num=3
It is very nice for our project's visibility that Phoronix uses JtR, but
other than that people should be taking Phoronix's benchmark results
(not only JtR, but in general) with a grain of salt. I tried contacting
Michael on a few occasions to point out issues and better ways to use
JtR in Phoronix benchmarks, but I never heard back. I gave up.
> I asked about that author of this article and he kindly replied that
> results were not swapped.
Wow. You got a reply? This does not match my experience. By what
means did you contact Michael?
> Briefly I see some obvious flavs there:
Oh, there must be another one: somehow the gcc options given for the
"MD5" benchmark include "-fopenmp", but the actual speeds are clearly
for a single core. Something must have gone very wrong.
> -it uses 1.7.9-jumbo-7 but none of its features (for now it could be
> better to test on 1.8.0)
The core tree, including version 1.8.0, does not include SIMD support
for md5crypt, so using a jumbo is OK for this benchmark.
> -not using -format option during -test, so all formats are benchmarked
> but only 1 value is parsed from the results
Not a big deal. Moreover, this gives some systems time to switch to the
higher clock rate before the format we're interested in is benchmarked.
> -custom build is not a good idea - we have proper architecture
> dependent make targets
Sure. A reason to use custom builds is comparing C compilers at
different optimization levels, but unfortunately the resulting numbers
are misleading if (mis)interpreted as reflecting JtR performance for
actual use.
> -openssl deps should be avoided to make results consistent between
> platforms (not important in des,bf,md5crypt but someday someone might
> want to add another format)
Hopefully, Phoronix benchmarks only formats for which all of the crypto
code is in JtR itself. This has been the case so far.
> -MD5, Blowfish are our internal names what is beeing benchmarked are
> md5crypt and bcrypt.
Indeed. Hopefully, Phoronix Test Suite will be updated to a 1.8-based
jumbo eventually (we got to release one first!) These formats have been
renamed in 1.8.
> -maybe now we should use -fork instead of openmp
JtR 1.8 does not output cumulative speed numbers for --fork yet, and
using OpenMP is a better benchmark for C compilers' OpenMP support.
So I think that Phoronix benchmarking these formats with OpenMP is fine.
It is an improvement over their previous single-core benchmarks.
> I also ran this test on my 3770k with the following results:
> Blowfish 5817 Real C/S
> MD5 41761 Real C/S -
OK. Yes, they have a ~25% performance hit for md5crypt (even relative
to the expected single-core speeds), perhaps because of custom CFLAGS
and maybe also because of possible regressions with gcc 4.8 (and need
for tuning of the interleave factor by us for this gcc version, which we
have not done yet...)
> I believe people here know the best how to use JtR as a benchmark and
> we could help fix that.
Well, you may try figuring out and then pointing out to Michael the
issue with OpenMP somehow not taking effect in the md5crypt benchmark.
The many points you listed are not to be brought up, I think, except
that we may make the "no custom CFLAGS unless required for specific
reason" suggestion.
Once we have a 1.8-based jumbo, we may also inform Michael of this fact
and of the format renames.
Thanks,
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
