john-dev - Re: [GSoC] building JtR for MIC

Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-Id: <47235F75-EE55-4CAC-8A54-DF59F93F4DBE@gmail.com>
Date: Tue, 24 Mar 2015 20:07:37 +0800
From: Lei Zhang <zhanglei.april@...il.com>
To: john-dev@...ts.openwall.com
Subject: Re: [GSoC] building JtR for MIC

Hi,

I was trying to figure out why some of the benchmark tests of john-jumbo FAILED on MIC, and found something probably worth discussing here.

I focused on one of the failing test, GPG. When I trace its execution using gdb, I found that at one point a invocation of SHA1_Update() produces a different value on MIC than on the host. I simplified the code to reproduce the abnormal behavior:
----------------------------------------------------
#include <stdio.h>
#include <openssl/sha.h>
#define BUFLEN 64

int main() {
	SHA_CTX ctx;
	SHA1_Init(&ctx);

	char keybuf[BUFLEN];
	FILE *file = fopen("dump", "rb");
	fread(keybuf, 1, BUFLEN, file);

	SHA1_Update(&ctx, keybuf, BUFLEN);
	printf("%lu\n", ctx.h0);
}
---------------------------------------------------- 
The input file used in the code is attached below.

The code above prints 748926355 when running on the host, but prints 2625354361 on MIC. There's seems to be something wrong with the cross-compiled OpenSSL, but I'm not sure yet. 

Maybe you have some insights on this?


Thanks,
Lei


Download attachment "dump" of type "application/octet-stream" (64 bytes)


> On Mar 18, 2015, at 6:38 PM, Solar Designer <solar@...nwall.com> wrote:
> 
> On Wed, Mar 18, 2015 at 05:27:45PM +0800, Lei Zhang wrote:
>> I tweaked with OMP_SCALE in several formats, according to magnum's advices, and their memory usage on MIC is much more reasonable now. With OpenMP enabled, john-jumbo won't abort anymore when running the benchmark, although 126 of the tests still FAILED.
> 
> That's a huge number.  How many of the failed tests are for dynamic*
> formats?  Perhaps there's one or a handful of bugs that are causing most
> of the failures.
> 
> I notice that in jumbo my original mic.h is modified to use autoconf'ed
> ARCH_* settings:
> 
> #if AC_BUILT
> #include "autoconfig.h"
> #else
> #define ARCH_WORD                       long
> #define ARCH_SIZE                       8
> #define ARCH_BITS                       64
> #define ARCH_BITS_LOG                   6
> #define ARCH_BITS_STR                   "64"
> #define ARCH_LITTLE_ENDIAN              1
> #define ARCH_INT_GT_32                  0
> #endif
> 
> This might be wrong since we're cross-compiling.  I doubt it's causing
> trouble now, though, since the host system is almost certainly x86_64,
> for which these settings just happen to be the same.
> 
>> Next I want to figure out how those tests failed, but I have a few questions first: (forgive me if they look stupid...)
>> 1. What exactly does it mean for a format to FAIL the test?
> 
> See formats.c: fmt_self_test().  It's usually indicated after a call to
> which of the format methods the test failed, but you'll most commonly
> see get_hash[0](0) or something, which means that the actual failure was
> likely in a preceding crypt_all() or set_key() or set_salt().  The test
> vectors are in the tests[] array in each format's *_fmt*.c file.
> In some cases, it's possible to narrow the problem down by temporarily
> commenting out some of the test vectors.  But usually not.
> 
>> 2. There're two kind of numbers in the output, real and virtual. What's their difference?
> 
> This is in the FAQ:
> 
> Q: What are the "real" and "virtual" c/s rates as reported by "--test"
> (on Unix-like operating systems)?
> A: These correspond to real and virtual (processor) time, respectively.
> The two results would differ when the system is under other load, with
> the "virtual" c/s rate indicating roughly what you could expect to get
> from the same machine if it were not loaded.
> 
> ... but this FAQ entry is outdated and needs to be revised.  As written,
> it applies to single-threaded builds only (I wrote it before we
> introduced OpenMP support).  Clearly, the numbers also differ greatly in
> multi-threaded builds.  When running a multi-threaded build on an
> otherwise idle system, the "real" speed will be roughly equal to the
> "virtual" speed times the number of threads.
> 
> Alexander
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.