[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20120108204121.GA1590@openwall.com>
Date: Mon, 9 Jan 2012 00:41:21 +0400
From: Solar Designer <solar@...nwall.com>
To: Alexey Dokuchaev <danfe@....ru>
Cc: john-users@...ts.openwall.com
Subject: Re: 1.7.9-jumbo5 fails to build with OMPFLAGS = -fopenmp -msse2
On Mon, Jan 09, 2012 at 01:56:29AM +0700, Alexey Dokuchaev wrote:
> On Sun, Jan 08, 2012 at 09:25:16PM +0400, Solar Designer wrote:
> > Did you get the FreeBSD port fixed? The best fix would be to get it to
> > stop patching the Makefile - I doubt that it actually needs to do that.
>
> Yes, I've just committed update to 1.7.9-jumbo-5 (I currently maintain the
> port). Also I indeed "patch" src/Makefile with sed(1) to uncomment relevant
> OMPFLAGS; I will see if I can pass it via MAKE_ARGS instead. I know that
> you're generally against patching makefiles and param.h and fully support
> you here. Maybe you can take a look at what it currently looks like and
> suggest what can/should be improved.
Oh, I did not realize you're the port maintainer now.
Well, there are a lot of things I'd change in the port. In arbitrary order:
1. Offer non-jumbo builds. I read this comment:
# We prefer -jumbo (community-enhanced) versions instead of vanilla releases
# due to the sad fact that `-jumbo-???' suffixed versions *precede* official
# ones per pkg_version(1) -t check. So, when Alexander rolls out new "plain"
# release, just hold the horses and wait for corresponding -jumbo one, which
# usually does not take too long to appear.
but I don't see how the version precedence is relevant. Regardless of
which kind of versions take precedence, it is a FreeBSD system admin's
decision which kind of versions to use. Someone will prefer non-jumbo
(less functionality, but fewer bugs and more consistent documentation),
someone else will prefer -jumbo.
I think it should be a with/without setting, just like with OpenMP.
You may have it default to including -jumbo.
2. Stop moving john.conf to /etc. You're not doing it for other *.conf
files from -jumbo anyway (the include files). This will also eliminate
the need to patch params.h.
3. The list of documentation files in PORTDOCS is incomplete (does not
include those added in -jumbo). You need to have it complete for
whatever version of JtR is being built. Better yet, just have the port
install everything from doc/ - is there a way to do that?
4. sparc64 builds are disabled - why? They should work just fine. If
not, please report that as a bug (with proper detail).
5. Stop replacing -O2 with ${CFLAGS}. I dare to do something like this
in the Owl package, but I always re-test (on multiple systems) and
reconsider when we update gcc in Owl. Clearly, no one does that in
FreeBSD (for John). JtR's Makefile is aware of which gcc options work
best. Besides, you're not overriding -Os in OPT_INLINE anyway, so
things are currently inconsistent.
6. For OMPFLAGS - yes, you may just pass them via make's command-line.
Owl's john.spec does. No need to patch the Makefile.
7. Why remove the _XOPEN_SOURCE from recovery.c? Did things not build
otherwise? Please report this as a bug if so.
8. "xtract" hasn't been part of JtR for more than a decade; thus,
you're probably installing a meaningless symlink for it.
9. You're not installing many other symlinks: unafs, unique, other un*
and *2john programs (symlinks to john) added in -jumbo. Please fix that.
10. Please install the relbench script. Maybe along with documentation.
11. You're installing the *.pl, *.rb, and *.py scripts into DATADIR,
along with the various files that JtR may use at runtime. I suggest
that you install those scripts along with the documentation instead.
Thus, I think that DATADIR should contain: *.conf (including the main
john.conf), *.chr, password.lst. But nothing else.
un* and *2john symlinks to john should be in the main bin directory
(where "john" itself is).
The various scripts should be in the same directory with the
documentation files.
12. You could introduce AVX and XOP builds. These are faster on CPUs
introduced in 2011. However, I do not know what the status of FreeBSD's
support for AVX is - do you? This requires recent gcc, binutils, and in
some cases also support for XSAVE in the kernel. (My understanding is
that JtR currently only depends on OSXSAVE in 32-bit builds with AVX,
but not in -x86-64-avx and -x86-64-xop builds. That's because it only
uses 256-bit AVX vectors in 32-bit builds. However, JtR 1.7.9 checks
for OSXSAVE in its CPU detection code enabled in any -avx and -xop
builds, including in -x86-64-avx and -x86-64-xop builds. This may be
something for me to relax in a future version.)
13. You could introduce binary fallbacks - that is, build multiple
binaries and have the more-demanding ones invoke fallback ones until
their demands for CPU capabilities are met and OpenMP support is not
being wasted (it has some overhead, so is better disabled when only one
thread would be run). Owl's package has these:
On 32-bit x86: XOP -> AVX -> SSE2 -> MMX -> any
On x86-64: XOP -> AVX -> SSE2 (implied for x86-64)
...and there are also OpenMP to non-OpenMP runtime fallbacks.
With these, you wouldn't need to have the user (FreeBSD system admin)
choose CPU type and OpenMP or not.
The biggest drawback is that this stuff gets very complicated and hard
to fully test. You can see this implemented here:
http://cvsweb.openwall.com/cgi/cvsweb.cgi/Owl/packages/john/john.spec?rev=HEAD
I am considering introducing a script into the JtR tree itself to avoid
such code duplication between packages of JtR.
Thanks,
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
