[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20120108215317.GA90289@regency.nsu.ru>
Date: Mon, 9 Jan 2012 04:53:17 +0700
From: Alexey Dokuchaev <danfe@....ru>
To: Solar Designer <solar@...nwall.com>
Cc: john-users@...ts.openwall.com
Subject: Re: 1.7.9-jumbo5 fails to build with OMPFLAGS = -fopenmp -msse2
On Mon, Jan 09, 2012 at 12:41:21AM +0400, Solar Designer wrote:
> On Mon, Jan 09, 2012 at 01:56:29AM +0700, Alexey Dokuchaev wrote:
> Well, there are a lot of things I'd change in the port. In arbitrary order:
>
> 1. Offer non-jumbo builds. I read this comment:
>
> # We prefer -jumbo (community-enhanced) versions instead of vanilla releases
> # due to the sad fact that `-jumbo-???' suffixed versions *precede* official
> # ones per pkg_version(1) -t check. [...]
>
> but I don't see how the version precedence is relevant.
It is relevant in that sense that if vanilla version would be committed
upon release, than corresponding -jumbo one would be considered by the
ports framework as older one. Frankly speaking, I've always preferred to
work with two distfiles: vanilla + jumbo.diff, but AFAIR I had to switch
to single distfile circa john-1.7.7-jumbo-6 as I've could not locate the
diff anymore: the only two options were either the vanilla one or already
patched -jumbo. Apologies in advance if it was just mistake on my behalf.
> Regardless of which kind of versions take precedence, it is a FreeBSD
> system admin's decision which kind of versions to use. Someone will
> prefer non-jumbo (less functionality, but fewer bugs and more consistent
> documentation), someone else will prefer -jumbo.
>
> I think it should be a with/without setting, just like with OpenMP.
> You may have it default to including -jumbo.
Fair enough. If -jumbo diff would be consistently available, I would
happily provide JUMBO option and utilize our handy PATCH_SITES/PATCHFILES
feature.
> 2. Stop moving john.conf to /etc. You're not doing it for other *.conf
> files from -jumbo anyway (the include files). This will also eliminate
> the need to patch params.h.
This was done long before I took over maintainership of the port, and I
simply did not want to break POLA. It seems that it was done as most
users expect some kind of default config be put in $localbase/etc; hence
the copying. What would you suggest instead? Please consider that we try
to make sure our ports are more-or-less usable (work) out of the box.
> 3. The list of documentation files in PORTDOCS is incomplete (does not
> include those added in -jumbo). You need to have it complete for
> whatever version of JtR is being built. Better yet, just have the port
> install everything from doc/ - is there a way to do that?
Thanks, good catch. Yes, there is a way. I will review and take care of
this.
> 4. sparc64 builds are disabled - why? They should work just fine. If
> not, please report that as a bug (with proper detail).
Unfortunately, this is the problem we have here at FreeBSD. Our build
cluster has sparc64/ppc boxes in order to generate packages for these arch
releases, however, no sparc64 nor ppc machines are available for regular
developers to log in and test their ports (I am not even talking about
getting virtualized root access). Quick search over
http://pointyhat.freebsd.org/errorlogs/ did not reveal the JtR's log.
I will try to find a sparc64 box and see what's going on there.
> 5. Stop replacing -O2 with ${CFLAGS}. I dare to do something like this
> in the Owl package, but I always re-test (on multiple systems) and
> reconsider when we update gcc in Owl. Clearly, no one does that in
> FreeBSD (for John). JtR's Makefile is aware of which gcc options work
> best. Besides, you're not overriding -Os in OPT_INLINE anyway, so
> things are currently inconsistent.
Our bylaws dictate that we should uphold users' choice of CFLAGS and CC,
which explains why we patch it. As John's author, I suppose you indeed
know what you're doing so I can make an exception here, given that John's
nature is a bit special when in comes to compilers and optimizations.
> 7. Why remove the _XOPEN_SOURCE from recovery.c? Did things not build
> otherwise? Please report this as a bug if so.
Yes, it fails to build. I believe it's pretty trivial to reproduce; just
comment that line out and try to "make" it.
> 8. "xtract" hasn't been part of JtR for more than a decade; thus,
> you're probably installing a meaningless symlink for it.
>
> 9. You're not installing many other symlinks: unafs, unique, other un*
> and *2john programs (symlinks to john) added in -jumbo. Please fix that.
Thanks, will do.
> 10. Please install the relbench script. Maybe along with documentation.
I do as of most recent commit (distinfo r1.15).
> 11. You're installing the *.pl, *.rb, and *.py scripts into DATADIR,
> along with the various files that JtR may use at runtime. I suggest
> that you install those scripts along with the documentation instead.
>
> Thus, I think that DATADIR should contain: *.conf (including the main
> john.conf), *.chr, password.lst. But nothing else.
>
> un* and *2john symlinks to john should be in the main bin directory
> (where "john" itself is).
>
> The various scripts should be in the same directory with the
> documentation files.
I will think it over. What I can say right now that DATADIR seemed like a
better choice to me vs. DOCSDIR as scripts are not really documentation
from normal user's perspective.
> 12. You could introduce AVX and XOP builds. These are faster on CPUs
> introduced in 2011. However, I do not know what the status of FreeBSD's
> support for AVX is - do you? This requires recent gcc, binutils, and in
> some cases also support for XSAVE in the kernel. (My understanding is
> that JtR currently only depends on OSXSAVE in 32-bit builds with AVX,
> but not in -x86-64-avx and -x86-64-xop builds. That's because it only
> uses 256-bit AVX vectors in 32-bit builds. However, JtR 1.7.9 checks
> for OSXSAVE in its CPU detection code enabled in any -avx and -xop
> builds, including in -x86-64-avx and -x86-64-xop builds. This may be
> something for me to relax in a future version.)
It's on my list, but as you probably know, there are certain problems with
putting recent GCC and binutils in the FreeBSD base due to GPLv3.
> 13. You could introduce binary fallbacks - that is, build multiple
> binaries and have the more-demanding ones invoke fallback ones until
> their demands for CPU capabilities are met and OpenMP support is not
> being wasted (it has some overhead, so is better disabled when only one
> thread would be run). Owl's package has these: [...]
Also on my list, I've seen your john.spec (exactly via the link you post
below). :-)
> The biggest drawback is that this stuff gets very complicated and hard
> to fully test. You can see this implemented here:
>
> http://cvsweb.openwall.com/cgi/cvsweb.cgi/Owl/packages/john/john.spec?rev=HEAD
>
> I am considering introducing a script into the JtR tree itself to avoid
> such code duplication between packages of JtR.
Awesome. Thanks a bunch for the feedback, I will send you a diff for
review once I wrap it up.
./danfe
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
