[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20120109034819.GA2822@openwall.com>
Date: Mon, 9 Jan 2012 07:48:19 +0400
From: Solar Designer <solar@...nwall.com>
To: Alexey Dokuchaev <danfe@....ru>
Cc: john-users@...ts.openwall.com
Subject: Re: 1.7.9-jumbo5 fails to build with OMPFLAGS = -fopenmp -msse2
On Mon, Jan 09, 2012 at 04:53:17AM +0700, Alexey Dokuchaev wrote:
> On Mon, Jan 09, 2012 at 12:41:21AM +0400, Solar Designer wrote:
> > On Mon, Jan 09, 2012 at 01:56:29AM +0700, Alexey Dokuchaev wrote:
> > Well, there are a lot of things I'd change in the port. In arbitrary order:
> >
> > 1. Offer non-jumbo builds. I read this comment:
> >
> > # We prefer -jumbo (community-enhanced) versions instead of vanilla releases
> > # due to the sad fact that `-jumbo-???' suffixed versions *precede* official
> > # ones per pkg_version(1) -t check. [...]
> >
> > but I don't see how the version precedence is relevant.
>
> It is relevant in that sense that if vanilla version would be committed
> upon release, than corresponding -jumbo one would be considered by the
> ports framework as older one.
If you treat vanilla and jumbo as just two versions on the same branch,
then indeed one will be considered newer and the other older, which is
wrong no matter which one is "newer" and which one is "older". They're
just different. Ideally, they should be treated as different branches,
or it can be a build setting.
> Frankly speaking, I've always preferred to
> work with two distfiles: vanilla + jumbo.diff, but AFAIR I had to switch
> to single distfile circa john-1.7.7-jumbo-6 as I've could not locate the
> diff anymore: the only two options were either the vanilla one or already
> patched -jumbo. Apologies in advance if it was just mistake on my behalf.
-jumbo remained available as a patch as well:
http://www.openwall.com/john/#contrib
http://download.openwall.net/pub/projects/john/1.7.9/
It's just that the pre-created tarball is so easy to find now that few
people look any further.
> > Regardless of which kind of versions take precedence, it is a FreeBSD
> > system admin's decision which kind of versions to use. Someone will
> > prefer non-jumbo (less functionality, but fewer bugs and more consistent
> > documentation), someone else will prefer -jumbo.
> >
> > I think it should be a with/without setting, just like with OpenMP.
> > You may have it default to including -jumbo.
>
> Fair enough. If -jumbo diff would be consistently available, I would
> happily provide JUMBO option and utilize our handy PATCH_SITES/PATCHFILES
> feature.
Yes, please do. I have no intent to stop producing -jumbo .diff's any
time soon.
> > 2. Stop moving john.conf to /etc. You're not doing it for other *.conf
> > files from -jumbo anyway (the include files). This will also eliminate
> > the need to patch params.h.
>
> This was done long before I took over maintainership of the port, and I
> simply did not want to break POLA. It seems that it was done as most
> users expect some kind of default config be put in $localbase/etc; hence
> the copying. What would you suggest instead? Please consider that we try
> to make sure our ports are more-or-less usable (work) out of the box.
The package will indeed work out of the box with all of these files
installed in JOHN_SYSTEMWIDE_HOME.
> > 4. sparc64 builds are disabled - why? They should work just fine. If
> > not, please report that as a bug (with proper detail).
>
> Unfortunately, this is the problem we have here at FreeBSD. Our build
> cluster has sparc64/ppc boxes in order to generate packages for these arch
> releases, however, no sparc64 nor ppc machines are available for regular
> developers to log in and test their ports (I am not even talking about
> getting virtualized root access). Quick search over
> http://pointyhat.freebsd.org/errorlogs/ did not reveal the JtR's log.
> I will try to find a sparc64 box and see what's going on there.
Understood. I actually have a couple of sparc64 machines, but I haven't
powered them up in a long time and neither has FreeBSD installed. One
of them has NetBSD/sparc64, though, and last time I checked John and
even -jumbo built there just fine (this was before 1.7.9, though). The
1.7.9 release was also tested on Solaris/sparc64.
> > 5. Stop replacing -O2 with ${CFLAGS}. I dare to do something like this
> > in the Owl package, but I always re-test (on multiple systems) and
> > reconsider when we update gcc in Owl. Clearly, no one does that in
> > FreeBSD (for John). JtR's Makefile is aware of which gcc options work
> > best. Besides, you're not overriding -Os in OPT_INLINE anyway, so
> > things are currently inconsistent.
>
> Our bylaws dictate that we should uphold users' choice of CFLAGS and CC,
> which explains why we patch it. As John's author, I suppose you indeed
> know what you're doing so I can make an exception here, given that John's
> nature is a bit special when in comes to compilers and optimizations.
Yes, I think it should be one of the exceptions. I imagine that you're
making a similar exception e.g. for mplayer (I think you should be).
> > 7. Why remove the _XOPEN_SOURCE from recovery.c? Did things not build
> > otherwise? Please report this as a bug if so.
>
> Yes, it fails to build. I believe it's pretty trivial to reproduce; just
> comment that line out and try to "make" it.
Sure, but I was hoping that you'd just post those compiler error messages.
> > 11. You're installing the *.pl, *.rb, and *.py scripts into DATADIR,
> > along with the various files that JtR may use at runtime. I suggest
> > that you install those scripts along with the documentation instead.
> >
> > Thus, I think that DATADIR should contain: *.conf (including the main
> > john.conf), *.chr, password.lst. But nothing else.
> >
> > un* and *2john symlinks to john should be in the main bin directory
> > (where "john" itself is).
> >
> > The various scripts should be in the same directory with the
> > documentation files.
>
> I will think it over. What I can say right now that DATADIR seemed like a
> better choice to me vs. DOCSDIR as scripts are not really documentation
> from normal user's perspective.
Frankly, I think the bin directory would be the right choice, but those
scripts are a bit too numerous and they're named in different ways
(e.g., if all of their names started with "john", it would make more
sense to just have them installed along with the main "john" binary).
Also, the mailer script assumes that it will be customized before use.
> > 12. You could introduce AVX and XOP builds. These are faster on CPUs
> > introduced in 2011. However, I do not know what the status of FreeBSD's
> > support for AVX is - do you? This requires recent gcc, binutils, and in
> > some cases also support for XSAVE in the kernel. (My understanding is
> > that JtR currently only depends on OSXSAVE in 32-bit builds with AVX,
> > but not in -x86-64-avx and -x86-64-xop builds. That's because it only
> > uses 256-bit AVX vectors in 32-bit builds. However, JtR 1.7.9 checks
> > for OSXSAVE in its CPU detection code enabled in any -avx and -xop
> > builds, including in -x86-64-avx and -x86-64-xop builds. This may be
> > something for me to relax in a future version.)
>
> It's on my list, but as you probably know, there are certain problems with
> putting recent GCC and binutils in the FreeBSD base due to GPLv3.
Yes, I am just unaware of the status re: AVX support specifically -
e.g., you might have newer gcc and binutils available in ports, and the
FreeBSD kernel might already include XSAVE support. If so, JtR would be
buildable and usable with AVX and XOP (on proper CPUs) after installing
those ports (and by having the build invoke those alternate build
tools). I am just guessing, though.
> Awesome. Thanks a bunch for the feedback, I will send you a diff for
> review once I wrap it up.
Sounds good. Thank you for your work on the port!
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
