[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20120716113708.GA22139@openwall.com>
Date: Mon, 16 Jul 2012 15:37:08 +0400
From: Solar Designer <solar@...nwall.com>
To: john-dev@...ts.openwall.com
Subject: Re: problem with disc space for shared files in MJohn
Aleksey -
On Sun, Jul 15, 2012 at 03:24:18PM +0400, Aleksey Cherepanov wrote:
> I heard that some users have about 40gb of wordlists individually.
> Currently it would be a problem if MJohn would copy all files to the
> server.
I think those huge-wordlist runs should be separate from MJohn
currently, if MJohn is such that it'd require wordlist uploads (I don't
know whether and why it does, I haven't been watching the discussions
around it closely enough).
> Problems are
> 1) exhausted disk space,
> 2) exhausted traffic limit,
> 3) just slow.
>
> Hence the questions:
> Alexander, could contest server have more space?
A little bit more, yes. Say up to 100 GB total for the 2-day contest is
possible on the current server. It has 2x146 GB SAS disks in RAID-1.
It would be nice to be able to easily exclude unimportant yet large
files from backups. Can you have them placed in some subdirectories of
a fixed name (we typically use the name "nobackup" for this purpose)?
I think MJohn shouldn't be tied to a particular server and its capacity,
though. It should be a generally useful tool. I understand the need to
try it out on our current contest server in the upcoming contest, though.
> Would not we exhaust traffic limit?
There's no traffic limit on the current server, except for whatever the
100 Mbps switch port imposes.
> Possible improvements:
> - compress files (I expect wordlists to give good ratio),
> - drop git and use something much easier that allows to download one
> file instead of the full repository (I'll do that for other reasons
> too: for instance staging of files to check checksums could be done
> easier),
> - allow attacks with just sha1 instead of the real files: user does not
> upload files but shares sha1, so he could stop and restart this
> attack on his own cores while other users could check they do not
> run the same attacks, but for modification of attack users should ask
> the owner (also it is useful because not all users want to share their
> files),
> - use other file hosting and/or distributed system for that (many
> servers and/or torrent (though this needs our own tracker)).
The SHA-1 approach makes sense to me. Maybe replace actual files with
SHA-1s when a certain size threshold is exceeded (e.g., 100 MB).
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
