[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20210218191618.GA15226@openwall.com>
Date: Thu, 18 Feb 2021 20:16:18 +0100
From: Solar Designer <solar@...nwall.com>
To: announce@...ts.openwall.com, passwdqc-users@...ts.openwall.com
Subject: passwdqc 2.0.0
Hi,
This is to announce passwdqc 2.0.0, a new version of our
password/passphrase strength checking and enforcement tool set:
https://www.openwall.com/passwdqc/
There's also a corresponding update of passwdqc for Windows:
https://www.openwall.com/passwdqc/windows/
The upstream repository for passwdqc has recently been moved to Git and
onto GitHub, and all of the changes since the previously announced
release (1.4.0 in December 2019) have been made in Git:
https://github.com/openwall/passwdqc
The major change, corresponding to the major version number update, is
addition of support for external wordlist, denylist, and binary filter
files. With these, passwdqc can be configured to deny passwords and
passphrases that are based on lines of a tiny external text file (the
"wordlist" option), directly appear in a tiny external text file (the
"denylist" option), or/and directly appear in a maybe huge binary
filter file (the "filter" option).
While usage of larger external text files is inefficient, the binary
filters are very efficient.
Under the hood passwdqc binary filters are improved cuckoo filters, a
modern probabilistic data structure. They never produce false negatives
(that is, all passwords that are meant to be denied will always be).
They occasionally produce false positives (that is, a previously unseen
password might be unintentionally denied), but with our settings and
algorithm this probability is negligible (fewer than 1 in a billion).
Looking up a password against a filter requires at most two tiny random
disk reads, which is very quick and lightweight for the server.
The binary filters can be created and otherwise managed with the newly
added pwqfilter(1) program. It can create a binary filter from a list
of plaintexts or from MD4 or NTLM hashes. The latter are supported in a
way that enables importing of HIBP (Pwned Passwords) database revisions
into passwdqc binary filters. A lot of effort went into optimizing
pwqfilter for speed, compactness of the resulting filters, and lower
false positive rate. For example, creating a 98.00% load factor cuckoo
filter from the pwned-passwords-ntlm-ordered-by-hash-v7.txt file, which
is 21 GiB (22 GB) and contains 613+ million lines, takes around 8
minutes on i7-4770K with 32 GiB RAM. The resulting binary filter file
is 2.3 GiB (2.5 GB), and (with the right optimization option to
pwqfilter) it has a false positive rate of around 1 in 1.15 billion.
With a lower target load factor, the file can be created much quicker
and have an even lower false positive rate, but it would then be larger.
As you can see from the website, we also use this as an opportunity to
sell pre-generated binary filter files. Given that the pwqfilter
program is free, users have a choice between generating filters on their
own and purchasing ours - or maybe both, for different input data sets.
(If the use of binary filters is desired in a given environment at all.)
pwqfilter works on arbitrary plain text strings or hex-encoded hashes,
and it can also be reused in lieu of grep(1) for many purposes, even
unrelated to passphrases and security. Considering such possible reuse
and similarity to grep, pwqfilter includes several command-line options
that match grep's, doesn't use command-line option names that would
conflict with grep's, and its exit codes are compatible with grep's.
For applications where false positives would be highly undesirable,
much lower false positive rates like 1 in a billion billion can be
achieved by using much lower target load factors like those below 44%.
(A classic cuckoo filter wouldn't achieve that much improvement in false
positive rate at lower load, but ours is adaptive. Might be worth
writing an Openwall article on.)
To separate the major change above from other changes, a few
intermediate releases have also been made, without them having been
announced separately. I'll summarize other significant changes here:
Changes since 1.5.0 to 2.0.0:
Added support for external wordlist, denylist, and binary filter.
(Described above in much detail, won't repeat here.)
Merged changes needed for building with Visual Studio on Windows. This
includes a refactoring of the random passphrase generator code to make
it shared between platforms. In other words, the main passwdqc project,
which is free software, now includes changes (freely licensed, indeed)
that had been made to its source files in the non-free Windows product
(without hurting portability). This simplifies further maintenance for
us and might be handy for someone else wanting to reuse passwdqc source
code in a Visual Studio or another native Windows project (maybe even
competing with ours).
Changes since 1.4.1 to 1.5.0:
Updated the included wordlist to avoid some inappropriate words in
randomly generated passphrases while not removing any words from the
"word-based" check, and also to have plenty of extra words for
subsequent removal of more words that might be considered inappropriate
from the initial 4096 that are used for randomly generated passphrases.
Most of the added words came from EFF Diceware, BIP-0039, and our own
processing of Project Gutenberg Australia books. (This change has
already resulted in constructive discussion and a likely upcoming
contribution on the very first GitHub issue. Arguably, without a prior
change like this our project was not ready for GitHub: we'd be getting
pull requests removing words from the list of exactly 4096, which we'd
have to be closing without action, causing frustration for everyone.)
Changes since 1.4.0 to 1.4.1:
Set default for "max" to 72 (was 40). The previous setting was based on
a reading of RFC 1939, which in practice did not matter. The new one is
based on bcrypt's truncation at 72, which actually still matters.
Documented "similar" in pwqcheck(1) help message and manual page. This
is a setting that was supported before and documented for other passwdqc
components before, but was apparently erroneously omitted from here.
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
