[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <56095123.6080907@openwall.net>
Date: Mon, 28 Sep 2015 09:39:31 -0500
From: jfoug <jfoug@...nwall.net>
To: john-dev@...ts.openwall.com
Subject: Proposal for --test-full logic
Solar,
I know this has been a pet project which you have spear-headed for
GSOC. I would like to propose this change in logic.
currently, -test and -test-full both perform tests, and then perform
benchmarks. I would propose that for -test-full, we remove that
benchmark logic, and instead use the optional param to give indication
for more tests to be run, where some of the 'deeper' tests may not be
fully fixed in all formats.
So --test-full or --test-full=0 would run a specific level. This would
be all tests which we would 'expect' to run without errors (possibly
with a format or 2 in whitelist). This would be very useful for adding
to our CI environments, to use this vs using the --test=0 which is used
today.
Then, --test-full=1 would be all the tests from --test-full=0, and then
some new tests, which may not have had all the bugs worked out of the
formats yet. This would be very useful allowing someone to easily
continue to develop improvements to the formats which are failing the
test, but be controlled by the optional switch, so it would not impact
the robot runs on the CI's.
There could be as many numbers as were needed. They could be done in an
independent manner, or in a sequential manner, so --test-full=3 may be
--test-full=0 and --test-full=3 or, it could be 0, 1, 2 and 3. I really
do not know which would be better, but if it was sequential, then
someone could simply do --test-full=99999 and pretty much know that they
will test ALL possible test-full cases.
At some point, the logic would get fixed for all formats, and that
'extra deep' test code could then be moved into the --test-full=0 to be
a default check. From that point on, we would expect no formats at all
to be allowed to have that particular defect.
I think this logic would greatly help to automate the CI tools to force
developers to submit code or build new formats in a proper without
allowing 'known' latent problems. But also joining benchmarking into
the --test-full logic, really does not seem to fit well.
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
