Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <90454692ddbe4b7f2587ed221a9d2bfd@smtp.hushmail.com>
Date: Mon, 27 Jul 2015 11:33:31 +0200
From: magnum <john.magnum@...hmail.com>
To: john-dev@...ts.openwall.com
Subject: Re: auditing our use of FMT_* flags

On 2015-07-27 10:27, Kai Zhao wrote:
> formats.h: 51~60
>
> /*
>   * This flag must be set for formats that do UCS-2, UTF-16 or some other
> wide
>   * encoding internally (eg. most Microsoft formats). The most common problem
>   * with formats not fully Unicode-aware is when a format like this is
> hard-coded
>   * to convert from ISO-8859-1 (ie. by just inserting 0x00, effectively just
>   * casting every char to a short). Such formats MUST set FMT_UNICODE and
> MUST
>   * NOT set FMT_UTF8, or users will get false negatives when using UTF-8 or
>   * codepages.
>   */
> #define FMT_UNICODE                     0x00000004
>
> "Such formats MUST set FMT_UNICODE and MUST NOT set FMT_UTF8"
> Here "Such formats" points to "The most common problem with formats..." ?
>
> Can anyone explain this ? Is there  any example ?
>
> I have found that there are only two formats which set FMT_UNICODE and
> NOT set FMT_UTF8. They are:
>
> opencl_mscash_fmt_plug.c,  opencl_nt_fmt_plug.c

This means they can currently only convert ISO-8859-1 to UTF-16/UCS-2. 
This is how Hashcat and most other crackers do it, because it's simple 
and fast code.

> There are 27 formats which both set FMT_UNICODE and SET FMT_UTF8.

This means they can convert any supported encoding to UTF-16/UCS-2. That 
involves a lot more code.

The flags ensure we bail with error instead of chugging along with false 
negatives:

$ ../run/john -form:nt-opencl test.in -w:rockyou_utf8.lst -enc:utf8
This format does not yet support other encodings than ISO-8859-1

$ ../run/john -form:nt-opencl test.in -w:rockyou_8859_1.lst -enc:8859-1
Loaded 1 password hash (nt-opencl, NT [MD4 OpenCL])
Press 'q' or Ctrl-C to abort, almost any other key for status
0g 0:00:00:00 DONE (2015-07-27 11:29) 0g/s 7708p/s 7708c/s 7708C/s 
elles1992..Vk122503
Session completed

magnum

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.