[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAKGWAO9bq7jyuNDQCWSzmTDXTA87T6KnEXLij23vNf6k+nmrGw@mail.gmail.com>
Date: Wed, 16 May 2018 12:22:36 -0500
From: Will Dietz <w@...z.org>
To: musl@...ts.openwall.com
Subject: iconv UTF-8 <--> CP1255 roundtrip possible bug?
I admit to being a bit unsure, but the behavior shown below doesn't
seem obviously right --LMK if I'm missing something :).
Input file attached for inspection without relying on it getting
through byte-identical to what I have--
indeed I'm not sure copy+paste into this is working correctly (the
characters look different in my terminal :)). Anyway:
$ cat cp1255-snippet.xxd
00000000: efac b3d6 b8d7 9d0a ........
$ xxd -r cp1255-snippet.xxd
דָּם
Attempt to round-trip this from UTF-8 to CP1255 and back,
first with glibc's iconv (2.26):
$ xxd -r cp1255-snippet.xxd|iconv -f UTF-8 -t CP1255|iconv -f CP1255
-t UTF-8 | xxd
00000000: efac b3d6 b8d7 9d0a
Looks good, same as what was sent in.
Using musl-based iconv utility (1.1.19):
$ xxd -r cp1255-snippet.xxd|$ICONV -f UTF-8 -t CP1255|$ICONV -f CP1255
-t UTF-8 | xxd
00000000: 2ad6 b8d7 9d0a *.....
Indeed, the result looks different than what was started with:
*ָם
(again apologies if that doesn't survive mailing and such)
This input was taken from gnu libiconv's test suite, in particular the
first line of tests/CP1255-snippet.UTF-8. Since it's 2 characters,
and test data, I hope there's no problem re:licensing O:).
I've reproduced the same behavior using iconv() directly, I can share
that if that would be preferable. It's the same code from earlier
iconv threads on the ML.
--------------
Hopefully this is useful!
On the subject, a question or two if it's not too much trouble:
* is the above what's meant by "round-trip" as discussed in[1]?
* What sorts of "round-trip" conversions are expected to work? And
over what inputs should round-trip conversions work-- for any 'valid"
UTF-8 or so?
Armed with some insights regarding these questions, I'm hoping to
scope out something that can be tested or (no promises!) perhaps
pushed through some formal verification goodness. But also I'm just
curious :).
Thanks!
~Will
[1] http://www.openwall.com/lists/musl/2018/02/27/2
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
