[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <87sgl7zal1.fsf@mid.deneb.enyo.de>
Date: Thu, 26 Dec 2019 10:43:06 +0100
From: Florian Weimer <fw@...eb.enyo.de>
To: Rich Felker <dalias@...c.org>
Cc: JeanHeyd Meneide <phdofthehouse@...il.com>, musl@...ts.openwall.com
Subject: Re: [ Guidance ] Potential New Routines; Requesting Help
* Rich Felker:
> On Wed, Dec 25, 2019 at 09:07:05PM +0100, Florian Weimer wrote:
>> * JeanHeyd Meneide:
>>
>> > I hope this e-mail finds you doing well this Holiday Season! I am
>> > interested in developing a few fast routines for text encoding for
>> > musl after the positive reception of a paper for the C Standard
>> > related to fast conversion routines:
>> >
>> > https://thephd.github.io/vendor/future_cxx/papers/source/C%20-%20Efficient%20Character%20Conversions.html
>>
>> I'm somewhat concerned that the C multibyte functions are too broken
>> to be useful. There is a at least one widely implemented character
>> set (Big5 as specified for HTML5) which does not fit the model implied
>> by the standard. Big5 does not have shift states, but a C
>> implementation using UTF-32 for wchar_t has to pretend it has because
>> correct conversion from Unicode to Big5 needs lookahead and cannot be
>> performed one character at a time.
>
> I don't think this can be modeled with shift states. C explicitly
> forbids a stateful wchar_t encoding/multi-wchar_t-characters. Shift
> states would be meaningful for the other direction.
The intent of the standard appears to be to support this as an
extension.
It's hard to tell because actual users of the interfaces with legacy
charsets do not seem to be represented on the standards committee
anymore (see the mblen behavioral change in C11 as evidence supporting
this theory).
> In any case I don't think it really matters. There are no existing
> implementations with this version of Big5 (with the offending HKSCS
> characters included) as the locale charset, since it can't work, and
> there really is no good reason to be adding *new* locale encodings.
Do you mean in musl?
> The reason we (speaking of the larger community; musl doesn't) have
> non-UTF-8 locales is legacy compatibility for users who need or insist
> on keeping them.
That is true.
> If there really is an insistence on using this version of Big5, the
> characters should be added to Unicode as <compat> characters so that
> there's an unambiguous one-to-one correspondence, and the people who
> care about it working should take responsibility for doing that.
Yes, I was very surprised this wasn't done for TSCII and HKSCS/Big5.
I think even for Big5, it would solve the issue because the decoding
process only needs to look at a single multibyte character at the
time (I may have suggested otherwise in the past).
A succint description of what is going on for Big 5 is here:
<https://encoding.spec.whatwg.org/#big5-decoder>, under step 3.3. The
conversion is actually fairly simple, it's just hard to fit it into
the C interfaces.
>> This would at least affect the proposed c8rtomb function.
>>
>> I posted a brief review of the problematic charsets in glibc here:
>>
>> <https://sourceware.org/ml/libc-alpha/2019-05/msg00079.html>
>
> I've read it but seemingly not in enough detail to gather what parts
> are relevant to this conversation.
It names a few character sets that have fake shift states because the
C interfaces cannot otherwise be used with them. Some of the new
interfaces are problematic in this context (whether or not UTF-32 is
used for wchar_t). I think new interfaces should be compatible with
existing implementation practice.
The other thing I found surprising is that there are no
ASCII-transparent charsets with traditional shift states in glibc.
The ASCII-transparent charsets with shift states have these fake shift
states.
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
