[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <as7oRh2WqMGdge5m@raf.org>
Date: Wed, 14 Oct 2026 13:26:14 +1100
From: raf <musl@....org>
To: musl@...ts.openwall.com
Subject: Re: fnmatch: invalid multibyte sequence in the string is
taken as end of string ("" matches "\xff" in UTF-8 locales)
On Fri, Oct 09, 2026 at 09:08:25AM -0400, Rich Felker <dalias@...c.org> wrote:
> On Thu, Oct 08, 2026 at 02:27:31PM +1100, raf wrote:
> > That's interesting. I've made more changes to my local copy of musl's fnmatch()
> > so that it supports non-UTF8 bytes (not all filenames are UTF-8, and they're not
> > necessarily all even text). With my current version, your test program prints
> > 1, 1, 1, 1. Its str_next() never returns -1. A patch is attached.
>
> Getting back to the technical aspects of this, I think you should
> check how your patch makes ${foo%?} behave. I would expect shells
> implement this functionality by stepping back from the end of the
> string one byte at a time and calling fnmatch until they get a
> successful match. If this is the case, your patch will make them rip
> apart a final multibyte character and produce malformed output.
>
> Note that this does not happen with *. The shortest match of a lone *
> is always the empty string, so ${foo%*} is just $foo. And the longest
> match of a lone * is the full string. And if you have anything other
> than a * prior to the *, it will necessarily anchor to a character
> boundary -- as long as patterns matching a single character only match
> a single character and don't wrongly match a single byte of a
> multibyte character.
>
> The same should also apply from the beginning of a string using
> ${foo#?} etc.
>
> Rich
Hi Rich, Thanks.
I'm not using my modified version of fnmatch for a
shell, so the ${foo%?} syntax isn't possible to test.
I'm only using it for my rawhide (rh) program which is
an alternative to find(1).
Glob patterns only appear in a single command line
argument that is the search criteria expression. It
doesn't get used in combination with all the
possibilities of shell syntax.
Having said that, I did take care to make sure that,
when finding the final tailcnt chars of str, it would
only back up over a complete UTF-8 character, or over a
single non-UTF-8 byte, at a time.
The real version only handles UTF-8 bytes, and is no
doubt faster as a result.
But I needed it to work with more than just UTF-8.
Rawhide works on lots of operating systems, not just
UTF-8-only ones. And on most operating systems (except
Windows and macOS), filenames can contain bytes that
aren't valid text in the current user's locale's
encoding.
By default, the local system implementation is used,
but several implementations seem to have bugs. That's
why I'm providing a modified musl implementation as an
alternative if the user requests it.
cheers,
raf
P.S. I'm looking forward to trying out your upcoming
fnmatch changes. But I need to finish something else
first.
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
