Subject: Re: UTF-8 and non-characters
From: Geoff Clare <[email-masked]>
To: [email-masked]
Date: Wed, 1 Jul 2015 10:55:14 +0100

Stephane Chazelas <[email-masked]> wrote, on 30 Jun 2015:
>
> Speaking of which, would a pseudo-UTF-8 locale where bytes that
> don't form valid characters are mapped to a character like
> U+FFFD (�) be POSIX compliant.
> 
> Like c3 a9 is é, but c3 41 a9 is �A�
> 
> or if not all mapped to a single character, mapped to dedicated
> unassigned code points (0x7fffff80 to 0x7fffffff for instance)? 
> 
> For instance, above c3 41 a9 being <U+7fffffc3>A<U+7fffffa9>
> 
> If allowed, would that not be desirable (I can see it
> potentially be a problem when processing partial input)?

I think this would cause inconsistency between btowc() and the various
multi-byte to wide-character conversion functions.

If btowc(0xc3) returns a wide character, then mbtowc() on c3 a9 ought
to convert the c3 to that wide character and return 1, instead of
converting c3 a9 to a wide é and returning 2.

Conversely, if btowc(0xc3) returns WEOF, then mbtowc() on c3 41 a9
ought not to convert the c3 to a wide character.

> A common source of bugs and security vulnerabilities with
> UTF-8 is that fact that not all sequences of bytes map to
> characters and in particular that they're not matched by RE's
> "." or ".*" or fnmatch()'s ? or *.
> 
> That's a common problem when you can't guarantee the input is
> valid text for instance for arbitrary file names from the file
> system. That's quite common when dealing with file names that
> were written in a single-byte character set in UTF-8 locales.
> 
> For instance,
> 
> find . -name '*'
> 
> With GNU find at least doesn't match on $'St\xe9phane.txt'
> (Stéphane.txt in the iso8859-1 charset).
> 
> An example of a more serious problem:
> 
> find . ! -name "* *" -exec cmd-that-would-break-with-spaces {} +

It looks like the pattern matching sections of the standard have
some problems with the use of the terms character and string.

2.13.1 says * matches "multiple characters", but 2.13.2 says it
matches "any string" in item 1 and then says it matches "a string
of zero or more characters" (i.e. any character string) in item 3.

> GNU sed even went as far as defining a new command for emptying
> the pattern space to work around that problem:
> 
> `z'
>      This command empties the content of pattern space.  It is usually
>      the same as `s/.*//', but is more efficient and works in the
>      presence of invalid multibyte sequences in the input stream.
>      POSIX mandates that such sequences are _not_ matched by `.', so
>      that there is no portable way to clear `sed''s buffers in the
>      middle of the script in most multibyte locales (including UTF-8
>      locales).
> 
> Is that claim (about it being a POSIX requirement) true?

I think it's true for regexec(), but not for sed.

(Perhaps we should add a REG_EILSEQ error return for regexec().)

> I'd expect the behaviour to be unspecified if the input is not
> text (as would be the case if there are invalid multi-byte
> sequences).

Exactly.

> See also
> http://unix.stackexchange.com/questions/6516/filtering-invalid-utf8
> where we wondered whether grep -vx '.*' was required to report
> lines with invalid multi-byte sequences.

Unspecified, for the same reason as for sed.

> There was also a discussion earlier here about shells' ? and *
> on invalid byte sequences and most shells seem to match
> individual bytes from invalid multibyte sequences as one
> character (except for yash that won't deal with those at all)
> which seem to me like the safest thing to do.
> 
> What's the OpenGroup position on that?

2.13.1 is clear that ? matches a character.

The requirements for * are ambiguous because of the conflicting text
I pointed out above.

-- 
Geoff Clare <[email-masked]>
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England