GNU bug report logs - #11621
questionable locale sorting order (especially as related to char ranges in REs)

Previous Next

To reply to this bug, email your comments to 11621 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Linda Walsh <coreutils <at> tlinx.org>
To: bug-coreutils <at> gnu.org
Subject: questionable locale sorting order (especially as related to char
	ranges in REs)
Date: Sun, 03 Jun 2012 15:13:19 -0700

Within in the past few years, use of ranges in RE's has become
unreliable due to some locale changes sorting their native character
sets such that a<A<b<B<y<Y<z<Z (vs. 'C' ordering A<B<Y<Z<a<b<y<z).

Additionally many distro's have switched to UTF-8 resulting in
localizations like en_GB.UTF-8, en_US.UTF-8, etc...

There seems to be a problem in when a user has set their system to use
Unicode, it is no longer using the locale specific character set 
(iso-8859-x,
or others).

In Unicode, it is recommended that upper case be uniformly sorted
below lower case (section 6.6, http://www.unicode.org/reports/tr10/).

A chart, including accent variations is at

http://unicode.org/charts/case/chart_Latin.htm.

Temporarily ignoring accents, only talking about lower and upper
case letters, you will note that the sorting order of A=41, B=42, C=43,
while the lower case letters from 'a', have weights a=61, b=62, c=63.

This uniformly puts all lower case letters "after" any upper case letters.

Thus -- I am asserting, that any computer using a local for country
preferences, BUT is also using a unicode character set (e.g. UTF-8),
should return sorted results as specified by the character set.

I.e. the utility 'sort' (and any programs that use the collation/sorting
order specified in the core-utils libs) should return A-Z < a-z.


This is currently not the case and is leading to erroneous results
in programs written before locales were considered.  The thing is --
in many cases, within some short period of locales being implemented,
many or most distro's also switched to UTF-8.

Unfortunately it's collation order has not been respected.

I would assert this is a serious bug that should be addressed ASAP...


Thanks,
Linda W.

Message #8 received at 11621 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Linda Walsh <coreutils <at> tlinx.org>
Cc: 11621 <at> debbugs.gnu.org
Subject: Re: bug#11621: questionable locale sorting order (especially as
	related to char ranges in REs)
Date: Sun, 03 Jun 2012 23:57:04 +0100

On 06/03/2012 11:13 PM, Linda Walsh wrote:
> Within in the past few years, use of ranges in RE's has become
> unreliable due to some locale changes sorting their native character
> sets such that a<A<b<B<y<Y<z<Z (vs. 'C' ordering A<B<Y<Z<a<b<y<z).
> 
> Additionally many distro's have switched to UTF-8 resulting in
> localizations like en_GB.UTF-8, en_US.UTF-8, etc...
> 
> There seems to be a problem in when a user has set their system to use
> Unicode, it is no longer using the locale specific character set (iso-8859-x,
> or others).

It's not specific to "unicode". Sorting in a iso-8859-1 charset
results in locale ordering:

$ printf "%s\n" A b a á | iconv -t iso-8859-1 | LC_ALL=en_US sort | iconv -f iso-8859-1
a
A
á
b

> In Unicode, it is recommended that upper case be uniformly sorted
> below lower case (section 6.6, http://www.unicode.org/reports/tr10/).
> 
> A chart, including accent variations is at
> 
> http://unicode.org/charts/case/chart_Latin.htm.

http://unicode.org/charts/case/chart_Latin.html

> Temporarily ignoring accents, only talking about lower and upper
> case letters, you will note that the sorting order of A=41, B=42, C=43,
> while the lower case letters from 'a', have weights a=61, b=62, c=63.
> 
> This uniformly puts all lower case letters "after" any upper case letters.
> 
> Thus -- I am asserting, that any computer using a locale for country
> preferences, BUT is also using a unicode character set (e.g. UTF-8),
> should return sorted results as specified by the character set.
> 
> I.e. the utility 'sort' (and any programs that use the collation/sorting
> order specified in the core-utils libs) should return A-Z < a-z.

Well case comparison is a complicated area.

For the special case of discounting accented chars etc.
you can use an attribute of the well designed UTF-8.
Enabling traditional byte comparison on (normalized) UTF-8 data
will result in data sorted in Unicode code point order:

$ printf "%s\n" A b a á | LC_ALL=C sort
A
a
b
á

> This is currently not the case and is leading to erroneous results
> in programs written before locales were considered.  The thing is --
> in many cases, within some short period of locales being implemented,
> many or most distro's also switched to UTF-8.
> 
> Unfortunately it's collation order has not been respected.
> 
> I would assert this is a serious bug that should be addressed ASAP...

As for the question in the subject for handling ranges in REs,
there has been recent work in changing as you suggest:

http://lists.gnu.org/archive/html/bug-gnulib/2011-06/threads.html#00105

cheers,
Pádraig.

Message #11 received at 11621 <at> debbugs.gnu.org (full text, mbox):

From: "Linda A. Walsh" <lkml <at> tlinx.org>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: 11621 <at> debbugs.gnu.org
Subject: Re: bug#11621: questionable locale sorting order (especially as
	related	to char ranges in REs)
Date: Sun, 03 Jun 2012 22:03:38 -0700

Pádraig Brady wrote:
> On 06/03/2012 11:13 PM, Linda Walsh wrote:
>> Within in the past few years, use of ranges in RE's has become
>> unreliable due to some locale changes sorting their native character
>> sets such that a<A<b<B<y<Y<z<Z (vs. 'C' ordering A<B<Y<Z<a<b<y<z).
>>
>> There seems to be a problem in when a user has set their system to use
>> Unicode, it is no longer using the locale specific character set (iso-8859-x,
>> or others).
----
	To clarify my above statement:


   There seems to be a problem in when a user has set their system to use
Unicode: It is no longer using the locale specific character set (iso-8859-x,
or others) -- ***or*** *their* *orderings*.  I.e. Unicode defines a collation
order -- I don't know that they others do ('C' does, but I don't know about
other locale-specific character sets).


> It's not specific to "unicode". Sorting in a iso-8859-1 charset
> results in locale ordering:
----
	Can you cite a source specifying the sort/collation order of the
iso-8859-1 charset that would prove that it is not-conforming to the collation
specification for that charset?

	I.e. If there is no official source, then the order with that charset
is "undefined", and while it may not be desirable, returning a<A<b<B, would not
be "an error".




>> http://unicode.org/charts/case/chart_Latin.htm.
> 
> http://unicode.org/charts/case/chart_Latin.html
---
	^^Correct^^ (typho)

>> Temporarily ignoring accents, only talking about lower and upper
>> case letters, ...
> 
> Well case comparison is a complicated area.
----
	A bit, but it's mostly just wrong in the gnu library concerning unicode, and,
as you are pointing out -- the 'C' encoding as well.
the 'C' locale was the original charset used by the 'C' language -- only 8 bits
wide.

	So how can it sort characters beyond the lower 256?
This would seem to be meaningless and bugs output.


Is it?...   When the case comparison ordering is specified in a
standard, it makes it fairly clear that one is either compliant with the standard
or not.

	In this case, the Gnu sort/collation lib is not Unicode/UTF-8 compliant.

	What happens in other charsets may or may not be covered under some
other standard -- e.g. the 'C'/ascii ordering is specified.  But I don't know
if others have relevant standards or not.

> 
> For the special case of discounting accented chars etc.
> you can use an attribute of the well designed UTF-8.
---
	This is not exactly the point -- the point is that the core sort
DOESN'T use that ordering.  That's the bug I am reporting.

	In reporting this, I'm trying to keep the argument 'simple' and focus on
the problem of widely used ranges in the first 256 code-points of
Unicode.

	Unicode gives a fairly extensive algorithm for handling accents,
but I didn't want to complicate the discussion by "going there".  Please
focus this bug on the lower 128 code points, as full unicode compliance
with the full collation algorithm that is specified is likely to be a
larger task.  HOWEVER, fixing the sorting/collation order of the lower
127 code points, is, comparatively a small task that conceivably could be
fixed in the next release.


> Enabling traditional byte comparison on (normalized) UTF-8 data
> will result in data sorted in Unicode code point order:
> A b a á => A a b á

But you are missing the point (as well as raising an interesting 'feature'(?bug?)).

How is it that 'C' collation collates characters that are outside the ascii range?

I.e. -- you can't interpret input data as 'unicode' in the 'C' locale.
So how does this work in the 'C' local?  AND more importantly -- it SHOULD work
when charset is unicode (UTF-8)... and does not.  Test prog:
---------------
#!/bin/bash
set -m
# vals to test:
declare -a vals=( A a B b X x Y y Z z Ⅷ  Ⅴ Ⅲ Ⅰ Ⅿ Ⅽ ⅶ  ⅼ ⅲ )
COLLATE_ORDER=C

function isatty {
	local fd=${1:-1} ;
	0<&$fd tty -s
}

function ord {
  local nl="";
	isatty && nl="\n"
	printf "%d$nl" "'$1"
}

function background_print {
	readarray -t inp
	for ch in "${inp[@]}"; {
		printf "%s   (U+%x)\n" "$ch" "$(ord "$ch")"
	}
}


printf "%s\n" "${vals[@]}" |
		LC_COLLATE=$COLLATE_ORDER sort |
		background_print

------------------------------------

Note, that the above produces:

/tmp/stest
Ⅷ   (U+2167)
Ⅴ   (U+2164)
Ⅲ   (U+2162)
Ⅰ   (U+2160)
Ⅿ   (U+216f)
Ⅽ   (U+216d)
ⅶ   (U+2176)
ⅼ   (U+217c)
ⅲ   (U+2172)
a   (U+61)
A   (U+41)
b   (U+62)
B   (U+42)
x   (U+78)
X   (U+58)
y   (U+79)
Y   (U+59)
z   (U+7a)
Z   (U+5a)

NOT the output you showed...Seems there's a bug in the C collation order?

Changing collation order to UTF-8:

Same thing:
 /tmp/stest
Ⅷ   (U+2167)
Ⅴ   (U+2164)
Ⅲ   (U+2162)
Ⅰ   (U+2160)
Ⅿ   (U+216f)
Ⅽ   (U+216d)
ⅶ   (U+2176)
ⅼ   (U+217c)
ⅲ   (U+2172)
a   (U+61)
A   (U+41)
b   (U+62)
B   (U+42)
x   (U+78)
X   (U+58)
y   (U+79)
Y   (U+59)
z   (U+7a)
Z   (U+5a)


>> I would assert this is a serious bug that should be addressed ASAP...
> 
> As for the question in the subject for handling ranges in REs,
> there has been recent work in changing as you suggest:
> 
> http://lists.gnu.org/archive/html/bug-gnulib/2011-06/threads.html#00105
----

	Recent?
The most recent posts on that thread look to be from June of last year.
I.e. a year ago.

I'm trying to stay focused on specific problems -- UTF-8 ordering is defined.
the gnu library doesn't follow it.

Major problem with so many progs relying on the lib!...

Message #14 received at 11621 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: "Linda A. Walsh" <lkml <at> tlinx.org>
Cc: 11621 <at> debbugs.gnu.org
Subject: Re: bug#11621: questionable locale sorting order (especially as
	related to char ranges in REs)
Date: Mon, 04 Jun 2012 09:48:52 +0100

On 06/04/2012 06:03 AM, Linda A. Walsh wrote:
> 
> 
> Pádraig Brady wrote:
>> On 06/03/2012 11:13 PM, Linda Walsh wrote:
>>> Within in the past few years, use of ranges in RE's has become
>>> unreliable due to some locale changes sorting their native character
>>> sets such that a<A<b<B<y<Y<z<Z (vs. 'C' ordering A<B<Y<Z<a<b<y<z).
>>>
>>> There seems to be a problem in when a user has set their system to use
>>> Unicode, it is no longer using the locale specific character set (iso-8859-x,
>>> or others).
> ----
>     To clarify my above statement:
> 
> 
>    There seems to be a problem in when a user has set their system to use
> Unicode: It is no longer using the locale specific character set (iso-8859-x,
> or others) -- ***or*** *their* *orderings*.  I.e. Unicode defines a collation
> order -- I don't know that they others do ('C' does, but I don't know about
> other locale-specific character sets).
> 
> 
>> It's not specific to "unicode". Sorting in a iso-8859-1 charset
>> results in locale ordering:
> ----
>     Can you cite a source specifying the sort/collation order of the
> iso-8859-1 charset that would prove that it is not-conforming to the collation
> specification for that charset?
> 
>     I.e. If there is no official source, then the order with that charset
> is "undefined", and while it may not be desirable, returning a<A<b<B, would not
> be "an error".

It's a charset. Of course the order is defined. Try: man iso-8859-1

The relative ordering can be trivially inferred from the command I presented.
But to be explicit:

$ printf "%s\n" A b a á | iconv -t iso-8859-1 | LC_ALL=en_US sort | iconv -f iso-8859-1
a
A
á
b

$ printf "%s\n" A b a á | iconv -t iso-8859-1 | LC_ALL=C sort | iconv -f iso-8859-1
A
a
b
á

> 
> 
> 
> 
>>> http://unicode.org/charts/case/chart_Latin.htm.
>>
>> http://unicode.org/charts/case/chart_Latin.html
> ---
>     ^^Correct^^ (typho)
> 
>>> Temporarily ignoring accents, only talking about lower and upper
>>> case letters, ...
>>
>> Well case comparison is a complicated area.
> ----
>     A bit, but it's mostly just wrong in the gnu library concerning unicode, and,
> as you are pointing out -- the 'C' encoding as well.
> the 'C' locale was the original charset used by the 'C' language -- only 8 bits
> wide.
> 
>     So how can it sort characters beyond the lower 256?
> This would seem to be meaningless and bugs output.

http://www.pixelbeat.org/docs/utf8_programming.html

> Is it?...   When the case comparison ordering is specified in a
> standard, it makes it fairly clear that one is either compliant with the standard
> or not.
> 
>     In this case, the Gnu sort/collation lib is not Unicode/UTF-8 compliant.
> 
>     What happens in other charsets may or may not be covered under some
> other standard -- e.g. the 'C'/ascii ordering is specified.  But I don't know
> if others have relevant standards or not.
> 
>>
>> For the special case of discounting accented chars etc.
>> you can use an attribute of the well designed UTF-8.
> ---
>     This is not exactly the point -- the point is that the core sort
> DOESN'T use that ordering.  That's the bug I am reporting.

Well you can't generally exclude accents.

> 
>     In reporting this, I'm trying to keep the argument 'simple' and focus on
> the problem of widely used ranges in the first 256 code-points of
> Unicode.
> 
>     Unicode gives a fairly extensive algorithm for handling accents,
> but I didn't want to complicate the discussion by "going there".  Please
> focus this bug on the lower 128 code points, as full unicode compliance
> with the full collation algorithm that is specified is likely to be a
> larger task.  HOWEVER, fixing the sorting/collation order of the lower
> 127 code points, is, comparatively a small task that conceivably could be
> fixed in the next release.

lower 127 = ASCII. If your input data is ASCII, just use LC_ALL=C.

>> Enabling traditional byte comparison on (normalized) UTF-8 data
>> will result in data sorted in Unicode code point order:
>> A b a á => A a b á
> 
> But you are missing the point (as well as raising an interesting 'feature'(?bug?)).
> 
> How is it that 'C' collation collates characters that are outside the ascii range?

Well whether C should be a "unicode" or "ascii" charset is a whole different
kettle of fish. I was just referring (as per the link above), that
UTF8 is well designed so that it works with many traditional single byte functions.

> I.e. -- you can't interpret input data as 'unicode' in the 'C' locale.
> So how does this work in the 'C' local?  AND more importantly -- it SHOULD work
> when charset is unicode (UTF-8)... and does not.  Test prog:
> ---------------
> #!/bin/bash
> set -m
> # vals to test:
> declare -a vals=( A a B b X x Y y Z z Ⅷ  Ⅴ Ⅲ Ⅰ Ⅿ Ⅽ ⅶ  ⅼ ⅲ )
> COLLATE_ORDER=C
> 
> function isatty {
>     local fd=${1:-1} ;
>     0<&$fd tty -s
> }
> 
> function ord {
>   local nl="";
>     isatty && nl="\n"
>     printf "%d$nl" "'$1"
> }
> 
> function background_print {
>     readarray -t inp
>     for ch in "${inp[@]}"; {
>         printf "%s   (U+%x)\n" "$ch" "$(ord "$ch")"
>     }
> }
> 
> 
> printf "%s\n" "${vals[@]}" |
>         LC_COLLATE=$COLLATE_ORDER sort |
>         background_print
> 
> ------------------------------------
> 
> Note, that the above produces:
> 
> /tmp/stest
> Ⅷ   (U+2167)
> Ⅴ   (U+2164)
> Ⅲ   (U+2162)
> Ⅰ   (U+2160)
> Ⅿ   (U+216f)
> Ⅽ   (U+216d)
> ⅶ   (U+2176)
> ⅼ   (U+217c)
> ⅲ   (U+2172)
> a   (U+61)
> A   (U+41)
> b   (U+62)
> B   (U+42)
> x   (U+78)
> X   (U+58)
> y   (U+79)
> Y   (U+59)
> z   (U+7a)
> Z   (U+5a)
> 
> NOT the output you showed...Seems there's a bug in the C collation order?

Note C doesn't use a collation order, it's simple byte comparison.
Seems there may be a bug in your script?
Also ensure that LC_ALL is not set, which will override LC_COLLATE.

$ printf "%s\n" A a B b 2 1 Ⅷ  ⅶ ⅲ | LC_COLLATE=C sort
1
2
A
B
a
b
Ⅷ
ⅲ
ⅶ

> 
> Changing collation order to UTF-8:
> 
> Same thing:
>  /tmp/stest
> Ⅷ   (U+2167)
> Ⅴ   (U+2164)
> Ⅲ   (U+2162)
> Ⅰ   (U+2160)
> Ⅿ   (U+216f)
> Ⅽ   (U+216d)
> ⅶ   (U+2176)
> ⅼ   (U+217c)
> ⅲ   (U+2172)
> a   (U+61)
> A   (U+41)
> b   (U+62)
> B   (U+42)
> x   (U+78)
> X   (U+58)
> y   (U+79)
> Y   (U+59)
> z   (U+7a)
> Z   (U+5a)
> 
> 
>>> I would assert this is a serious bug that should be addressed ASAP...
>>
>> As for the question in the subject for handling ranges in REs,
>> there has been recent work in changing as you suggest:
>>
>> http://lists.gnu.org/archive/html/bug-gnulib/2011-06/threads.html#00105
> ----
> 
>     Recent?

?

> The most recent posts on that thread look to be from June of last year.
> I.e. a year ago.
> 
> I'm trying to stay focused on specific problems -- UTF-8 ordering is defined.
> the gnu library doesn't follow it.
> 
> Major problem with so many progs relying on the lib!...

cheers,
Pádraig.

Message #17 received at 11621 <at> debbugs.gnu.org (full text, mbox):

From: "Linda A. Walsh" <lkml <at> tlinx.org>
To: 11621 <at> debbugs.gnu.org, Pádraig Brady <P <at> draigBrady.com>
Subject: Re: bug#11621: questionable locale sorting order (especially as
	related	to char ranges in REs)
Date: Wed, 06 Jun 2012 18:16:02 -0700

[Message part 1 (text/plain, inline)]

Pádraig Brady wrote:
> On 06/04/2012 06:03 AM, Linda A. Walsh wrote:
>   
>> Pádraig Brady wrote:
>>     
>>> On 06/03/2012 11:13 PM, Linda Walsh wrote:
>>>       
>>>> Within in the past few years, use of ranges in RE's has become
>>>> unreliable due to some locale changes sorting their native character
>>>> sets such that a<A<b<B<y<Y<z<Z (vs. 'C' ordering A<B<Y<Z<a<b<y<z).
>>>>
>>>> There seems to be a problem in when a user has set their system to use
>>>> Unicode, it is no longer using the locale specific character set (iso-8859-x,
>>>> or others).
>>>>         
>> ----
>>     To clarify my above statement:
>>
>>
>>    There seems to be a problem in when a user has set their system to use
>> Unicode: It is no longer using the locale specific character set (iso-8859-x,
>> or others) -- ***or*** *their* *orderings*.  I.e. Unicode defines a collation
>> order -- I don't know that they others do ('C' does, but I don't know about
>> other locale-specific character sets).
>>
>>
>>     
>>> It's not specific to "unicode". Sorting in a iso-8859-1 charset
>>> results in locale ordering:
>>>       
>> ----
>>     Can you cite a source specifying the sort/collation order of the
>> iso-8859-1 charset that would prove that it is not-conforming to the collation specification for that charset?
>>
>>     I.e. If there is no official source, then the order with that charset
>> is "undefined", and while it may not be desirable, returning a<A<b<B, would not be "an error".
>>     
>
> It's a charset. Of course the order is defined. Try: man iso-8859-1
>
> The relative ordering can be trivially inferred from the command I presented.
> But to be explicit:
>
> $ printf "%s\n" A b a á | iconv -t iso-8859-1 | LC_ALL=en_US [sic] sort | iconv -f iso-8859-1
> a
> A
> á
> b
>   
----
Your example doesn't show the collation order of iso-8859-1.   You are 
setting it to 'en_US' (as LC_ALL overrides all other LC vars; LANG sets 
the default, but individual settings in the LC variables can override it.

A corrected example:

$ (Charset=iso-8859-1; printf "%s\n" A b B a á | iconv -t $Charset | 
LANG=en_US LC_CHARSET=$Charset LC_COLLATE=$Charset sort | iconv -f 
$Charset |tr "\n" " ";echo "") 
A B a b á

(I used 'Charset' to hold the charset name, added parens, printed them 
in the same orientation as input, and added a 2nd capital letter to make 
upper/lower case ordering clear.)

   I might note how "trivial" it was to arrive at incorrect output.  
People often think me a pain because I ask them to explain what they 
perceive to be
obvious.  Unfortunately, what is obvious to 1 person may not be so to 
another.

   The 'á' is not ASCII (original charset for C locale, coming from 
unix & C programming language -- a reason why POSIX renamed the 'C' 
local to the POSIX
locale.

   However, as 'á' is in the 1st 256 chars (above the ASCII range), it 
can still work if you remove the iconv stuff (and note, I have no other 
locale vars
set:

$ echo ${!LC_*} ${!LAN*}
LC_COLLATE LC_CTYPE

$ (Charset=ASCII; printf "%s\n" A B b a á |  LC_CHARSET=$Charset 
LC_COLLATE=$Charset sort |tr "\n" " ";echo "")         
A B a b á

   To bring this to completion -- most linux systems today use the UTF-8
character set.  It shows an *identical* collation order for the above 
chars as the iso-8859-1 charset.

   It appears that the collating functions are confused by the notation 
that has been adopted in many distributions...namely <locale>.charset.   
In such a notation, where the charset has been explicitly specified, and 
where the charset has explicit COLLATION and case folding rules (those 
for Unicode are extensive and handle accents as well as other forms like 
ſȘșʂȿᵴᶊṠṡṢṣṤṥṦṧṨṩẛẜẝẞⱾꞨꞩSsߌśŜŝŞşŠšˢ...etc.

   Therefore, I would like to see the character set's collation and 
folding rules used where they are officially specified (as in the case 
of Unicode or POSIX).

   Are you the person responsible for the libicuXXX files?

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.