Subject: bug#12907: Possible Bug in sort core utility

From: "Coffey, Terrence (Terrence) **CTR**" <[email-masked]>
Date: Fri, 23 Nov 2012 16:17:27 +0000

Hi Bob, Eric and Paul,
Thank you all for your quick responses. Due to time zone differences, I had gone home so I did not see
your emails until this morning. I'm based in Galway, Ireland. I'm 5 to 8 hours ahead of you depending where
in USA you are.  I started putting together a response to your email but then got switched to another task
and only got a chance to work on it again today. I realize that you may now be out of office due to Thanks
Giving holiday.

I've pulled together all your responses into this one email.
I've run most of the commands you supplied and have attached them to this email.

Unfortunately, none of your suggestions worked for me.
The result I was expecting was
4 drwx------. 2 WFUser WFUser 4096 Nov  9 17:39 .esd-500
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:14 .tc
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:14 .tc2
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:15 .tc4
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:18 .tc3
4 drwxrwxrwt. 2 root   root   4096 Nov 19 16:05 .ICE-unix
4 -r--r--r--. 1 root   root     11 Nov 19 16:05 .X0-lock
4 drwxrwxrwt. 2 root   root   4096 Nov 19 16:05 .X11-unix
4 drwx------. 2 WFUser WFUser 4096 Nov  9 17:39 pulse-g9SlRO3UEqGg
4 drwx------. 2 gdm    gdm    4096 Nov 19 16:05 pulse-Jkx9hhtHj4dy
4 drwx------. 2 gdm    gdm    4096 Nov 19 16:06 orbit-gdm

See test data in attached files.

The --time-style=long.iso seem to alter the way in which the time field is handled in the sort command. From
my results, it appears the XX:YY (XX=Hours YY=Minutes) are treated as separate fields without this option.
See attached file for this.

I have a few questions
1) Should LC_ALL be set to 'C' or C or are they equivalent      export LC_ALL='C'
   I've tried both way and it appeared not to alter the result.

Please find attached the commands I ran with the locale unset , set to C and set to 'C'.

Eric, you mentioned that RH 6.3 is an old release. Where can I get the latest approved RH release ?
RH 6.3 is the latest release that Avaya has approved for internal use but given my reported issue
It would be useful to use --debug to see what is happening or not happening.

Thanks for all of your assistance,
Terence Coffey

=======
Paul Eggert [[email-masked]]
Most likely it's a locale problem.
Try "sort --debug" and try setting LC_ALL='C'
in your environment.

========

tag 12907 notabug
thanks

On 11/16/2012 11:06 AM, Coffey, Terrence (Terrence) **CTR** wrote:
> Hi,
> I think I might have located a bug. I'm using Redhat 6.3.

Thanks for the report.  However, I suspect that you are hitting this FAQ, and not a bug in sort itself:
https://www.gnu.org/software/coreutils/faq/#Sort-does-not-sort-in-normal-order_0021

Unfortunately, the 'sort' version shippted in RHEL 6.3 is too old to support the --debug option, but upgrading to a newer version will help shed some insight into your problems.

> [WFUser@RHEL63TEMP3942 tmp]$ ls -Asl  /tmp |sort -k11 -k7M -k8n -k9,10

As I don't have the same 'ls -Asl /tmp' output, I took your output and put in in a temp file of my own.  Now, on that file, and with the sort
8.15 in Fedora 17, I get:

$ LC_ALL=en_US.UTF-8 sort --debug -k11 -k7M -k8n -k9,10 tmpfile
sort: using `en_US.UTF-8' sorting rules
sort: leading blanks are significant in key 1; consider also specifying `b'
sort: key 3 is numeric and spans multiple fields
sort: leading blanks are significant in key 4; consider also specifying `b'
total 76
        ^ no match for key
        ^ no match for key
        ^ no match for key
        ^ no match for key
________
0 -rw-------. 1 root   root      0 Nov  9 15:06 yum.log
                                                       ^ no match for key
                                   ___
                                        _
                                         ______________ _______________________________________________________
4 -r--r--r--. 1 root   root     11 Nov  9 17:37 .X0-lock
                                                        ^ no match for key
                                   ___
                                        _
                                         _______________ ________________________________________________________

...

Wow - just that much information is quite useful.  You aren't sorting what you thought you were sorting.

> 
> The filename is key 11 so I was expecting to see all the dot files 
> group together

The filename is NOT field 11, but field 10 (fields start with 1, not 0).
 Furthermore, the debug option is warning you that unterminated -k specifications are risky; for example, it is better to use -k7,7M than -k7M.  Also, it is warning you that -k11 includes leading space, which may affect the sort (depending on your locale).  Paul already pointed out that your choice of locale may be important (en_US.UTF-8 typically does a dictionary sort, ignoring punctuation, but '.' is punctuation).

> I'm try to get a listing of dot file sorted by date and all other files sorted by date. I'd like all the dot file to appears before all other files.

Here's something closer, but it still has a fatal flaw - the moment you request to sort by dot files first, you have specified that key takes priority over dates.

$ LC_ALL=C sort -k10b -k7,7M -k8,8nb -k9,9 tmpfile total 76
4 drwxrwxrwt. 2 root   root   4096 Nov  9 18:35 .ICE-unix
4 -r--r--r--. 1 root   root     11 Nov  9 17:37 .X0-lock
4 drwxrwxrwt. 2 root   root   4096 Nov  9 18:35 .X11-unix
4 -r--r--r--. 1 root   root     11 Nov  9 18:35 .X2-lock
4 drwx------. 2 WFUser WFUser 4096 Nov  9 17:39 .esd-500
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:14 .tc
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:14 .tc2
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:18 .tc3
4 -rw-rw-r--. 1 WFUser WFUser    5 Nov 16 18:15 .tc4
4 drwx------. 2 root   root   4096 Nov  9 18:35 keyring-kjLYun
4 drwx------. 2 WFUser WFUser 4096 Nov  9 17:39 keyring-nQ6lMb
4 drwx------. 2 WFUser WFUser 4096 Nov  9 18:20 orbit-WFUser
4 drwx------. 2 gdm    gdm    4096 Nov  9 17:39 orbit-gdm
4 drwx------. 2 root   root   4096 Nov  9 18:35 orbit-root
4 drwx------. 2 gdm    gdm    4096 Nov  9 17:40 pulse-Jkx9hhtHj4dy
4 drwx------. 2 root   root   4096 Nov  9 18:35 pulse-TSaTZvcbd1kc
4 drwx------. 2 WFUser WFUser 4096 Nov  9 17:39 pulse-g9SlRO3UEqGg
4 drwx------. 2 root   root   4096 Nov  9 18:35 ssh-BRSEI10523
4 drwx------. 2 WFUser WFUser 4096 Nov  9 17:39 virtual-WFUser.AVmQkN
0 -rw-------. 1 root   root      0 Nov  9 15:06 yum.log

Really, the ONLY way you are going to sort two independent chunks of your list by date is to instead concatenate two independently sorted chunks, and realize that ls already does the date sorting that you desire.  Perhaps you are really looking for:

(set -f; ls -lsd --sort=date /tmp/.[!.] /tmp/.?*; ls -ls --sort=date /tmp)

-- 
Eric Blake   [email-masked]    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

-----Original Message-----
From: Bob Proulx [mailto:[email-masked]] 
Sent: 16 November 2012 18:59
To: Coffey, Terrence (Terrence) **CTR**
Cc: [email-masked]
Subject: Re: bug#12907: Possible Bug in sort core utility

tag 12907 + moreinfo
thanks

Coffey, Terrence (Terrence) **CTR** wrote:
> I think I might have located a bug. I'm using Redhat 6.3.

What locale are you using?  You can print out your locale settings with the locale command.

  $ locale

Very often the locale is set to a "human" locale where case is folded and punctuation is ignored in "dictionary" style.  This ordering is a libc system library definition.

> I'm try to get a listing of dot file sorted by date and all other 
> files sorted by date. I'd like all the dot file to appears before all 
> other files.

I personally would do this with two different commands.  First list the dot files using shell file glob matching only dot files.  Second list non-dot files.  I would also use the --time-style=long-iso format so that the dates are printed consistently.

  $ ls -Ald /tmp/.[!.]* ; ls -Ald /tmp/*

But I realize that sets different columns for the two different commands.

> Initially I was testing on 6.2 and noticed that the .(dot) files were 
> not passed to sort. However this issue is fixed in 6.3.

I have no idea what you are talking about.  I don't recall any bug in that area.  And if it is there then bringing up a fixed bug muddies the water.

> However it does not appear to be able to sort dot files. The filename 
> is key 11

In your example the filename is key 10 not 11.

  0 -rw-------. 1 root   root      0 Nov  9 15:06 yum.log
  1 2           3 4      5         6 7    8 9     10

That might be the most of your problem.

> If you need some additional data, just let me know.

Try this:

  $ ls -Asl --time-style=long-iso / | env LC_ALL=C sort -k9,9 -k7,8

Then please report back in if your problem is resolved.  Please keep the bug log in the recipient list.

See also the FAQ entry:

  http://www.gnu.org/software/coreutils/faq/#Sort-does-not-sort-in-normal-order_0021

Bob