GNU bug report logs - #25388
ls-quotes: kills existing scripts reading "ls" -1 as input

Previous Next

Full log

View this message in rfc822 format

From: Eric Blake <eblake <at> redhat.com>
To: L A Walsh <coreutils <at> tlinx.org>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, Andreas Schwab <schwab <at> linux-m68k.org>, 25388 <at> debbugs.gnu.org
Subject: bug#25388: Bug in ls, kills existing scripts reading "ls" -1 as input
Date: Mon, 9 Jan 2017 16:57:22 -0600

[Message part 1 (text/plain, inline)]

On 01/09/2017 03:22 PM, L A Walsh wrote:
> 
> 
> Eric Blake wrote:
>> On 01/09/2017 02:35 PM, L A Walsh wrote:
>>>     I also stand for it being strongly against GNU
>>> standards to add more such behaviors.
>>
>> 'ls' did not recently add any more cases where tty output differs from
>> non-tty output when all other things are equal in the default state.
>> All that changed was that tty output is formatted differently than it
>> has been in the past. 
> ----
>     That is not the case.
> 
> Created dir with common files w/spaces in them from people's
> Doc dirs.  I also replaced the space in "My Pictures w/a tab to
> see the effect -- though it wasn't necessary.

Okay, I'm going to follow along, using two different versions of ls:
8.25 (Fedora 25) and 8.4 (RHEL 6).

$ mkdir -p /tmp/ls
$ cd /tmp/ls
$ touch 'a b' c

Now, let's check whether output differed based on tty in the old
version, 8.4:

$ \ls
a b  c
$ \ls | cat
a b
c
$ \ls -1 | cat
a b
c
$ \ls --version | head -n1
ls (GNU coreutils) 8.4

Yep, the output is different based on whether stdout was a tty.

Now, let's check again with a modern version:

$ \ls
'a b'  c
$ \ls | cat
a b
c
$ \ls -1 | cat
a b
c
$ \ls --version | head -n1
ls (GNU coreutils) 8.25

Yep, the output STILL differs based on whether stdout was a tty.  But
note, that in BOTH cases, the output of '\ls -1' is identical when
stdout is NOT a tty.  If you are parsing the output of ls -1 in a
script, chances are you are NOT creating a pty, and therefore the output
you are parsing did NOT change.  The only content that changed between
ls versions is the CONTENT of the tty output, but NOT the logic of
whether to use tty output, nor the content when tty output is not involved.

The GNU Coding Standards do NOT forbid the content of tty output to
change between versions (that is, the tty output of version x.1 and x.2
need not be the same), they only state that if version x.1 outputs a
particular text to a non-tty, it should default to outputting that same
text to a tty, IF that difference in behavior was not already
grandfathered in.

Let's go read the actual GNU Coding Standards, instead of paraphrasing it:

https://www.gnu.org/prep/standards/standards.html#User-Interfaces

"Likewise, please don’t make the behavior of a command-line program
depend on the type of output device it gets as standard output or
standard input. Device independence is an important principle of the
system’s design; do not compromise it merely to save someone from typing
an option now and then. (Variation in error message syntax when using a
terminal is ok, because that is a side issue that people do not depend on.)

If you think one behavior is most useful when the output is to a
terminal, and another is most useful when the output is a file or a
pipe, then it is usually best to make the default behavior the one that
is useful with output to a terminal, and have an option for the other
behavior. You can also build two different versions of the program with
different names.

There is an exception for programs whose output in certain cases is
binary data. Sending such output to a terminal is useless and can cause
trouble. If such a program normally sends its output to stdout, it
should detect, in these cases, when the output is a terminal and give an
error message instead. The -f option should override this exception,
thus permitting the output to go to the terminal.

Compatibility requires certain programs to depend on the type of output
device. It would be disastrous if ls or sh did not do so in the way all
users expect. In some of these cases, we supplement the program with a
preferred alternate version that does not depend on the output device
type. For example, we provide a dir program much like ls except that its
default output format is always multi-column format."


Nothing in there says that we can't change what ls outputs to a termianl
between versions of ls, merely that the reason that ls DEFAULTS to
different output to terminal than to non-terminal is historical.
Furthermore, it _encourages_ output to a terminal to default to being
safe in the presence of binary data (filenames can contain control
characters, which if not properly quoted are indeed disastrous to the
terminal), and does not forbid us from using a version of quoting that
is less ambiguous in newer versions of ls than it was in previous versions.

>> your claim that ls' behavior is non-standard has been proven to be
>> false; I quoted the line from POSIX that explicitly permits ls
> ---
>     By permitting it as an exception, you are admitting it
> is non-standard compared to other programs.

Huh? The fact that ls is allowed to have implementation-defined output
when writing to a tty is standard.  It may not be consistent with other
applications, but the point being discussed here wasn't consistency, but
standards-compliant.  You can't be non-standard by comparing to another
app - either the behavior of ls is permitted by the standard or it is
not permitted, regardless of what other apps do.

>     
>> Now you are excluding a fact.  POSIX _does_ state that the output of
>> 'ls' when targetting a tty is implementation-defined,
> ---
>     I'm not quoting posix -- I'm quoting the GNU
> standards you mention.

I actually quoted it above.  So far, you have only paraphrased it, and I
think you are trying to claim that the GNU Coding Standards do not let
us change the tty-only output between versions, but that is not what it
says.

> 
> 
> Anyway, as demonstrated above, the output is not the same and
> varies based on whether or not the destination is a tty.

Indeed - and we've been telling you that default ls output has ALWAYS
varied based on whether or not the destination is a tty.  The fact that
it STILL varies (although with different content than in older versions
of ls) is nothing new.  We are not violating the requirement that no NEW
code should have a difference based on tty, because we already HAVE a
difference based on tty.

> 
> Your claim that the output is the same is demonstrably false.

No, you have only managed to prove that the tty-only output of ls has
changed between versions, and NOT that the tty-only and non-tty output
used to be the same but now differ.  The tty-only and non-tty output
have ALWAYS been different, and I was not claiming that they have been
the same.

> They are not functionally equivalent.

You are trying to feed tty output into a shell script. In general, this
is not portable.  But this is not ls' fault.  It has NEVER been portable
to try and parse the tty-only output of ls in a shell script.  Just
because the outcome of trying to parse non-portable output has changed
doesn't mean that the output is broken - rather, your attempt to feed
non-portable output to the shell in the first place is broken.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.