GNU bug report logs - #25553
How many coreutils rely on tabs to align things (other, than 'du'?

Previous Next

To reply to this bug, email your comments to 25553 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: bug-coreutils <at> gnu.org
Subject: How many coreutils rely on tabs to align things (other, than 'du'?
Date: Thu, 26 Jan 2017 22:35:03 -0800

Are there other utils that rely on tabs to align
output as much as 'du'?

What would be objections to fixing 'du' or others
that rely on such?

Message #8 received at 25553 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: L A Walsh <coreutils <at> tlinx.org>, 25553 <at> debbugs.gnu.org
Subject: Re: bug#25553: How many coreutils rely on tabs to align things
 (other, than 'du'?
Date: Fri, 27 Jan 2017 14:26:05 -0800

L A Walsh wrote:
> Are there other utils that rely on tabs to align
> output as much as 'du'?

Sure. 'ls', 'fmt', 'cut', 'cat', 'pr',....

> What would be objections to fixing 'du' or others
> that rely on such?

It would depend on the proposed change.

Message #11 received at 25553 <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 25553 <at> debbugs.gnu.org
Subject: Re: bug#25553: How many coreutils rely on tabs to align things (other, 
 than 'du'?
Date: Fri, 27 Jan 2017 18:33:57 -0800

Paul Eggert wrote:
> L A Walsh wrote:
>> Are there other utils that rely on tabs to align
>> output as much as 'du'?
> 
> Sure. 'ls', 'fmt', 'cut', 'cat', 'pr',....
> 
>> What would be objections to fixing 'du' or others
>> that rely on such?
> 
> It would depend on the proposed change.
----
I'm working on some loose ends, but basically, 

First, it would default to providing no change.  ;-)

Through use of a switch it can expand the tabs to spaces
using a default of every 8th column (after 1) (using a
switch value of 'ExpandTo').

And with 2 env vars, TTY_TABSIZE & TTY_TABSET
it can re-entabulate the output to use the user's 
tabsize as related through the env vars.

The env vars cause no change in program behavior unless
the switch specifies 'AlignTo'.

I developed the 'tabout' feature in a 'tabout.c+h' file
and added it to 'du' -- but noted it was not the best way
to test, so wrote a driver-program, 'retab', to allow focus
testing on the retabbing code that does the retabbing on
files or stdin and sends the result to stdout.

...that's a rough description...

Will let you know as I finish off some loose ends and code
refactoring/tidying...

Message #14 received at 25553 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: L A Walsh <coreutils <at> tlinx.org>
Cc: 25553 <at> debbugs.gnu.org
Subject: Re: bug#25553: How many coreutils rely on tabs to align things
 (other, than 'du'?
Date: Sat, 28 Jan 2017 02:41:24 -0800

L A Walsh wrote:
> I'm working on some loose ends, but basically,
> First, it would default to providing no change.  ;-)
>
> Through use of a switch it can expand the tabs to spaces
> using a default of every 8th column (after 1) (using a
> switch value of 'ExpandTo').
>
> And with 2 env vars, TTY_TABSIZE & TTY_TABSET
> it can re-entabulate the output to use the user's tabsize as related through the
> env vars.

It sounds like this could be done via 'du | expand', so why change 'du'?

Also, environment variables cause too many problems. If new functionality is 
needed here (which is not yet clear), it should be via new options.

Message #17 received at submit <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: bug-coreutils <at> gnu.org
Subject: Re: bug#25553: How many coreutils rely on tabs to align things (other, 
 than 'du'?
Date: Sat, 28 Jan 2017 14:45:24 -0800

Paul Eggert wrote:
> L A Walsh wrote:
>> I'm working on some loose ends, but basically,
>> First, it would default to providing no change.  ;-)
>>
>> Through use of a switch it can expand the tabs to spaces
>> using a default of every 8th column (after 1) (using a
>> switch value of 'ExpandTo').
>>
>> And with 2 env vars, TTY_TABSIZE & TTY_TABSET
>> it can re-entabulate the output to use the user's tabsize as related 
>> through the
>> env vars.
> 
> It sounds like this could be done via 'du | expand', so why change 'du'?
> 
> Also, environment variables cause too many problems. If new 
> functionality is needed here (which is not yet clear), it should be via 
> new options.
---
	The idea would be to have it be transparent. Adding
on various output filters is hardly transparent.  So would you 
care to show me how you would invoke expand to change 'du's 
tab-output to that on my output device (terminal seems a bit of
a misnomer these days), but it is a tty-type window compatible
with xterm, the linux console, vt100's/vt102's as well some 
standards.

	As for env-vars.  What problems are you envisioning.
As I mentioned, both env-vars and program options are required --
they serve different purposes.  Alternatively, you could have
settings files (i.e. /etc/gnurc & "~/.gnurc") that could store
settings designed to apply across multiple tools, but like the
ENV settings I'm conceiving of would be like the envvar "TERM"
or an LC_CTYPE to describe the output device's character encoding.

	Indeed, in this case, TTY_TAB settings are specific
to a person's terminal -- just like TERM and LC_CTYPE

	What type of problems outweighing the benefits 
do you see with env vars like TERM and LC_CTYPE.

	As for 'expand' handling the job -- how would
expand convert tabstops from 'du' into the tabstops used by
an arbitrary terminal?  For that matter, since directory and
file sizes often exceed 8-characters forcing du's output 
to not align, how can a user have 'du' align it's columns
automatically, as my additions can do?

Message #20 received at 25553 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: L A Walsh <coreutils <at> tlinx.org>
Cc: 25553 <at> debbugs.gnu.org
Subject: Re: bug#25553: How many coreutils rely on tabs to align things
 (other, than 'du'?
Date: Sat, 28 Jan 2017 16:20:07 -0800

L A Walsh wrote:
>     The idea would be to have it be transparent. Adding
> on various output filters is hardly transparent.

Nor is having behavior depend on environment variables.

>     As for env-vars.  What problems are you envisioning.

Shell scripts might stop working because the environment variables are not set 
up in the usual or desired way. This is a common problem with other environment 
variables. In the past, we've relied on the environment too often for 
relatively-unimportant things. We should not repeat this mistake.

>     As for 'expand' handling the job -- how would
> expand convert tabstops from 'du' into the tabstops used by
> an arbitrary terminal?  For that matter, since directory and
> file sizes often exceed 8-characters forcing du's output to not align, how can a
> user have 'du' align it's columns
> automatically, as my additions can do?

Send du's output to a file, look for the longest number in the file, and adjust 
column width accordingly. Whatever tabstops you like, you can tell to 'expand'. 
Expanding tabs is the job of 'expand', not of every program that outputs tabs.

Message #25 received at submit <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: bug-coreutils <at> gnu.org
Subject: Re: bug#25553: How many coreutils rely on tabs to align things (other, 
 than 'du'?
Date: Sat, 28 Jan 2017 22:13:02 -0800

Paul Eggert wrote:
> L A Walsh wrote:
>>     The idea would be to have it be transparent. Adding
>> on various output filters is hardly transparent.
> 
> Nor is having behavior depend on environment variables.
---
	The behavior isn't dependent on environment variables.
The behavior is selected by the run-time switch values.

	The run-time switch, like ls's --color flag get color
values from LS_COLORS (which came from the user's TERM settings).
There, you have core utils relying on a 2nd ENV var to cache the
results of the 1st (TERM) env var.



>>     As for env-vars.  What problems are you envisioning.
> 
> Shell scripts might stop working because the environment variables are 
> not set up in the usual or desired way.
---
	This is already a problem with functions and aliases as well
as PATH differences.  Though, for that matter, scripts *RELY* 
on environment variables already.  This usage could conceivable be
no different than the case of 'ls' altering its screen output if
output is not used in a script.  How would that not address your concern?


> This is a common problem with 
> other environment variables. In the past, we've relied on the 
> environment too often for relatively-unimportant things. We should not 
> repeat this mistake.
----
	I've never had the problem, but what mistakes are you claiming
were made by use of environment variables for TERMinal settings, character
set settings or language settings?  In fact, doesn't gnu make heavy use
of LC_specific variables -- and would fail miserably in localization if
those variables were disallowed?

	You can't argue that there are no valid uses for ENV vars.
At the same time, you skirted the issue of using system-wide and per-user
config files which might also be used for similar purpose.



>>     As for 'expand' handling the job -- how would
>> expand convert tabstops from 'du' into the tabstops used by
>> an arbitrary terminal?  For that matter, since directory and
>> file sizes often exceed 8-characters forcing du's output to not align, 
>> how can a user have 'du' align it's columns
>> automatically, as my additions can do?
> 
> Send du's output to a file,
----

	And if the program doesn't generate the same output
to a file as to a 'TTY'?  However, this is for TTY-output 
devices -- it's not intended for inter-program communications.
In fact, as you point out, that should likely be excluded by default.  

	You mention 'ls' -- as an example of another tab-using
program.  How can I send ls's default output through a filter
and have it be the same as to a TTY?  By default -- one can't.
The same happens for users of Gnu's libc.  It, also dumps output 
to a terminal that, by default, is not redirectable. 



> look for the longest number in the file, and 
> adjust column width accordingly. Whatever tabstops you like, you can 
> tell to 'expand'. Expanding tabs is the job of 'expand', not of every 
> program that outputs tabs.
----
	Expanding tabs, for *some* gnu programs, could be done w/expand, 
but you didn't mention how the output would be re-encoded for a user's
tab settings on output, not to mention how you would redirect
'ls's output through a filter or to a file to any effect.

Message #28 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: L A Walsh <coreutils <at> tlinx.org>
Cc: bug-coreutils <at> gnu.org
Subject: Re: bug#25553: How many coreutils rely on tabs to align things
 (other, than 'du'?
Date: Mon, 30 Jan 2017 15:56:33 -0800

On 01/28/2017 10:13 PM, L A Walsh wrote:
>
>
>     This is already a problem

Of course. All I am saying is that we should not make it worse.

> you skirted the issue of using system-wide and per-user
> config files which might also be used for similar purpose.

That's even worse. Coreutils doesn't do that now, thank goodness.

> How can I send ls's default output through a filter
> and have it be the same as to a TTY?

'ls -C'.

> you didn't mention how the output would be re-encoded for a user's
> tab settings on output

That could be the job of 'expand'. The user's tab settings can be 
specified as arguments to 'expand'. If 'expand' doesn't have the desired 
functionality now, perhaps we should improve 'expand'. But we should not 
be modifying every program to do expansion; just 'expand'. That's the 
software tools design philosophy.

Message #31 received at submit <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: bug-coreutils <at> gnu.org
Subject: Re: bug#25553: How many coreutils rely on tabs to align things (other, 
 than 'du'?
Date: Mon, 30 Jan 2017 19:56:47 -0800

Paul Eggert wrote:
> On 01/28/2017 10:13 PM, L A Walsh wrote:
>>
>>
>>     This is already a problem
> 
> Of course. All I am saying is that we should not make it worse.
---
	"Do as I say, not as I do?"  
	Wasn't 'ls' just change recently?  You can come
up with arguments that it really wasn't made worse, but I
and others noticed it -- where as before, it wasn't
that noticeable, in part, because ls ***already*** has an
option to expand tabs in it that some have resorted to to
get reasonable output.  It _does NOT_ have an option to
make use of a user's tab settings, nor does it have an option
to always generate the same output whether to a TTY or 
a pipe/file.

	Just as 'ls' already added options for user
control of color tab-expansion and such (up to a point --
still can't have it generate # of columns I want as it
reads those in from the TTY/Env), I'm adding the ability
to customize tab-usage on output.


>> you skirted the issue of using system-wide and per-user
>> config files which might also be used for similar purpose.
> 
> That's even worse. Coreutils doesn't do that now, thank goodness.
----
	Well, on that I'm fine -- ENV vars are cleaner for
terminal settings anyway, but thought I'd offer an
alternative. ;-)


> 
>> How can I send ls's default output through a filter
>> and have it be the same as to a TTY?
> 
> 'ls -C'.
---	
	Doesn't work.  I type 'ls' and get (abbreviated):

'['         id      set-fields.c
b2sum          id.c      set-fields.h
basename.o       join.o      sha512sum
blake2         kill      shred
cat          kill.c      shred.c
chcon.c        libstdbuf.c   shuf.o
chcon.o        libstdbuf.so    single-binary.mk
cp.c         mknod     sum
cp.o         mknod.c     sum.c
csplit         mknod.o     sum.o
csplit.o       mktemp.c      sync.c
cu-progs.mk        mktemp.o      sync.o
cut          mv      system.h
cut.c          mv.c      t
date         nice      tabout.c.000
date.o         nice.o      tabout.c.002
---
I type 'ls -C' and it doesn't look anything like that. 


>> you didn't mention how the output would be re-encoded for a user's
>> tab settings on output
> 
> That could be the job of 'expand'. The user's tab settings can be 
> specified as arguments to 'expand'. If 'expand' doesn't have the desired 
> functionality now, perhaps we should improve 'expand'.
---
	Expand would need to run in background and only apply
changes to output that is going to a user's terminal.  If you
think that is possible I'd be happy to see/test/try it.

> But we should not 
> be modifying every program to do expansion; just 'expand'. That's the 
> software tools design philosophy.
----

	The various GNU and coreutil tools have mods in them
to support different user terminals as well as different 
user languages (and locales) -- there is no generic filter done
to take program output and translate it on the fly -- it's built
into each program.  But we **ARE NOT** talking piped or streamed 
programs -- but ones that alter the 1-char=>1 space whether
to tty or file paradigm.  They can't be filtered because they generate
non-linear output (1 char in = 1 char advanced on the TTY).  Even
tabs don't really adhere to that as they go forward a variable amount
of spacing depending on what came before and how the output device
is setup to emulate tabs.


	This is the same thing -- but like translations, can't
be done in a "post-filter" stage if for no other reason that
programs like 'ls' don't generate the same output to TTY's as
to 'filters'.  There is no way for you to alter a general
purpose expansion program like 'entab' to show you a filtered
view of 'ls' because 'ls' generates different output to
the TTY vs. filters.

	Even *if* the output was the same -- it is already the 
case that the tools don't have the option for a user to 
customize their output, say, to always expand tabs, or to
always use user-specified tabsizes.

	The only solution to fix having different end-user-output
devices / languages was to provide a common interface or Terminal
API -- or to modify each program for translation into the the 
number of tabs used in each users "locale".  Currently, all of the
programs have some modifications for each end user's locale.  Some
adapt to the users output device, if ONLY, in-so-much as adapting
to the user's TTY width.

	I don't think all the programs need modification, but if
you really want to not touch all programs, then you are talking about
adding a library module to filter stdout & maybe stderr.  But
having it work across all tools vs. a half-a-dozen is a very
different scope and amount of work.

	I'm only proposing to "refilter" (retab)
a handful of programs -- and only if the user includes a switch
for that program.  They can easily toss it into an alias and 
forget about it -- and if the changes are in those programs, they
can be, optionally limited to only changing terminal output --
not pipe output.

Message #34 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: L A Walsh <coreutils <at> tlinx.org>
Cc: bug-coreutils <at> gnu.org
Subject: Re: bug#25553: How many coreutils rely on tabs to align things
 (other, than 'du'?
Date: Mon, 30 Jan 2017 22:05:24 -0800

L A Walsh wrote:
>> 'ls -C'.
> ---
>     Doesn't work.

It works for me. If it doesn't work for you, perhaps you need to investigate 
why. This alone might solve whatever problem you're having with 'ls'.

As for the rest of the proposal, it sounds too complicated for what would be 
only a minor improvement.

Message #37 received at submit <at> debbugs.gnu.org (full text, mbox):

From: L A Walsh <coreutils <at> tlinx.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>, bug-coreutils <at> gnu.org
Subject: Re: bug#25553: How many coreutils rely on tabs to align things (other, 
 than 'du'?
Date: Tue, 31 Jan 2017 12:10:33 -0800

Paul Eggert wrote:
> L A Walsh wrote:
>>> 'ls -C'.
>> ---
>>     Doesn't work.
> 
> It works for me. 
---
	Ah, I tested it the wrong window where I 
hadn't unset LS_OPTIONS...  If only this was
all "automatic" like locale settings or choosing
terminal-specific color codes... ;^/

	It still doesn't align the columns.

'['         id      set-fields.c
b2sum          id.c      set-fields.h
chown-core.o       logname     src_b2sum-md5sum.o
chown.c        logname.c     src_base32-base64.o
chroot.c       ls      src_ginstall-cp-hash.o
comm         ls-vdir.c     src_libstdbuf_so-libstdbuf.o
core         make-prime-list.c   src_sha512sum-md5sum.o
expand-common.h      printenv      truncate.c
extent-scan.c        prog-fprintf.h    tty.o
extent-scan.h        prog-fprintf.o    uname
find-mount-point.o   realpath.o    uniq.o

	I guess I need to look at ls next.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.