Subject: bug#34488: Add sort --limit, or document workarounds for sort|head error messages

From: Eric Blake <[email-masked]>
Date: Fri, 15 Feb 2019 09:20:58 -0600

On 2/15/19 8:43 AM, 積丹尼 Dan Jacobson wrote:
> Things start out cheery, but quickly get ugly,
> 
> $ for i in 9 99 999 9999 99999; do seq $i|sort -n|sed 5q|wc -l; done
> 5
> 5
> 5
> 5
> sort: write failed: 'standard output': Broken pipe
> sort: write error
> 5
> sort: write failed: 'standard output': Broken pipe
> sort: write error

Your code is demonstrating what happens when sed ends processing without
consuming all of sort's output, to the point that sort is producing more
output than what stdio can buffer, and thus with enough pending output
will trigger a situation of SIGPIPE - but when SIGPIPE is ignored, that
instead turns into an EPIPE failure to write(), and sort treats all
write() failures as worthy of error reporting.  If you want sort to die
silently, then don't ignore SIGPIPE, so that sort won't see EPIPE and
thus won't be noisy.

$ (trap '' PIPE; seq 9999 | sort -n | sed 5q | wc -l)
5
sort: write failed: 'standard output': Broken pipe
sort: write error
$ (trap - PIPE; seq 9999 | sort -n | sed 5q | wc -l)
5

Except that POSIX has the nasty requirement that sh started with an
inherited ignored SIGPIPE must silently ignore all attempts from within
the shell to restore SIGPIPE handling to child processes of the shell:

$ (trap '' PIPE; bash -c 'trap - PIPE; \
   seq 9999 | sort -n | sed 5q | wc -l')
5
sort: write failed: 'standard output': Broken pipe
sort: write error

And unfortunately, there are several common cases of badly-behaved
environment setups that leave SIGPIPE ignored in the child (for example,
a quick google search found this:
https://blog.nelhage.com/2010/02/a-very-subtle-bug/)

You HAVE to use some other intermediate program if you want to override
an inherited ignored SIGPIPE in sh into an inherited default-behavior
SIGPIPE in sort.

Perhaps coreutils should teach 'env' a command-line option to forcefully
reset SIGPIPE back to default behavior (or add a new coreutil that does
the same idea), as a way to work around POSIX' requirement on sh.  If we
did that, then even if your sh is started with SIGPIPE ignored (so that
the shell itself can't restore default behavior), you could do this
theoretical invocation:

$ seq 9999 | env --default-signal PIPE sort -n | sed 5q | wc -l
5

and avoid the EPIPE failures because sort is forced to start with
SIGPIPE handling rather than ignored.

> 
> Therefore, kindly add a sort --limit=n,

Not scalable.  The problem you encountered is NOT the fault of sort, but
is common to ALL utilities which dutifully report write() failures on
EPIPE errors, which in turn happens when SIGPIPE is ignored.  Adding an
option to every such utility that produces outtput is not a good use of
time.  If you want to change things universally, perhaps you should
petition POSIX to change the requirements to allow applications the
option to silently exit without an error message on EPIPE failures to
write(), instead of the current wording that all write() errors must be
diagnosed.

> and/or on (info "(coreutils) sort invocation")
> admit the problem, and give some workarounds, lest
> our scripts occasionally spew error messages seemingly randomly,
> just when the boss is looking.

The problem is not in sort, but in the fact that your environment is
ignoring SIGPIPE. Documenting something in sort doesn't scale, but
perhaps the documentation could mention SIGPIPE considerations in a more
global chapter covering all of the coreutils.

> 
> And no fair saying "just save the output" (could be big) "into a file
> first, and do head(1) or sed(1) on that."
> 

If you have an app that exits noisily on write failures to an early-exit
pipe, your solutions are to quit ignoring SIGPIPE, or to change the
pipeline to consume all of the output instead of exiting early and
causing write failure.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org