GNU bug report logs - #22195
deviation from POSIX in tee

Previous Next

Package: coreutils;

Reported by: Eric Renouf <erenouf <at> skaion.com>

Date: Thu, 17 Dec 2015 20:35:01 UTC

Severity: normal

Tags: notabug

Done: Assaf Gordon <assafgordon <at> gmail.com>

Bug is archived. No further changes may be made.

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 22195 in the body.
You can then email your comments to 22195 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

View this report as an mbox folder, status mbox, maintainer mbox


Report forwarded to bug-coreutils <at> gnu.org:
bug#22195; Package coreutils. (Thu, 17 Dec 2015 20:35:01 GMT) Full text and rfc822 format available.

Acknowledgement sent to Eric Renouf <erenouf <at> skaion.com>:
New bug report received and forwarded. Copy sent to bug-coreutils <at> gnu.org. (Thu, 17 Dec 2015 20:35:01 GMT) Full text and rfc822 format available.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Eric Renouf <erenouf <at> skaion.com>
To: bug-coreutils <at> gnu.org
Subject: deviation from POSIX in tee
Date: Thu, 17 Dec 2015 15:17:29 -0500 (Eastern Standard Time)
When using tee with process substitution, if the process exits early it 
will cause tee to get a SIGPIPE after trying to write to the closed named 
pipe (per man 2 write).  This causes tee to exit without finishing writing 
to any additional files or processes, which seems to be a violation of the 
POSIX standard at 
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/tee.html

specifically the Consequences of Errors section which states

"If a write to any successfully opened file operand fails, writes to other 
successfully opened file operands and standard output shall continue, but 
the exit status shall be non-zero. Otherwise, the default actions 
specified in Utility Description Defaults apply."

To recreate the problem I ran the following under bash:

for i in {1..300}; do
    echo "$i"
    echo "$i" >&2
    sleep 1
done | tee >(head -1 > h.txt; echo "Head done") \
 >(tail -1 > t.txt) >/dev/null

And I see output like the following:

1
Head done
2

and then it exits with echo "${PIPESTATUS[@]}" showing

141 141

and t.txt from the second process substitution only having "1" in it

This was found from a Stack Overflow question at
http://stackoverflow.com/questions/34340706/incorrect-results-with-bash-process-substitution-and-tail

I have verified this behavior with tee (GNU coreutils) 8.5 and tee (GNU 
coreutils) 8.13




Information forwarded to bug-coreutils <at> gnu.org:
bug#22195; Package coreutils. (Fri, 18 Dec 2015 03:23:01 GMT) Full text and rfc822 format available.

Message #8 received at 22195 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eric Renouf <erenouf <at> skaion.com>, 22195 <at> debbugs.gnu.org
Subject: Re: bug#22195: deviation from POSIX in tee
Date: Thu, 17 Dec 2015 19:22:01 -0800
Eric Renouf wrote:
> If a write to any successfully opened file operand fails

But the write didn't fail here. Instead, a signal was sent to 'tee'. If you 
don't want the signal, trap it. E.g.:

trap '' PIPE
for i in {1..300}; do
    echo "$i"
    echo "$i" >&2
    sleep 1
done | tee >(head -1 > h.txt; echo "Head done") \
 >(tail -1 > t.txt) >/dev/null

will give the behavior you want.

So there is no deviation from POSIX here.




Information forwarded to bug-coreutils <at> gnu.org:
bug#22195; Package coreutils. (Fri, 18 Dec 2015 07:20:01 GMT) Full text and rfc822 format available.

Message #11 received at 22195 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: Paul Eggert <eggert <at> cs.ucla.edu>, Eric Renouf <erenouf <at> skaion.com>,
 22195 <at> debbugs.gnu.org
Subject: Re: bug#22195: deviation from POSIX in tee
Date: Fri, 18 Dec 2015 08:19:35 +0100
On 12/18/2015 04:22 AM, Paul Eggert wrote:
> Eric Renouf wrote:
>> If a write to any successfully opened file operand fails
> 
> But the write didn't fail here. Instead, a signal was sent to 'tee'. If you 
> don't want the signal, trap it. E.g.:
> 
> trap '' PIPE
> for i in {1..300}; do
>      echo "$i"
>      echo "$i" >&2
>      sleep 1
> done | tee >(head -1 > h.txt; echo "Head done") \
>   >(tail -1 > t.txt) >/dev/null
> 
> will give the behavior you want.
> 
> So there is no deviation from POSIX here.

Furthermore, tee got the new --output-error=MODE option in
version 8.24 to have more control over this:

  $ src/tee --help
  ...
    -p                        diagnose errors writing to non pipes
      --output-error[=MODE]   set behavior on write error.  See MODE below
  ...
  MODE determines behavior with write errors on the outputs:
    'warn'         diagnose errors writing to any output
    'warn-nopipe'  diagnose errors writing to any output not a pipe
    'exit'         exit on error writing to any output
    'exit-nopipe'  exit on error writing to any output not a pipe
  The default MODE for the -p option is 'warn-nopipe'.
  The default operation when --output-error is not specified, is to
  exit immediately on error writing to a pipe, and diagnose errors
  writing to non pipe outputs.

A minor note to your command: instead of redirecting tee's output to
/dev/null, you could use another pipe to save the write operations
in tee to it:

  producer \
    | tee --output-error=warn \
        >(head -1 > h.txt; echo "Head done") \
        | tail -1 > t.txt

Have a nice day,
Berny




Information forwarded to bug-coreutils <at> gnu.org:
bug#22195; Package coreutils. (Fri, 18 Dec 2015 11:14:01 GMT) Full text and rfc822 format available.

Message #14 received at 22195 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>, Eric Renouf <erenouf <at> skaion.com>,
 22195 <at> debbugs.gnu.org
Subject: Re: bug#22195: deviation from POSIX in tee
Date: Fri, 18 Dec 2015 11:13:10 +0000
tag 22195 notabug
close 22195
stop

On 18/12/15 03:22, Paul Eggert wrote:
> Eric Renouf wrote:
>> If a write to any successfully opened file operand fails
> 
> But the write didn't fail here. Instead, a signal was sent to 'tee'. If you 
> don't want the signal, trap it. E.g.:
> 
> trap '' PIPE
> for i in {1..300}; do
>      echo "$i"
>      echo "$i" >&2
>      sleep 1
> done | tee >(head -1 > h.txt; echo "Head done") \
>   >(tail -1 > t.txt) >/dev/null
> 
> will give the behavior you want.
> 
> So there is no deviation from POSIX here.

Generally you don't want to ignore SIGPIPE.
http://pixelbeat/programming/sigpipe_handling.html
as then you have to deal with EPIPE from write():

  $ trap '' PIPE
  $ seq 100000 | tee >(head -n1) > >(sed -n '/10000/{p;q}')
  1
  10000
  seq: write error

As Bernhard said, the new -p option is a solution to this:

  $ seq 100000 | tee >(head -n1) > >(tail -n1)
  1
  14139

  $ seq 100000 | tee -p >(head -n1) > >(tail -n1)
  1
  100000

Note that's an option, rather than the default behavior
as POSIX states to handle SIGPIPE in the default manner,
and more importantly to support existing cases like the
following (yes we don't support both modes of operation):

  $ yes | tee log | timeout process

Since this is somewhat of a subjective default,
and this issue pops up so frequently, I've just added it to:
http://www.pixelbeat.org/docs/coreutils-gotchas.html#tee

cheers,
Pádraig




Information forwarded to bug-coreutils <at> gnu.org:
bug#22195; Package coreutils. (Fri, 18 Dec 2015 18:39:02 GMT) Full text and rfc822 format available.

Message #17 received at 22195 <at> debbugs.gnu.org (full text, mbox):

From: Bob Proulx <bob <at> proulx.com>
To: Eric Renouf <erenouf <at> skaion.com>, 22195 <at> debbugs.gnu.org
Subject: Re: bug#22195: deviation from POSIX in tee
Date: Fri, 18 Dec 2015 11:38:53 -0700
Pádraig Brady wrote:
> Paul Eggert wrote:
> > trap '' PIPE
> 
> Generally you don't want to ignore SIGPIPE.
> http://pixelbeat/programming/sigpipe_handling.html
> as then you have to deal with EPIPE from write():

I wanted to add emphasis to this.  Ignoring SIGPIPE causes a cascade
of associated problems.  Best not to do it.

Bob

P.S. Typo alert:
  http://pixelbeat/programming/sigpipe_handling.html
Should be:
  http://www.pixelbeat.org/programming/sigpipe_handling.html




Added tag(s) notabug. Request was from Assaf Gordon <assafgordon <at> gmail.com> to control <at> debbugs.gnu.org. (Wed, 24 Oct 2018 21:37:02 GMT) Full text and rfc822 format available.

bug closed, send any further explanations to 22195 <at> debbugs.gnu.org and Eric Renouf <erenouf <at> skaion.com> Request was from Assaf Gordon <assafgordon <at> gmail.com> to control <at> debbugs.gnu.org. (Wed, 24 Oct 2018 21:37:02 GMT) Full text and rfc822 format available.

bug archived. Request was from Debbugs Internal Request <help-debbugs <at> gnu.org> to internal_control <at> debbugs.gnu.org. (Thu, 22 Nov 2018 12:24:10 GMT) Full text and rfc822 format available.

This bug report was last modified 6 years and 262 days ago.

Previous Next


GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.