GNU bug report logs - #8961
stdbuf has no effect on some programs

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 8961 in the body.
You can then email your comments to 8961 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: bug-coreutils <at> gnu.org
Subject: stdbuf has no effect on some programs
Date: Wed, 29 Jun 2011 22:59:13 +0200

Hi,

The glibc 'iconv' program buffers its input, and some people don't like this.
I thought that the 'stdbuf' program could remove the buffering, but it does not
work.

How to reproduce:

Create this script and make it executable:
================================== producer ==================================
#!/bin/sh
echo Hello
/bin/sleep 3
echo World
==============================================================================

$ ./producer | /usr/bin/iconv -f ASCII
Hello
World

All the output comes at the end.

$ stdbuf -o 0 ./producer | /usr/bin/iconv -f ASCII
Hello
World

All the output comes at the end.

$ ./producer | stdbuf -i 0 /usr/bin/iconv -f ASCII
Hello
World

All the output comes at the end.

$ stdbuf -o 0 ./producer | stdbuf -i 0 /usr/bin/iconv -f ASCII
Hello
World

All the output comes at the end.

What do I need to do to get the output of the first line immediately?

/usr/bin/iconv is from glibc, but I get the same behaviour from libiconv's
'iconv' program too.

In $ ./producer | /bin/cat
I get the first line immediately, but the coreutils documentation
<http://www.gnu.org/software/coreutils/manual/html_node/stdbuf-invocation.html>
makes me think that 'stdbuf' is meant for those programs that do not work like
'cat'.

Bruno
-- 
In memoriam José Olaya <http://es.wikipedia.org/wiki/José_Olaya>

Message #8 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bruno Haible <bruno <at> clisp.org>
Cc: 8961 <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Thu, 30 Jun 2011 00:05:19 +0100

On 29/06/11 21:59, Bruno Haible wrote:
> Hi,
> 
> The glibc 'iconv' program buffers its input, and some people don't like this.
> I thought that the 'stdbuf' program could remove the buffering, but it does not
> work.

The following shows I think that iconv is bypassing stdio and buffering internally?

(echo; sleep 3; echo) | ltrace iconv -f ASCII

The stdbuf man page notes that:

NOTE: If COMMAND adjusts the buffering of its standard  streams  (`tee'
does  for  e.g.) then that will override corresponding settings changed
by `stdbuf'.  Also some filters (like `dd' and `cat'  etc.)  don't  use
streams for I/O, and are thus unaffected by `stdbuf' settings.

cheers,
Pádraig.

Message #13 received at 8961-done <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bruno Haible <bruno <at> clisp.org>
Cc: 8961-done <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Thu, 30 Jun 2011 00:29:22 +0100

On 30/06/11 00:05, Pádraig Brady wrote:
> On 29/06/11 21:59, Bruno Haible wrote:
>> Hi,
>>
>> The glibc 'iconv' program buffers its input, and some people don't like this.
>> I thought that the 'stdbuf' program could remove the buffering, but it does not
>> work.
> 
> The following shows I think that iconv is bypassing stdio and buffering internally?
> 
> (echo; sleep 3; echo) | ltrace iconv -f ASCII
> 
> The stdbuf man page notes that:
> 
> NOTE: If COMMAND adjusts the buffering of its standard  streams  (`tee'
> does  for  e.g.) then that will override corresponding settings changed
> by `stdbuf'.  Also some filters (like `dd' and `cat'  etc.)  don't  use
> streams for I/O, and are thus unaffected by `stdbuf' settings.

In fact iconv seems to buffer for ever and so it not scalable,
as demonstrated with this consuming all of memory:

  yes | iconv

It would be OK to treat '\n' simply in all
unibyte encodings and utf8 for example,
but that would introduce an inconsistency I suppose.
Though maybe iconv could employ more scalable
buffering at least, for all encodings?

cheers,
Pádraig.

Message #16 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: Pádraig Brady <P <at> draigbrady.com>
Cc: 8961 <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Thu, 30 Jun 2011 01:34:12 +0200

Hi Pádraig,

> The following shows I think that iconv is bypassing stdio and buffering internally?
> 
> (echo; sleep 3; echo) | ltrace iconv -f ASCII

Indeed, the glibc 'iconv' does not use stdio apparently:

__libc_start_main(0x402a00, 3, 0x7fff76b92d68, 0x409c90, 0x409c80 <unfinished ...>
setlocale(6, "")                                                                                 = "de_DE.UTF-8"
textdomain("libc")                                                                               = "libc"
argp_parse(0x60d2e0, 3, 0x7fff76b92d68, 0, 0x7fff76b92c5c)                                       = 0
strchr("ASCII", '/')                                                                             = NULL
strchr("", '/')                                                                                  = NULL
iconv_open(0x409daf, 0x7fff76b946e6, 0, 0x53494c4300494900, 0xfefefefefefefeff)                  = 0x613080
fileno(0x7f4c6e9726a0)                                                                           = 0
realloc(NULL, 32768)                                                                             = 0x00615100
read(0, "\n", 32768)                                                                             = 1
read(0, "\n", 32767)                                                                             = 1
read(0, "", 32766)                                                                               = 0
iconv(0x613080, 0x7fff76b8ab38, 0x7fff76b8ab30, 0x7fff76b92b48, 0x7fff76b92b40)                  = 0
...

> The stdbuf man page notes that:
> 
> NOTE: If COMMAND adjusts the buffering of its standard  streams  (`tee'
> does  for  e.g.) then that will override corresponding settings changed
> by `stdbuf'.  Also some filters (like `dd' and `cat'  etc.)  don't  use
> streams for I/O, and are thus unaffected by `stdbuf' settings.

This paragraph is written in a way that does not make the implications
very clear. How about the attached patch, meant to show the restrictions more
prominently?


2011-06-29  Bruno Haible  <bruno <at> clisp.org>

	* doc/coreutils.texi (stdbuf invocation): Mention the restrictions on
	the command more prominently.

--- doc/coreutils.texi.orig	Thu Jun 30 01:31:13 2011
+++ doc/coreutils.texi	Thu Jun 30 01:30:54 2011
@@ -15398,6 +15398,17 @@
 stdbuf @var{option}@dots{} @var{command}
 @end example
 
+@var{command} must start with the name of a program that
+@enumerate
+@item
+uses the ISO C @code{FILE} streams for input/output (note the
+programs @command{dd} and @command{cat} don't do that),
+
+@item
+does not adjust the buffering of its standard streams (note the
+program @command{tee} is not in this category).
+@end enumerate
+
 Any additional @var{arg}s are passed as additional arguments to the
 @var{command}.
 
@@ -15446,12 +15457,6 @@
 
 @end table
 
-NOTE: If @var{command} adjusts the buffering of its standard streams
-(@command{tee} does for e.g.) then that will override corresponding settings
-changed by @command{stdbuf}.  Also some filters (like @command{dd} and
-@command{cat} etc.) don't use streams for I/O, and are thus unaffected
-by @command{stdbuf} settings.
-
 @cindex exit status of @command{stdbuf}
 Exit status:
 
-- 
In memoriam José Olaya <http://es.wikipedia.org/wiki/José_Olaya>

Message #19 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: Pádraig Brady <P <at> draigbrady.com>
Cc: 8961 <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Thu, 30 Jun 2011 01:42:16 +0200

Pádraig Brady wrote:
> The following shows I think that iconv is bypassing stdio and buffering internally?
> 
> (echo; sleep 3; echo) | ltrace iconv -f ASCII

This is true for the glibc 'iconv' program. But stdbuf also does not work with
'iconv' from GNU libiconv, and this program uses stdio in a very simple form:
It reads from stdin using fread(). It does *not* call setvbuf explicitly.

$ (echo; sleep 3; echo) | ltrace iconv -f ASCII
__libc_start_main(0x804a250, 3, 0xffe20264, 0x804b3c0, 0x804b420 <unfinished ...>
strrchr("iconv", '/')                                                                            = NULL
setlocale(0, "")                                                                                 = "de_DE.UTF-8"
setlocale(5, "")                                                                                 = "de_DE.UTF-8"
bindtextdomain("libiconv", "/packages/gnu/share/locale")                                         = "/packages/gnu/share/locale"
textdomain("libiconv")                                                                           = "libiconv"
strlen("-f")                                                                                     = 2
libiconv_open(0x804c262, 0xffe206d7, 0xffe206d4, 0xffe206d4, 0)                                  = 0x804f4a0
iconv_canonicalize(0x804c262, 0xffe206d7, 0xffe206d4, 0xffe206d4, 0)                             = 0xf7effecc
iconv_canonicalize(0xffe206d7, 0xffe206d7, 0xffe206d4, 0xffe206d4, 0)                            = 0xf7efff38
libiconvctl(0x804f4a0, 6, 0xffe201a0, 0xffe206d7, 0)                                             = 0
libiconvctl(0x804f4a0, 5, 0xffe20190, 0x804f4a0, 0)                                              = 0
dcgettext(0, 0x804c27e, 5, 0, 0)                                                                 = 0x8057b50
libiconv(0x804f4a0, 0, 0, 0, 0)                                                                  = 0
fread(0xffe1f110, 1, 4096, 0xf7df5420)                                                           = 2
libiconv(0x804f4a0, 0xffe1d100, 0xffe1d104, 0xffe1d108, 0xffe1d10c)                              = 0
__errno_location()                                                                               = 0xf7cb5688
fwrite("\n\n", 1, 2, 0xf7df54c0

)                                                                 = 2
fread(0xffe1f110, 1, 4096, 0xf7df5420)                                                           = 0
libiconv(0x804f4a0, 0, 0, 0xffe1d0f4, 0xffe1d0f8)                                                = 0
ferror(0xf7df5420)                                                                               = 0
libiconv_close(0x804f4a0, 0xf7df5420, 0x8057b50, 0, 0)                                           = 0
ferror(0xf7df54c0)                                                                               = 0
fclose(0xf7df54c0)                                                                               = 0
exit(0 <unfinished ...>
+++ exited (status 0) +++

$ (echo Hello; sleep 3; echo World) | stdbuf iconv -f ASCII
Hello
World

All output comes after 3 seconds - stdbuf has no effect.

I'm using glibc-2.8.

Bruno
-- 
In memoriam José Olaya <http://es.wikipedia.org/wiki/José_Olaya>

Message #22 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: Pádraig Brady <P <at> draigbrady.com>
Cc: 8961 <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Thu, 30 Jun 2011 01:50:14 +0200

Pádraig Brady wrote:
> In fact iconv seems to buffer for ever and so it not scalable,
> as demonstrated with this consuming all of memory:
> 
>   yes | iconv

Indeed, yes. It would be a good idea to file a glibc bug report about it.

Bruno
-- 
In memoriam José Olaya <http://es.wikipedia.org/wiki/José_Olaya>

Message #25 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bruno Haible <bruno <at> clisp.org>
Cc: 8961 <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Thu, 30 Jun 2011 12:15:03 +0100

On 30/06/11 00:42, Bruno Haible wrote:
> Pádraig Brady wrote:
>> The following shows I think that iconv is bypassing stdio and buffering internally?
>>
>> (echo; sleep 3; echo) | ltrace iconv -f ASCII
> 
> This is true for the glibc 'iconv' program. But stdbuf also does not work with
> 'iconv' from GNU libiconv, and this program uses stdio in a very simple form:
> It reads from stdin using fread(). It does *not* call setvbuf explicitly.
> 
> $ (echo; sleep 3; echo) | ltrace iconv -f ASCII

> fread(0xffe1f110, 1, 4096, 0xf7df5420)                                                           = 2

> $ (echo Hello; sleep 3; echo World) | stdbuf iconv -f ASCII
> Hello
> World

From stdbuf.c /* FIXME: Should we mandate at least one option?  */

Anway I don't think that this works even if you specify -i0
because fread() only seems to return after feof() or ferror()
as demonstrated by interacting with the following run with ltrace.

#include <stdio.h>
int main(void)
{
    setvbuf (stdin, NULL, _IONBF, 0);
    setvbuf (stdout, NULL, _IONBF, 0);
    char buf[BUFSIZ];
    for (;;) {
      size_t count = fread (buf,1,BUFSIZ,stdin);
      fwrite (buf,1,count,stdin);
      if (feof (stdin))
        break;
    }
    return 0;
}

I guess glibc could return early if there were no partial records
(which there never will be with a size of 1).
It's a bit surprising it doesn't even given the
wording of the man page.

cheers,
Pádraig.

Message #28 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bruno Haible <bruno <at> clisp.org>
Cc: 8961 <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Thu, 30 Jun 2011 14:52:20 +0100

On 30/06/11 00:34, Bruno Haible wrote:
> Hi Pádraig,
> 
>> The following shows I think that iconv is bypassing stdio and buffering internally?
>>
>> (echo; sleep 3; echo) | ltrace iconv -f ASCII
> 
> Indeed, the glibc 'iconv' does not use stdio apparently:
> 
> __libc_start_main(0x402a00, 3, 0x7fff76b92d68, 0x409c90, 0x409c80 <unfinished ...>
> setlocale(6, "")                                                                                 = "de_DE.UTF-8"
> textdomain("libc")                                                                               = "libc"
> argp_parse(0x60d2e0, 3, 0x7fff76b92d68, 0, 0x7fff76b92c5c)                                       = 0
> strchr("ASCII", '/')                                                                             = NULL
> strchr("", '/')                                                                                  = NULL
> iconv_open(0x409daf, 0x7fff76b946e6, 0, 0x53494c4300494900, 0xfefefefefefefeff)                  = 0x613080
> fileno(0x7f4c6e9726a0)                                                                           = 0
> realloc(NULL, 32768)                                                                             = 0x00615100
> read(0, "\n", 32768)                                                                             = 1
> read(0, "\n", 32767)                                                                             = 1
> read(0, "", 32766)                                                                               = 0
> iconv(0x613080, 0x7fff76b8ab38, 0x7fff76b8ab30, 0x7fff76b92b48, 0x7fff76b92b40)                  = 0
> ...
> 
>> The stdbuf man page notes that:
>>
>> NOTE: If COMMAND adjusts the buffering of its standard  streams  (`tee'
>> does  for  e.g.) then that will override corresponding settings changed
>> by `stdbuf'.  Also some filters (like `dd' and `cat'  etc.)  don't  use
>> streams for I/O, and are thus unaffected by `stdbuf' settings.
> 
> This paragraph is written in a way that does not make the implications
> very clear. How about the attached patch, meant to show the restrictions more
> prominently?
> 
> 
> 2011-06-29  Bruno Haible  <bruno <at> clisp.org>
> 
> 	* doc/coreutils.texi (stdbuf invocation): Mention the restrictions on
> 	the command more prominently.
> 
> --- doc/coreutils.texi.orig	Thu Jun 30 01:31:13 2011
> +++ doc/coreutils.texi	Thu Jun 30 01:30:54 2011
> @@ -15398,6 +15398,17 @@
>  stdbuf @var{option}@dots{} @var{command}
>  @end example
>  
> +@var{command} must start with the name of a program that
> +@enumerate
> +@item
> +uses the ISO C @code{FILE} streams for input/output (note the
> +programs @command{dd} and @command{cat} don't do that),
> +
> +@item
> +does not adjust the buffering of its standard streams (note the
> +program @command{tee} is not in this category).
> +@end enumerate

I'll apply this.
I'll might also add a note about fread() blocking (which POSIX requires)
and that setting no _buffering_ doesn't impact that.

cheers,
Pádraig.

Message #31 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: Pádraig Brady <P <at> draigbrady.com>
Cc: 8961 <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Thu, 30 Jun 2011 21:27:40 +0200

Hi Pádraig,

> > $ (echo Hello; sleep 3; echo World) | stdbuf iconv -f ASCII
> > Hello
> > World
> 
> From stdbuf.c /* FIXME: Should we mandate at least one option?  */

Oops, I meant this:

$ (echo Hello; sleep 3; echo World) | stdbuf -i 0 iconv -f ASCII
Hello
World

Here too, the output comes only at the end, all at once.

> Anway I don't think that this works even if you specify -i0
> because fread() only seems to return after feof() or ferror()

Exactly. So, on which kinds of programs can -i0 work?
  - Not in programs that use read(0,...) because they are not affected
    by LD_PRELOADs.
  - Not in programs that use fread().
  - Not in programs that use getc() repeatedly - because the buffering
    behaviour of stdio is independent of whether you use fread() or [f]getc().
Can you name a single program on which -i0 works?

> int main(void)
> {
>     setvbuf (stdin, NULL, _IONBF, 0);
>     setvbuf (stdout, NULL, _IONBF, 0);

But most program's _don't_ call setvbuf. If a program, like 'grep', has
an option like --line-buffered, then I don't need setvbuf.

> I'll might also add a note about fread() blocking (which POSIX requires)
> and that setting no _buffering_ doesn't impact that.

Yes, that seems to be the point. "no buffering" != "non-blocking".

Could 'stdbuf -i0' be changed to put file descriptor 0 into non-blocking state
before invoking the child process? Is that the solution?

Bruno
-- 
In memoriam Jürgen Ponto <http://en.wikipedia.org/wiki/Jürgen_Ponto>

Message #34 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bruno Haible <bruno <at> clisp.org>
Cc: 8961 <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Thu, 30 Jun 2011 23:54:10 +0100

On 30/06/11 20:27, Bruno Haible wrote:
> Hi Pádraig,
> 
>>> $ (echo Hello; sleep 3; echo World) | stdbuf iconv -f ASCII
>>> Hello
>>> World
>>
>> From stdbuf.c /* FIXME: Should we mandate at least one option?  */
> 
> Oops, I meant this:
> 
> $ (echo Hello; sleep 3; echo World) | stdbuf -i 0 iconv -f ASCII
> Hello
> World
> 
> Here too, the output comes only at the end, all at once.
> 
>> Anway I don't think that this works even if you specify -i0
>> because fread() only seems to return after feof() or ferror()
> 
> Exactly. So, on which kinds of programs can -i0 work?
>   - Not in programs that use read(0,...) because they are not affected
>     by LD_PRELOADs.
>   - Not in programs that use fread().
>   - Not in programs that use getc() repeatedly - because the buffering
>     behaviour of stdio is independent of whether you use fread() or [f]getc().
> Can you name a single program on which -i0 works?

Programs that use getline() for example (like sed).
They will change to reading 1 char at a time,
as will programs that use [f]getc().
I'll update the docs with details.

> 
>> int main(void)
>> {
>>     setvbuf (stdin, NULL, _IONBF, 0);
>>     setvbuf (stdout, NULL, _IONBF, 0);
> 
> But program's _don't_ call setvbuf. If a program, like 'grep', has
> an option like --line-buffered, then I don't need setvbuf.

Right, that was just a test program to simulate what `stdbuf fread_prog` does.
The raison d'être of stdbuf is to not add --line-buffered etc. to all progs
that might need it.

> 
>> I'll might add a note about fread() blocking (which POSIX requires)
>> and that setting no _buffering_ doesn't impact that.
> 
> Yes, that seems to be the point. "no buffering" != "non-blocking> Could 'stdbuf -i0' be changed to put file descriptor 0 into non-blocking state
> before invoking the child process? Is that the solution?

Well apps would need to handle EAGAIN sensibly,
and most probably don't do that.

Essentially, setting non buffered input is to limit what
the app reads (so that a subsequent app may further process stdin),
rather than changing the responsiveness to input.

cheers,
Pádraig.

Message #37 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: Pádraig Brady <P <at> draigbrady.com>
Cc: 8961 <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Fri, 1 Jul 2011 11:03:46 +0200

Hi Pádraig,

> Essentially, setting non buffered input is to limit what
> the app reads (so that a subsequent app may further process stdin),
> rather than changing the responsiveness to input.

Oh, really? This ought to be documented and emphasized.

I was under the impression - because "stdbuf -i 0" translates into a
call to setvbuf (stdin, ..., _IONBF) and because the man page of setvbuf
says
     "When an output stream is unbuffered, information
      appears on the destination file or terminal as soon as written"
- that asking for no buffering would also imply immediate responsiveness.

> > Can you name a single program on which -i0 works?
> 
> Programs that use getline() for example (like sed).
> They will change to reading 1 char at a time,
> as will programs that use [f]getc().

I see. So in summary we can say:
  - stdbuf works only on programs that use stdio, because setvbuf calls
    have no effect on the behaviour of read() and write(),
  - If a program only calls [f]getc and processes input immediately,
    then stdbuf -i 0 will have the desired effect (unbuffered input,
    implying no blocking).
  - If a program only calls [f]getc or getline() and processes input
    immediately, then stdbuf -i L will have the desired effect
    (line-buffered input, implying no blocking after a line ends).
  - If a program calls fread() of an entire block, or has logic to call
    [f]getc() until a fixed-size buffer is filled, then stdbuf -i 0
    and stdbuf -i L will have no effect.

Is that right?

Bruno
-- 
In memoriam Karl Gröger <http://en.wikipedia.org/wiki/Karl_Gröger>

Message #40 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bruno Haible <bruno <at> clisp.org>
Cc: 8961 <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Fri, 01 Jul 2011 10:53:32 +0100

On 01/07/11 10:03, Bruno Haible wrote:
> Hi Pádraig,
> 
>> Essentially, setting non buffered input is to limit what
>> the app reads (so that a subsequent app may further process stdin),
>> rather than changing the responsiveness to input.
> 
> Oh, really? This ought to be documented and emphasized.
> 
> I was under the impression - because "stdbuf -i 0" translates into a
> call to setvbuf (stdin, ..., _IONBF) and because the man page of setvbuf
> says
>      "When an output stream is unbuffered, information
>       appears on the destination file or terminal as soon as written"
> - that asking for no buffering would also imply immediate responsiveness.
> 
>>> Can you name a single program on which -i0 works?
>>
>> Programs that use getline() for example (like sed).
>> They will change to reading 1 char at a time,
>> as will programs that use [f]getc().
> 
> I see. So in summary we can say:
>   - stdbuf works only on programs that use stdio, because setvbuf calls
>     have no effect on the behaviour of read() and write(),
>   - If a program only calls [f]getc and processes input immediately,
>     then stdbuf -i 0 will have the desired effect (unbuffered input,
>     implying no blocking).
>   - If a program only calls [f]getc or getline() and processes input
>     immediately, then stdbuf -i L will have the desired effect
>     (line-buffered input, implying no blocking after a line ends).
>   - If a program calls fread() of an entire block, or has logic to call
>     [f]getc() until a fixed-size buffer is filled, then stdbuf -i 0
>     and stdbuf -i L will have no effect.

All right, except that we disabled -iL as it's ineffective
and would only add more confusion IMHO.

I was going with this patch:

    doc: detail the effect of disabling input buffering with stdbuf

    * docs/coreutils.texi (stdbuf invocation): Expand on the different
    reasons for disabling buffering on input and output.

diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index f74dd1c..ad65cff 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -15437,8 +15437,13 @@ This option is invalid with standard input.

 @item 0
 Disable buffering of the selected stream.
-In this mode data is output immediately and only the
+In this mode, data is output immediately and only the
 amount of data requested is read from input.
+Note the difference in function for input and output.
+Disabling buffering for input will not influence the responsiveness
+or blocking behavior of the stream input functions.
+For example @code{fread} will still block until @code{EOF} or error,
+even if the underlying @code{read} returns less data than requested.

 @item @var{size}
 Specify the size of the buffer to use in fully buffered mode.

Message #43 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: Pádraig Brady <P <at> draigbrady.com>
Cc: 8961 <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Sun, 3 Jul 2011 14:17:54 +0200

Pádraig Brady wrote:
> +Disabling buffering for input will not influence the responsiveness
> +or blocking behavior of the stream input functions.

OK, but then what is the effect of "-i0"?

It has only an effect on programs that use stdio. On these programs, it has no
effect on the interface between the program and the FILE buffers, only an effect
on the interface between the FILE buffers and the kernel. More precisely:
  - Without -i0, for refilling the buffers, the stdio makes a call to
    read(0,buf,N) and receives the available n bytes (n <= N).
  - With -i0, the stdio makes repeated calls to read(0,..,1) and
    receives the available n bytes one after the other.

The only change in behaviour that I can see in such programs is that its
throughput is reduced, from 35 MB/sec to 2 MB/sec in my tests.

So, "-i0" has no useful effect on any program. I therefore suggest that you
remove this option. Likewise for "-o0". Because if you keep it, other users will
- like me - spend time, trying to make use of this option. When in fact they
are always useless.

Bruno
-- 
In memoriam Yuri Shchekochikhin <http://en.wikipedia.org/wiki/Yuri_Shchekochikhin>

Message #46 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Bruno Haible <bruno <at> clisp.org>
To: Pádraig Brady <P <at> draigbrady.com>
Cc: 8961 <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Sun, 3 Jul 2011 19:50:39 +0200

I wrote:
> Likewise for "-o0".

Oops, please forget the statement about -o0. With the modified GNU libiconv
'iconv' program, -o0 does make a difference:
  $ (echo -n abc; sleep 1; echo def) | iconv
vs.
  $ (echo -n abc; sleep 1; echo def) | stdbuf -o0 iconv
In the first case, output is line buffered. In the second case, "abc"
occurs immediately.

Bruno
-- 
In memoriam Yuri Shchekochikhin <http://en.wikipedia.org/wiki/Yuri_Shchekochikhin>

Message #49 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Bruno Haible <bruno <at> clisp.org>
Cc: 8961 <at> debbugs.gnu.org
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Sun, 03 Jul 2011 23:38:49 +0100

On 03/07/11 18:50, Bruno Haible wrote:
> I wrote:
>> Likewise for "-o0".
> 
> Oops, please forget the statement about -o0. With the modified GNU libiconv
> 'iconv' program, -o0 does make a difference:
>   $ (echo -n abc; sleep 1; echo def) | iconv
> vs.
>   $ (echo -n abc; sleep 1; echo def) | stdbuf -o0 iconv
> In the first case, output is line buffered. In the second case, "abc"
> occurs immediately.
> 
> Bruno

Also "-i0" is useful if a command doesn't consume all of input,
and needs a subsequent command to process the rest, like:

producer | (stdbuf -i0 cmd1; cmd2)

I'll expand the info to make this function apparent.

cheers,
Pádraig.

Message #52 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Paolo Bonzini <bonzini <at> gnu.org>
Cc: 8961 <at> debbugs.gnu.org, Bruno Haible <bruno <at> clisp.org>
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Mon, 04 Jul 2011 10:08:06 +0100

[Message part 1 (text/plain, inline)]

On 03/07/11 23:38, Pádraig Brady wrote:
> On 03/07/11 18:50, Bruno Haible wrote:
>> I wrote:
>>> Likewise for "-o0".
>>
>> Oops, please forget the statement about -o0. With the modified GNU libiconv
>> 'iconv' program, -o0 does make a difference:
>>   $ (echo -n abc; sleep 1; echo def) | iconv
>> vs.
>>   $ (echo -n abc; sleep 1; echo def) | stdbuf -o0 iconv
>> In the first case, output is line buffered. In the second case, "abc"
>> occurs immediately.
>>
>> Bruno
> 
> Also "-i0" is useful if a command doesn't consume all of input,
> and needs a subsequent command to process the rest, like:
> 
> producer | (stdbuf -i0 cmd1; cmd2)
> 
> I'll expand the info to make this function apparent.

Hmm, this especially useful with sed, where one
can [Qq]uit early depending on the input.

But I notice that sed no longer supports `setbuf -i0`.
This is because of 3a8e165a which discards the
original stdin and instead fdopens() a new one
to support reading in binary mode on windos.
http://git.sv.gnu.org/gitweb/?p=sed.git;a=commit;h=3a8e165a

Note in sed 4.2.2 the -u option was enhanced to
support this internally (the documentation had already
said that -u did this), so only sed 4.2 and 4.2.1
have no way to control this.

In any case there is no point I think opening
a new stream for stdin on non windos platforms
as it adds a bit of overhead, and precludes the
use of setbuf -i...

So how about the attached patch?

cheers,
Pádraig.

[sed-i0.diff (text/x-patch, attachment)]

Message #55 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Paolo Bonzini <bonzini <at> gnu.org>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: 8961 <at> debbugs.gnu.org, Bruno Haible <bruno <at> clisp.org>
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Mon, 04 Jul 2011 12:44:31 +0200

On 07/04/2011 11:08 AM, Pádraig Brady wrote:
> * execute.c (open_next_file): Only reopen stdin on windos.

Applied, thanks.

Paolo

Message #58 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: 8961 <at> debbugs.gnu.org, Paolo Bonzini <bonzini <at> gnu.org>,
	Bruno Haible <bruno <at> clisp.org>
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Tue, 05 Jul 2011 09:41:35 -0600

[Message part 1 (text/plain, inline)]

On 07/04/2011 03:08 AM, Pádraig Brady wrote:
> But I notice that sed no longer supports `setbuf -i0`.
> This is because of 3a8e165a which discards the
> original stdin and instead fdopens() a new one
> to support reading in binary mode on windos.
> http://git.sv.gnu.org/gitweb/?p=sed.git;a=commit;h=3a8e165a

This seems to point to the fact that using freopen() to force binary
mode on stdin is the wrong approach, and that we may be better served by
using lower-level idioms that change the binary mode of the underlying
file descriptor without also altering stdin with unintended side-effects.

-- 
Eric Blake   eblake <at> redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #61 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Eric Blake <eblake <at> redhat.com>
Cc: 8961 <at> debbugs.gnu.org, Paolo Bonzini <bonzini <at> gnu.org>,
	Bruno Haible <bruno <at> clisp.org>
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Tue, 05 Jul 2011 20:13:43 +0100

On 05/07/11 16:41, Eric Blake wrote:
> On 07/04/2011 03:08 AM, Pádraig Brady wrote:
>> But I notice that sed no longer supports `setbuf -i0`.
>> This is because of 3a8e165a which discards the
>> original stdin and instead fdopens() a new one
>> to support reading in binary mode on windos.
>> http://git.sv.gnu.org/gitweb/?p=sed.git;a=commit;h=3a8e165a
> 
> This seems to point to the fact that using freopen() to force binary
> mode on stdin is the wrong approach, and that we may be better served by
> using lower-level idioms that change the binary mode of the underlying
> file descriptor without also altering stdin with unintended side-effects.

Good point.

On the other hand all uses are protected by O_BINARY?...
and so only apply on windos. Does stdbuf work there?
Also od and head are the only commands that
can consume part of stdin, but they handle this
internally with setvbuf and read() respectively.
So this would only be a performance issue,
discarding the users buf size and reinitialization overhead.

If this was equivalent though, then it would be better:

ret = O_BINARY ? fcntl(fileno (stdin), F_SETFL, O_BINARY): 0;

cheers,
Pádraig.

Message #64 received at 8961 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: 8961 <at> debbugs.gnu.org, Paolo Bonzini <bonzini <at> gnu.org>,
	Bruno Haible <bruno <at> clisp.org>
Subject: Re: bug#8961: stdbuf has no effect on some programs
Date: Tue, 05 Jul 2011 15:32:04 -0600

[Message part 1 (text/plain, inline)]

On 07/05/2011 01:13 PM, Pádraig Brady wrote:
> On 05/07/11 16:41, Eric Blake wrote:
>> On 07/04/2011 03:08 AM, Pádraig Brady wrote:
>>> But I notice that sed no longer supports `setbuf -i0`.
>>> This is because of 3a8e165a which discards the
>>> original stdin and instead fdopens() a new one
>>> to support reading in binary mode on windos.
>>> http://git.sv.gnu.org/gitweb/?p=sed.git;a=commit;h=3a8e165a
>>
>> This seems to point to the fact that using freopen() to force binary
>> mode on stdin is the wrong approach, and that we may be better served by
>> using lower-level idioms that change the binary mode of the underlying
>> file descriptor without also altering stdin with unintended side-effects.
> 
> Good point.
> 
> On the other hand all uses are protected by O_BINARY?...
> and so only apply on windos. Does stdbuf work there?
> Also od and head are the only commands that
> can consume part of stdin, but they handle this
> internally with setvbuf and read() respectively.
> So this would only be a performance issue,
> discarding the users buf size and reinitialization overhead.
> 
> If this was equivalent though, then it would be better:
> 
> ret = O_BINARY ? fcntl(fileno (stdin), F_SETFL, O_BINARY): 0;

For cygwin, I believe that this _is_ equivalent to changing just the
mode but nothing else in stdin - fcntl changes the fd mode, then calls
into an internal stdio hook function that walks all open FILE *, and if
any of those FILE * are visiting fd, then also updates that FILE * to
reflect the new operating mode.

But I don't know if it works for all other platforms with non-zero O_BINARY.

-- 
Eric Blake   eblake <at> redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.