From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 29 16:59:32 2011 Received: (at submit) by debbugs.gnu.org; 29 Jun 2011 20:59:33 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qc1r9-0007Li-VK for submit@debbugs.gnu.org; Wed, 29 Jun 2011 16:59:32 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qc1r8-0007LX-OR for submit@debbugs.gnu.org; Wed, 29 Jun 2011 16:59:31 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Qc1r1-00052n-PV for submit@debbugs.gnu.org; Wed, 29 Jun 2011 16:59:25 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_NONE, T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([140.186.70.17]:53634) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qc1r1-00052j-Bx for submit@debbugs.gnu.org; Wed, 29 Jun 2011 16:59:23 -0400 Received: from eggs.gnu.org ([140.186.70.92]:55779) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qc1qz-0003cN-0j for bug-coreutils@gnu.org; Wed, 29 Jun 2011 16:59:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Qc1qv-00051t-V7 for bug-coreutils@gnu.org; Wed, 29 Jun 2011 16:59:20 -0400 Received: from mo-p00-ob.rzone.de ([81.169.146.162]:50941) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qc1qv-00051V-6v for bug-coreutils@gnu.org; Wed, 29 Jun 2011 16:59:17 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1309381155; l=1718; s=domk; d=haible.de; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Date:Subject:To: From:X-RZG-CLASS-ID:X-RZG-AUTH; bh=PPhwhjSR7Ph7LZx/E2KG60zNHOs=; b=sohKnI8H34hYNrAoJccKohyPjLWSxE2iLqJWfb+vFKcn76C7XDOdcBkYZsuLukHFXkX dp4K6fvIagh+N2ZeM37/J9Nj3lyogjb0cX1xe9/XInwwLViI6bNU2Y8EDRnvCgKTAVlht xNg3YEn/s+mEHRCz+sNBJEUeqzwf0cEj4bo= X-RZG-AUTH: :Ln4Re0+Ic/6oZXR1YgKryK8brksyK8dozXDwHXjf9hj/zDNRbP444xzQwQ== X-RZG-CLASS-ID: mo00 Received: from linuix.haible.de (dslb-088-068-058-108.pools.arcor-ip.net [88.68.58.108]) by smtp.strato.de (jimi mo17) (RZmta 26.0) with ESMTPA id w032f3n5TKwg0x ; Wed, 29 Jun 2011 22:59:15 +0200 (MEST) From: Bruno Haible To: bug-coreutils@gnu.org Subject: stdbuf has no effect on some programs Date: Wed, 29 Jun 2011 22:59:13 +0200 User-Agent: KMail/1.9.9 MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <201106292259.14133.bruno@clisp.org> X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 140.186.70.17 X-Spam-Score: -5.5 (-----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.5 (-----) Hi, The glibc 'iconv' program buffers its input, and some people don't like thi= s. I thought that the 'stdbuf' program could remove the buffering, but it does= not work. How to reproduce: Create this script and make it executable: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D producer =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D #!/bin/sh echo Hello /bin/sleep 3 echo World =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D $ ./producer | /usr/bin/iconv -f ASCII Hello World All the output comes at the end. $ stdbuf -o 0 ./producer | /usr/bin/iconv -f ASCII Hello World All the output comes at the end. $ ./producer | stdbuf -i 0 /usr/bin/iconv -f ASCII Hello World All the output comes at the end. $ stdbuf -o 0 ./producer | stdbuf -i 0 /usr/bin/iconv -f ASCII Hello World All the output comes at the end. What do I need to do to get the output of the first line immediately? /usr/bin/iconv is from glibc, but I get the same behaviour from libiconv's 'iconv' program too. In $ ./producer | /bin/cat I get the first line immediately, but the coreutils documentation makes me think that 'stdbuf' is meant for those programs that do not work l= ike 'cat'. Bruno =2D-=20 In memoriam Jos=C3=A9 Olaya From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 29 19:06:07 2011 Received: (at 8961) by debbugs.gnu.org; 29 Jun 2011 23:06:07 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qc3pf-0000Go-7M for submit@debbugs.gnu.org; Wed, 29 Jun 2011 19:06:07 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1Qc3pd-0000GL-HO for 8961@debbugs.gnu.org; Wed, 29 Jun 2011 19:06:06 -0400 Received: (qmail 54045 invoked from network); 29 Jun 2011 23:05:59 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 29 Jun 2011 23:05:59 -0000 Message-ID: <4E0BAFAF.2090907@draigBrady.com> Date: Thu, 30 Jun 2011 00:05:19 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Bruno Haible Subject: Re: bug#8961: stdbuf has no effect on some programs References: <201106292259.14133.bruno@clisp.org> In-Reply-To: <201106292259.14133.bruno@clisp.org> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.7 (--) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.7 (--) On 29/06/11 21:59, Bruno Haible wrote: > Hi, > > The glibc 'iconv' program buffers its input, and some people don't like this. > I thought that the 'stdbuf' program could remove the buffering, but it does not > work. The following shows I think that iconv is bypassing stdio and buffering internally? (echo; sleep 3; echo) | ltrace iconv -f ASCII The stdbuf man page notes that: NOTE: If COMMAND adjusts the buffering of its standard streams (`tee' does for e.g.) then that will override corresponding settings changed by `stdbuf'. Also some filters (like `dd' and `cat' etc.) don't use streams for I/O, and are thus unaffected by `stdbuf' settings. cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 29 19:30:10 2011 Received: (at 8961-done) by debbugs.gnu.org; 29 Jun 2011 23:30:10 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qc4Cw-0001lo-5d for submit@debbugs.gnu.org; Wed, 29 Jun 2011 19:30:10 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1Qc4Cu-0001c3-LL for 8961-done@debbugs.gnu.org; Wed, 29 Jun 2011 19:30:09 -0400 Received: (qmail 57064 invoked from network); 29 Jun 2011 23:30:02 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 29 Jun 2011 23:30:02 -0000 Message-ID: <4E0BB552.5010303@draigBrady.com> Date: Thu, 30 Jun 2011 00:29:22 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Bruno Haible Subject: Re: bug#8961: stdbuf has no effect on some programs References: <201106292259.14133.bruno@clisp.org> <4E0BAFAF.2090907@draigBrady.com> In-Reply-To: <4E0BAFAF.2090907@draigBrady.com> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.7 (--) X-Debbugs-Envelope-To: 8961-done Cc: 8961-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.7 (--) On 30/06/11 00:05, Pádraig Brady wrote: > On 29/06/11 21:59, Bruno Haible wrote: >> Hi, >> >> The glibc 'iconv' program buffers its input, and some people don't like this. >> I thought that the 'stdbuf' program could remove the buffering, but it does not >> work. > > The following shows I think that iconv is bypassing stdio and buffering internally? > > (echo; sleep 3; echo) | ltrace iconv -f ASCII > > The stdbuf man page notes that: > > NOTE: If COMMAND adjusts the buffering of its standard streams (`tee' > does for e.g.) then that will override corresponding settings changed > by `stdbuf'. Also some filters (like `dd' and `cat' etc.) don't use > streams for I/O, and are thus unaffected by `stdbuf' settings. In fact iconv seems to buffer for ever and so it not scalable, as demonstrated with this consuming all of memory: yes | iconv It would be OK to treat '\n' simply in all unibyte encodings and utf8 for example, but that would introduce an inconsistency I suppose. Though maybe iconv could employ more scalable buffering at least, for all encodings? cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 29 19:34:23 2011 Received: (at 8961) by debbugs.gnu.org; 29 Jun 2011 23:34:24 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qc4H1-0002Vi-EL for submit@debbugs.gnu.org; Wed, 29 Jun 2011 19:34:23 -0400 Received: from mo-p00-ob.rzone.de ([81.169.146.160]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qc4Gz-0002VV-Bt for 8961@debbugs.gnu.org; Wed, 29 Jun 2011 19:34:22 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1309390455; l=2714; s=domk; d=haible.de; h=Content-Transfer-Encoding:Content-Type:MIME-Version:In-Reply-To: References:Cc:Date:Subject:To:From:X-RZG-CLASS-ID:X-RZG-AUTH; bh=2R+XplInS9JYYDGgZKRTMlxqGcc=; b=ZWBJ3Xrej8VIOwKh6WXfJLcej5yBbXRa4+Egn1BfBC1FwsN3tkqcRi95ybOfC6m57U8 IKUlJS+Q3ppbahjqZPW5t31MKB3xamkgT/iVTt1/vOv+7vRw80/obJFaZQIt8g5YtaEPH d1ZhjH94fU6sd5xz/BPdBDQ5Udmdt4J9c40= X-RZG-AUTH: :Ln4Re0+Ic/6oZXR1YgKryK8brksyK8dozXDwHXjf9hj/zDNRbP444xzQwQ== X-RZG-CLASS-ID: mo00 Received: from linuix.haible.de (dslb-088-068-058-108.pools.arcor-ip.net [88.68.58.108]) by smtp.strato.de (klopstock mo35) (RZmta 26.0) with ESMTPA id a06aa6n5TLA1AB ; Thu, 30 Jun 2011 01:34:14 +0200 (MEST) From: Bruno Haible To: =?utf-8?q?P=C3=A1draig_Brady?= Subject: Re: bug#8961: stdbuf has no effect on some programs Date: Thu, 30 Jun 2011 01:34:12 +0200 User-Agent: KMail/1.9.9 References: <201106292259.14133.bruno@clisp.org> <4E0BAFAF.2090907@draigBrady.com> In-Reply-To: <4E0BAFAF.2090907@draigBrady.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <201106300134.13635.bruno@clisp.org> X-Spam-Score: -3.5 (---) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -3.5 (---) Hi P=C3=A1draig, > The following shows I think that iconv is bypassing stdio and buffering i= nternally? >=20 > (echo; sleep 3; echo) | ltrace iconv -f ASCII Indeed, the glibc 'iconv' does not use stdio apparently: __libc_start_main(0x402a00, 3, 0x7fff76b92d68, 0x409c90, 0x409c80 setlocale(6, "") = =3D "de_DE.UTF-8" textdomain("libc") = =3D "libc" argp_parse(0x60d2e0, 3, 0x7fff76b92d68, 0, 0x7fff76b92c5c) = =3D 0 strchr("ASCII", '/') = =3D NULL strchr("", '/') = =3D NULL iconv_open(0x409daf, 0x7fff76b946e6, 0, 0x53494c4300494900, 0xfefefefefefef= eff) =3D 0x613080 fileno(0x7f4c6e9726a0) = =3D 0 realloc(NULL, 32768) = =3D 0x00615100 read(0, "\n", 32768) = =3D 1 read(0, "\n", 32767) = =3D 1 read(0, "", 32766) = =3D 0 iconv(0x613080, 0x7fff76b8ab38, 0x7fff76b8ab30, 0x7fff76b92b48, 0x7fff76b92= b40) =3D 0 =2E.. > The stdbuf man page notes that: >=20 > NOTE: If COMMAND adjusts the buffering of its standard streams (`tee' > does for e.g.) then that will override corresponding settings changed > by `stdbuf'. Also some filters (like `dd' and `cat' etc.) don't use > streams for I/O, and are thus unaffected by `stdbuf' settings. This paragraph is written in a way that does not make the implications very clear. How about the attached patch, meant to show the restrictions mo= re prominently? 2011-06-29 Bruno Haible * doc/coreutils.texi (stdbuf invocation): Mention the restrictions on the command more prominently. =2D-- doc/coreutils.texi.orig Thu Jun 30 01:31:13 2011 +++ doc/coreutils.texi Thu Jun 30 01:30:54 2011 @@ -15398,6 +15398,17 @@ stdbuf @var{option}@dots{} @var{command} @end example =20 +@var{command} must start with the name of a program that +@enumerate +@item +uses the ISO C @code{FILE} streams for input/output (note the +programs @command{dd} and @command{cat} don't do that), + +@item +does not adjust the buffering of its standard streams (note the +program @command{tee} is not in this category). +@end enumerate + Any additional @var{arg}s are passed as additional arguments to the @var{command}. =20 @@ -15446,12 +15457,6 @@ =20 @end table =20 =2DNOTE: If @var{command} adjusts the buffering of its standard streams =2D(@command{tee} does for e.g.) then that will override corresponding sett= ings =2Dchanged by @command{stdbuf}. Also some filters (like @command{dd} and =2D@command{cat} etc.) don't use streams for I/O, and are thus unaffected =2Dby @command{stdbuf} settings. =2D @cindex exit status of @command{stdbuf} Exit status: =20 =2D-=20 In memoriam Jos=C3=A9 Olaya From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 29 19:42:26 2011 Received: (at 8961) by debbugs.gnu.org; 29 Jun 2011 23:42:27 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qc4Oo-0002i2-IX for submit@debbugs.gnu.org; Wed, 29 Jun 2011 19:42:26 -0400 Received: from mo-p00-ob.rzone.de ([81.169.146.160]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qc4Om-0002hq-RM for 8961@debbugs.gnu.org; Wed, 29 Jun 2011 19:42:25 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1309390939; l=2099; s=domk; d=haible.de; h=Content-Transfer-Encoding:Content-Type:MIME-Version:In-Reply-To: References:Cc:Date:Subject:To:From:X-RZG-CLASS-ID:X-RZG-AUTH; bh=tSN5elQ9EWaAM+PGAyfU2E5oM4g=; b=xnMjXTI756kPhYivkXTV/sMaH4QU7urT/TiauuTmX3rSXiko/TeT3fAyaF7sMHZ1VMx dkUMYfxFypaqyQCvoUGhlBwAOGV0awrz1pjSK90/o80tp7Y/Tj5xjgnLegAA+IZrGzkmh 7Itj8fKj1lWTyX5KqBhSjA66yArGugpYXcY= X-RZG-AUTH: :Ln4Re0+Ic/6oZXR1YgKryK8brksyK8dozXDwHXjf9hj/zDNRbP444xzQwQ== X-RZG-CLASS-ID: mo00 Received: from linuix.haible.de (dslb-088-068-058-108.pools.arcor-ip.net [88.68.58.108]) by smtp.strato.de (cohen mo19) (RZmta 26.0) with ESMTPA id C073fen5TMu86N ; Thu, 30 Jun 2011 01:42:18 +0200 (MEST) From: Bruno Haible To: =?utf-8?q?P=C3=A1draig_Brady?= Subject: Re: bug#8961: stdbuf has no effect on some programs Date: Thu, 30 Jun 2011 01:42:16 +0200 User-Agent: KMail/1.9.9 References: <201106292259.14133.bruno@clisp.org> <4E0BAFAF.2090907@draigBrady.com> In-Reply-To: <4E0BAFAF.2090907@draigBrady.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <201106300142.18111.bruno@clisp.org> X-Spam-Score: -3.5 (---) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -3.4 (---) P=C3=A1draig Brady wrote: > The following shows I think that iconv is bypassing stdio and buffering i= nternally? >=20 > (echo; sleep 3; echo) | ltrace iconv -f ASCII This is true for the glibc 'iconv' program. But stdbuf also does not work w= ith 'iconv' from GNU libiconv, and this program uses stdio in a very simple for= m: It reads from stdin using fread(). It does *not* call setvbuf explicitly. $ (echo; sleep 3; echo) | ltrace iconv -f ASCII __libc_start_main(0x804a250, 3, 0xffe20264, 0x804b3c0, 0x804b420 strrchr("iconv", '/') = =3D NULL setlocale(0, "") = =3D "de_DE.UTF-8" setlocale(5, "") = =3D "de_DE.UTF-8" bindtextdomain("libiconv", "/packages/gnu/share/locale") = =3D "/packages/gnu/share/locale" textdomain("libiconv") = =3D "libiconv" strlen("-f") = =3D 2 libiconv_open(0x804c262, 0xffe206d7, 0xffe206d4, 0xffe206d4, 0) = =3D 0x804f4a0 iconv_canonicalize(0x804c262, 0xffe206d7, 0xffe206d4, 0xffe206d4, 0) = =3D 0xf7effecc iconv_canonicalize(0xffe206d7, 0xffe206d7, 0xffe206d4, 0xffe206d4, 0) = =3D 0xf7efff38 libiconvctl(0x804f4a0, 6, 0xffe201a0, 0xffe206d7, 0) = =3D 0 libiconvctl(0x804f4a0, 5, 0xffe20190, 0x804f4a0, 0) = =3D 0 dcgettext(0, 0x804c27e, 5, 0, 0) = =3D 0x8057b50 libiconv(0x804f4a0, 0, 0, 0, 0) = =3D 0 fread(0xffe1f110, 1, 4096, 0xf7df5420) = =3D 2 libiconv(0x804f4a0, 0xffe1d100, 0xffe1d104, 0xffe1d108, 0xffe1d10c) = =3D 0 __errno_location() = =3D 0xf7cb5688 fwrite("\n\n", 1, 2, 0xf7df54c0 ) =3D 2 fread(0xffe1f110, 1, 4096, 0xf7df5420) = =3D 0 libiconv(0x804f4a0, 0, 0, 0xffe1d0f4, 0xffe1d0f8) = =3D 0 ferror(0xf7df5420) = =3D 0 libiconv_close(0x804f4a0, 0xf7df5420, 0x8057b50, 0, 0) = =3D 0 ferror(0xf7df54c0) = =3D 0 fclose(0xf7df54c0) = =3D 0 exit(0 +++ exited (status 0) +++ $ (echo Hello; sleep 3; echo World) | stdbuf iconv -f ASCII Hello World All output comes after 3 seconds - stdbuf has no effect. I'm using glibc-2.8. Bruno =2D-=20 In memoriam Jos=C3=A9 Olaya From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 29 19:50:25 2011 Received: (at 8961) by debbugs.gnu.org; 29 Jun 2011 23:50:26 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qc4WX-0003fY-31 for submit@debbugs.gnu.org; Wed, 29 Jun 2011 19:50:25 -0400 Received: from mo-p00-ob.rzone.de ([81.169.146.161]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qc4WU-0003fL-OR for 8961@debbugs.gnu.org; Wed, 29 Jun 2011 19:50:23 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1309391416; l=339; s=domk; d=haible.de; h=Content-Transfer-Encoding:Content-Type:MIME-Version:In-Reply-To: References:Cc:Date:Subject:To:From:X-RZG-CLASS-ID:X-RZG-AUTH; bh=uOYUS84eDnkLqLL8VgHXD3TmI9Y=; b=gNw6VmIqc5QY2hVb9Q1RQabdVkB0AUBdMgygUZ7r9Xiu765O1ozFH0hbCUKShmAdhwi R5wXpdTMBPyS28KpkZOWgES+UHNpoa528Kzj/mE+hBL5i0oT+H4I2f30c6cQHQ7PGTzYL C8FFh01I0AJTBb9NdRQBZ09RMT5x9z3DOtw= X-RZG-AUTH: :Ln4Re0+Ic/6oZXR1YgKryK8brksyK8dozXDwHXjf9hj/zDNRbP444xzQwQ== X-RZG-CLASS-ID: mo00 Received: from linuix.haible.de (dslb-088-068-058-108.pools.arcor-ip.net [88.68.58.108]) by smtp.strato.de (klopstock mo1) (RZmta 26.0) with ESMTPA id e061d0n5TLkXt8 ; Thu, 30 Jun 2011 01:50:16 +0200 (MEST) From: Bruno Haible To: =?utf-8?q?P=C3=A1draig_Brady?= Subject: Re: bug#8961: stdbuf has no effect on some programs Date: Thu, 30 Jun 2011 01:50:14 +0200 User-Agent: KMail/1.9.9 References: <201106292259.14133.bruno@clisp.org> <4E0BAFAF.2090907@draigBrady.com> <4E0BB552.5010303@draigBrady.com> In-Reply-To: <4E0BB552.5010303@draigBrady.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <201106300150.15447.bruno@clisp.org> X-Spam-Score: -3.4 (---) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -3.4 (---) P=C3=A1draig Brady wrote: > In fact iconv seems to buffer for ever and so it not scalable, > as demonstrated with this consuming all of memory: >=20 > yes | iconv Indeed, yes. It would be a good idea to file a glibc bug report about it. Bruno =2D-=20 In memoriam Jos=C3=A9 Olaya From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 30 07:15:55 2011 Received: (at 8961) by debbugs.gnu.org; 30 Jun 2011 11:15:56 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QcFDv-0004HI-2s for submit@debbugs.gnu.org; Thu, 30 Jun 2011 07:15:55 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1QcFDs-0004H6-SD for 8961@debbugs.gnu.org; Thu, 30 Jun 2011 07:15:53 -0400 Received: (qmail 56861 invoked from network); 30 Jun 2011 11:15:46 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 30 Jun 2011 11:15:46 -0000 Message-ID: <4E0C5AB7.2030104@draigBrady.com> Date: Thu, 30 Jun 2011 12:15:03 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Bruno Haible Subject: Re: bug#8961: stdbuf has no effect on some programs References: <201106292259.14133.bruno@clisp.org> <4E0BAFAF.2090907@draigBrady.com> <201106300142.18111.bruno@clisp.org> In-Reply-To: <201106300142.18111.bruno@clisp.org> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.7 (--) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.7 (--) On 30/06/11 00:42, Bruno Haible wrote: > Pádraig Brady wrote: >> The following shows I think that iconv is bypassing stdio and buffering internally? >> >> (echo; sleep 3; echo) | ltrace iconv -f ASCII > > This is true for the glibc 'iconv' program. But stdbuf also does not work with > 'iconv' from GNU libiconv, and this program uses stdio in a very simple form: > It reads from stdin using fread(). It does *not* call setvbuf explicitly. > > $ (echo; sleep 3; echo) | ltrace iconv -f ASCII > fread(0xffe1f110, 1, 4096, 0xf7df5420) = 2 > $ (echo Hello; sleep 3; echo World) | stdbuf iconv -f ASCII > Hello > World >From stdbuf.c /* FIXME: Should we mandate at least one option? */ Anway I don't think that this works even if you specify -i0 because fread() only seems to return after feof() or ferror() as demonstrated by interacting with the following run with ltrace. #include int main(void) { setvbuf (stdin, NULL, _IONBF, 0); setvbuf (stdout, NULL, _IONBF, 0); char buf[BUFSIZ]; for (;;) { size_t count = fread (buf,1,BUFSIZ,stdin); fwrite (buf,1,count,stdin); if (feof (stdin)) break; } return 0; } I guess glibc could return early if there were no partial records (which there never will be with a size of 1). It's a bit surprising it doesn't even given the wording of the man page. cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 30 09:53:11 2011 Received: (at 8961) by debbugs.gnu.org; 30 Jun 2011 13:53:11 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QcHg7-0007AX-6H for submit@debbugs.gnu.org; Thu, 30 Jun 2011 09:53:11 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1QcHg5-0007AM-6I for 8961@debbugs.gnu.org; Thu, 30 Jun 2011 09:53:10 -0400 Received: (qmail 86672 invoked from network); 30 Jun 2011 13:53:03 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 30 Jun 2011 13:53:03 -0000 Message-ID: <4E0C7F94.4030205@draigBrady.com> Date: Thu, 30 Jun 2011 14:52:20 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Bruno Haible Subject: Re: bug#8961: stdbuf has no effect on some programs References: <201106292259.14133.bruno@clisp.org> <4E0BAFAF.2090907@draigBrady.com> <201106300134.13635.bruno@clisp.org> In-Reply-To: <201106300134.13635.bruno@clisp.org> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.7 (--) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.7 (--) On 30/06/11 00:34, Bruno Haible wrote: > Hi Pádraig, > >> The following shows I think that iconv is bypassing stdio and buffering internally? >> >> (echo; sleep 3; echo) | ltrace iconv -f ASCII > > Indeed, the glibc 'iconv' does not use stdio apparently: > > __libc_start_main(0x402a00, 3, 0x7fff76b92d68, 0x409c90, 0x409c80 > setlocale(6, "") = "de_DE.UTF-8" > textdomain("libc") = "libc" > argp_parse(0x60d2e0, 3, 0x7fff76b92d68, 0, 0x7fff76b92c5c) = 0 > strchr("ASCII", '/') = NULL > strchr("", '/') = NULL > iconv_open(0x409daf, 0x7fff76b946e6, 0, 0x53494c4300494900, 0xfefefefefefefeff) = 0x613080 > fileno(0x7f4c6e9726a0) = 0 > realloc(NULL, 32768) = 0x00615100 > read(0, "\n", 32768) = 1 > read(0, "\n", 32767) = 1 > read(0, "", 32766) = 0 > iconv(0x613080, 0x7fff76b8ab38, 0x7fff76b8ab30, 0x7fff76b92b48, 0x7fff76b92b40) = 0 > ... > >> The stdbuf man page notes that: >> >> NOTE: If COMMAND adjusts the buffering of its standard streams (`tee' >> does for e.g.) then that will override corresponding settings changed >> by `stdbuf'. Also some filters (like `dd' and `cat' etc.) don't use >> streams for I/O, and are thus unaffected by `stdbuf' settings. > > This paragraph is written in a way that does not make the implications > very clear. How about the attached patch, meant to show the restrictions more > prominently? > > > 2011-06-29 Bruno Haible > > * doc/coreutils.texi (stdbuf invocation): Mention the restrictions on > the command more prominently. > > --- doc/coreutils.texi.orig Thu Jun 30 01:31:13 2011 > +++ doc/coreutils.texi Thu Jun 30 01:30:54 2011 > @@ -15398,6 +15398,17 @@ > stdbuf @var{option}@dots{} @var{command} > @end example > > +@var{command} must start with the name of a program that > +@enumerate > +@item > +uses the ISO C @code{FILE} streams for input/output (note the > +programs @command{dd} and @command{cat} don't do that), > + > +@item > +does not adjust the buffering of its standard streams (note the > +program @command{tee} is not in this category). > +@end enumerate I'll apply this. I'll might also add a note about fread() blocking (which POSIX requires) and that setting no _buffering_ doesn't impact that. cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 30 15:27:55 2011 Received: (at 8961) by debbugs.gnu.org; 30 Jun 2011 19:27:55 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QcMu2-000878-No for submit@debbugs.gnu.org; Thu, 30 Jun 2011 15:27:55 -0400 Received: from mo-p00-ob.rzone.de ([81.169.146.162]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QcMtz-00086u-1e for 8961@debbugs.gnu.org; Thu, 30 Jun 2011 15:27:52 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1309462063; l=1548; s=domk; d=haible.de; h=Content-Transfer-Encoding:Content-Type:MIME-Version:In-Reply-To: References:Cc:Date:Subject:To:From:X-RZG-CLASS-ID:X-RZG-AUTH; bh=iyAG0aPV3cTTlZUXzD+qQ1nz3kA=; b=c+mMT695PdbMahc97FM9lR6wvIwzFuShsxQC42z75fWytOzgytAf0Wp5wh8yjCKp+JA wpfacZDOcA6DyLk+di9T2cdZIBDlA4K2WL2gSaEBbDxGIphunpMBLrCXKCW730ficuCmt qrDyVfAOlPJt+1di4uPA56RL35ruBMd42oc= X-RZG-AUTH: :Ln4Re0+Ic/6oZXR1YgKryK8brksyK8dozXDwHXjf9hj/zDNRbP444xzQwQ== X-RZG-CLASS-ID: mo00 Received: from linuix.haible.de (dslb-088-068-058-108.pools.arcor-ip.net [88.68.58.108]) by smtp.strato.de (cohen mo38) (RZmta 26.0) with ESMTPA id j00766n5UIETu2 ; Thu, 30 Jun 2011 21:27:43 +0200 (MEST) From: Bruno Haible To: =?utf-8?q?P=C3=A1draig_Brady?= Subject: Re: bug#8961: stdbuf has no effect on some programs Date: Thu, 30 Jun 2011 21:27:40 +0200 User-Agent: KMail/1.9.9 References: <201106292259.14133.bruno@clisp.org> <201106300142.18111.bruno@clisp.org> <4E0C5AB7.2030104@draigBrady.com> In-Reply-To: <4E0C5AB7.2030104@draigBrady.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <201106302127.42178.bruno@clisp.org> X-Spam-Score: -3.4 (---) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -3.4 (---) Hi P=C3=A1draig, > > $ (echo Hello; sleep 3; echo World) | stdbuf iconv -f ASCII > > Hello > > World >=20 > From stdbuf.c /* FIXME: Should we mandate at least one option? */ Oops, I meant this: $ (echo Hello; sleep 3; echo World) | stdbuf -i 0 iconv -f ASCII Hello World Here too, the output comes only at the end, all at once. > Anway I don't think that this works even if you specify -i0 > because fread() only seems to return after feof() or ferror() Exactly. So, on which kinds of programs can -i0 work? - Not in programs that use read(0,...) because they are not affected by LD_PRELOADs. - Not in programs that use fread(). - Not in programs that use getc() repeatedly - because the buffering behaviour of stdio is independent of whether you use fread() or [f]getc= (). Can you name a single program on which -i0 works? > int main(void) > { > setvbuf (stdin, NULL, _IONBF, 0); > setvbuf (stdout, NULL, _IONBF, 0); But most program's _don't_ call setvbuf. If a program, like 'grep', has an option like --line-buffered, then I don't need setvbuf. > I'll might also add a note about fread() blocking (which POSIX requires) > and that setting no _buffering_ doesn't impact that. Yes, that seems to be the point. "no buffering" !=3D "non-blocking". Could 'stdbuf -i0' be changed to put file descriptor 0 into non-blocking st= ate before invoking the child process? Is that the solution? Bruno =2D-=20 In memoriam J=C3=BCrgen Ponto From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 30 18:55:05 2011 Received: (at 8961) by debbugs.gnu.org; 30 Jun 2011 22:55:05 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QcQ8W-0004Pe-RU for submit@debbugs.gnu.org; Thu, 30 Jun 2011 18:55:05 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1QcQ8U-0004P9-76 for 8961@debbugs.gnu.org; Thu, 30 Jun 2011 18:55:03 -0400 Received: (qmail 77568 invoked from network); 30 Jun 2011 22:54:55 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 30 Jun 2011 22:54:55 -0000 Message-ID: <4E0CFE92.7070007@draigBrady.com> Date: Thu, 30 Jun 2011 23:54:10 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Bruno Haible Subject: Re: bug#8961: stdbuf has no effect on some programs References: <201106292259.14133.bruno@clisp.org> <201106300142.18111.bruno@clisp.org> <4E0C5AB7.2030104@draigBrady.com> <201106302127.42178.bruno@clisp.org> In-Reply-To: <201106302127.42178.bruno@clisp.org> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.6 (--) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.6 (--) On 30/06/11 20:27, Bruno Haible wrote: > Hi Pádraig, > >>> $ (echo Hello; sleep 3; echo World) | stdbuf iconv -f ASCII >>> Hello >>> World >> >> From stdbuf.c /* FIXME: Should we mandate at least one option? */ > > Oops, I meant this: > > $ (echo Hello; sleep 3; echo World) | stdbuf -i 0 iconv -f ASCII > Hello > World > > Here too, the output comes only at the end, all at once. > >> Anway I don't think that this works even if you specify -i0 >> because fread() only seems to return after feof() or ferror() > > Exactly. So, on which kinds of programs can -i0 work? > - Not in programs that use read(0,...) because they are not affected > by LD_PRELOADs. > - Not in programs that use fread(). > - Not in programs that use getc() repeatedly - because the buffering > behaviour of stdio is independent of whether you use fread() or [f]getc(). > Can you name a single program on which -i0 works? Programs that use getline() for example (like sed). They will change to reading 1 char at a time, as will programs that use [f]getc(). I'll update the docs with details. > >> int main(void) >> { >> setvbuf (stdin, NULL, _IONBF, 0); >> setvbuf (stdout, NULL, _IONBF, 0); > > But program's _don't_ call setvbuf. If a program, like 'grep', has > an option like --line-buffered, then I don't need setvbuf. Right, that was just a test program to simulate what `stdbuf fread_prog` does. The raison d'être of stdbuf is to not add --line-buffered etc. to all progs that might need it. > >> I'll might add a note about fread() blocking (which POSIX requires) >> and that setting no _buffering_ doesn't impact that. > > Yes, that seems to be the point. "no buffering" != "non-blocking> Could 'stdbuf -i0' be changed to put file descriptor 0 into non-blocking state > before invoking the child process? Is that the solution? Well apps would need to handle EAGAIN sensibly, and most probably don't do that. Essentially, setting non buffered input is to limit what the app reads (so that a subsequent app may further process stdin), rather than changing the responsiveness to input. cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 01 05:03:57 2011 Received: (at 8961) by debbugs.gnu.org; 1 Jul 2011 09:03:57 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QcZdl-00031M-BQ for submit@debbugs.gnu.org; Fri, 01 Jul 2011 05:03:57 -0400 Received: from mo-p00-ob.rzone.de ([81.169.146.160]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QcZdi-000319-BF for 8961@debbugs.gnu.org; Fri, 01 Jul 2011 05:03:55 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1309511028; l=1629; s=domk; d=haible.de; h=Content-Transfer-Encoding:Content-Type:MIME-Version:In-Reply-To: References:Cc:Date:Subject:To:From:X-RZG-CLASS-ID:X-RZG-AUTH; bh=cZlzYex7qEuK3ALrfydQg3t3FC8=; b=ydCkNEpxan0STEK0PoLqyDlLhhghNYh9Cf+oFkV5QPjZw2DMSyBvhTB4j2LHxWlFjJm gTCKf5dKDDODXz0HO/AYmt7qal4Oo90NYAip0cMaxg7+of5apaHVOlJHCELJ0obo28967 1wWyBDkDZTYu3mIWbOFYAAOd4oPE1OaIi7M= X-RZG-AUTH: :Ln4Re0+Ic/6oZXR1YgKryK8brksyK8dozXDwHXjf9hj/zDNRbP444xzQwQ== X-RZG-CLASS-ID: mo00 Received: from linuix.haible.de (dslb-088-068-058-108.pools.arcor-ip.net [88.68.58.108]) by smtp.strato.de (jimi mo56) (RZmta 26.0) with ESMTPA id 5041e5n617AiWQ ; Fri, 1 Jul 2011 11:03:47 +0200 (MEST) From: Bruno Haible To: =?utf-8?q?P=C3=A1draig_Brady?= Subject: Re: bug#8961: stdbuf has no effect on some programs Date: Fri, 1 Jul 2011 11:03:46 +0200 User-Agent: KMail/1.9.9 References: <201106292259.14133.bruno@clisp.org> <201106302127.42178.bruno@clisp.org> <4E0CFE92.7070007@draigBrady.com> In-Reply-To: <4E0CFE92.7070007@draigBrady.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <201107011103.47004.bruno@clisp.org> X-Spam-Score: -3.4 (---) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -3.4 (---) Hi P=C3=A1draig, > Essentially, setting non buffered input is to limit what > the app reads (so that a subsequent app may further process stdin), > rather than changing the responsiveness to input. Oh, really? This ought to be documented and emphasized. I was under the impression - because "stdbuf -i 0" translates into a call to setvbuf (stdin, ..., _IONBF) and because the man page of setvbuf says "When an output stream is unbuffered, information appears on the destination file or terminal as soon as written" =2D that asking for no buffering would also imply immediate responsiveness. > > Can you name a single program on which -i0 works? >=20 > Programs that use getline() for example (like sed). > They will change to reading 1 char at a time, > as will programs that use [f]getc(). I see. So in summary we can say: - stdbuf works only on programs that use stdio, because setvbuf calls have no effect on the behaviour of read() and write(), - If a program only calls [f]getc and processes input immediately, then stdbuf -i 0 will have the desired effect (unbuffered input, implying no blocking). - If a program only calls [f]getc or getline() and processes input immediately, then stdbuf -i L will have the desired effect (line-buffered input, implying no blocking after a line ends). - If a program calls fread() of an entire block, or has logic to call [f]getc() until a fixed-size buffer is filled, then stdbuf -i 0 and stdbuf -i L will have no effect. Is that right? Bruno =2D-=20 In memoriam Karl Gr=C3=B6ger From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 01 05:55:43 2011 Received: (at 8961) by debbugs.gnu.org; 1 Jul 2011 09:55:43 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QcaRr-0004BO-C7 for submit@debbugs.gnu.org; Fri, 01 Jul 2011 05:55:43 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1QcaRp-0004BB-3I for 8961@debbugs.gnu.org; Fri, 01 Jul 2011 05:55:42 -0400 Received: (qmail 75194 invoked from network); 1 Jul 2011 09:54:19 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 1 Jul 2011 09:54:19 -0000 Message-ID: <4E0D991C.6020200@draigBrady.com> Date: Fri, 01 Jul 2011 10:53:32 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Bruno Haible Subject: Re: bug#8961: stdbuf has no effect on some programs References: <201106292259.14133.bruno@clisp.org> <201106302127.42178.bruno@clisp.org> <4E0CFE92.7070007@draigBrady.com> <201107011103.47004.bruno@clisp.org> In-Reply-To: <201107011103.47004.bruno@clisp.org> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.6 (--) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.6 (--) On 01/07/11 10:03, Bruno Haible wrote: > Hi Pádraig, > >> Essentially, setting non buffered input is to limit what >> the app reads (so that a subsequent app may further process stdin), >> rather than changing the responsiveness to input. > > Oh, really? This ought to be documented and emphasized. > > I was under the impression - because "stdbuf -i 0" translates into a > call to setvbuf (stdin, ..., _IONBF) and because the man page of setvbuf > says > "When an output stream is unbuffered, information > appears on the destination file or terminal as soon as written" > - that asking for no buffering would also imply immediate responsiveness. > >>> Can you name a single program on which -i0 works? >> >> Programs that use getline() for example (like sed). >> They will change to reading 1 char at a time, >> as will programs that use [f]getc(). > > I see. So in summary we can say: > - stdbuf works only on programs that use stdio, because setvbuf calls > have no effect on the behaviour of read() and write(), > - If a program only calls [f]getc and processes input immediately, > then stdbuf -i 0 will have the desired effect (unbuffered input, > implying no blocking). > - If a program only calls [f]getc or getline() and processes input > immediately, then stdbuf -i L will have the desired effect > (line-buffered input, implying no blocking after a line ends). > - If a program calls fread() of an entire block, or has logic to call > [f]getc() until a fixed-size buffer is filled, then stdbuf -i 0 > and stdbuf -i L will have no effect. All right, except that we disabled -iL as it's ineffective and would only add more confusion IMHO. I was going with this patch: doc: detail the effect of disabling input buffering with stdbuf * docs/coreutils.texi (stdbuf invocation): Expand on the different reasons for disabling buffering on input and output. diff --git a/doc/coreutils.texi b/doc/coreutils.texi index f74dd1c..ad65cff 100644 --- a/doc/coreutils.texi +++ b/doc/coreutils.texi @@ -15437,8 +15437,13 @@ This option is invalid with standard input. @item 0 Disable buffering of the selected stream. -In this mode data is output immediately and only the +In this mode, data is output immediately and only the amount of data requested is read from input. +Note the difference in function for input and output. +Disabling buffering for input will not influence the responsiveness +or blocking behavior of the stream input functions. +For example @code{fread} will still block until @code{EOF} or error, +even if the underlying @code{read} returns less data than requested. @item @var{size} Specify the size of the buffer to use in fully buffered mode. From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 03 08:18:05 2011 Received: (at 8961) by debbugs.gnu.org; 3 Jul 2011 12:18:05 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QdLcj-0002BL-0O for submit@debbugs.gnu.org; Sun, 03 Jul 2011 08:18:05 -0400 Received: from mo-p00-ob.rzone.de ([81.169.146.161]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QdLcg-0002As-RW for 8961@debbugs.gnu.org; Sun, 03 Jul 2011 08:18:04 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1309695476; l=1210; s=domk; d=haible.de; h=Content-Transfer-Encoding:Content-Type:MIME-Version:In-Reply-To: References:Cc:Date:Subject:To:From:X-RZG-CLASS-ID:X-RZG-AUTH; bh=4ABrVYdoYKMQt8oAlPAECDvpU8k=; b=tln0rA0qqItjCX1cy7R693VASkJAh0gsqiccSqcc2Rl8brAAzp/GD49IUTbahOJJbTg 2HCe5iofRMjOjke5zNCydr5BwaOtrx4R3O5BoMKng8p0CstQVaSpLAVibbxW4AirpADqU B5aahYjqN0W5xxNjONIu6vGN0BTL0eL+wSM= X-RZG-AUTH: :Ln4Re0+Ic/6oZXR1YgKryK8brksyK8dozXDwHXjf9hj/zDNRbP444xzQwQ== X-RZG-CLASS-ID: mo00 Received: from linuix.haible.de (dslb-088-068-058-108.pools.arcor-ip.net [88.68.58.108]) by smtp.strato.de (fruni mo27) (RZmta 26.0) with ESMTPA id k04d0bn639gpfX ; Sun, 3 Jul 2011 14:17:55 +0200 (MEST) From: Bruno Haible To: =?utf-8?q?P=C3=A1draig_Brady?= Subject: Re: bug#8961: stdbuf has no effect on some programs Date: Sun, 3 Jul 2011 14:17:54 +0200 User-Agent: KMail/1.9.9 References: <201106292259.14133.bruno@clisp.org> <201107011103.47004.bruno@clisp.org> <4E0D991C.6020200@draigBrady.com> In-Reply-To: <4E0D991C.6020200@draigBrady.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <201107031417.55083.bruno@clisp.org> X-Spam-Score: -3.4 (---) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -3.4 (---) P=C3=A1draig Brady wrote: > +Disabling buffering for input will not influence the responsiveness > +or blocking behavior of the stream input functions. OK, but then what is the effect of "-i0"? It has only an effect on programs that use stdio. On these programs, it has= no effect on the interface between the program and the FILE buffers, only an e= ffect on the interface between the FILE buffers and the kernel. More precisely: - Without -i0, for refilling the buffers, the stdio makes a call to read(0,buf,N) and receives the available n bytes (n <=3D N). - With -i0, the stdio makes repeated calls to read(0,..,1) and receives the available n bytes one after the other. The only change in behaviour that I can see in such programs is that its throughput is reduced, from 35 MB/sec to 2 MB/sec in my tests. So, "-i0" has no useful effect on any program. I therefore suggest that you remove this option. Likewise for "-o0". Because if you keep it, other users= will =2D like me - spend time, trying to make use of this option. When in fact t= hey are always useless. Bruno =2D-=20 In memoriam Yuri Shchekochikhin From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 03 13:50:50 2011 Received: (at 8961) by debbugs.gnu.org; 3 Jul 2011 17:50:50 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QdQoj-0001d9-U3 for submit@debbugs.gnu.org; Sun, 03 Jul 2011 13:50:50 -0400 Received: from mo-p00-ob.rzone.de ([81.169.146.160]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QdQoh-0001cx-Rc for 8961@debbugs.gnu.org; Sun, 03 Jul 2011 13:50:49 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1309715442; l=453; s=domk; d=haible.de; h=Content-Transfer-Encoding:Content-Type:MIME-Version:In-Reply-To: References:Cc:Date:Subject:To:From:X-RZG-CLASS-ID:X-RZG-AUTH; bh=Oy8v8PyUh5zEhvLZgUcG24c+qLA=; b=PRNhnN0rQ/JCGeTkMfYCIOoPpc1rPkYqShCeNEUliQWZwmIfyPopVMLpIK9hqy6NXET QaH2EzLTX46q1wzDYsz/q4dMfDJQM382JqZJFFemAlRWdT9prtdINgen3zms8OmpzwuZK lpLhffTP0lJNzOGZUJas3TNS4VyLbYjJJHQ= X-RZG-AUTH: :Ln4Re0+Ic/6oZXR1YgKryK8brksyK8dozXDwHXjf9hj/zDNRbP444xzQwQ== X-RZG-CLASS-ID: mo00 Received: from linuix.haible.de (dslb-088-068-058-108.pools.arcor-ip.net [88.68.58.108]) by smtp.strato.de (cohen mo43) (RZmta 26.0) with ESMTPA id Q009b9n63FjbRy ; Sun, 3 Jul 2011 19:50:41 +0200 (MEST) From: Bruno Haible To: =?utf-8?q?P=C3=A1draig_Brady?= Subject: Re: bug#8961: stdbuf has no effect on some programs Date: Sun, 3 Jul 2011 19:50:39 +0200 User-Agent: KMail/1.9.9 References: <201106292259.14133.bruno@clisp.org> <4E0D991C.6020200@draigBrady.com> <201107031417.55083.bruno@clisp.org> In-Reply-To: <201107031417.55083.bruno@clisp.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <201107031950.41129.bruno@clisp.org> X-Spam-Score: -3.4 (---) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -3.4 (---) I wrote: > Likewise for "-o0". Oops, please forget the statement about -o0. With the modified GNU libiconv 'iconv' program, -o0 does make a difference: $ (echo -n abc; sleep 1; echo def) | iconv vs. $ (echo -n abc; sleep 1; echo def) | stdbuf -o0 iconv In the first case, output is line buffered. In the second case, "abc" occurs immediately. Bruno -- In memoriam Yuri Shchekochikhin From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 03 18:39:57 2011 Received: (at 8961) by debbugs.gnu.org; 3 Jul 2011 22:39:57 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QdVKX-0004kO-7t for submit@debbugs.gnu.org; Sun, 03 Jul 2011 18:39:57 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1QdVKU-0004jv-Az for 8961@debbugs.gnu.org; Sun, 03 Jul 2011 18:39:55 -0400 Received: (qmail 51719 invoked from network); 3 Jul 2011 22:39:48 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 3 Jul 2011 22:39:48 -0000 Message-ID: <4E10EF79.7010103@draigBrady.com> Date: Sun, 03 Jul 2011 23:38:49 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Bruno Haible Subject: Re: bug#8961: stdbuf has no effect on some programs References: <201106292259.14133.bruno@clisp.org> <4E0D991C.6020200@draigBrady.com> <201107031417.55083.bruno@clisp.org> <201107031950.41129.bruno@clisp.org> In-Reply-To: <201107031950.41129.bruno@clisp.org> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.6 (--) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.6 (--) On 03/07/11 18:50, Bruno Haible wrote: > I wrote: >> Likewise for "-o0". > > Oops, please forget the statement about -o0. With the modified GNU libiconv > 'iconv' program, -o0 does make a difference: > $ (echo -n abc; sleep 1; echo def) | iconv > vs. > $ (echo -n abc; sleep 1; echo def) | stdbuf -o0 iconv > In the first case, output is line buffered. In the second case, "abc" > occurs immediately. > > Bruno Also "-i0" is useful if a command doesn't consume all of input, and needs a subsequent command to process the rest, like: producer | (stdbuf -i0 cmd1; cmd2) I'll expand the info to make this function apparent. cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 04 05:09:17 2011 Received: (at 8961) by debbugs.gnu.org; 4 Jul 2011 09:09:17 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qdf9Y-0007ZH-J8 for submit@debbugs.gnu.org; Mon, 04 Jul 2011 05:09:16 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1Qdf9W-0007Z7-0Z for 8961@debbugs.gnu.org; Mon, 04 Jul 2011 05:09:15 -0400 Received: (qmail 23501 invoked from network); 4 Jul 2011 09:09:07 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 4 Jul 2011 09:09:07 -0000 Message-ID: <4E1182F6.1010307@draigBrady.com> Date: Mon, 04 Jul 2011 10:08:06 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Paolo Bonzini Subject: Re: bug#8961: stdbuf has no effect on some programs References: <201106292259.14133.bruno@clisp.org> <4E0D991C.6020200@draigBrady.com> <201107031417.55083.bruno@clisp.org> <201107031950.41129.bruno@clisp.org> <4E10EF79.7010103@draigBrady.com> In-Reply-To: <4E10EF79.7010103@draigBrady.com> X-Enigmail-Version: 1.0.1 Content-Type: multipart/mixed; boundary="------------020008020202010604050700" X-Spam-Score: -2.6 (--) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org, Bruno Haible X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.6 (--) This is a multi-part message in MIME format. --------------020008020202010604050700 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On 03/07/11 23:38, Pádraig Brady wrote: > On 03/07/11 18:50, Bruno Haible wrote: >> I wrote: >>> Likewise for "-o0". >> >> Oops, please forget the statement about -o0. With the modified GNU libiconv >> 'iconv' program, -o0 does make a difference: >> $ (echo -n abc; sleep 1; echo def) | iconv >> vs. >> $ (echo -n abc; sleep 1; echo def) | stdbuf -o0 iconv >> In the first case, output is line buffered. In the second case, "abc" >> occurs immediately. >> >> Bruno > > Also "-i0" is useful if a command doesn't consume all of input, > and needs a subsequent command to process the rest, like: > > producer | (stdbuf -i0 cmd1; cmd2) > > I'll expand the info to make this function apparent. Hmm, this especially useful with sed, where one can [Qq]uit early depending on the input. But I notice that sed no longer supports `setbuf -i0`. This is because of 3a8e165a which discards the original stdin and instead fdopens() a new one to support reading in binary mode on windos. http://git.sv.gnu.org/gitweb/?p=sed.git;a=commit;h=3a8e165a Note in sed 4.2.2 the -u option was enhanced to support this internally (the documentation had already said that -u did this), so only sed 4.2 and 4.2.1 have no way to control this. In any case there is no point I think opening a new stream for stdin on non windos platforms as it adds a bit of overhead, and precludes the use of setbuf -i... So how about the attached patch? cheers, Pádraig. --------------020008020202010604050700 Content-Type: text/x-patch; name="sed-i0.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="sed-i0.diff" >From 80d5c3a49b7819b14abf3ad728bb6253689ed1ee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?P=C3=A1draig=20Brady?= Date: Mon, 4 Jul 2011 08:24:21 +0100 Subject: [PATCH] avoid reopening stdin This adjusts the change made in 3a8e165a, to not reopen stdin on platforms where there is no separate "binary mode". That added some overhead and also precludes the use of tools like `stdbuf -i...`. * execute.c (open_next_file): Only reopen stdin on windos. --- sed/execute.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/sed/execute.c b/sed/execute.c index befc8f9..076ed0a 100644 --- a/sed/execute.c +++ b/sed/execute.c @@ -701,7 +701,11 @@ open_next_file(name, input) if (name[0] == '-' && name[1] == '\0' && !in_place_extension) { clearerr(stdin); /* clear any stale EOF indication */ +#if defined(WIN32) || defined(_WIN32) || defined(__CYGWIN__) || defined(MSDOS) || defined(__EMX__) input->fp = ck_fdopen (fileno (stdin), "stdin", read_mode, false); +#else + input->fp = stdin; +#endif } else if ( ! (input->fp = ck_fopen(name, read_mode, false)) ) { -- 1.7.5.2 --------------020008020202010604050700-- From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 04 06:44:43 2011 Received: (at 8961) by debbugs.gnu.org; 4 Jul 2011 10:44:43 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qdgdv-0001Eo-Ad for submit@debbugs.gnu.org; Mon, 04 Jul 2011 06:44:43 -0400 Received: from mail-ww0-f46.google.com ([74.125.82.46]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qdgds-0001Eb-5Z for 8961@debbugs.gnu.org; Mon, 04 Jul 2011 06:44:41 -0400 Received: by wwf25 with SMTP id 25so4745523wwf.15 for <8961@debbugs.gnu.org>; Mon, 04 Jul 2011 03:44:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=1T6/h4yrtx7kY1QmXjKp9NIL2mn4Nfbpr8pbTBg3U74=; b=TYRbKOs7gFIlbh/m9sH+rOEFPiRwC8k9mbsUpx6RhHbYZdj+C2ZYtlu3Ue4kTqmJ3s 1T6LFoKCK2oX+g8tUdOZrHyfivZLOqXCYyoCaEhg3aN9HTQSK4qFn2Y1VPImUGVEbClZ G2jrESi5ONwVhuPT6BCmKktMzUpen3B++47Ro= Received: by 10.227.37.83 with SMTP id w19mr5213615wbd.101.1309776274288; Mon, 04 Jul 2011 03:44:34 -0700 (PDT) Received: from yakj.usersys.redhat.com (93-34-199-31.ip51.fastwebnet.it [93.34.199.31]) by mx.google.com with ESMTPS id fh12sm953832wbb.35.2011.07.04.03.44.32 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 04 Jul 2011 03:44:33 -0700 (PDT) Message-ID: <4E11998F.4080802@gnu.org> Date: Mon, 04 Jul 2011 12:44:31 +0200 From: Paolo Bonzini User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110428 Fedora/3.1.10-1.fc15 Mnenhy/0.8.3 Thunderbird/3.1.10 MIME-Version: 1.0 To: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= Subject: Re: bug#8961: stdbuf has no effect on some programs References: <201106292259.14133.bruno@clisp.org> <4E0D991C.6020200@draigBrady.com> <201107031417.55083.bruno@clisp.org> <201107031950.41129.bruno@clisp.org> <4E10EF79.7010103@draigBrady.com> <4E1182F6.1010307@draigBrady.com> In-Reply-To: <4E1182F6.1010307@draigBrady.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: -4.8 (----) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org, Bruno Haible X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.4 (----) On 07/04/2011 11:08 AM, Pádraig Brady wrote: > * execute.c (open_next_file): Only reopen stdin on windos. Applied, thanks. Paolo From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 05 11:41:46 2011 Received: (at 8961) by debbugs.gnu.org; 5 Jul 2011 15:41:46 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qe7kv-0002gF-NA for submit@debbugs.gnu.org; Tue, 05 Jul 2011 11:41:46 -0400 Received: from mx1.redhat.com ([209.132.183.28]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qe7kt-0002fn-IN for 8961@debbugs.gnu.org; Tue, 05 Jul 2011 11:41:44 -0400 Received: from int-mx01.intmail.prod.int.phx2.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p65FfaRW000454 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 5 Jul 2011 11:41:36 -0400 Received: from [10.3.113.151] (ovpn-113-151.phx2.redhat.com [10.3.113.151]) by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id p65FfZEO031917; Tue, 5 Jul 2011 11:41:35 -0400 Message-ID: <4E1330AF.9050104@redhat.com> Date: Tue, 05 Jul 2011 09:41:35 -0600 From: Eric Blake Organization: Red Hat User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110428 Fedora/3.1.10-1.fc14 Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.10 MIME-Version: 1.0 To: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= Subject: Re: bug#8961: stdbuf has no effect on some programs References: <201106292259.14133.bruno@clisp.org> <4E0D991C.6020200@draigBrady.com> <201107031417.55083.bruno@clisp.org> <201107031950.41129.bruno@clisp.org> <4E10EF79.7010103@draigBrady.com> <4E1182F6.1010307@draigBrady.com> In-Reply-To: <4E1182F6.1010307@draigBrady.com> X-Enigmail-Version: 1.1.2 OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enigB386A28FF0641BA176E693B5" X-Scanned-By: MIMEDefang 2.67 on 10.5.11.11 X-Spam-Score: -10.3 (----------) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org, Paolo Bonzini , Bruno Haible X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -10.3 (----------) This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigB386A28FF0641BA176E693B5 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 07/04/2011 03:08 AM, P=C3=A1draig Brady wrote: > But I notice that sed no longer supports `setbuf -i0`. > This is because of 3a8e165a which discards the > original stdin and instead fdopens() a new one > to support reading in binary mode on windos. > http://git.sv.gnu.org/gitweb/?p=3Dsed.git;a=3Dcommit;h=3D3a8e165a This seems to point to the fact that using freopen() to force binary mode on stdin is the wrong approach, and that we may be better served by using lower-level idioms that change the binary mode of the underlying file descriptor without also altering stdin with unintended side-effects.= --=20 Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org --------------enigB386A28FF0641BA176E693B5 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBCAAGBQJOEzCvAAoJEKeha0olJ0Nq9E8H/0xBD6h4+BmPHFrFQn7HMgbR O7GP2T2C+nEPLucAiRbYEXO90gbzOKNbMwoa1ZNjIenWBXoBiuzt8dzw9ObdTefo OKv9+o5FH5uL7/YHcvol/Zxacs3KfDXutLOUnzYEfTe11iyW2qfKnM9OV2EfnP1G 13KCWsCi46aNKvJxvmrmFwiBfTQJNizNqQ8nGDgtzFOLWoDZ0vrABnDH03qCVBfP IcZ8ySdODsu0okhw8WBK2ws20xfaNJ4e6SRwwZpA4FQ/Wpbm0OATCjx0CevDRx2e yPDUTkG0NTyKcfKTW958xOLeWIty0qCjaEduNKLYDRZLWhcn1uWQgxvmLUh+Vb8= =AMp4 -----END PGP SIGNATURE----- --------------enigB386A28FF0641BA176E693B5-- From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 05 15:14:59 2011 Received: (at 8961) by debbugs.gnu.org; 5 Jul 2011 19:15:00 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QeB5H-0001Vj-9Q for submit@debbugs.gnu.org; Tue, 05 Jul 2011 15:14:59 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1QeB5F-0001VU-C6 for 8961@debbugs.gnu.org; Tue, 05 Jul 2011 15:14:58 -0400 Received: (qmail 65944 invoked from network); 5 Jul 2011 19:14:51 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 5 Jul 2011 19:14:51 -0000 Message-ID: <4E136267.90103@draigBrady.com> Date: Tue, 05 Jul 2011 20:13:43 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Eric Blake Subject: Re: bug#8961: stdbuf has no effect on some programs References: <201106292259.14133.bruno@clisp.org> <4E0D991C.6020200@draigBrady.com> <201107031417.55083.bruno@clisp.org> <201107031950.41129.bruno@clisp.org> <4E10EF79.7010103@draigBrady.com> <4E1182F6.1010307@draigBrady.com> <4E1330AF.9050104@redhat.com> In-Reply-To: <4E1330AF.9050104@redhat.com> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.6 (--) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org, Paolo Bonzini , Bruno Haible X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.6 (--) On 05/07/11 16:41, Eric Blake wrote: > On 07/04/2011 03:08 AM, Pádraig Brady wrote: >> But I notice that sed no longer supports `setbuf -i0`. >> This is because of 3a8e165a which discards the >> original stdin and instead fdopens() a new one >> to support reading in binary mode on windos. >> http://git.sv.gnu.org/gitweb/?p=sed.git;a=commit;h=3a8e165a > > This seems to point to the fact that using freopen() to force binary > mode on stdin is the wrong approach, and that we may be better served by > using lower-level idioms that change the binary mode of the underlying > file descriptor without also altering stdin with unintended side-effects. Good point. On the other hand all uses are protected by O_BINARY?... and so only apply on windos. Does stdbuf work there? Also od and head are the only commands that can consume part of stdin, but they handle this internally with setvbuf and read() respectively. So this would only be a performance issue, discarding the users buf size and reinitialization overhead. If this was equivalent though, then it would be better: ret = O_BINARY ? fcntl(fileno (stdin), F_SETFL, O_BINARY): 0; cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 05 17:32:17 2011 Received: (at 8961) by debbugs.gnu.org; 5 Jul 2011 21:32:17 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QeDE8-0006OF-O0 for submit@debbugs.gnu.org; Tue, 05 Jul 2011 17:32:17 -0400 Received: from mx1.redhat.com ([209.132.183.28]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QeDE5-0006O2-1u for 8961@debbugs.gnu.org; Tue, 05 Jul 2011 17:32:15 -0400 Received: from int-mx01.intmail.prod.int.phx2.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p65LW5J7010498 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 5 Jul 2011 17:32:05 -0400 Received: from [10.3.113.151] (ovpn-113-151.phx2.redhat.com [10.3.113.151]) by int-mx01.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id p65LW4u5010268; Tue, 5 Jul 2011 17:32:04 -0400 Message-ID: <4E1382D4.7050408@redhat.com> Date: Tue, 05 Jul 2011 15:32:04 -0600 From: Eric Blake Organization: Red Hat User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.17) Gecko/20110428 Fedora/3.1.10-1.fc14 Lightning/1.0b3pre Mnenhy/0.8.3 Thunderbird/3.1.10 MIME-Version: 1.0 To: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= Subject: Re: bug#8961: stdbuf has no effect on some programs References: <201106292259.14133.bruno@clisp.org> <4E0D991C.6020200@draigBrady.com> <201107031417.55083.bruno@clisp.org> <201107031950.41129.bruno@clisp.org> <4E10EF79.7010103@draigBrady.com> <4E1182F6.1010307@draigBrady.com> <4E1330AF.9050104@redhat.com> <4E136267.90103@draigBrady.com> In-Reply-To: <4E136267.90103@draigBrady.com> X-Enigmail-Version: 1.1.2 OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig134FFEF59948A67C8F515E43" X-Scanned-By: MIMEDefang 2.67 on 10.5.11.11 X-Spam-Score: -10.3 (----------) X-Debbugs-Envelope-To: 8961 Cc: 8961@debbugs.gnu.org, Paolo Bonzini , Bruno Haible X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -10.3 (----------) This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig134FFEF59948A67C8F515E43 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 07/05/2011 01:13 PM, P=C3=A1draig Brady wrote: > On 05/07/11 16:41, Eric Blake wrote: >> On 07/04/2011 03:08 AM, P=C3=A1draig Brady wrote: >>> But I notice that sed no longer supports `setbuf -i0`. >>> This is because of 3a8e165a which discards the >>> original stdin and instead fdopens() a new one >>> to support reading in binary mode on windos. >>> http://git.sv.gnu.org/gitweb/?p=3Dsed.git;a=3Dcommit;h=3D3a8e165a >> >> This seems to point to the fact that using freopen() to force binary >> mode on stdin is the wrong approach, and that we may be better served = by >> using lower-level idioms that change the binary mode of the underlying= >> file descriptor without also altering stdin with unintended side-effec= ts. >=20 > Good point. >=20 > On the other hand all uses are protected by O_BINARY?... > and so only apply on windos. Does stdbuf work there? > Also od and head are the only commands that > can consume part of stdin, but they handle this > internally with setvbuf and read() respectively. > So this would only be a performance issue, > discarding the users buf size and reinitialization overhead. >=20 > If this was equivalent though, then it would be better: >=20 > ret =3D O_BINARY ? fcntl(fileno (stdin), F_SETFL, O_BINARY): 0; For cygwin, I believe that this _is_ equivalent to changing just the mode but nothing else in stdin - fcntl changes the fd mode, then calls into an internal stdio hook function that walks all open FILE *, and if any of those FILE * are visiting fd, then also updates that FILE * to reflect the new operating mode. But I don't know if it works for all other platforms with non-zero O_BINA= RY. --=20 Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org --------------enig134FFEF59948A67C8F515E43 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBCAAGBQJOE4LUAAoJEKeha0olJ0NqWHEH/2ydz++BZoG0uGOjO9PC0RjH tdBZg2MHnQmDuW5Z6A097LUKJycRsUyO9isb+g5xvb4z/GkBidxPQJfFS5ngoSsW vNE8CGL/saBju80JSY/BbvbFJKCg0ptvNXmAszqafVOUdztcURrofpVkqJIpVgJh atXIwMhvhqlkWE1U0fgJYPnfKC5HkwXzpARssRAdqu6p2se/7tgoY9wg8/wTxXcU wjZ4Ly1AxDuiqPc2hkfVr28qI4+QcnZ0eKOAWKIl7p/5+COfWxJ1SfUnaXcwfiRO TJRNpZHo3rZTOUiEPuWrH4y8fe0jDEKUMqzzvko3bRrtPM92xIoMi4tKUjc+wUE= =tIEI -----END PGP SIGNATURE----- --------------enig134FFEF59948A67C8F515E43-- From unknown Fri Aug 08 22:15:21 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Wed, 03 Aug 2011 11:24:05 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator