From unknown Mon Jun 23 04:12:22 2025 X-Loop: help-debbugs@gnu.org Subject: bug#13498: "cut -f" lags a line Resent-From: Scott Lamb Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Sat, 19 Jan 2013 17:27:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 13498 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: 13498@debbugs.gnu.org X-Debbugs-Original-To: bug-coreutils@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.135861638112097 (code B ref -1); Sat, 19 Jan 2013 17:27:01 +0000 Received: (at submit) by debbugs.gnu.org; 19 Jan 2013 17:26:21 +0000 Received: from localhost ([127.0.0.1]:40510 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TwcBP-000392-1g for submit@debbugs.gnu.org; Sat, 19 Jan 2013 12:26:20 -0500 Received: from eggs.gnu.org ([208.118.235.92]:49947) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TwTuR-0006FW-F0 for submit@debbugs.gnu.org; Sat, 19 Jan 2013 03:36:17 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TwTtb-0000PG-2E for submit@debbugs.gnu.org; Sat, 19 Jan 2013 03:35:24 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-102.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, T_DKIM_INVALID,USER_IN_WHITELIST autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([208.118.235.17]:52768) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TwTta-0000PC-Ut for submit@debbugs.gnu.org; Sat, 19 Jan 2013 03:35:22 -0500 Received: from eggs.gnu.org ([208.118.235.92]:33072) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TwTtZ-00048i-Nd for bug-coreutils@gnu.org; Sat, 19 Jan 2013 03:35:22 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TwTtX-0000Oj-Ph for bug-coreutils@gnu.org; Sat, 19 Jan 2013 03:35:21 -0500 Received: from mail-ob0-f173.google.com ([209.85.214.173]:61782) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TwTtX-0000OT-Jo for bug-coreutils@gnu.org; Sat, 19 Jan 2013 03:35:19 -0500 Received: by mail-ob0-f173.google.com with SMTP id xn12so4405665obc.4 for ; Sat, 19 Jan 2013 00:35:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=slamb.org; s=google; h=mime-version:x-received:date:message-id:subject:from:to :content-type; bh=3TNEXMCwkTIj5E9XpWe4NgyvA7HwKHA3YfNgAnEVack=; b=dPxG013y6ePleyacK86mIowtYvbSVh0HbJj/oGbqOVxCFYu3CGNx2Z4Nj8KWxyAT67 gECDh+e1rE+J+F9kMSi9bbcrZvxy45PaCtvW+6ievKUthc3L1Y0Hc2gvUhrx1rMfk+Yz R+OEeYeZ5rIPprlYxGWcoHp7jOei4rMuX5j2Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=3TNEXMCwkTIj5E9XpWe4NgyvA7HwKHA3YfNgAnEVack=; b=CsZB0K3zFtp7mbT1OhHzWmb0emtKmK1QeqmPhgFbNH+WLNBUPoV+OSM3kKOZzlUBqt op6Rie1FkCzQVYkJBrknymRWmifJAENdVUY2ZerkI2vzZ32J5/PC/sqRxoHzxt1sh3Iz 7+AiE5GO6byS3nS3wXqZS2+hEsuuGavBUuMG/2K55DoEhfkGZx90dpt12Y0c4dDeRvYN qHKZAOPYrv2WQxHKoD0hhcfKauaQdV4s1ol8WTNydd6EQFJXhZDjfpuX9bjBcVtQ+r+S w/OkUsgzl/5P7R3vrxXg09T5uqQT7kL7Z3ZkcyV7h4cPNZKXNO21XD8BceSMHK4eUfrd Tw9w== MIME-Version: 1.0 X-Received: by 10.182.188.69 with SMTP id fy5mr9090436obc.74.1358584518495; Sat, 19 Jan 2013 00:35:18 -0800 (PST) Received: by 10.76.88.19 with HTTP; Sat, 19 Jan 2013 00:35:18 -0800 (PST) Date: Sat, 19 Jan 2013 00:35:18 -0800 Message-ID: From: Scott Lamb Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQnuzayFGBqsfeXSJ73OalwBQculZzk5ciUbuZf4i5i0QhN95HndKUB7kKLdqIpbkC+sRaTH X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 208.118.235.17 X-Spam-Score: -4.2 (----) X-Mailman-Approved-At: Sat, 19 Jan 2013 12:26:16 -0500 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.5 (-----) "cut -f" has an apparently long-standing behavior that I'd consider a bug: it does not fully send line N to stdout until the first character of line N+1 has been read on stdin. This is confusing when stdin comes from "tail -f" or the like. The exact behavior varies slightly. If stdin is a tty, all but the trailing newline will be flushed immediately and then the trailing newline will be flushed when the next character shows up. If stdin is not a tty, there's no flush at all until the next character shows up. For example, if I type the following into a shell on Ubuntu 12.04.1, meaning cut from coreutils 8.13 and glibc package version 2.15-0ubuntu10.3: cut -f1- foo bar baz ^D I will see the following: $ cut -f1- foo foobar barbaz baz $ and if I instead use "cat | cut -f1-" in the first line, I will see the following: $ cat | cut -f1- foo bar foo baz bar baz $ (coreutils's cut -c does not have the same laggy behavior. Neither does BSD cut on my OS X machine in either -c or -f mode.) This code in cut_fields (still found in trunk tip) is responsible for delaying the newline; it runs between the newline being read and being written: if (c == '\n') { c = getc (stream); if (c != EOF) { ungetc (c, stream); c = '\n'; } } I believe that code is there to avoid turning one newline at EOF into two, but that goal could be accomplished in another way. I don't know exactly why the behavior differs based on stdin being a tty or not. My best guess is that glibc might have some logic that, if stdin is a tty, automatically flushes stdout any time the program blocks on stdin. glibc's stdio internals are a bit hard for me to follow, so I haven't found the code in question. Apparently this is a vaguely standardized behavior; I see a stackoverflow post mentioning the following: """ The input and output dynamics of interactive devices shall take place as specified in 7.19.3. The intent of these requirements is that unbuffered or line-buffered output appear as soon as possible, to ensure that prompting messages actually appear prior to a program waiting for input. (ISO/IEC 9899:TC2 Committee Draft -- May 6, 2005, page 14). """ -- Scott Lamb From unknown Mon Jun 23 04:12:22 2025 X-Loop: help-debbugs@gnu.org Subject: bug#13498: "cut -f" lags a line Resent-From: Andreas Schwab Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Sat, 19 Jan 2013 20:06:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 13498 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: Scott Lamb Cc: 13498@debbugs.gnu.org Received: via spool by 13498-submit@debbugs.gnu.org id=B13498.135862596026659 (code B ref 13498); Sat, 19 Jan 2013 20:06:01 +0000 Received: (at 13498) by debbugs.gnu.org; 19 Jan 2013 20:06:00 +0000 Received: from localhost ([127.0.0.1]:40554 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Twefv-0006vw-MX for submit@debbugs.gnu.org; Sat, 19 Jan 2013 15:05:59 -0500 Received: from mail-out.m-online.net ([212.18.0.10]:32918) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Twefs-0006vn-Ci for 13498@debbugs.gnu.org; Sat, 19 Jan 2013 15:05:57 -0500 Received: from frontend1.mail.m-online.net (frontend1.mail.intern.m-online.net [192.168.8.180]) by mail-out.m-online.net (Postfix) with ESMTP id 3YpVP940vTz3hhfM; Sat, 19 Jan 2013 21:05:01 +0100 (CET) Received: from localhost (dynscan1.mnet-online.de [192.168.6.68]) by mail.m-online.net (Postfix) with ESMTP id 3YpVP92NbFzbbjj; Sat, 19 Jan 2013 21:05:01 +0100 (CET) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.180]) by localhost (dynscan1.mail.m-online.net [192.168.6.68]) (amavisd-new, port 10024) with ESMTP id z9bKNFlLx1jZ; Sat, 19 Jan 2013 21:04:35 +0100 (CET) X-Auth-Info: 8T0j+N3KIHtYGc7ULEe45XJZJ7XuNkM+5bE6zTYk84Y= Received: from igel.home (ppp-88-217-126-207.dynamic.mnet-online.de [88.217.126.207]) by mail.mnet-online.de (Postfix) with ESMTPA; Sat, 19 Jan 2013 21:05:00 +0100 (CET) Received: by igel.home (Postfix, from userid 501) id 92F23CA2A1; Sat, 19 Jan 2013 21:04:59 +0100 (CET) From: Andreas Schwab References: X-Yow: It's strange, but I'm only TRULY ALIVE when I'm covered in POLKA DOTS and TACO SAUCE... Date: Sat, 19 Jan 2013 21:04:59 +0100 In-Reply-To: (Scott Lamb's message of "Sat, 19 Jan 2013 00:35:18 -0800") Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.2.92 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.5 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) Scott Lamb writes: > I don't know exactly why the behavior differs based on stdin being a > tty or not. My best guess is that glibc might have some logic that, if > stdin is a tty, automatically flushes stdout any time the program > blocks on stdin. When a new buffer is read for a line buffered or unbuffered stream, stdout is flushed. This is traditional Unix behaviour, but AFAIK not required by any standard. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." From unknown Mon Jun 23 04:12:22 2025 X-Loop: help-debbugs@gnu.org Subject: bug#13498: "cut -f" lags a line Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Sun, 20 Jan 2013 12:43:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 13498 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: Scott Lamb Cc: 13498@debbugs.gnu.org Received: via spool by 13498-submit@debbugs.gnu.org id=B13498.135868573530801 (code B ref 13498); Sun, 20 Jan 2013 12:43:01 +0000 Received: (at 13498) by debbugs.gnu.org; 20 Jan 2013 12:42:15 +0000 Received: from localhost ([127.0.0.1]:41164 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TwuE3-00080i-51 for submit@debbugs.gnu.org; Sun, 20 Jan 2013 07:42:15 -0500 Received: from mail3.vodafone.ie ([213.233.128.45]:62125) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TwuE0-00080Z-OL for 13498@debbugs.gnu.org; Sun, 20 Jan 2013 07:42:14 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApMBAFzl+1BtTIdf/2dsb2JhbAANN75PgxEBAQEEMgEFQRALDQEKCRYPCQMCAQIBRQYNAQcBAYghqSiRbJEmEwOXKIRxjTGBZgc Received: from unknown (HELO [192.168.1.79]) ([109.76.135.95]) by mail3.vodafone.ie with ESMTP; 20 Jan 2013 12:41:13 +0000 Message-ID: <50FBE5E8.5020606@draigBrady.com> Date: Sun, 20 Jan 2013 12:41:12 +0000 From: =?UTF-8?Q?P=C3=A1draig?= Brady User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1 MIME-Version: 1.0 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: 0.8 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) On 01/19/2013 08:35 AM, Scott Lamb wrote: > "cut -f" has an apparently long-standing behavior that I'd consider a > bug: it does not fully send line N to stdout until the first character > of line N+1 has been read on stdin. This is confusing when stdin comes > from "tail -f" or the like. The exact behavior varies slightly. If > stdin is a tty, all but the trailing newline will be flushed > immediately and then the trailing newline will be flushed when the > next character shows up. If stdin is not a tty, there's no flush at > all until the next character shows up. > > For example, if I type the following into a shell on Ubuntu 12.04.1, > meaning cut from coreutils 8.13 and glibc package version > 2.15-0ubuntu10.3: > > cut -f1- > foo > bar > baz > ^D > > I will see the following: > > $ cut -f1- > foo > foobar > > barbaz > > baz > $ > > and if I instead use "cat | cut -f1-" in the first line, I will see > the following: > > $ cat | cut -f1- > foo > bar > foo > baz > bar > baz > $ > > (coreutils's cut -c does not have the same laggy behavior. Neither > does BSD cut on my OS X machine in either -c or -f mode.) > > This code in cut_fields (still found in trunk tip) is responsible for > delaying the newline; it runs between the newline being read and being > written: > > if (c == '\n') > { > c = getc (stream); > if (c != EOF) > { > ungetc (c, stream); > c = '\n'; > } > } > > I believe that code is there to avoid turning one newline at EOF into > two, but that goal could be accomplished in another way. > > I don't know exactly why the behavior differs based on stdin being a > tty or not. My best guess is that glibc might have some logic that, if > stdin is a tty, automatically flushes stdout any time the program > blocks on stdin. glibc's stdio internals are a bit hard for me to > follow, so I haven't found the code in question. Apparently this is a > vaguely standardized behavior; I see a stackoverflow post mentioning > the following: > > """ > The input and output dynamics of interactive devices shall take place > as specified in 7.19.3. The intent of these requirements is that > unbuffered or line-buffered output appear as soon as possible, to > ensure that prompting messages actually appear prior to a program > waiting for input. > > (ISO/IEC 9899:TC2 Committee Draft -- May 6, 2005, page 14). > """ For my reference: http://comments.pixelbeat.org/programming/stdio_buffering/#comment-250521 Yes the use of ungetc() is awkward in cut. I notice that pr is the only other util using ungetc. Also the i18n version of cut on my system has a rewritten cut_fields() function that doesn't exhibit the behavior. ungetc() is coupled with the use of getndelim2(), but I'll have a look at addressing this. thanks, Pádraig. From unknown Mon Jun 23 04:12:22 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.428 (Entity 5.428) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Scott Lamb Subject: bug#13498: closed (Re: bug#13498: "cut -f" lags a line) Message-ID: References: <50FDFB82.6070200@draigBrady.com> X-Gnu-PR-Message: they-closed 13498 X-Gnu-PR-Package: coreutils Reply-To: 13498@debbugs.gnu.org Date: Tue, 22 Jan 2013 02:40:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1358822402-27013-1" This is a multi-part message in MIME format... ------------=_1358822402-27013-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #13498: "cut -f" lags a line which was filed against the coreutils package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 13498@debbugs.gnu.org. --=20 13498: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D13498 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1358822402-27013-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 13498-done) by debbugs.gnu.org; 22 Jan 2013 02:39:08 +0000 Received: from localhost ([127.0.0.1]:43824 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TxTlT-00070a-Nr for submit@debbugs.gnu.org; Mon, 21 Jan 2013 21:39:08 -0500 Received: from mail3.vodafone.ie ([213.233.128.45]:38712) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TxTlR-00070Q-GY for 13498-done@debbugs.gnu.org; Mon, 21 Jan 2013 21:39:06 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AuEBALX6/VBtTUAA/2dsb2JhbAANN4V+iG2vXoMSAQEEAQI1QRALDQETFAIPCQMCAQIBFi8GDQEHAQGIIaktgzCPKJE2A5Neg0qEcY0kDQ Received: from unknown (HELO [192.168.1.79]) ([109.77.64.0]) by mail3.vodafone.ie with ESMTP; 22 Jan 2013 02:37:55 +0000 Message-ID: <50FDFB82.6070200@draigBrady.com> Date: Tue, 22 Jan 2013 02:37:54 +0000 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1 MIME-Version: 1.0 To: Scott Lamb Subject: Re: bug#13498: "cut -f" lags a line References: <50FBE5E8.5020606@draigBrady.com> In-Reply-To: <50FBE5E8.5020606@draigBrady.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 13498-done Cc: 13498-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: 0.8 (/) Proposed patch at: http://lists.gnu.org/archive/html/coreutils/2013-01/msg00076.html ------------=_1358822402-27013-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 19 Jan 2013 17:26:21 +0000 Received: from localhost ([127.0.0.1]:40510 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TwcBP-000392-1g for submit@debbugs.gnu.org; Sat, 19 Jan 2013 12:26:20 -0500 Received: from eggs.gnu.org ([208.118.235.92]:49947) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1TwTuR-0006FW-F0 for submit@debbugs.gnu.org; Sat, 19 Jan 2013 03:36:17 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TwTtb-0000PG-2E for submit@debbugs.gnu.org; Sat, 19 Jan 2013 03:35:24 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-102.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_LOW, T_DKIM_INVALID,USER_IN_WHITELIST autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([208.118.235.17]:52768) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TwTta-0000PC-Ut for submit@debbugs.gnu.org; Sat, 19 Jan 2013 03:35:22 -0500 Received: from eggs.gnu.org ([208.118.235.92]:33072) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TwTtZ-00048i-Nd for bug-coreutils@gnu.org; Sat, 19 Jan 2013 03:35:22 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1TwTtX-0000Oj-Ph for bug-coreutils@gnu.org; Sat, 19 Jan 2013 03:35:21 -0500 Received: from mail-ob0-f173.google.com ([209.85.214.173]:61782) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1TwTtX-0000OT-Jo for bug-coreutils@gnu.org; Sat, 19 Jan 2013 03:35:19 -0500 Received: by mail-ob0-f173.google.com with SMTP id xn12so4405665obc.4 for ; Sat, 19 Jan 2013 00:35:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=slamb.org; s=google; h=mime-version:x-received:date:message-id:subject:from:to :content-type; bh=3TNEXMCwkTIj5E9XpWe4NgyvA7HwKHA3YfNgAnEVack=; b=dPxG013y6ePleyacK86mIowtYvbSVh0HbJj/oGbqOVxCFYu3CGNx2Z4Nj8KWxyAT67 gECDh+e1rE+J+F9kMSi9bbcrZvxy45PaCtvW+6ievKUthc3L1Y0Hc2gvUhrx1rMfk+Yz R+OEeYeZ5rIPprlYxGWcoHp7jOei4rMuX5j2Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=3TNEXMCwkTIj5E9XpWe4NgyvA7HwKHA3YfNgAnEVack=; b=CsZB0K3zFtp7mbT1OhHzWmb0emtKmK1QeqmPhgFbNH+WLNBUPoV+OSM3kKOZzlUBqt op6Rie1FkCzQVYkJBrknymRWmifJAENdVUY2ZerkI2vzZ32J5/PC/sqRxoHzxt1sh3Iz 7+AiE5GO6byS3nS3wXqZS2+hEsuuGavBUuMG/2K55DoEhfkGZx90dpt12Y0c4dDeRvYN qHKZAOPYrv2WQxHKoD0hhcfKauaQdV4s1ol8WTNydd6EQFJXhZDjfpuX9bjBcVtQ+r+S w/OkUsgzl/5P7R3vrxXg09T5uqQT7kL7Z3ZkcyV7h4cPNZKXNO21XD8BceSMHK4eUfrd Tw9w== MIME-Version: 1.0 X-Received: by 10.182.188.69 with SMTP id fy5mr9090436obc.74.1358584518495; Sat, 19 Jan 2013 00:35:18 -0800 (PST) Received: by 10.76.88.19 with HTTP; Sat, 19 Jan 2013 00:35:18 -0800 (PST) Date: Sat, 19 Jan 2013 00:35:18 -0800 Message-ID: Subject: "cut -f" lags a line From: Scott Lamb To: bug-coreutils@gnu.org Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQnuzayFGBqsfeXSJ73OalwBQculZzk5ciUbuZf4i5i0QhN95HndKUB7kKLdqIpbkC+sRaTH X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 208.118.235.17 X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Sat, 19 Jan 2013 12:26:16 -0500 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.5 (-----) "cut -f" has an apparently long-standing behavior that I'd consider a bug: it does not fully send line N to stdout until the first character of line N+1 has been read on stdin. This is confusing when stdin comes from "tail -f" or the like. The exact behavior varies slightly. If stdin is a tty, all but the trailing newline will be flushed immediately and then the trailing newline will be flushed when the next character shows up. If stdin is not a tty, there's no flush at all until the next character shows up. For example, if I type the following into a shell on Ubuntu 12.04.1, meaning cut from coreutils 8.13 and glibc package version 2.15-0ubuntu10.3: cut -f1- foo bar baz ^D I will see the following: $ cut -f1- foo foobar barbaz baz $ and if I instead use "cat | cut -f1-" in the first line, I will see the following: $ cat | cut -f1- foo bar foo baz bar baz $ (coreutils's cut -c does not have the same laggy behavior. Neither does BSD cut on my OS X machine in either -c or -f mode.) This code in cut_fields (still found in trunk tip) is responsible for delaying the newline; it runs between the newline being read and being written: if (c == '\n') { c = getc (stream); if (c != EOF) { ungetc (c, stream); c = '\n'; } } I believe that code is there to avoid turning one newline at EOF into two, but that goal could be accomplished in another way. I don't know exactly why the behavior differs based on stdin being a tty or not. My best guess is that glibc might have some logic that, if stdin is a tty, automatically flushes stdout any time the program blocks on stdin. glibc's stdio internals are a bit hard for me to follow, so I haven't found the code in question. Apparently this is a vaguely standardized behavior; I see a stackoverflow post mentioning the following: """ The input and output dynamics of interactive devices shall take place as specified in 7.19.3. The intent of these requirements is that unbuffered or line-buffered output appear as soon as possible, to ensure that prompting messages actually appear prior to a program waiting for input. (ISO/IEC 9899:TC2 Committee Draft -- May 6, 2005, page 14). """ -- Scott Lamb ------------=_1358822402-27013-1-- From debbugs-submit-bounces@debbugs.gnu.org Thu May 29 21:29:24 2014 Received: (at control) by debbugs.gnu.org; 30 May 2014 01:29:24 +0000 Received: from localhost ([127.0.0.1]:36641 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1WqBdM-00021H-14 for submit@debbugs.gnu.org; Thu, 29 May 2014 21:29:24 -0400 Received: from mx1.redhat.com ([209.132.183.28]:32124) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1WqBdI-000215-Vs for control@debbugs.gnu.org; Thu, 29 May 2014 21:29:23 -0400 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s4U1TJaB004257 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 29 May 2014 21:29:19 -0400 Received: from [10.3.113.192] (ovpn-113-192.phx2.redhat.com [10.3.113.192]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s4U1TIxU007062 for ; Thu, 29 May 2014 21:29:19 -0400 Message-ID: <5387DEEE.5060600@redhat.com> Date: Thu, 29 May 2014 19:29:18 -0600 From: Eric Blake Organization: Red Hat, Inc. User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 To: GNU bug tracker automated control server Subject: Re: Archived problem report bug#13498 (bug#13498: "cut -f" lags a line) References: <5387B6AC.2060902@redhat.com> In-Reply-To: X-Enigmail-Version: 1.6 OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="8fXHKVKuWNUrow3nCabnrsKfMb3f6h1Tt" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Spam-Score: -5.7 (-----) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.7 (-----) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --8fXHKVKuWNUrow3nCabnrsKfMb3f6h1Tt Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable unarchive 13498 stop On 05/29/2014 04:38 PM, GNU bug Tracking System wrote: > You sent a message to the GNU bug tracking system, relating to bug#1349= 8. >=20 > Your message was dated Thu, 29 May 2014 16:37:32 -0600 and was sent to > 13498-submit@debbugs.gnu.org. It had > Message-ID: <5387B6AC.2060902@redhat.com> > and Subject: bug#13498: "cut -f" lags a line >=20 > This bug is currently in a read-only state. This is because it has bee= n > closed and has received no comments for more than 28 days, until now. >=20 > If you still wish to comment on or change this report, you must first > "unarchive" it. You can do this by sending a message to the control-se= rver > (control@debbugs.gnu.org), with body containing: >=20 > unarchive 13498 >=20 > This does not reopen the report. You might also want to do that, > but there is no need to if you just want to add a comment. >=20 > For more information, see http://debbugs.gnu.org/server-control.html, > or contact help-debbugs@gnu.org if you need assistance. >=20 > (If you believe you have received this mail in error, please contact > help-debbugs@gnu.org.) >=20 --=20 Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --8fXHKVKuWNUrow3nCabnrsKfMb3f6h1Tt Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCAAGBQJTh97uAAoJEKeha0olJ0NqGOoH/3Fg9Xtu5ZAEpt3bG5VMN0c2 R40stp9zNj+pwFYZomxJlnTusQtbJS+E/V3ISE5IT+m35HcJCIeYo7g7wf4e7hqp o22gW1ZnkxvqP0XRDPzfxGSxioKORx/WX1y23HXbLz2X3wFmSE/y3IxyZUPhaDFA pM1/n6TcgceFnqQuW1ahFTyDn7z7RghF32PwNktlaulhBnlfUXWvhE4vPj0/dbOH URrEsXYXXQ69WF1Z/A9ekLCXr0qrzfLWL580A8qACSHE4ItL/rii/wFfKoBYecKG CH4LMtyOtkdqHQjzDMC0SqSOOTPUZsjSmxSQao2C1eOBtDVowozVJcEdOBzTZsA= =8AYe -----END PGP SIGNATURE----- --8fXHKVKuWNUrow3nCabnrsKfMb3f6h1Tt-- From unknown Mon Jun 23 04:12:22 2025 X-Loop: help-debbugs@gnu.org Subject: bug#13498: "cut -f" lags a line Resent-From: Eric Blake Original-Sender: "Debbugs-submit" Resent-CC: bug-coreutils@gnu.org Resent-Date: Fri, 30 May 2014 01:32:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 13498 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: Cc: 13498@debbugs.gnu.org Received: via spool by 13498-submit@debbugs.gnu.org id=B13498.14014134968149 (code B ref 13498); Fri, 30 May 2014 01:32:02 +0000 Received: (at 13498) by debbugs.gnu.org; 30 May 2014 01:31:36 +0000 Received: from localhost ([127.0.0.1]:36646 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1WqBfT-00027M-Uv for submit@debbugs.gnu.org; Thu, 29 May 2014 21:31:36 -0400 Received: from mx1.redhat.com ([209.132.183.28]:13559) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1WqBfR-00027A-1u for 13498@debbugs.gnu.org; Thu, 29 May 2014 21:31:34 -0400 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id s4U1VWhf005142 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for <13498@debbugs.gnu.org>; Thu, 29 May 2014 21:31:32 -0400 Received: from [10.3.113.192] (ovpn-113-192.phx2.redhat.com [10.3.113.192]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id s4U1VWHY008169 for <13498@debbugs.gnu.org>; Thu, 29 May 2014 21:31:32 -0400 Message-ID: <5387DF73.5030307@redhat.com> Date: Thu, 29 May 2014 19:31:31 -0600 From: Eric Blake Organization: Red Hat, Inc. User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.5.0 MIME-Version: 1.0 References: In-Reply-To: X-Enigmail-Version: 1.6 OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="l2T58PHM2t29PbfAftu4OSPMWrlchKaNk" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Spam-Score: -4.4 (----) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.4 (----) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --l2T58PHM2t29PbfAftu4OSPMWrlchKaNk Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 01/19/2013 01:04 PM, Andreas Schwab wrote: [revisiting an old bug, since I just noticed it] > Scott Lamb writes: >=20 >> I don't know exactly why the behavior differs based on stdin being a >> tty or not. My best guess is that glibc might have some logic that, if= >> stdin is a tty, automatically flushes stdout any time the program >> blocks on stdin. >=20 > When a new buffer is read for a line buffered or unbuffered stream, > stdout is flushed. This is traditional Unix behaviour, but AFAIK not > required by any standard. Actually, POSIX requires it: http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#= tag_15_05 > When a stream is "unbuffered", bytes are intended to appear from the > source or at the destination as soon as possible; otherwise, bytes may > be accumulated and transmitted as a block. When a stream is "fully > buffered", bytes are intended to be transmitted as a block when a buffe= r > is filled. When a stream is "line buffered", bytes are intended to be > transmitted as a block when a byte is encountered. > Furthermore, bytes are intended to be transmitted as a block when a > buffer is filled, when input is requested on an unbuffered stream, or > when input is requested on a line-buffered stream that requires the > transmission of bytes. stdout is required to be buffered, and when stdin is the same terminal as stdout, then stdin is line-buffered and it is sufficient that an input line on stdin forces stdout to be flushed. --=20 Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --l2T58PHM2t29PbfAftu4OSPMWrlchKaNk Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCAAGBQJTh99zAAoJEKeha0olJ0Nqh2sH/3YQRMUYx4qu3eN1ochDEPSZ KS+ACzJYFxtSbn2h8CLejW5T+nClAbNSqdBcYu65E8gXxmIVOyqtFY1qE4v9Tgm8 DX05XerlmRXtMRIhjzBlHV7ibJ54qRujG/Hi/nXs6x8hOk+3uOxUnRLYGGuo8bPJ GkPbvnhKoLlNDR956vqwFyjrc3Hulu7kfziHFpm3QJb9Klh7A/nN+3Qm2G5oWfUn XLGwkYjhdfvO3vqgoyp8bEMOrPcIffvFqRVm+81UiTD5spKEXgVYn0HcRPsyNFc1 uS9zm1fy4+8Nt8NFZ0YL7rlzjoLLqL0jZWc8SOhV4sP4M5bMZSsTTYWZUub1lpk= =uQID -----END PGP SIGNATURE----- --l2T58PHM2t29PbfAftu4OSPMWrlchKaNk--