From unknown Wed Jun 25 00:20:36 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#20029 <20029@debbugs.gnu.org> To: bug#20029 <20029@debbugs.gnu.org> Subject: Status: 'yes' surprisingly slow Reply-To: bug#20029 <20029@debbugs.gnu.org> Date: Wed, 25 Jun 2025 07:20:36 +0000 retitle 20029 'yes' surprisingly slow reassign 20029 coreutils submitter 20029 Ole Tange severity 20029 normal thanks From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 07 06:49:44 2015 Received: (at submit) by debbugs.gnu.org; 7 Mar 2015 11:49:44 +0000 Received: from localhost ([127.0.0.1]:38171 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YUDEl-0002Ut-Qz for submit@debbugs.gnu.org; Sat, 07 Mar 2015 06:49:44 -0500 Received: from eggs.gnu.org ([208.118.235.92]:51447) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YUDEj-0002Ug-9V for submit@debbugs.gnu.org; Sat, 07 Mar 2015 06:49:42 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YUDEd-0003PH-8K for submit@debbugs.gnu.org; Sat, 07 Mar 2015 06:49:35 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=BAYES_40,FREEMAIL_FROM, T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:60051) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YUDEd-0003P6-5j for submit@debbugs.gnu.org; Sat, 07 Mar 2015 06:49:35 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53164) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YUDEc-0001Tb-5Q for bug-coreutils@gnu.org; Sat, 07 Mar 2015 06:49:34 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YUDEb-0003Ot-CM for bug-coreutils@gnu.org; Sat, 07 Mar 2015 06:49:34 -0500 Received: from mail-ig0-x22c.google.com ([2607:f8b0:4001:c05::22c]:34505) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YUDEb-0003Op-6H for bug-coreutils@gnu.org; Sat, 07 Mar 2015 06:49:33 -0500 Received: by igal13 with SMTP id l13so9440148iga.1 for ; Sat, 07 Mar 2015 03:49:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:from:date:message-id:subject:to:content-type; bh=M+lh0LM0EfQVujRHg8oCHVblgL2z3PLK1z8GsGOA/bU=; b=FfTAtL60eBF7qdS5sgho8/xQPODFxLedN8bpEgmzmbge4C4DiMx6JU8r4ItdTkfE+O ZZnNErCQ6V+tYdOsAWJ1UAOP7m8aEsLjxGK1HSgNZF3wPc48RCwN5wTZiEsWEsC99Zqq IB4qEzkLl9o6JtV/RtUmsPC/lo9WQHi/GHO+N6xtVkKsJgtr5fOdaIGQkaNLIzJtTVKv CXr8tnpLve9rGBNpWRn+kLo0YPSy9eAPJpcUeXyJY2o4oWB7Y6kPzR9gqNH35ZoaXnDO Z1hngFClJIpkZOgM3Oi00tj+QxFTU6gHBJ6d54TMk+tkMsRkavWEWm6fHd6IpBgq1dVR MChA== X-Received: by 10.107.134.160 with SMTP id q32mr34863601ioi.70.1425728971924; Sat, 07 Mar 2015 03:49:31 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.15.202 with HTTP; Sat, 7 Mar 2015 03:49:10 -0800 (PST) From: Ole Tange Date: Sat, 7 Mar 2015 12:49:10 +0100 X-Google-Sender-Auth: MVUIW5pM8ttCNV3vIQzrI58XZZE Message-ID: Subject: 'yes' surprisingly slow To: bug-coreutils@gnu.org Content-Type: text/plain; charset=UTF-8 X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.0 (----) These two commands give the same output: $ yes `echo {1..1000}` | head -c 2300M | md5sum a0241f2247e9a37db60e7def3e4f7038 - $ yes "`echo {1..1000}`" | head -c 2300M | md5sum a0241f2247e9a37db60e7def3e4f7038 - But the time to run is quite different: $ time yes "`echo {1..1000}`" | head -c 2300M >/dev/null real 0m0.897s user 0m0.384s sys 0m1.343s $ time yes `echo {1..1000}` | head -c 2300M >/dev/null real 0m11.352s user 0m10.571s sys 0m2.590s WTF?! I imagine 'yes' spends a lot of time collecting the 1000 args. But why does it do that more than once? /Ole From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 07 07:10:51 2015 Received: (at 20029) by debbugs.gnu.org; 7 Mar 2015 12:10:51 +0000 Received: from localhost ([127.0.0.1]:38178 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YUDZC-00034P-Ug for submit@debbugs.gnu.org; Sat, 07 Mar 2015 07:10:51 -0500 Received: from mail4.vodafone.ie ([213.233.128.170]:27443) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YUDZA-00033t-2V for 20029@debbugs.gnu.org; Sat, 07 Mar 2015 07:10:48 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AiUHANHp+lRtT8QO/2dsb2JhbABcgwaENqxSlU+CUQKBJk0BAQEBAQF8hBABBSMPAVYLDQsCAgUWCwICCQMCAQIBRQYBDAgBAYgvAbBrhWmUXAELIIEhiXaEdYJogUMFoF2MZiOCAxuBUD6CdAEBAQ Received: from unknown (HELO localhost.localdomain) ([109.79.196.14]) by mail3.vodafone.ie with ESMTP; 07 Mar 2015 12:10:39 +0000 Message-ID: <54FAEABF.1090104@draigBrady.com> Date: Sat, 07 Mar 2015 12:10:39 +0000 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Ole Tange , 20029@debbugs.gnu.org Subject: Re: bug#20029: 'yes' surprisingly slow References: In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 20029 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 07/03/15 11:49, Ole Tange wrote: > These two commands give the same output: > > $ yes `echo {1..1000}` | head -c 2300M | md5sum > a0241f2247e9a37db60e7def3e4f7038 - > > $ yes "`echo {1..1000}`" | head -c 2300M | md5sum > a0241f2247e9a37db60e7def3e4f7038 - > > But the time to run is quite different: > > $ time yes "`echo {1..1000}`" | head -c 2300M >/dev/null > > real 0m0.897s > user 0m0.384s > sys 0m1.343s > > $ time yes `echo {1..1000}` | head -c 2300M >/dev/null > > real 0m11.352s > user 0m10.571s > sys 0m2.590s > > WTF?! > > I imagine 'yes' spends a lot of time collecting the 1000 args. But why > does it do that more than once? The stdio interactions dominate here. The slow case has 1000 times more fputs_unlocked() calls. Yes we could build the line up once and output that. If doing that we could also build up a BUFSIZ of complete lines to output at a time, in which case you'd probably avoid stdio altogether. BTW I noticed tee uses stdio calls which is redundant overhead currently. It wouldn't if we added a --buffered call to tee so that it might honor stdbuf(1), though I'm not sure it's worth that flexibility in tee. I'll look at improving these. thanks, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 09 15:48:28 2015 Received: (at 20029) by debbugs.gnu.org; 9 Mar 2015 19:48:28 +0000 Received: from localhost ([127.0.0.1]:40947 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV3f9-0004Tw-K8 for submit@debbugs.gnu.org; Mon, 09 Mar 2015 15:48:28 -0400 Received: from mail2.vodafone.ie ([213.233.128.44]:24102) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV3f5-0004Te-Cp for 20029@debbugs.gnu.org; Mon, 09 Mar 2015 15:48:25 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvYKAH/4/VRtTRx//2dsb2JhbABcgwZSVQWDCq0pkjiFcQECAoErTQEBAQEBAXyEDwEBAQQjZgsNBAMBAgEJFgsCAgkDAgECAT0IBgEMBgIBAQWIKgEDpleFaZVcAQEBAQEFAQEBAQEBAQEaixeEXRiCaIFFBY9tggeBLlGHCTmFOIZagk2DQiOCAxuBUD4xgQQkgRsBAQE Received: from unknown (HELO localhost.localdomain) ([109.77.28.127]) by mail2.vodafone.ie with ESMTP; 09 Mar 2015 19:47:44 +0000 Message-ID: <54FDF8DF.7010109@draigBrady.com> Date: Mon, 09 Mar 2015 19:47:43 +0000 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Ole Tange , 20029@debbugs.gnu.org Subject: Re: bug#20029: 'yes' surprisingly slow References: <54FAEABF.1090104@draigBrady.com> In-Reply-To: <54FAEABF.1090104@draigBrady.com> Content-Type: multipart/mixed; boundary="------------000600080208040101070005" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 20029 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) This is a multi-part message in MIME format. --------------000600080208040101070005 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit On 07/03/15 12:10, Pádraig Brady wrote: > On 07/03/15 11:49, Ole Tange wrote: >> These two commands give the same output: >> >> $ yes `echo {1..1000}` | head -c 2300M | md5sum >> a0241f2247e9a37db60e7def3e4f7038 - >> >> $ yes "`echo {1..1000}`" | head -c 2300M | md5sum >> a0241f2247e9a37db60e7def3e4f7038 - >> >> But the time to run is quite different: >> >> $ time yes "`echo {1..1000}`" | head -c 2300M >/dev/null >> >> real 0m0.897s >> user 0m0.384s >> sys 0m1.343s >> >> $ time yes `echo {1..1000}` | head -c 2300M >/dev/null >> >> real 0m11.352s >> user 0m10.571s >> sys 0m2.590s >> >> WTF?! >> >> I imagine 'yes' spends a lot of time collecting the 1000 args. But why >> does it do that more than once? > > The stdio interactions dominate here. > The slow case has 1000 times more fputs_unlocked() calls. > Yes we could build the line up once and output that. > If doing that we could also build up a BUFSIZ of complete lines > to output at a time, in which case you'd probably avoid stdio altogether. The attached should make things more efficient here. thanks, Pádraig. --------------000600080208040101070005 Content-Type: text/x-patch; name="yes-efficiency.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="yes-efficiency.patch" >From 7959bbf19307705e98f08cfa32a9dcf67672590c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?P=C3=A1draig=20Brady?= Date: Mon, 9 Mar 2015 19:27:32 +0000 Subject: [PATCH] yes: output data more efficiently yes(1) may be used to generate repeating patterns of text for test inputs etc., so adjust to be more efficient. Profiling the case where yes(1) is outputting small items through stdio (which was the default case), shows the overhead continuously processing small items in main() and in stdio: $ yes >/dev/null & perf top -p $! 31.02% yes [.] main 27.36% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5 14.51% libc-2.20.so [.] fputs_unlocked 13.50% libc-2.20.so [.] strlen 10.66% libc-2.20.so [.] __GI___mempcpy 1.98% yes [.] fputs_unlocked@plta Sending more data per stdio call improves the situation, but still, there is significant stdio overhead due to memory copies, and the repeated string length checking: $ yes "`echo {1..1000}`" >/dev/null & perf top -p $! 42.26% libc-2.20.so [.] __GI___mempcpy 17.38% libc-2.20.so [.] strlen 5.21% [kernel] [k] __srcu_read_lock 4.58% [kernel] [k] __srcu_read_unlock 4.27% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5 2.50% libc-2.20.so [.] __GI___libc_write 2.45% [kernel] [k] system_call 2.40% [kernel] [k] system_call_after_swapgs 2.27% [kernel] [k] vfs_write 2.09% libc-2.20.so [.] _IO_do_write@@GLIBC_2.2.5 2.01% [kernel] [k] fsnotify 1.95% libc-2.20.so [.] _IO_file_write@@GLIBC_2.2.5 1.44% yes [.] main We can avoid all stdio overhead by building up the buffer _once_ and outputting that, and the profile below shows the bottleneck moved to the kernel: $ src/yes >/dev/null & perf top -p $! 15.42% [kernel] [k] __srcu_read_lock 12.98% [kernel] [k] __srcu_read_unlock 9.41% libc-2.20.so [.] __GI___libc_write 9.11% [kernel] [k] vfs_write 8.35% [kernel] [k] fsnotify 8.02% [kernel] [k] system_call 5.84% [kernel] [k] system_call_after_swapgs 4.54% [kernel] [k] __fget_light 3.98% [kernel] [k] sys_write 3.65% [kernel] [k] selinux_file_permission 3.44% [kernel] [k] rw_verify_area 2.94% [kernel] [k] __fsnotify_parent 2.76% [kernel] [k] security_file_permission 2.39% yes [.] main 2.17% [kernel] [k] __fdget_pos 2.13% [kernel] [k] sysret_check 0.81% [kernel] [k] write_null 0.36% yes [.] write@plt Note this change also ensures that yes will only write complete lines for lines softer than BUFSIZ. * src/yes.c (main): Build up a BUFSIZ buffer of lines, and output that, rather than having stdio process each item. * tests/misc/yes.sh: Add a new test for various buffer sizes. * tests/local.mk: Reference the new test. Fixes http://bugs.gnu.org/20029 --- src/yes.c | 43 +++++++++++++++++++++++++++++++++++++++++-- tests/local.mk | 1 + tests/misc/yes.sh | 28 ++++++++++++++++++++++++++++ 3 files changed, 70 insertions(+), 2 deletions(-) create mode 100755 tests/misc/yes.sh diff --git a/src/yes.c b/src/yes.c index b35b13f..91dea11 100644 --- a/src/yes.c +++ b/src/yes.c @@ -58,6 +58,10 @@ Repeatedly output a line with all specified STRING(s), or 'y'.\n\ int main (int argc, char **argv) { + char buf[BUFSIZ]; + char *pbuf = buf; + int i; + initialize_main (&argc, &argv); set_program_name (argv[0]); setlocale (LC_ALL, ""); @@ -77,9 +81,44 @@ main (int argc, char **argv) argv[argc++] = bad_cast ("y"); } - while (true) + /* Buffer data locally once, rather than having the + large overhead of stdio buffering each item. */ + for (i = optind; i < argc; i++) + { + size_t len = strlen (argv[i]); + if (BUFSIZ < len || BUFSIZ - len <= pbuf - buf) + break; + memcpy (pbuf, argv[i], len); + pbuf += len; + *pbuf++ = i == argc - 1 ? '\n' : ' '; + } + if (i < argc) + pbuf = NULL; + else + { + size_t line_len = pbuf - buf; + size_t lines = BUFSIZ / line_len; + while (--lines) + { + memcpy (pbuf, pbuf - line_len, line_len); + pbuf += line_len; + } + } + + /* The normal case is to continuously output the local buffer. */ + while (pbuf) + { + if (write (STDOUT_FILENO, buf, pbuf - buf) == -1) + { + error (0, errno, _("standard output")); + return EXIT_FAILURE; + } + } + + /* If the data doesn't fit in BUFSIZ then it's large + and not too onerous to output using stdio in any case. */ + while (! pbuf) { - int i; for (i = optind; i < argc; i++) if (fputs (argv[i], stdout) == EOF || putchar (i == argc - 1 ? '\n' : ' ') == EOF) diff --git a/tests/local.mk b/tests/local.mk index 9a52080..56cba69 100644 --- a/tests/local.mk +++ b/tests/local.mk @@ -402,6 +402,7 @@ all_tests = \ tests/misc/uniq.pl \ tests/misc/uniq-perf.sh \ tests/misc/xattr.sh \ + tests/misc/yes.sh \ tests/tail-2/wait.sh \ tests/tail-2/retry.sh \ tests/tail-2/symlink.sh \ diff --git a/tests/misc/yes.sh b/tests/misc/yes.sh new file mode 100755 index 0000000..8828030 --- /dev/null +++ b/tests/misc/yes.sh @@ -0,0 +1,28 @@ +#!/bin/sh +# Validate yes buffer handling + +# Copyright (C) 2015 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +. "${srcdir=.}/tests/init.sh"; path_prepend_ ./src +print_ver_ yes + +for size in 1 4095 4096 8191 8192 16383 16384; do + printf "%${size}s\n" '' > out.1 + yes "$(printf %${size}s '')" | head -n2 | uniq > out.2 + compare out.1 out.2 || fail=1 +done + +Exit $fail -- 2.1.0 --------------000600080208040101070005-- From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 09 16:02:22 2015 Received: (at 20029) by debbugs.gnu.org; 9 Mar 2015 20:02:22 +0000 Received: from localhost ([127.0.0.1]:40951 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV3sb-0004q3-V1 for submit@debbugs.gnu.org; Mon, 09 Mar 2015 16:02:22 -0400 Received: from mx1.redhat.com ([209.132.183.28]:55911) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV3sZ-0004pu-I2 for 20029@debbugs.gnu.org; Mon, 09 Mar 2015 16:02:20 -0400 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id t29K2HdX012844 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 9 Mar 2015 16:02:17 -0400 Received: from [10.3.113.79] (ovpn-113-79.phx2.redhat.com [10.3.113.79]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t29K2GVU014118; Mon, 9 Mar 2015 16:02:16 -0400 Message-ID: <54FDFC48.9010104@redhat.com> Date: Mon, 09 Mar 2015 14:02:16 -0600 From: Eric Blake Organization: Red Hat, Inc. User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , Ole Tange , 20029@debbugs.gnu.org Subject: Re: bug#20029: 'yes' surprisingly slow References: <54FAEABF.1090104@draigBrady.com> <54FDF8DF.7010109@draigBrady.com> In-Reply-To: <54FDF8DF.7010109@draigBrady.com> OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="jixiNo2XxIxLHHgh3TwQQdc9sJ9JJnWmR" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 20029 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --jixiNo2XxIxLHHgh3TwQQdc9sJ9JJnWmR Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 03/09/2015 01:47 PM, P=C3=A1draig Brady wrote: >=20 > Note this change also ensures that yes will only write complete lines > for lines softer than BUFSIZ. s/softer/smaller/ >=20 > * src/yes.c (main): Build up a BUFSIZ buffer of lines, > and output that, rather than having stdio process each item. > * tests/misc/yes.sh: Add a new test for various buffer sizes. > * tests/local.mk: Reference the new test. > Fixes http://bugs.gnu.org/20029 > --- > src/yes.c | 43 +++++++++++++++++++++++++++++++++++++++++-- > tests/local.mk | 1 + > tests/misc/yes.sh | 28 ++++++++++++++++++++++++++++ > 3 files changed, 70 insertions(+), 2 deletions(-) > create mode 100755 tests/misc/yes.sh >=20 > diff --git a/src/yes.c b/src/yes.c > index b35b13f..91dea11 100644 > --- a/src/yes.c > +++ b/src/yes.c > @@ -58,6 +58,10 @@ Repeatedly output a line with all specified STRING(s= ), or 'y'.\n\ > int > main (int argc, char **argv) > { > + char buf[BUFSIZ]; Do you really want this stack-allocated? BUFSIZ can be larger than a page, which can then interfere with stack overflow detection. > +. "${srcdir=3D.}/tests/init.sh"; path_prepend_ ./src > +print_ver_ yes > + > +for size in 1 4095 4096 8191 8192 16383 16384; do Should you also test 4097 8193 and 16385 (that is, a likely 1-more-than-BUFSIZ in the mix)? --=20 Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --jixiNo2XxIxLHHgh3TwQQdc9sJ9JJnWmR Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCAAGBQJU/fxIAAoJEKeha0olJ0Nq9PoH/RQHsVGTicYZNfDVmQz50+bY O+3rj4PcZeFzUWqt6lCRWQgtgWuy/o9NRtiWIxaTo0UYIJafuK27e4TAy2YLrn0A ejBYc5IzdWfRfA37NfxQwRrgzudBBRCb+cwNozDobKmH7UkKdT88RPv6hKP8xB9z fErkEOnDaRKgjknZktUMWoE6CMJOGbwLBXq+U+YtCPvpEPYDzlLHOt1k2ojkbqEt afB8LzZO2gWTXVbW8raFhBMZP86jI5iqpokVSJY1/JIUZLuU+4HAnWBABOazetkj j+jIPNjNguQpB3hkRlI9oFTY/geJnRi5qhUDkVFLnoiMXC+693AVy9vTXMpYnBU= =Q7Dp -----END PGP SIGNATURE----- --jixiNo2XxIxLHHgh3TwQQdc9sJ9JJnWmR-- From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 09 18:15:08 2015 Received: (at 20029) by debbugs.gnu.org; 9 Mar 2015 22:15:08 +0000 Received: from localhost ([127.0.0.1]:40980 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV5x5-0008HU-DO for submit@debbugs.gnu.org; Mon, 09 Mar 2015 18:15:08 -0400 Received: from mail2.vodafone.ie ([213.233.128.44]:14062) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV5x2-0008Gf-Oo for 20029@debbugs.gnu.org; Mon, 09 Mar 2015 18:15:05 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AtIIAIMa/lRtTRx//2dsb2JhbABcgwZSVQWDCr9khXEBAgKBK00BAQEBAQF8hBABBSMPAVYLDQEKAgIFFgsCAgkDAgECAUUGAQwIAQGILwEDpCeFaZV/AQEIAgEfgSGJdoR1gmiBRQWTc4dChTiGWoYPI4NuPjGCQwEBAQ Received: from unknown (HELO localhost.localdomain) ([109.77.28.127]) by mail2.vodafone.ie with ESMTP; 09 Mar 2015 22:14:58 +0000 Message-ID: <54FE1B61.7080406@draigBrady.com> Date: Mon, 09 Mar 2015 22:14:57 +0000 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Eric Blake , Ole Tange , 20029@debbugs.gnu.org Subject: Re: bug#20029: 'yes' surprisingly slow References: <54FAEABF.1090104@draigBrady.com> <54FDF8DF.7010109@draigBrady.com> <54FDFC48.9010104@redhat.com> In-Reply-To: <54FDFC48.9010104@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 20029 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 09/03/15 20:02, Eric Blake wrote: > On 03/09/2015 01:47 PM, Pádraig Brady wrote: > >> >> Note this change also ensures that yes will only write complete lines >> for lines softer than BUFSIZ. > > s/softer/smaller/ > >> >> * src/yes.c (main): Build up a BUFSIZ buffer of lines, >> and output that, rather than having stdio process each item. >> * tests/misc/yes.sh: Add a new test for various buffer sizes. >> * tests/local.mk: Reference the new test. >> Fixes http://bugs.gnu.org/20029 >> --- >> src/yes.c | 43 +++++++++++++++++++++++++++++++++++++++++-- >> tests/local.mk | 1 + >> tests/misc/yes.sh | 28 ++++++++++++++++++++++++++++ >> 3 files changed, 70 insertions(+), 2 deletions(-) >> create mode 100755 tests/misc/yes.sh >> >> diff --git a/src/yes.c b/src/yes.c >> index b35b13f..91dea11 100644 >> --- a/src/yes.c >> +++ b/src/yes.c >> @@ -58,6 +58,10 @@ Repeatedly output a line with all specified STRING(s), or 'y'.\n\ >> int >> main (int argc, char **argv) >> { >> + char buf[BUFSIZ]; > > Do you really want this stack-allocated? BUFSIZ can be larger than a > page, which can then interfere with stack overflow detection. Well we do such stack buffers elsewhere: $ git grep char.*\\[BUFSIZ\\] src/head.c: char buf[BUFSIZ]; src/head.c: char buffer[BUFSIZ]; src/head.c: char buffer[BUFSIZ]; src/head.c: char buffer[BUFSIZ]; src/head.c: char buffer[BUFSIZ]; src/ls.c: char smallbuf[BUFSIZ]; src/od.c: char buf[BUFSIZ]; src/tail.c: char buffer[BUFSIZ]; src/tail.c: char buffer[BUFSIZ]; src/tail.c: char buffer[BUFSIZ]; src/tail.c: char buffer[BUFSIZ]; src/tail.c: char buffer[BUFSIZ]; src/tail.c: char buffer[BUFSIZ]; src/tee.c: char buffer[BUFSIZ]; src/tr.c:static char io_buf[BUFSIZ]; src/uptime.c: char buf[BUFSIZ]; src/yes.c: char buf[BUFSIZ]; We would probably change them all if this was thought to be a problem. >> +. "${srcdir=.}/tests/init.sh"; path_prepend_ ./src >> +print_ver_ yes >> + >> +for size in 1 4095 4096 8191 8192 16383 16384; do > > Should you also test 4097 8193 and 16385 (that is, a likely > 1-more-than-BUFSIZ in the mix)? The 1 more is implicit with the \n added by yes(1). thanks for the review! Pádraig From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 09 18:46:36 2015 Received: (at 20029) by debbugs.gnu.org; 9 Mar 2015 22:46:37 +0000 Received: from localhost ([127.0.0.1]:40988 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV6RY-0000st-Kq for submit@debbugs.gnu.org; Mon, 09 Mar 2015 18:46:36 -0400 Received: from smtp.cs.ucla.edu ([131.179.128.62]:45350) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV6RW-0000sf-Rp for 20029@debbugs.gnu.org; Mon, 09 Mar 2015 18:46:35 -0400 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id A4A2639E801D; Mon, 9 Mar 2015 15:46:28 -0700 (PDT) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lUEI5fnhmXfp; Mon, 9 Mar 2015 15:46:28 -0700 (PDT) Received: from [192.168.1.9] (pool-100-32-155-148.lsanca.fios.verizon.net [100.32.155.148]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 5125039E801C; Mon, 9 Mar 2015 15:46:28 -0700 (PDT) Message-ID: <54FE22C0.5040003@cs.ucla.edu> Date: Mon, 09 Mar 2015 15:46:24 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , Eric Blake , Ole Tange , 20029@debbugs.gnu.org Subject: Re: bug#20029: 'yes' surprisingly slow References: <54FAEABF.1090104@draigBrady.com> <54FDF8DF.7010109@draigBrady.com> <54FDFC48.9010104@redhat.com> <54FE1B61.7080406@draigBrady.com> In-Reply-To: <54FE1B61.7080406@draigBrady.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 20029 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Pádraig Brady wrote: > We would probably change them all if this was thought to be a problem. I don't think it's a real problem. I've never run into a system where a nonrecursive function can't declare a local variable of size BUFSIZ, and given the long tradition of software doing just that I would say that such a system has such a low quality of implementation that we needn't worry about it. From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 09 19:09:37 2015 Received: (at 20029) by debbugs.gnu.org; 9 Mar 2015 23:09:37 +0000 Received: from localhost ([127.0.0.1]:41012 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV6np-0003AQ-0i for submit@debbugs.gnu.org; Mon, 09 Mar 2015 19:09:37 -0400 Received: from mx1.redhat.com ([209.132.183.28]:38678) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV6nm-00039p-Jy for 20029@debbugs.gnu.org; Mon, 09 Mar 2015 19:09:35 -0400 Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id t29N9U87011209 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 9 Mar 2015 19:09:30 -0400 Received: from [10.3.113.79] (ovpn-113-79.phx2.redhat.com [10.3.113.79]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t29N9THp002779; Mon, 9 Mar 2015 19:09:29 -0400 Message-ID: <54FE2829.3060106@redhat.com> Date: Mon, 09 Mar 2015 17:09:29 -0600 From: Eric Blake Organization: Red Hat, Inc. User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , Ole Tange , 20029@debbugs.gnu.org Subject: Re: bug#20029: 'yes' surprisingly slow References: <54FAEABF.1090104@draigBrady.com> <54FDF8DF.7010109@draigBrady.com> <54FDFC48.9010104@redhat.com> <54FE1B61.7080406@draigBrady.com> In-Reply-To: <54FE1B61.7080406@draigBrady.com> OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="eC4GE7nEUDiAcn4uQo3r4hEEUM60hH65P" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: 20029 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --eC4GE7nEUDiAcn4uQo3r4hEEUM60hH65P Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 03/09/2015 04:14 PM, P=C3=A1draig Brady wrote: >>> + >>> +for size in 1 4095 4096 8191 8192 16383 16384; do >> >> Should you also test 4097 8193 and 16385 (that is, a likely >> 1-more-than-BUFSIZ in the mix)? >=20 > The 1 more is implicit with the \n added by yes(1). Then you should also test 4094 8190 and 16382, to make sure that a multiple of a non-aligned subset with a small unused tail also works (although I guess the 4097 tests the case of an unused tail if BUFSIZ is 8192, although in that case the unused tail is much larger than a few bytes). --=20 Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --eC4GE7nEUDiAcn4uQo3r4hEEUM60hH65P Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCAAGBQJU/igpAAoJEKeha0olJ0NqGpQH/i+A+BJUN2Kpqaq3EKtb4R+3 GDkOvVMXIVHcSsVxfKwmuPemFaSXaEv53sNybp47AIEQ1W09svRxqLUAGUNFxz5L cCYqswlFd1gJWoxW2ltSGFFiSPS6V1RZk1wQGf4wRlGGhrFkk6x13FVweGw1ZFrR X1yZizy8nxLmAbvkhgM7fqCK7rjokR9MkQP6LrjO3pHYyidPTRasvbFuDRVHQBNw weeL0IqqiC0psgAwWpxpyTA0uP4HP3fVdH+Hhpz5uujnlnjF62ism67awcAccU+y XsbosA9XYLjP/gJ1o+yE/MK53SR2lp6wXeo/u0h0LSYyLLV33IqyJMNA3BA9jYI= =PUQl -----END PGP SIGNATURE----- --eC4GE7nEUDiAcn4uQo3r4hEEUM60hH65P-- From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 09 20:31:46 2015 Received: (at 20029) by debbugs.gnu.org; 10 Mar 2015 00:31:46 +0000 Received: from localhost ([127.0.0.1]:41036 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV85J-0003oS-CK for submit@debbugs.gnu.org; Mon, 09 Mar 2015 20:31:46 -0400 Received: from mx1.redhat.com ([209.132.183.28]:50250) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV85F-0003mF-Tx for 20029@debbugs.gnu.org; Mon, 09 Mar 2015 20:31:43 -0400 Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id t2A0Vefq001384 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 9 Mar 2015 20:31:40 -0400 Received: from foobarbaz (vpn1-4-214.ams2.redhat.com [10.36.4.214]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t2A0VbCx001182 (version=TLSv1/SSLv3 cipher=AES128-GCM-SHA256 bits=128 verify=NO); Mon, 9 Mar 2015 20:31:39 -0400 From: Giuseppe Scrivano To: =?utf-8?Q?P=C3=A1draig?= Brady Subject: Re: bug#20029: 'yes' surprisingly slow References: <54FAEABF.1090104@draigBrady.com> <54FDF8DF.7010109@draigBrady.com> Date: Tue, 10 Mar 2015 01:31:36 +0100 In-Reply-To: <54FDF8DF.7010109@draigBrady.com> (=?utf-8?Q?=22P=C3=A1draig?= Brady"'s message of "Mon, 09 Mar 2015 19:47:43 +0000") Message-ID: <87pp8hhevr.fsf@redhat.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.4 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: 20029 Cc: 20029@debbugs.gnu.org, Ole Tange X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.0 (----) P=C3=A1draig Brady writes: > The attached should make things more efficient here. > > thanks, > P=C3=A1draig. > > > From 7959bbf19307705e98f08cfa32a9dcf67672590c Mon Sep 17 00:00:00 2001 > From: =3D?UTF-8?q?P=3DC3=3DA1draig=3D20Brady?=3D > Date: Mon, 9 Mar 2015 19:27:32 +0000 > Subject: [PATCH] yes: output data more efficiently > > yes(1) may be used to generate repeating patterns of text > for test inputs etc., so adjust to be more efficient. > > Profiling the case where yes(1) is outputting small items > through stdio (which was the default case), shows the overhead > continuously processing small items in main() and in stdio: > > $ yes >/dev/null & perf top -p $! > 31.02% yes [.] main > 27.36% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5 > 14.51% libc-2.20.so [.] fputs_unlocked > 13.50% libc-2.20.so [.] strlen > 10.66% libc-2.20.so [.] __GI___mempcpy > 1.98% yes [.] fputs_unlocked@plta > > Sending more data per stdio call improves the situation, > but still, there is significant stdio overhead due to memory copies, > and the repeated string length checking: > > $ yes "`echo {1..1000}`" >/dev/null & perf top -p $! > 42.26% libc-2.20.so [.] __GI___mempcpy > 17.38% libc-2.20.so [.] strlen > 5.21% [kernel] [k] __srcu_read_lock > 4.58% [kernel] [k] __srcu_read_unlock > 4.27% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5 > 2.50% libc-2.20.so [.] __GI___libc_write > 2.45% [kernel] [k] system_call > 2.40% [kernel] [k] system_call_after_swapgs > 2.27% [kernel] [k] vfs_write > 2.09% libc-2.20.so [.] _IO_do_write@@GLIBC_2.2.5 > 2.01% [kernel] [k] fsnotify > 1.95% libc-2.20.so [.] _IO_file_write@@GLIBC_2.2.5 > 1.44% yes [.] main > > We can avoid all stdio overhead by building up the buffer > _once_ and outputting that, and the profile below shows > the bottleneck moved to the kernel: > > $ src/yes >/dev/null & perf top -p $! > 15.42% [kernel] [k] __srcu_read_lock > 12.98% [kernel] [k] __srcu_read_unlock > 9.41% libc-2.20.so [.] __GI___libc_write > 9.11% [kernel] [k] vfs_write > 8.35% [kernel] [k] fsnotify > 8.02% [kernel] [k] system_call > 5.84% [kernel] [k] system_call_after_swapgs > 4.54% [kernel] [k] __fget_light > 3.98% [kernel] [k] sys_write > 3.65% [kernel] [k] selinux_file_permission > 3.44% [kernel] [k] rw_verify_area > 2.94% [kernel] [k] __fsnotify_parent > 2.76% [kernel] [k] security_file_permission > 2.39% yes [.] main > 2.17% [kernel] [k] __fdget_pos > 2.13% [kernel] [k] sysret_check > 0.81% [kernel] [k] write_null > 0.36% yes [.] write@plt > > Note this change also ensures that yes will only write complete lines > for lines softer than BUFSIZ. > > * src/yes.c (main): Build up a BUFSIZ buffer of lines, > and output that, rather than having stdio process each item. > * tests/misc/yes.sh: Add a new test for various buffer sizes. > * tests/local.mk: Reference the new test. > Fixes http://bugs.gnu.org/20029 > --- > src/yes.c | 43 +++++++++++++++++++++++++++++++++++++++++-- > tests/local.mk | 1 + > tests/misc/yes.sh | 28 ++++++++++++++++++++++++++++ > 3 files changed, 70 insertions(+), 2 deletions(-) > create mode 100755 tests/misc/yes.sh > > diff --git a/src/yes.c b/src/yes.c > index b35b13f..91dea11 100644 > --- a/src/yes.c > +++ b/src/yes.c > @@ -58,6 +58,10 @@ Repeatedly output a line with all specified STRING(s),= or 'y'.\n\ > int > main (int argc, char **argv) > { > + char buf[BUFSIZ]; > + char *pbuf =3D buf; > + int i; > + > initialize_main (&argc, &argv); > set_program_name (argv[0]); > setlocale (LC_ALL, ""); > @@ -77,9 +81,44 @@ main (int argc, char **argv) > argv[argc++] =3D bad_cast ("y"); > } >=20=20 > - while (true) > + /* Buffer data locally once, rather than having the > + large overhead of stdio buffering each item. */ > + for (i =3D optind; i < argc; i++) > + { > + size_t len =3D strlen (argv[i]); > + if (BUFSIZ < len || BUFSIZ - len <=3D pbuf - buf) > + break; > + memcpy (pbuf, argv[i], len); > + pbuf +=3D len; > + *pbuf++ =3D i =3D=3D argc - 1 ? '\n' : ' '; > + } > + if (i < argc) > + pbuf =3D NULL; since the buffer is partly filled, wouldn't be better to reuse it? Something like this (barely tested): diff --git a/src/yes.c b/src/yes.c index 91dea11..ac690ce 100644 --- a/src/yes.c +++ b/src/yes.c @@ -92,9 +92,7 @@ main (int argc, char **argv) pbuf +=3D len; *pbuf++ =3D i =3D=3D argc - 1 ? '\n' : ' '; } - if (i < argc) - pbuf =3D NULL; - else + if (i =3D=3D argc) { size_t line_len =3D pbuf - buf; size_t lines =3D BUFSIZ / line_len; @@ -106,7 +104,7 @@ main (int argc, char **argv) } =20 /* The normal case is to continuously output the local buffer. */ - while (pbuf) + while (i =3D=3D argc) { if (write (STDOUT_FILENO, buf, pbuf - buf) =3D=3D -1) { @@ -117,11 +115,17 @@ main (int argc, char **argv) =20 /* If the data doesn't fit in BUFSIZ then it's large and not too onerous to output using stdio in any case. */ - while (! pbuf) + while (i !=3D argc) { - for (i =3D optind; i < argc; i++) - if (fputs (argv[i], stdout) =3D=3D EOF - || putchar (i =3D=3D argc - 1 ? '\n' : ' ') =3D=3D EOF) + int j; + if ((pbuf - buf) && fwrite (buf, pbuf - buf, 1, stdout) =3D=3D 0) + { + error (0, errno, _("standard output")); + return EXIT_FAILURE; + } + for (j =3D i; j < argc; j++) + if (fputs (argv[j], stdout) =3D=3D EOF + || putchar (j =3D=3D argc - 1 ? '\n' : ' ') =3D=3D EOF) { error (0, errno, _("standard output")); return EXIT_FAILURE; The difference seems to be (without the patch): $ time src/yes `echo {1..2000}` | head -c 2000M | md5sum 55c293324aa6ecce14f0bf30da5a4686 - real 0m7.994s user 0m11.093s sys 0m2.953s versus (with the patch): $ time src/yes `echo {1..2000}` | head -c 2000M | md5sum 55c293324aa6ecce14f0bf30da5a4686 - real 0m3.534s user 0m4.164s sys 0m1.803s Regards, Giuseppe From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 09 20:50:52 2015 Received: (at 20029-done) by debbugs.gnu.org; 10 Mar 2015 00:50:53 +0000 Received: from localhost ([127.0.0.1]:41048 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV8No-0006GF-2I for submit@debbugs.gnu.org; Mon, 09 Mar 2015 20:50:52 -0400 Received: from mail2.vodafone.ie ([213.233.128.44]:23840) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YV8Nk-0006G0-Ue for 20029-done@debbugs.gnu.org; Mon, 09 Mar 2015 20:50:50 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvgKABI//lRtTRx//2dsb2JhbABcgwZSVQWDCq0ukjiFcQECAoEsTQEBAQEBAXyEDwEBAQMBIw8BRhALDQEDAwECAQICBRYLAgIJAwIBAgE9CAYNAQUCAQGIIwwBA6YPhWmWFQEBAQEBAQQBAQEBAQEBARqBIYl2hG4HgmiBRQWTc4dChTiGWoYPI4IDG4FQPjGCQwEBAQ Received: from unknown (HELO localhost.localdomain) ([109.77.28.127]) by mail2.vodafone.ie with ESMTP; 10 Mar 2015 00:50:42 +0000 Message-ID: <54FE3FE1.2010100@draigBrady.com> Date: Tue, 10 Mar 2015 00:50:41 +0000 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Giuseppe Scrivano Subject: Re: bug#20029: 'yes' surprisingly slow References: <54FAEABF.1090104@draigBrady.com> <54FDF8DF.7010109@draigBrady.com> <87pp8hhevr.fsf@redhat.com> In-Reply-To: <87pp8hhevr.fsf@redhat.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 20029-done Cc: 20029-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 10/03/15 00:31, Giuseppe Scrivano wrote: > Pádraig Brady writes: > >> The attached should make things more efficient here. >> >> thanks, >> Pádraig. >> >> >> From 7959bbf19307705e98f08cfa32a9dcf67672590c Mon Sep 17 00:00:00 2001 >> From: =?UTF-8?q?P=C3=A1draig=20Brady?= >> Date: Mon, 9 Mar 2015 19:27:32 +0000 >> Subject: [PATCH] yes: output data more efficiently >> >> yes(1) may be used to generate repeating patterns of text >> for test inputs etc., so adjust to be more efficient. >> >> Profiling the case where yes(1) is outputting small items >> through stdio (which was the default case), shows the overhead >> continuously processing small items in main() and in stdio: >> >> $ yes >/dev/null & perf top -p $! >> 31.02% yes [.] main >> 27.36% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5 >> 14.51% libc-2.20.so [.] fputs_unlocked >> 13.50% libc-2.20.so [.] strlen >> 10.66% libc-2.20.so [.] __GI___mempcpy >> 1.98% yes [.] fputs_unlocked@plta >> >> Sending more data per stdio call improves the situation, >> but still, there is significant stdio overhead due to memory copies, >> and the repeated string length checking: >> >> $ yes "`echo {1..1000}`" >/dev/null & perf top -p $! >> 42.26% libc-2.20.so [.] __GI___mempcpy >> 17.38% libc-2.20.so [.] strlen >> 5.21% [kernel] [k] __srcu_read_lock >> 4.58% [kernel] [k] __srcu_read_unlock >> 4.27% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5 >> 2.50% libc-2.20.so [.] __GI___libc_write >> 2.45% [kernel] [k] system_call >> 2.40% [kernel] [k] system_call_after_swapgs >> 2.27% [kernel] [k] vfs_write >> 2.09% libc-2.20.so [.] _IO_do_write@@GLIBC_2.2.5 >> 2.01% [kernel] [k] fsnotify >> 1.95% libc-2.20.so [.] _IO_file_write@@GLIBC_2.2.5 >> 1.44% yes [.] main >> >> We can avoid all stdio overhead by building up the buffer >> _once_ and outputting that, and the profile below shows >> the bottleneck moved to the kernel: >> >> $ src/yes >/dev/null & perf top -p $! >> 15.42% [kernel] [k] __srcu_read_lock >> 12.98% [kernel] [k] __srcu_read_unlock >> 9.41% libc-2.20.so [.] __GI___libc_write >> 9.11% [kernel] [k] vfs_write >> 8.35% [kernel] [k] fsnotify >> 8.02% [kernel] [k] system_call >> 5.84% [kernel] [k] system_call_after_swapgs >> 4.54% [kernel] [k] __fget_light >> 3.98% [kernel] [k] sys_write >> 3.65% [kernel] [k] selinux_file_permission >> 3.44% [kernel] [k] rw_verify_area >> 2.94% [kernel] [k] __fsnotify_parent >> 2.76% [kernel] [k] security_file_permission >> 2.39% yes [.] main >> 2.17% [kernel] [k] __fdget_pos >> 2.13% [kernel] [k] sysret_check >> 0.81% [kernel] [k] write_null >> 0.36% yes [.] write@plt >> >> Note this change also ensures that yes will only write complete lines >> for lines softer than BUFSIZ. >> >> * src/yes.c (main): Build up a BUFSIZ buffer of lines, >> and output that, rather than having stdio process each item. >> * tests/misc/yes.sh: Add a new test for various buffer sizes. >> * tests/local.mk: Reference the new test. >> Fixes http://bugs.gnu.org/20029 >> --- >> src/yes.c | 43 +++++++++++++++++++++++++++++++++++++++++-- >> tests/local.mk | 1 + >> tests/misc/yes.sh | 28 ++++++++++++++++++++++++++++ >> 3 files changed, 70 insertions(+), 2 deletions(-) >> create mode 100755 tests/misc/yes.sh >> >> diff --git a/src/yes.c b/src/yes.c >> index b35b13f..91dea11 100644 >> --- a/src/yes.c >> +++ b/src/yes.c >> @@ -58,6 +58,10 @@ Repeatedly output a line with all specified STRING(s), or 'y'.\n\ >> int >> main (int argc, char **argv) >> { >> + char buf[BUFSIZ]; >> + char *pbuf = buf; >> + int i; >> + >> initialize_main (&argc, &argv); >> set_program_name (argv[0]); >> setlocale (LC_ALL, ""); >> @@ -77,9 +81,44 @@ main (int argc, char **argv) >> argv[argc++] = bad_cast ("y"); >> } >> >> - while (true) >> + /* Buffer data locally once, rather than having the >> + large overhead of stdio buffering each item. */ >> + for (i = optind; i < argc; i++) >> + { >> + size_t len = strlen (argv[i]); >> + if (BUFSIZ < len || BUFSIZ - len <= pbuf - buf) >> + break; >> + memcpy (pbuf, argv[i], len); >> + pbuf += len; >> + *pbuf++ = i == argc - 1 ? '\n' : ' '; >> + } >> + if (i < argc) >> + pbuf = NULL; > > since the buffer is partly filled, wouldn't be better to reuse it? > > Something like this (barely tested): > > diff --git a/src/yes.c b/src/yes.c > index 91dea11..ac690ce 100644 > --- a/src/yes.c > +++ b/src/yes.c > @@ -92,9 +92,7 @@ main (int argc, char **argv) > pbuf += len; > *pbuf++ = i == argc - 1 ? '\n' : ' '; > } > - if (i < argc) > - pbuf = NULL; > - else > + if (i == argc) > { > size_t line_len = pbuf - buf; > size_t lines = BUFSIZ / line_len; > @@ -106,7 +104,7 @@ main (int argc, char **argv) > } > > /* The normal case is to continuously output the local buffer. */ > - while (pbuf) > + while (i == argc) > { > if (write (STDOUT_FILENO, buf, pbuf - buf) == -1) > { > @@ -117,11 +115,17 @@ main (int argc, char **argv) > > /* If the data doesn't fit in BUFSIZ then it's large > and not too onerous to output using stdio in any case. */ > - while (! pbuf) > + while (i != argc) > { > - for (i = optind; i < argc; i++) > - if (fputs (argv[i], stdout) == EOF > - || putchar (i == argc - 1 ? '\n' : ' ') == EOF) > + int j; > + if ((pbuf - buf) && fwrite (buf, pbuf - buf, 1, stdout) == 0) > + { > + error (0, errno, _("standard output")); > + return EXIT_FAILURE; > + } > + for (j = i; j < argc; j++) > + if (fputs (argv[j], stdout) == EOF > + || putchar (j == argc - 1 ? '\n' : ' ') == EOF) > { > error (0, errno, _("standard output")); > return EXIT_FAILURE; > > > The difference seems to be (without the patch): > > $ time src/yes `echo {1..2000}` | head -c 2000M | md5sum > 55c293324aa6ecce14f0bf30da5a4686 - > > real 0m7.994s > user 0m11.093s > sys 0m2.953s > > versus (with the patch): > > $ time src/yes `echo {1..2000}` | head -c 2000M | md5sum > 55c293324aa6ecce14f0bf30da5a4686 - > > real 0m3.534s > user 0m4.164s > sys 0m1.803s A fair point for the many small arguments case, though doesn't help the few large arguments case. Given this is only slightly more complicated, we might as well do this also. thanks, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 10 17:18:52 2015 Received: (at 20029) by debbugs.gnu.org; 10 Mar 2015 21:18:52 +0000 Received: from localhost ([127.0.0.1]:41973 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YVRYB-0003F1-QW for submit@debbugs.gnu.org; Tue, 10 Mar 2015 17:18:52 -0400 Received: from mail-ig0-f171.google.com ([209.85.213.171]:34486) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YVRY9-0003El-Hl for 20029@debbugs.gnu.org; Tue, 10 Mar 2015 17:18:50 -0400 Received: by igal13 with SMTP id l13so32882298iga.1 for <20029@debbugs.gnu.org>; Tue, 10 Mar 2015 14:18:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=r9DKLPLKLb+i8AGQ3GHJNZOiyLtpeCk8wilrc0emqIw=; b=sfa0JGpkobjf985oB4h88/lJ6/e5Z/Rx7AU3awyfcgmaBRfOht5wgFcBACXqStcipe kqGnIyqFP8wFIoEW5D9nbpYybZMDiVx5pkbnWKYFu9jmTmCh0zpDWYFcL2ZLBosausXv YIxWxO7ZiE+WFNfhWPRRPnqZ9sly+gNA7bSGI9JA7IKpTE/+mivc6YE/Lp2HyWipdN/N mnnUIFwMCTKwGIh/6DicKag/QpA9l7prGTcwxxNCv9wIx5th6dIrNFn5FVhVAE6jn+0t H/lOYNSMAHZkBXJ7ZawafS5TEpVfrtXfdMwgpB7YMG/9b24K438W93gTf2mKiOu6ATcB hJ2w== X-Received: by 10.107.150.149 with SMTP id y143mr1342864iod.22.1426022324023; Tue, 10 Mar 2015 14:18:44 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.15.202 with HTTP; Tue, 10 Mar 2015 14:18:23 -0700 (PDT) In-Reply-To: <87pp8hhevr.fsf@redhat.com> References: <54FAEABF.1090104@draigBrady.com> <54FDF8DF.7010109@draigBrady.com> <87pp8hhevr.fsf@redhat.com> From: Ole Tange Date: Tue, 10 Mar 2015 22:18:23 +0100 X-Google-Sender-Auth: jmxRxL-2h3RVBdWDj9xhFQc0904 Message-ID: Subject: Re: bug#20029: 'yes' surprisingly slow To: Giuseppe Scrivano Content-Type: text/plain; charset=UTF-8 X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 20029 Cc: 20029@debbugs.gnu.org, =?UTF-8?Q?P=C3=A1draig_Brady?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) On Tue, Mar 10, 2015 at 1:31 AM, Giuseppe Scrivano wrote: > $ time src/yes `echo {1..2000}` | head -c 2000M | md5sum > 55c293324aa6ecce14f0bf30da5a4686 - > > real 0m7.994s > user 0m11.093s > sys 0m2.953s > > versus (with the patch): > > $ time src/yes `echo {1..2000}` | head -c 2000M | md5sum > 55c293324aa6ecce14f0bf30da5a4686 - > > real 0m3.534s > user 0m4.164s > sys 0m1.803s Are you sure your are not limited by md5sum? $ time yes "`echo {1..2000}`" | head -c 2000M >/dev/null real 0m0.660s user 0m0.180s sys 0m1.115s The solution should perform no worse than that. /Ole From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 10 19:59:27 2015 Received: (at 20029) by debbugs.gnu.org; 10 Mar 2015 23:59:28 +0000 Received: from localhost ([127.0.0.1]:42038 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YVU3a-0000kq-FP for submit@debbugs.gnu.org; Tue, 10 Mar 2015 19:59:27 -0400 Received: from mail2.vodafone.ie ([213.233.128.44]:24530) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YVU3X-0000kX-Td for 20029@debbugs.gnu.org; Tue, 10 Mar 2015 19:59:24 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AkUFAPmD/1RtTfQJ/2dsb2JhbABcgwaBLIEpgRNOrWiYMAKBNU0BAQEBAQF8hBABBSMPAUYQCw0LAgIFFgsCAgkDAgECAUUGAQwBBQIBAReIGAGqWoVplVMBAQEBBgEBAQEBHYEhiXaEFlgHgmiBRQWbDoVxiSmDQyOCAxuBUD4xgQKBQQEBAQ Received: from unknown (HELO localhost.localdomain) ([109.77.244.9]) by mail2.vodafone.ie with ESMTP; 10 Mar 2015 23:59:16 +0000 Message-ID: <54FF8554.5070009@draigBrady.com> Date: Tue, 10 Mar 2015 23:59:16 +0000 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Ole Tange , Giuseppe Scrivano Subject: Re: bug#20029: 'yes' surprisingly slow References: <54FAEABF.1090104@draigBrady.com> <54FDF8DF.7010109@draigBrady.com> <87pp8hhevr.fsf@redhat.com> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 20029 Cc: 20029@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 10/03/15 21:18, Ole Tange wrote: > On Tue, Mar 10, 2015 at 1:31 AM, Giuseppe Scrivano wrote: > >> $ time src/yes `echo {1..2000}` | head -c 2000M | md5sum >> 55c293324aa6ecce14f0bf30da5a4686 - >> >> real 0m7.994s >> user 0m11.093s >> sys 0m2.953s >> >> versus (with the patch): >> >> $ time src/yes `echo {1..2000}` | head -c 2000M | md5sum >> 55c293324aa6ecce14f0bf30da5a4686 - >> >> real 0m3.534s >> user 0m4.164s >> sys 0m1.803s > > Are you sure your are not limited by md5sum? > > $ time yes "`echo {1..2000}`" | head -c 2000M >/dev/null > > real 0m0.660s > user 0m0.180s > sys 0m1.115s > > The solution should perform no worse than that. Two separate cases. Yours above is a single large argument. Giuseppe's example above, and your original problematic example were without quotes, and so with many arguments passed to yes. I've applied Giuseppe's patch with an augmented test. cheers, Pádraig. From unknown Wed Jun 25 00:20:36 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Wed, 08 Apr 2015 11:24:03 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator