From debbugs-submit-bounces@debbugs.gnu.org Wed Oct 09 19:20:04 2013 Received: (at submit) by debbugs.gnu.org; 9 Oct 2013 23:20:04 +0000 Received: from localhost ([127.0.0.1]:38804 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VU32w-0005X8-K4 for submit@debbugs.gnu.org; Wed, 09 Oct 2013 19:20:03 -0400 Received: from eggs.gnu.org ([208.118.235.92]:53632) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VU319-0005U8-Iw for submit@debbugs.gnu.org; Wed, 09 Oct 2013 19:18:11 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1VU317-0000HG-Im for submit@debbugs.gnu.org; Wed, 09 Oct 2013 19:18:11 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, HTML_MESSAGE,T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:46143) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VU317-0000HA-Fz for submit@debbugs.gnu.org; Wed, 09 Oct 2013 19:18:09 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55351) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VU316-0007VE-7J for bug-coreutils@gnu.org; Wed, 09 Oct 2013 19:18:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1VU315-0000Gv-3V for bug-coreutils@gnu.org; Wed, 09 Oct 2013 19:18:08 -0400 Received: from mail-qc0-x22f.google.com ([2607:f8b0:400d:c01::22f]:37734) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VU314-0000Gm-V0 for bug-coreutils@gnu.org; Wed, 09 Oct 2013 19:18:07 -0400 Received: by mail-qc0-f175.google.com with SMTP id v2so1151577qcr.34 for ; Wed, 09 Oct 2013 16:18:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=XxRWpeFfhuvnUlOIcWV/VrEBx4RQgtnhJa9qRSYHuyY=; b=KHXawKGAIcHUzsobDXdreyYedkrrduyOdy3cCy1EM5S8RztT+zAGheeeoI8g8jeUGO YMF3ctuWf/Oa0KmX4ANuE0wNsaxdvDfrmJq28S+dyzrfhB+UPwViJOPbq6nVzLBOPyEn FL5uhardome260heHXwflR0Amed3SsVC4otsVDg9/geS3wWoB3XH6NOTB/+Jxzf1DciJ YJwabfEQZlfMUEDGVh0ZODFBDwS1KDwM6tL7j9rU+kAOeF6Zuo18fCuGq5/snyPkURYu CCsrEqUs7sEYJUJWO/H4K+8M5uZkqFv3ibGNJmz4u4mKTPXnCHbeIXFVu98b7W6MdioC LECQ== MIME-Version: 1.0 X-Received: by 10.229.192.65 with SMTP id dp1mr2577278qcb.19.1381360685853; Wed, 09 Oct 2013 16:18:05 -0700 (PDT) Received: by 10.224.73.136 with HTTP; Wed, 9 Oct 2013 16:18:05 -0700 (PDT) Date: Wed, 9 Oct 2013 16:18:05 -0700 Message-ID: Subject: Parameter -d or --direct to open files with flag O_DIRECT? From: Kyle Sallee To: bug-coreutils@gnu.org Content-Type: multipart/mixed; boundary=001a11339846fcb33b04e857185b X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Wed, 09 Oct 2013 19:20:00 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.0 (----) --001a11339846fcb33b04e857185b Content-Type: multipart/alternative; boundary=001a11339846fcb33704e8571859 --001a11339846fcb33704e8571859 Content-Type: text/plain; charset=UTF-8 Please consider the attached gzipped patch file. It provides additional parameters for cat. The additional parameter allows cat to open files and read content without storing the content in the kernel's block cache. --001a11339846fcb33704e8571859 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Please consider the attached gzipped patch file.=
It provides additional parameters for cat.
The additional para= meter allows cat to open files and read content
without storing the cont= ent in the kernel's block cache.
--001a11339846fcb33704e8571859-- --001a11339846fcb33b04e857185b Content-Type: application/x-gzip; name="coreutils-8.21-O_DIRECT-cat.patch.gz" Content-Disposition: attachment; filename="coreutils-8.21-O_DIRECT-cat.patch.gz" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hml6cuct0 H4sICDnjVVIAA2NvcmV1dGlscy04LjIxLU9fRElSRUNULWNhdC5wYXRjaAClUstu2zAQPFdfMTUK xIlEW34IfgQGHLQ+FAjaonBuAQRKWttCJNIRKRup238vKT/kPtwW6B542N0Z7sxuki4WYAXYQxex LKjUaabYsNXttFURt2OuW/HFgsMYu1h81fU7PeZ3WM9HZzDuB+Ng1PKPAeYPfN9xXff/CKZTsFHg 9eGaN8B06gBgdx4YUyu5ZTzLUAc9l+mGZyQ0tATbzOaP4rFCRBYhyjyiggkpooyLJ4vYp2BSlK/1 C2Sp16VGlgpSHuSGiiJNSIEJy+QapsQyJWlBscZ5yDUJLNLMNG9TvcIi40t8DN+9/zx7O8dxDsLv 45fJj4DZSSqJRNWAJFXrjL/gDbiGKUEuQDxeVaNbsHUu6A+tdUF/dPJu51TwXeMnMxqeMSHkxbLM zRAePjzc33u4iq6+eVa2RexFX2hMbOMP1BcaxVmjei6JvhD70wTK9ldiBj2jwg0G9SG0b/CJF4rM jeU5NyZY8WYTOpVCtYCbtgOHAduV2QuazRgTLEmbhjCTYomm+S32YN6Nh0ZEQulyczebm1FsPTww 7We5vrZG/CNV8leuepWvJ2CdQ2K3lzrsVXsbBt7wIPUYUUH86dbIco+ZmBsDjP/jOlXdYWgvMsxl Qvg6OV3i7VnXgcut+fdcZLjOvrTXF1bXN4EuSrp1vgM3fNk3VgQAAA== --001a11339846fcb33b04e857185b-- From debbugs-submit-bounces@debbugs.gnu.org Wed Oct 09 23:08:36 2013 Received: (at 15578) by debbugs.gnu.org; 10 Oct 2013 03:08:36 +0000 Received: from localhost ([127.0.0.1]:39212 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VU6c7-0004pN-8S for submit@debbugs.gnu.org; Wed, 09 Oct 2013 23:08:35 -0400 Received: from mail5.vodafone.ie ([213.233.128.176]:11618) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VU6c4-0004pE-Kq for 15578@debbugs.gnu.org; Wed, 09 Oct 2013 23:08:33 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApQBAGUZVlJtT4U6/2dsb2JhbAANTYc5unWCfIEzgxkBAQEEIw8BRhALDQsCAgUUAgsCAgkDAgECAUUGDQEHAQGuHnWSO4EpjhwHgmqBOQOeYYNgimg Received: from unknown (HELO [192.168.1.79]) ([109.79.133.58]) by mail3.vodafone.ie with ESMTP; 10 Oct 2013 04:08:31 +0100 Message-ID: <52561A2E.5010101@draigBrady.com> Date: Thu, 10 Oct 2013 04:08:30 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: Kyle Sallee Subject: Re: bug#15578: Parameter -d or --direct to open files with flag O_DIRECT? References: In-Reply-To: X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 15578 Cc: 15578@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 10/10/2013 12:18 AM, Kyle Sallee wrote: > Please consider the attached gzipped patch file. > It provides additional parameters for cat. > The additional parameter allows cat to open files and read content > without storing the content in the kernel's block cache. > dd supports this already and is more sorted to lower level setting like this. Could you describe _why_ you would like this in cat particularly. thanks, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Wed Oct 09 23:11:54 2013 Received: (at control) by debbugs.gnu.org; 10 Oct 2013 03:11:54 +0000 Received: from localhost ([127.0.0.1]:39226 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VU6fJ-0004uZ-Ux for submit@debbugs.gnu.org; Wed, 09 Oct 2013 23:11:54 -0400 Received: from mx1.redhat.com ([209.132.183.28]:58195) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VU6fG-0004uL-Lj; Wed, 09 Oct 2013 23:11:52 -0400 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id r9A3BnFI029760 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 9 Oct 2013 23:11:49 -0400 Received: from [10.3.113.34] (ovpn-113-34.phx2.redhat.com [10.3.113.34]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id r9A3BnbG029088; Wed, 9 Oct 2013 23:11:49 -0400 Message-ID: <52561AF4.1050805@redhat.com> Date: Wed, 09 Oct 2013 21:11:48 -0600 From: Eric Blake Organization: Red Hat, Inc. User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130923 Thunderbird/17.0.9 MIME-Version: 1.0 To: Kyle Sallee Subject: Re: bug#15578: Parameter -d or --direct to open files with flag O_DIRECT? References: In-Reply-To: X-Enigmail-Version: 1.5.2 OpenPGP: url=http://people.redhat.com/eblake/eblake.gpg Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="Ap3tDlhXMInfWDFSPjPecTv6JIUabR4JV" X-Scanned-By: MIMEDefang 2.68 on 10.5.11.24 X-Spam-Score: -5.2 (-----) X-Debbugs-Envelope-To: control Cc: 15578-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.2 (-----) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --Ap3tDlhXMInfWDFSPjPecTv6JIUabR4JV Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable tag 15578 notabug thanks On 10/09/2013 05:18 PM, Kyle Sallee wrote: > Please consider the attached gzipped patch file. Thanks for the patch. Unless the patch is huger than 100k, it's best to post it inline rather than forcing readers to go through extra hoops to unpack it. > It provides additional parameters for cat. > The additional parameter allows cat to open files and read content > without storing the content in the kernel's block cache. However, adding an option to 'cat' is probably not the best option. We already have 'dd' that can request O_DIRECT on operating systems where that is defined, so you are probably better off converting your scripts to use dd than to try and retrofit cat and wait for the patches to percolate into your distro. Look for 'dd iflag=3Ddirect'. I'm closing this bug as we probably aren't going to change cat, but do feel free to add additional comments on this thread as appropriate. Who knows - maybe it's worth reopening and repurposing this bug report into a documentation patch. --=20 Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --Ap3tDlhXMInfWDFSPjPecTv6JIUabR4JV Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCAAGBQJSVhr0AAoJEKeha0olJ0Nq7OQH/1ox69UK9n3A4dGcpuVCaCAE QrBTxdGhPwxHVMKbGv2xAtlQWDM1cLehRb/0nhzTXK0T08fAVaz+oZFPObQMVlE8 0tZJEN6i8+4qYlOJna5Jdyz4GTqlfGxDuTNolMOjVch22LDx+OVb1Appw+6qSmMH 0RhYqG/D/tplPdC8UpE6khrPs59uqxZqCUBY9Zj+bAHAwP+Nd94IRajfPjguZRse pEkMTtEi1ws9KETAfEay4hn4N2t345GliMMqrGGeR8UF3gXBCbME7gvABXpYpfRy Xt4eLcNYvUf3+FT5pgk4qprD/jot0kGL0eSdLZ3/C1SQutlHv7fOrNPqpGipOzE= =O5Ol -----END PGP SIGNATURE----- --Ap3tDlhXMInfWDFSPjPecTv6JIUabR4JV-- From debbugs-submit-bounces@debbugs.gnu.org Thu Oct 10 00:49:25 2013 Received: (at 15578-done) by debbugs.gnu.org; 10 Oct 2013 04:49:25 +0000 Received: from localhost ([127.0.0.1]:39346 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VU8Bg-0008JJ-UI for submit@debbugs.gnu.org; Thu, 10 Oct 2013 00:49:25 -0400 Received: from mail-qc0-f170.google.com ([209.85.216.170]:59111) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VU8Be-0008JA-Dd for 15578-done@debbugs.gnu.org; Thu, 10 Oct 2013 00:49:22 -0400 Received: by mail-qc0-f170.google.com with SMTP id m20so1358802qcx.15 for <15578-done@debbugs.gnu.org>; Wed, 09 Oct 2013 21:49:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=PmS7/TSkUFD6I4Ikr12lw0cig3qwKp/Hk6++MoNoSWM=; b=mTK6R6v/hWoYJBrWat9hGYWcaIeoPK1dwYRqu68wFmRzDFHAtKVQNL9/l5H0T9r923 JBdiNaWfOBmWb27ALZCilfNFy4/9udBX1M19qhn9v6T+kf88cMx1bhyoehNxHwAuSYaD kWk1bDHZhM60ODqzAC/uX8Sh7Vd76KIJ+z0Y8gsAwwpcn0mfFDIj5BKBGYMWE46WEMkQ clYXz7IlhjGsY/MYGY5Lm8SF84/rCqL9BBgbl+EAPr4FUI0y/ciBd0+yS+OHplY6IROH d1ssNDZO6//XZsD1IH/CXgbdbMvSi1+LVktK1KzaAo5EIybtUNNoMDYCO88pkMwg/xLb IJWg== MIME-Version: 1.0 X-Received: by 10.229.12.197 with SMTP id y5mr5631260qcy.8.1381380561895; Wed, 09 Oct 2013 21:49:21 -0700 (PDT) Received: by 10.224.73.136 with HTTP; Wed, 9 Oct 2013 21:49:21 -0700 (PDT) In-Reply-To: <52561AF4.1050805@redhat.com> References: <52561AF4.1050805@redhat.com> Date: Wed, 9 Oct 2013 21:49:21 -0700 Message-ID: Subject: Re: bug#15578: Parameter -d or --direct to open files with flag O_DIRECT? From: Kyle Sallee To: Eric Blake Content-Type: multipart/alternative; boundary=001a11340566b1071804e85bb943 X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 15578-done Cc: 15578-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --001a11340566b1071804e85bb943 Content-Type: text/plain; charset=UTF-8 Please forgive the inconvenience. Attaching a gzipped patch file ensures the integrity of patch file content. I should have presented only the idea rather than a patch for the implementation. I concur dd is already suited to the task. Please consider | sed 's/^/if=/' | xargs -r --max-lines=1 dd iflag=direct # in contrast with | xargs -r --max-lines=4096 cat -d -- Invoking dd 466059 times costs only a slight performance decrease as compared with invoking cat 114 times. However, this example probably represents rare usage for cat. Thanks for granting the time and consideration. --001a11340566b1071804e85bb943 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Please forgive the inconvenience.
Attac= hing a gzipped patch file
ensures the integrity of patch file content.
I should have presented only the idea
rather than a patch f= or the implementation.

I concur dd is already suited to the task.

Please consider

| sed 's/^/if=3D/' | xargs -r --= max-lines=3D1 dd iflag=3Ddirect=C2=A0 # in contrast with
=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 | xargs -r --max-lines=3D4096 cat -d --

Invoking dd 466059 times costs only a slight performance dec= rease
as compared with invoking cat 114 times.
However, th= is example probably represents rare usage for cat.

Thanks for granting the time and consideration.
--001a11340566b1071804e85bb943-- From debbugs-submit-bounces@debbugs.gnu.org Thu Oct 10 04:09:13 2013 Received: (at 15578) by debbugs.gnu.org; 10 Oct 2013 08:09:13 +0000 Received: from localhost ([127.0.0.1]:39599 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VUBJ1-0004uo-Qt for submit@debbugs.gnu.org; Thu, 10 Oct 2013 04:09:12 -0400 Received: from mail1.vodafone.ie ([213.233.128.43]:33413) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VUBIz-0004uf-4n for 15578@debbugs.gnu.org; Thu, 10 Oct 2013 04:09:09 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApQBAIVfVlJtT3WW/2dsb2JhbAANTIM/g3q9LkqBNYMZAQEBBCMPAUYQCw0LAgIFFgsCAgkDAgECAUUGDQEHAQEFiAmlc3WSOoEpjh4HgmqBOQOeZINgimg Received: from unknown (HELO [192.168.1.79]) ([109.79.117.150]) by mail1.vodafone.ie with ESMTP; 10 Oct 2013 09:09:07 +0100 Message-ID: <5256609F.9050101@draigBrady.com> Date: Thu, 10 Oct 2013 09:09:03 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: Kyle Sallee Subject: Re: bug#15578: Parameter -d or --direct to open files with flag O_DIRECT? References: <52561AF4.1050805@redhat.com> In-Reply-To: X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 15578 Cc: 15578@debbugs.gnu.org, Eric Blake X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 10/10/2013 05:49 AM, Kyle Sallee wrote: > Please forgive the inconvenience. > Attaching a gzipped patch file > ensures the integrity of patch file content. > I should have presented only the idea > rather than a patch for the implementation. > > I concur dd is already suited to the task. > > Please consider > > | sed 's/^/if=/' | xargs -r --max-lines=1 dd iflag=direct # in contrast > with > | xargs -r --max-lines=4096 cat -d -- > > Invoking dd 466059 times costs only a slight performance decrease > as compared with invoking cat 114 times. > However, this example probably represents rare usage for cat. > > Thanks for granting the time and consideration. Fair point, but still not worth adding to cat(1) since it's not special in this regard. Something like this might be more appropriate: https://github.com/Feh/nocache Note that doesn't avoid the page cache completely, and so may be more performant/portable than O_DIRECT. (dd has this functionality too as described at 'nocache' at: http://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html) cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Thu Oct 10 19:21:52 2013 Received: (at 15578) by debbugs.gnu.org; 10 Oct 2013 23:21:52 +0000 Received: from localhost ([127.0.0.1]:41807 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VUPYF-0006r8-HG for submit@debbugs.gnu.org; Thu, 10 Oct 2013 19:21:51 -0400 Received: from mail1.vodafone.ie ([213.233.128.43]:50166) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VUPYC-0006qy-VX for 15578@debbugs.gnu.org; Thu, 10 Oct 2013 19:21:50 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApQBAIo1V1JtT3WW/2dsb2JhbAANTIM/g3u9VUuBNoMZAQEBBCMPAUYQCw0LAgIFFgsCAgkDAgECAUUGDQEHAQEFiAmnAnWSPYEpjh4HgmqBOQOeZINgimg Received: from unknown (HELO [192.168.1.79]) ([109.79.117.150]) by mail1.vodafone.ie with ESMTP; 11 Oct 2013 00:21:47 +0100 Message-ID: <5257368A.5050804@draigBrady.com> Date: Fri, 11 Oct 2013 00:21:46 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130110 Thunderbird/17.0.2 MIME-Version: 1.0 To: Kyle Sallee Subject: Re: bug#15578: Parameter -d or --direct to open files with flag O_DIRECT? References: <52561AF4.1050805@redhat.com> <5256609F.9050101@draigBrady.com> In-Reply-To: <5256609F.9050101@draigBrady.com> X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 15578 Cc: 15578@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 10/10/2013 09:09 AM, Pádraig Brady wrote: > On 10/10/2013 05:49 AM, Kyle Sallee wrote: >> Please forgive the inconvenience. >> Attaching a gzipped patch file >> ensures the integrity of patch file content. >> I should have presented only the idea >> rather than a patch for the implementation. >> >> I concur dd is already suited to the task. >> >> Please consider >> >> | sed 's/^/if=/' | xargs -r --max-lines=1 dd iflag=direct # in contrast >> with >> | xargs -r --max-lines=4096 cat -d -- >> >> Invoking dd 466059 times costs only a slight performance decrease >> as compared with invoking cat 114 times. >> However, this example probably represents rare usage for cat. >> >> Thanks for granting the time and consideration. > > Fair point, but still not worth adding to cat(1) > since it's not special in this regard. > > Something like this might be more appropriate: > https://github.com/Feh/nocache > > Note that doesn't avoid the page cache completely, > and so may be more performant/portable than O_DIRECT. > (dd has this functionality too as described at 'nocache' at: > http://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html) One possibility worth mentioning, would be to add a files0-from=F option to dd, like du,sort,wc already have. Now those have it because they need to operate on the complete input set, for accumulation or sorting, and thus can't resort to separated runs with xargs or whatever. dd might use it as it has a very different command syntax to the standard tools. So that would allow a general method to efficiently read many files. Another related thing to consider is the above would allow a single process to handle everything, but it might be better to split the load into a process per CPU. thanks, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Sat Oct 12 16:03:04 2013 Received: (at 15578) by debbugs.gnu.org; 12 Oct 2013 20:03:04 +0000 Received: from localhost ([127.0.0.1]:44674 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VV5Ox-0002nq-El for submit@debbugs.gnu.org; Sat, 12 Oct 2013 16:03:04 -0400 Received: from mail-qe0-f42.google.com ([209.85.128.42]:61468) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1VV5Ot-0002nI-K4 for 15578@debbugs.gnu.org; Sat, 12 Oct 2013 16:03:00 -0400 Received: by mail-qe0-f42.google.com with SMTP id gc15so4218488qeb.15 for <15578@debbugs.gnu.org>; Sat, 12 Oct 2013 13:02:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Ss96vt5s6t4VTm8JaeamON4Hrj27php9Xn0MHs12xBE=; b=mM08GSloqmrem48CPjlrCjR+FvFUVA9Akmv1dDADRc18aXRa+Lu6FMfTtzXAcqlLsD N4xVVfQgwvJwOkctPrtx7WnrjRi1saRwAVn3I+4YEX4YXrnGtJIqde0HZKZOOuXmLlHM gAcfqu3RM2ktkAo2EUELYk+dCbilqHQA70VcIdUxmped84QBoeut+JYCwVHk0NUda9Lm ngOJ5e7re+++WwE5y9Id23x/KxLJOY1XTTAVVNzD5mia8LOsKypgXdpjJCAsadJjqiJv TSB4x5wAUMMoqBd+1xD3bN+c7duKJnr6XjIvLGmWQXWf5ZlIj6sb0agmqkY9Ay9yR4Wq HLSw== MIME-Version: 1.0 X-Received: by 10.49.72.201 with SMTP id f9mr16668661qev.49.1381608174104; Sat, 12 Oct 2013 13:02:54 -0700 (PDT) Received: by 10.224.73.136 with HTTP; Sat, 12 Oct 2013 13:02:53 -0700 (PDT) In-Reply-To: <5257368A.5050804@draigBrady.com> References: <52561AF4.1050805@redhat.com> <5256609F.9050101@draigBrady.com> <5257368A.5050804@draigBrady.com> Date: Sat, 12 Oct 2013 13:02:53 -0700 Message-ID: Subject: Re: bug#15578: Parameter -d or --direct to open files with flag O_DIRECT? From: Kyle Sallee To: =?UTF-8?Q?P=C3=A1draig_Brady?= Content-Type: multipart/alternative; boundary=047d7b677be46fb35704e890b860 X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 15578 Cc: 15578@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --047d7b677be46fb35704e890b860 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Thu, Oct 10, 2013 at 4:21 PM, P=C3=A1draig Brady wrot= e: > > One possibility worth mentioning, would be to add a files0-from=3DF optio= n > to dd, > like du,sort,wc already have. > Wonderful compromise. I agree. I appreciate how it already continues an established and understood convention in parameters for coreutils utilities rather than creating something new. Oddly enough I might have never used the -files0-from=3D parameter for du sort wc However, that might sometimes save on an invocation of cat within a pipe. But I am not often working with null terminated lists of files. Merely creating that list might sometimes require an invocation of "tr" to translate the linefeed to null. However, I am aware that "find" can generate lists of null terminated output. > Now those have it because they need to operate on the complete input set, > for accumulation or sorting, and thus can't resort to separated runs > with xargs or whatever. That makes sense. > dd might use it as it has a very different command > syntax to the standard tools. So that would allow a general method > to efficiently read many files. > Since dd might be the only utility from coreutils that allows specification of additional open flags then dd would have been my second choice for modification after cat. > Another related thing to consider is the above would allow a single > process to handle everything, but it might be better to split the > load into a process per CPU. > The type of processing that dd does to transliterate character sets and do conversion can be processor intensive when the amount of data being processed is immense. Therefore, the benefit of concurrent threaded processing of blocks while outputting the processed blocks linearly could allow dd to expedite processing in proportion to the amount of processors allowed. However, I anticipate problems could occur when processing around block edges. Yet individual processing of files concurrently avoids that potential problem. Perhaps concurrent processing of files with dd would most often be accomplished with xargs? That is because the output might not be desired as a single data block. Therefore, if dd allowed specification of multiple input files then it should probably also allow the specification of a list of an equal amount of output files? The cost for invoking dd through xargs to gain concurrent processing of individual files is low in comparison for the boon gained from using two or more processors concurrently. Therefore, the added complexity might not be worth the additional speed when nearly the same benefit can be gained by invoking dd by xargs -P Aside from the rejected modification to cat, and the proposed modification to dd, and writing a small C helper program; I wonder what possibilities exist that have not yet been contemplated? Perhaps a BASH plugin can be created that reads a list of file names on stdin and a list of open flags as parameters and then opens those files and writes their output to stdout? Chet might accept a patch for a such a plugin. Because BASH does not install plugins by default and lacks an inter-POSIX suggested installation location; the availability and use of the plugins go unnoticed by almost everyone who lacks specialization in shell scripting. Or more succinctly stated few people might possess an opinion concerning BASH plugins. And if the plugin is rejected then I can still regress to the idea of writing a standalone C program for the task. Although my original suggestion might have seemed otherwise, I am not in favor of rapid mutation of coreutils as opposed to carefully considered evolution. I appreciate exploring different possibilities for implementing functionality and carefully weighing the boons and the banes to arrive at a conclusion that becomes best for the long term goals of coreutils. I am 100% satisfied with coreutils. I wanted an eloquent solution for a spliter of an idea. I arrogantly and hastily expected that coreutils should immediately accommodate. Please forgive my lack of objectivity, insistence, and persistence. Please pardon the delayed response. I wanted to avoid a rapid rate of communication becoming a burden or a cause for irritation. And I hoped that a day's worth of other activities would grant opportunity for a fresh perspective upon resuming consideration. > thanks, > P=C3=A1draig. > Thanks for the continued attention. I am satisfied with the outcome. >From the conversation I gained a better understanding of how to accomplish the task. The outcome is appreciated. I previously expected that "cat" --direct would suffice. Yet now I also want to mitigate the necessity of invoking "cat" using "xargs." I should have realized that before sending the first email. coreutils should be expected to do what coreutils does, but not expected to be the most expeditious implementation. I wanted unreasonable performance. Please forgive that I created a bother and selfishly utilized other P=C3=A1draig Brady's time instead of thinking for myself. That aside I should not omit saying; thanks for maintaining and sharing coreutils. Shell scripting would be insufferable without coreutils. :) --047d7b677be46fb35704e890b860 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On T= hu, Oct 10, 2013 at 4:21 PM, P=C3=A1draig Brady <P@draigbrady.com> wrote:

One possibility worth mentioning, would be to add a files0-from= =3DF option to dd,
like du,sort,wc already have.

Wonderful= compromise.=C2=A0 I agree.
I appreciate how it already conti= nues an established and understood convention
in parameters f= or coreutils utilities rather than creating something new.

Oddly enough I might have never used the -files0-from=3D par= ameter for du sort wc
However, that might sometimes save on a= n invocation of cat within a pipe.
But I am not often working= with null terminated lists of files.
Merely creating that list might sometimes require an invocation of "tr= "
to translate the linefeed to null.
However, I am aw= are that "find" can generate lists of null terminated output.

=C2=A0
dd might use it as it has a very different command
syntax to the standard tools. So that would allow a general method
to efficiently read many files.

Since d= d might be the only utility from coreutils that allows
specification of = additional open flags
then dd would have been my second choice for modif= ication after cat.

=C2=A0

Aside from the rejected modification to cat,=
and the proposed modification to dd,
and writing a small C helper pr= ogram;
I wonder what possibilities exist
that have not yet been conte= mplated?

Perhaps a BASH plugin can be created
that read= s a list of file names on stdin
and a list of open flags as p= arameters
and then opens those files and writes their output to stdout?<= /div>
Chet might accept a patch for a such a plugin.
Because B= ASH does not install plugins by default
and lacks an inter-POSIX suggest= ed installation location;
the availability and use of the plu= gins go unnoticed
by almost everyone who lacks specialization in shell scripting.
Or more succinctly stated few people might possess
an opin= ion concerning BASH plugins.
And if the plugin is rejected then I can still regress to the idea of writing
a standalone C program f= or the task.

Although my original suggestion m= ight have seemed otherwise,
I am not in favor of rapid mutati= on of coreutils
as opposed to carefully considered evolution.
I appreciate ex= ploring different possibilities for implementing functionality
and caref= ully weighing the boons and the banes to arrive at a conclusion
that bec= omes best for the long term goals of coreutils.

I am 100% satisfied with coreutils.
I wanted a= n eloquent solution for a spliter of an idea.
I arrogantly and hastily e= xpected that coreutils
should immediately accommodate.
Please forgive my lack of objectivity, insistence, and persisten= ce.

Please pardon the delayed response.
I wanted to av= oid a rapid rate of communication
becoming a burden or a cause for irrit= ation.
And I hoped that a day's worth of other activities
would = grant opportunity for a fresh perspective
upon resuming consideration.
=C2=A0
thanks,
P=C3=A1draig.

Thanks for the cont= inued attention.
I am satisfied with the outcome.
From the conversation I gained a better understanding
of h= ow to accomplish the task.
The outcome is appreciated.

<= div class=3D"gmail_extra">I previously expected that "cat" --dire= ct would suffice.
Yet now I also want t= o mitigate the necessity of invoking "cat" using "xargs.&quo= t;
I should have realized that before sending= the first email.

coreutils should be expected to do what coreutils does,but not expected to be the most expeditious implementation.
I wanted unreasonable performance.

=
Please forgive that I created a bother
= and selfishly utilized other P=C3=A1draig Brady's time
instead of thinking for myself.=
That aside I should not omit saying;
thanks for maintaining and sharing c= oreutils.
Shell scripting would be= insufferable without coreutils.=C2=A0 :)
--047d7b677be46fb35704e890b860-- From unknown Tue Jun 17 22:17:47 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sun, 10 Nov 2013 12:24:03 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator