From unknown Sat Aug 16 21:16:42 2025 X-Loop: help-debbugs@gnu.org Subject: bug#24399: FICLONE for coreutils; copy_file_range? Resent-From: Paul Eggert Original-Sender: "Debbugs-submit" Resent-CC: bug-coreutils@gnu.org Resent-Date: Fri, 09 Sep 2016 22:47:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 24399 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: 24399@debbugs.gnu.org X-Debbugs-Original-To: bug-coreutils@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.147346119020476 (code B ref -1); Fri, 09 Sep 2016 22:47:02 +0000 Received: (at submit) by debbugs.gnu.org; 9 Sep 2016 22:46:30 +0000 Received: from localhost ([127.0.0.1]:54971 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1biUZ4-0005KC-7T for submit@debbugs.gnu.org; Fri, 09 Sep 2016 18:46:30 -0400 Received: from eggs.gnu.org ([208.118.235.92]:60730) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1biUZ3-0005K0-3F for submit@debbugs.gnu.org; Fri, 09 Sep 2016 18:46:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1biUYx-0000eh-1a for submit@debbugs.gnu.org; Fri, 09 Sep 2016 18:46:23 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.0 required=5.0 tests=BAYES_40 autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:50461) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1biUYw-0000eb-Uu for submit@debbugs.gnu.org; Fri, 09 Sep 2016 18:46:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51690) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1biUYu-0001U2-RJ for bug-coreutils@gnu.org; Fri, 09 Sep 2016 18:46:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1biUYq-0000dt-JV for bug-coreutils@gnu.org; Fri, 09 Sep 2016 18:46:19 -0400 Received: from zimbra.cs.ucla.edu ([131.179.128.68]:43050) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1biUYq-0000dm-9Q for bug-coreutils@gnu.org; Fri, 09 Sep 2016 18:46:16 -0400 Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 69BEB161172 for ; Fri, 9 Sep 2016 15:46:15 -0700 (PDT) Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id RZjGI15zBO-h for ; Fri, 9 Sep 2016 15:46:14 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id B7F3B16123D for ; Fri, 9 Sep 2016 15:46:14 -0700 (PDT) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id aTf6xqbRAvlO for ; Fri, 9 Sep 2016 15:46:14 -0700 (PDT) Received: from Penguin.CS.UCLA.EDU (Penguin.CS.UCLA.EDU [131.179.64.200]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id A0B85161172 for ; Fri, 9 Sep 2016 15:46:14 -0700 (PDT) From: Paul Eggert Organization: UCLA Computer Science Department Message-ID: <574ba8cd-c661-1a5c-ae9b-1a0da921947a@cs.ucla.edu> Date: Fri, 9 Sep 2016 15:46:14 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------59309AA2A61F69113597310D" X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.0 (----) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.0 (----) This is a multi-part message in MIME format. --------------59309AA2A61F69113597310D Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit I installed the attached patch so that cp etc. can use the FICLONE API that appears to be what the Linux kernel has standardized on for this sort of thing. I notice that the Linux 4.5 kernel has added a copy_file_range syscall that is said to clone in some cases, but this isn't yet declared in the /usr/include headers on Fedora 24 (4.7.2 kernel). Also, its man page says it uses a size_t (not off_t) to count the number of bytes to be copied, which is a strange choice for a file-copying API. What's up with that? --------------59309AA2A61F69113597310D Content-Type: application/x-patch; name="0001-cp-use-FICLONE-instead-of-BTRFS_IOC_CLONE.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="0001-cp-use-FICLONE-instead-of-BTRFS_IOC_CLONE.patch" RnJvbSBkYjA0MTcyZDcyMjljNTBjY2NjMDlhOTIwNzU3MzM4OTJkM2E4ZDg2IE1vbiBTZXAg MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBQYXVsIEVnZ2VydCA8ZWdnZXJ0QGNzLnVjbGEuZWR1 PgpEYXRlOiBGcmksIDkgU2VwIDIwMTYgMTU6Mjg6MzEgLTA3MDAKU3ViamVjdDogW1BBVENI XSBjcDogdXNlIEZJQ0xPTkUgaW5zdGVhZCBvZiBCVFJGU19JT0NfQ0xPTkUKClRoaXMgZG9l c24ndCBhZmZlY3QgdGhlIGdlbmVyYXRlZCBjb2RlIG9uIG15IHN5c3RlbTsgaXQncyBtZXJl bHkKYSBjbGVhbmVyIHdheSB0byB1c2UgdGhlIHJlY2VudGx5LWludHJvZHVjZWQgTGludXgt c3BlY2lmaWMgQVBJLgoqIG00L2ptLW1hY3Jvcy5tNCAoZ2xfQ0hFQ0tfQUxMX0hFQURFUlMp OiBDaGVjayBmb3IgbGludXgvZnMuaC4KKiBzcmMvY29weS5jOiBJbmNsdWRlIDxsaW51eC5m cy5oPiBpZiBhdmFpbGFibGUuCihGSUNMT05FKSBbX19saW51eF9fXTogRGVmaW5lIGlmIG5v dCBhbHJlYWR5IGRlZmluZWQuCihjbG9uZV9maWxlKTogVXNlIEZJQ0xPTkUgaW5zdGVhZCBv ZiB0aGUgb2xkZXIgQlRSRlNfSU9DX0NMT05FLAphcyB0aGlzIGlvY3RsIGlzIG5vIGxvbmdl ciBidHJmcy1zcGVjaWZpYy4KLS0tCiBtNC9qbS1tYWNyb3MubTQgfCAgMyArKy0KIHNyYy9j b3B5LmMgICAgICB8IDE2ICsrKysrKysrKystLS0tLS0KIDIgZmlsZXMgY2hhbmdlZCwgMTIg aW5zZXJ0aW9ucygrKSwgNyBkZWxldGlvbnMoLSkKCmRpZmYgLS1naXQgYS9tNC9qbS1tYWNy b3MubTQgYi9tNC9qbS1tYWNyb3MubTQKaW5kZXggNDM1YzZmZi4uYTE5MWFmMSAxMDA2NDQK LS0tIGEvbTQvam0tbWFjcm9zLm00CisrKyBiL200L2ptLW1hY3Jvcy5tNApAQCAtMSw0ICsx LDQgQEAKLSNzZXJpYWwgMTEwICAgLSotIGF1dG9jb25mIC0qLQorI3NlcmlhbCAxMTEgICAt Ki0gYXV0b2NvbmYgLSotCiAKIGRubCBNaXNjIHR5cGUtcmVsYXRlZCBtYWNyb3MgZm9yIGNv cmV1dGlscy4KIApAQCAtMjA3LDYgKzIwNyw3IEBAIEFDX0RFRlVOKFtnbF9DSEVDS19BTExf SEVBREVSU10sCiAgIEFDX0NIRUNLX0hFQURFUlNfT05DRShbCiAgICAgaHVyZC5oCiAgICAg bGludXgvZmFsbG9jLmgKKyAgICBsaW51eC9mcy5oCiAgICAgcGF0aHMuaAogICAgIHByaXYu aAogICAgIHN0cm9wdHMuaApkaWZmIC0tZ2l0IGEvc3JjL2NvcHkuYyBiL3NyYy9jb3B5LmMK aW5kZXggMTkxY2NhYS4uOGNjNWM1YiAxMDA2NDQKLS0tIGEvc3JjL2NvcHkuYworKysgYi9z cmMvY29weS5jCkBAIC03NCw2ICs3NCwxNCBAQAogIyBpbmNsdWRlIDxsaW51eC9mYWxsb2Mu aD4KICNlbmRpZgogCisjaWZkZWYgSEFWRV9MSU5VWF9GU19ICisjIGluY2x1ZGUgPGxpbnV4 L2ZzLmg+CisjZW5kaWYKKworI2lmICFkZWZpbmVkIEZJQ0xPTkUgJiYgZGVmaW5lZCBfX2xp bnV4X18KKyMgZGVmaW5lIEZJQ0xPTkUgX0lPVyAoMHg5NCwgOSwgaW50KQorI2VuZGlmCisK ICNpZm5kZWYgSEFWRV9GQ0hPV04KICMgZGVmaW5lIEhBVkVfRkNIT1dOIGZhbHNlCiAjIGRl ZmluZSBmY2hvd24oZmQsIHVpZCwgZ2lkKSAoLTEpCkBAIC0zMjAsMTIgKzMyOCw4IEBAIHNw YXJzZV9jb3B5IChpbnQgc3JjX2ZkLCBpbnQgZGVzdF9mZCwgY2hhciAqYnVmLCBzaXplX3Qg YnVmX3NpemUsCiBzdGF0aWMgaW5saW5lIGludAogY2xvbmVfZmlsZSAoaW50IGRlc3RfZmQs IGludCBzcmNfZmQpCiB7Ci0jaWZkZWYgX19saW51eF9fCi0jIHVuZGVmIEJUUkZTX0lPQ1RM X01BR0lDCi0jIGRlZmluZSBCVFJGU19JT0NUTF9NQUdJQyAweDk0Ci0jIHVuZGVmIEJUUkZT X0lPQ19DTE9ORQotIyBkZWZpbmUgQlRSRlNfSU9DX0NMT05FIF9JT1cgKEJUUkZTX0lPQ1RM X01BR0lDLCA5LCBpbnQpCi0gIHJldHVybiBpb2N0bCAoZGVzdF9mZCwgQlRSRlNfSU9DX0NM T05FLCBzcmNfZmQpOworI2lmZGVmIEZJQ0xPTkUKKyAgcmV0dXJuIGlvY3RsIChkZXN0X2Zk LCBGSUNMT05FLCBzcmNfZmQpOwogI2Vsc2UKICAgKHZvaWQpIGRlc3RfZmQ7CiAgICh2b2lk KSBzcmNfZmQ7Ci0tIAoyLjcuNAoK --------------59309AA2A61F69113597310D-- From unknown Sat Aug 16 21:16:42 2025 X-Loop: help-debbugs@gnu.org Subject: bug#24399: FICLONE for coreutils; copy_file_range? Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady Original-Sender: "Debbugs-submit" Resent-CC: bug-coreutils@gnu.org Resent-Date: Sat, 10 Sep 2016 10:06:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 24399 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: Paul Eggert , 24399@debbugs.gnu.org Received: via spool by 24399-submit@debbugs.gnu.org id=B24399.147350190318783 (code B ref 24399); Sat, 10 Sep 2016 10:06:02 +0000 Received: (at 24399) by debbugs.gnu.org; 10 Sep 2016 10:05:03 +0000 Received: from localhost ([127.0.0.1]:55144 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bif9j-0004st-Bn for submit@debbugs.gnu.org; Sat, 10 Sep 2016 06:05:03 -0400 Received: from mail.magicbluesmoke.com ([82.195.144.49]:37648) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bif9h-0004sR-LX for 24399@debbugs.gnu.org; Sat, 10 Sep 2016 06:05:01 -0400 Received: from [192.168.1.80] (unknown [109.76.68.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.magicbluesmoke.com (Postfix) with ESMTPSA id 024C510B; Sat, 10 Sep 2016 11:05:00 +0100 (IST) References: <574ba8cd-c661-1a5c-ae9b-1a0da921947a@cs.ucla.edu> From: =?UTF-8?Q?P=C3=A1draig?= Brady Message-ID: <57D3DACC.3030201@draigBrady.com> Date: Sat, 10 Sep 2016 11:05:00 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <574ba8cd-c661-1a5c-ae9b-1a0da921947a@cs.ucla.edu> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 09/09/16 23:46, Paul Eggert wrote: > I installed the attached patch so that cp etc. can use the FICLONE API > that appears to be what the Linux kernel has standardized on for this > sort of thing. Nice thanks. This is just renaming as the kernel used the same ABI as for the BTRFS ioctl. > I notice that the Linux 4.5 kernel has added a copy_file_range syscall > that is said to clone in some cases, but this isn't yet declared in the > /usr/include headers on Fedora 24 (4.7.2 kernel). Also, its man page > says it uses a size_t (not off_t) to count the number of bytes to be > copied, which is a strange choice for a file-copying API. What's up with > that? I've been monitoring that syscall a bit. It will be very useful for efficient server side only copying, and can take advantage of reflinking where available. I was pushing for more control flags for dealing with reflinking and handling of holes, but that was rejected in the first iteration at least. One quite awkward thing at present is if the file system doesn't implement the syscall, it falls back to do_splice_direct which is basically sendfile(), and that will expand holes. thanks, Pádraig From unknown Sat Aug 16 21:16:42 2025 X-Loop: help-debbugs@gnu.org Subject: bug#24399: FICLONE for coreutils; copy_file_range? Resent-From: Paul Eggert Original-Sender: "Debbugs-submit" Resent-CC: bug-coreutils@gnu.org Resent-Date: Sun, 11 Sep 2016 02:19:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 24399 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: =?UTF-8?Q?P=C3=A1draig?= Brady , 24399@debbugs.gnu.org Received: via spool by 24399-submit@debbugs.gnu.org id=B24399.147356030516220 (code B ref 24399); Sun, 11 Sep 2016 02:19:01 +0000 Received: (at 24399) by debbugs.gnu.org; 11 Sep 2016 02:18:25 +0000 Received: from localhost ([127.0.0.1]:55867 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1biuLg-0004DX-In for submit@debbugs.gnu.org; Sat, 10 Sep 2016 22:18:25 -0400 Received: from zimbra.cs.ucla.edu ([131.179.128.68]:45838) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1biuLf-0004DL-BQ for 24399@debbugs.gnu.org; Sat, 10 Sep 2016 22:18:23 -0400 Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 1374E160D51; Sat, 10 Sep 2016 19:18:18 -0700 (PDT) Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id cLhSXS0_MIS8; Sat, 10 Sep 2016 19:18:17 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 6B9351611EA; Sat, 10 Sep 2016 19:18:17 -0700 (PDT) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id B80VqeV__VhC; Sat, 10 Sep 2016 19:18:17 -0700 (PDT) Received: from [192.168.1.9] (unknown [100.32.155.148]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id 4C91B160D51; Sat, 10 Sep 2016 19:18:17 -0700 (PDT) References: <574ba8cd-c661-1a5c-ae9b-1a0da921947a@cs.ucla.edu> <57D3DACC.3030201@draigBrady.com> From: Paul Eggert Organization: UCLA Computer Science Department Message-ID: Date: Sat, 10 Sep 2016 19:18:17 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <57D3DACC.3030201@draigBrady.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Spam-Score: -1.3 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.3 (-) P=C3=A1draig Brady wrote: > One quite awkward thing at present is if the file system doesn't > implement the syscall, it falls back to do_splice_direct > which is basically sendfile(), and that will expand holes. OK, thanks for the info. It sounds like copy_file_range is not ready for=20 prime-time, then. Its use of a size_t to represent a file size seems like= it is=20 obviously wrong, as well. From debbugs-submit-bounces@debbugs.gnu.org Sun Sep 11 13:22:38 2016 Received: (at control) by debbugs.gnu.org; 11 Sep 2016 17:22:38 +0000 Received: from localhost ([127.0.0.1]:56682 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bj8Sk-0003zu-BV for submit@debbugs.gnu.org; Sun, 11 Sep 2016 13:22:38 -0400 Received: from zimbra.cs.ucla.edu ([131.179.128.68]:36260) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bj8Sj-0003zi-1C for control@debbugs.gnu.org; Sun, 11 Sep 2016 13:22:37 -0400 Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 91A35160972 for ; Sun, 11 Sep 2016 10:22:31 -0700 (PDT) Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id n_epxzshIU4o for ; Sun, 11 Sep 2016 10:22:31 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id ED837160D51 for ; Sun, 11 Sep 2016 10:22:30 -0700 (PDT) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id IAQ0j_Ybm2yE for ; Sun, 11 Sep 2016 10:22:30 -0700 (PDT) Received: from [192.168.1.9] (unknown [100.32.155.148]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id D4E10160972 for ; Sun, 11 Sep 2016 10:22:30 -0700 (PDT) To: control@debbugs.gnu.org From: Paul Eggert Subject: 24399 is done Organization: UCLA Computer Science Department Message-ID: <82d35cc0-07db-614f-0e72-87cf73173788@cs.ucla.edu> Date: Sun, 11 Sep 2016 10:22:30 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -2.2 (--) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.2 (--) tags 24399 patch close 24399