From unknown Wed Jun 18 23:14:02 2025 X-Loop: help-debbugs@gnu.org Subject: bug#6048: [coreutils] Re: Feature: add ocfs2 reflink to cp(1) Resent-From: "jeff.liu" Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-To: owner@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Tue, 27 Apr 2010 09:32:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 6048 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: coreutils@gnu.org Cc: sunil.mushran@oracle.com, 6048@debbugs.gnu.org, jim@meyering.net, Joel.Becker@oracle.com, chris.mason@oracle.com, P@draigBrady.com, tao.ma@oracle.com X-Debbugs-Original-Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , Chris Mason , =?UTF-8?Q?P=C3=A1draig?= Brady , Tao Ma Received: via spool by submit@debbugs.gnu.org id=B.12723606793196 (code B ref -1); Tue, 27 Apr 2010 09:32:02 +0000 Received: (at submit) by debbugs.gnu.org; 27 Apr 2010 09:31:19 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O6h8Q-0000pV-Cg for submit@debbugs.gnu.org; Tue, 27 Apr 2010 05:31:18 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O6h8N-0000pQ-N7 for submit@debbugs.gnu.org; Tue, 27 Apr 2010 05:31:16 -0400 Received: from lists.gnu.org ([199.232.76.165]:57688) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1O6h8K-0008WD-Le for submit@debbugs.gnu.org; Tue, 27 Apr 2010 05:31:12 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1O6h8J-0003Mh-Nq for bug-coreutils@gnu.org; Tue, 27 Apr 2010 05:31:11 -0400 Received: from [140.186.70.92] (port=49954 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1O6h8D-0003K1-CS for bug-coreutils@gnu.org; Tue, 27 Apr 2010 05:31:11 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.0 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1O6fRE-0001KR-24 for bug-coreutils@gnu.org; Tue, 27 Apr 2010 03:42:37 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:51174) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1O6fQd-0001CZ-3W; Tue, 27 Apr 2010 03:42:00 -0400 Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o3R7fkEu029130 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 27 Apr 2010 07:41:47 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o3PLUHsf027555; Tue, 27 Apr 2010 07:41:45 GMT Received: from abhmt001.oracle.com by acsmt353.oracle.com with ESMTP id 192465331272354072; Tue, 27 Apr 2010 00:41:12 -0700 Received: from [10.182.121.53] (/10.182.121.53) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 27 Apr 2010 00:41:11 -0700 Message-ID: <4BD69513.6000404@oracle.com> Date: Tue, 27 Apr 2010 15:41:07 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 References: <4BA776E5.5030901@oracle.com> <20100322201808.GA14285@mail.oracle.com> <4BB8B3F0.7060703@oracle.com> <20100408015349.GE4573@mail.oracle.com> <4BBEC110.8080608@oracle.com> In-Reply-To: <4BBEC110.8080608@oracle.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet15.oracle.com [148.87.113.117] X-CT-RefId: str=0001.0A090209.4BD6953C.006D:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.9 (----) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.2 (------) > Joel Becker wrote: >> On Sun, Apr 04, 2010 at 11:44:48PM +0800, jeff.liu wrote: >>> +/* This function detects whether the source and destination files >>> + are all resides on OCFS2. If true, then perform the OCFS2 reflink >>> + operation. Otherwise, performing btrfs clone operation. */ >> I don't know that the correct behavior is btrfs' when ocfs2 is >> not detected. > Looks I should not implement 'is_ocfs2_file()' here, it add a bit extra indistinct noise. > Now my idea is just performing either btrfs clone or ocfs2 reflink operation if cp(1) invoked with > '--reflink=[WHEN]'. > If both of them failed no matter what the reason is, fall back to normal copy process when the > reflink mode is 'REFLINK_AUTO', or abort. > >>> +/* Perform the OCFS2 CoW reflink operation if possible. >>> + We do not attempt to preserve security attributes for a reference >>> + counted link. Instead, let 'x->preserve_xxxx' to process them if >>> + they are the user expected. >>> + Upon success, return 0, Otherwise, return -1 and set errno. */ >>> +static inline int >>> +reflink_file (char const *src_name, char const *dst_name, >>> + int src_fd, bool *new_dst) >>> +{ >>> +#ifdef __linux__ >>> +# ifndef REFLINK_ATTR_NONE >>> +# define REFLINK_ATTR_NONE 0 >>> +# endif >> If '-p' was specified, you should honor it with >> REFLINK_ATTR_PRESERVE. > Thanks for pointing this out. At first, I think the file attributes preserve should managed through > normal copy, its wrong, I will fix it in the next patch submit. > >> Joel >> > > Thanks, > -Jeff > Hello All, This is the revised version to add ocfs2 reflink feature to cp(1) based on Joel's comments. Changes: 1. Remove 'is_ocfs2_file()' function, it does not need to pre-check the file system type for reflink operation. If ocfs2 reflink ioctl(2) failed due to this issue or any other reason, proceed to do btrfs clone operation. 2. Do perserve attriutes if `cp' invoked with '-p' opition. In this case,'mode,ownership,timestamp' are reflinked to destination file; security_context will be reflinked as well if possible. Also, 'xattrs' are reflinked always no matter preserve_attr or preserve_none specified. 3. From the code layout point of view, I separated ocfs2 reflink and btrfs clone into 2 steps. the reason is, ocfs2 ioctl(2) play with the destination path name instead of a file descriptor. So I place its code block at the top of '*new_dst' check, otherwise, we have to unlink(2) the 'dst_desc' when trying ocfs2 reflink ioctl(2). Tests against preserve option: ============================= ACL: --- jeff@jeff-laptop:~/opensource_dev/coreutils$ setfacl -m user:oracle:r-- /ocfs2/a jeff@jeff-laptop:~/opensource_dev/coreutils$ ls -l /ocfs2/a -rw-r--r--+ 1 jeff jeff 5 Apr 27 14:04 /ocfs2/a jeff@jeff-laptop:~/opensource_dev/coreutils$ getfacl /ocfs2/a getfacl: Removing leading '/' from absolute path names # file: ocfs2/a # owner: jeff # group: jeff user::rw- user:oracle:r-- group::r-- mask::r-- other::r-- jeff@jeff-laptop:~/opensource_dev/coreutils$ ./src/cp --reflink=always -p /ocfs2/a /ocfs2/c jeff@jeff-laptop:~/opensource_dev/coreutils$ ls -l /ocfs2/c -rw-r--r--+ 1 jeff jeff 5 Apr 27 14:04 /ocfs2/c jeff@jeff-laptop:~/opensource_dev/coreutils$ getfacl /ocfs2/c getfacl: Removing leading '/' from absolute path names # file: ocfs2/c # owner: jeff # group: jeff user::rw- user:oracle:r-- group::r-- mask::r-- other::r-- mode,ownership,timestamp: ------------------------- root@jeff-laptop:/home/jeff/opensource_dev/coreutils# ./src/cp --reflink=always -p /ocfs2/a /ocfs2/a_preserve_ownership root@jeff-laptop:/home/jeff/opensource_dev/coreutils# ls -l /ocfs2/a -rw-r--r--+ 1 jeff jeff 5 Apr 27 14:04 /ocfs2/a root@jeff-laptop:/home/jeff/opensource_dev/coreutils# ls -l /ocfs2/a_preserve_ownership -rw-r--r--+ 1 jeff jeff 5 Apr 27 14:04 /ocfs2/a_preserve_ownership Security context: ----------------- root@jeff-laptop:~# chcon oracle:object_r:tmp_t:s0 /ocfs2/a root@jeff-laptop:~# ls -l /ocfs2/a -rw-r--r--+ 1 jeff jeff 5 Apr 27 14:04 /ocfs2/a root@jeff-laptop:~# ls -Z /ocfs2/a |grep oracle:object_r:tmp_t:s0 oracle:object_r:tmp_t:s0 /ocfs2/a jeff@jeff-laptop:~/opensource_dev/coreutils$ ./src/cp --reflink=always -p /ocfs2/a /ocfs2/e_preserve_security -rw-r--r--+ 1 jeff jeff 5 Apr 27 14:04 /ocfs2/e_preserve_security jeff@jeff-laptop:~/opensource_dev/coreutils$ ls -Z /ocfs2/e_preserve_security |grep oracle:object_r:tmp_t:s0 oracle:object_r:tmp_t:s0 /ocfs2/e_preserve_security Would you guys please review the patch which shown as following? I will continue to write the test script if you guys like the current implemention. Any comments are appreciated! >From 96d1d86b82ec101caeebe2c51fad6b2a52d093f5 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Tue, 27 Apr 2010 15:37:17 +0800 Subject: [PATCH 1/1] Add ocfs2 reflink to cp(1), it share the same '--reflink=[WHEN]' option with btrfs clone operation. If cp(1) invoked with '-p' option, and the underlaying file system is OCFS2 for both source and destination files, it will try to preserve the default attributes(i.e. 'mode,ownership,timestamp'), xattr as well as security context if possbile. Signed-off-by: Jie Liu --- src/copy.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 files changed, 76 insertions(+), 1 deletions(-) diff --git a/src/copy.c b/src/copy.c index d18fd45..d2ba306 100644 --- a/src/copy.c +++ b/src/copy.c @@ -54,6 +54,12 @@ #include "areadlink.h" #include "yesno.h" +#if HAVE_SYS_VFS_H +# include +#else +# include +#endif + #if USE_XATTR # include # include @@ -132,6 +138,47 @@ utimens_symlink (char const *file, struct timespec const *timespec) return err; } +/* Perform the OCFS2 CoW reflink ioctl(2) operation if possible. + When using '-p' option, the file's default attributes(i.e. mode,timestamp, + ownership and security context if possbile) are reflinked to the destination + file as well. We will then skip over the standard preserve process for such + attributes. Also, 'xattrs' are reflinked always even if 'REFLINK_ATTR_NONE'. + Upon success, return 0, Otherwise, return -1 and set errno. */ +static inline int +reflink_file (char const *src_name, char const *dst_name, + bool preserve_attrs, int src_fd) +{ +#ifdef __linux__ +# ifndef REFLINK_ATTR_NONE +# define REFLINK_ATTR_NONE 0 +# endif +# ifndef REFLINK_ATTR_PRESERVE +# define REFLINK_ATTR_PRESERVE 1 +# endif +# ifndef OCFS2_IOC_REFLINK + struct reflink_arguments { + uint64_t old_path; + uint64_t new_path; + uint64_t preserve; + }; +# define OCFS2_IOC_REFLINK _IOW ('o', 4, struct reflink_arguments) +# endif + struct reflink_arguments args = { + .old_path = (unsigned long) src_name, + .new_path = (unsigned long) dst_name, + .preserve = preserve_attrs ? REFLINK_ATTR_PRESERVE : REFLINK_ATTR_NONE, + }; + return ioctl (src_fd, OCFS2_IOC_REFLINK, &args); +#else + (void) src_name; + (void) dst_name; + (void) preserve_attrs; + (void) src_fd; + errno = ENOTSUP; + return -1; +#endif +} + /* Perform the O(1) btrfs clone operation, if possible. Upon success, return 0. Otherwise, return -1 and set errno. */ static inline int @@ -514,6 +561,31 @@ copy_reg (char const *src_name, char const *dst_name, goto close_src_desc; } + bool reflink_ok = false; + if (x->reflink_mode) + { + /* When cp is invoked with '--reflink=[WHEN]', try to do OCFS2 reflink ioctl(2) + first, if it fails, then try Btrfs clone once again at the later time. + The reason to perform those operations separately is because `cp' will create + the destination file if it is a '*new_dst'. In this case, we have to unlink(2) + it firstly, otherwise, OCFS2 reflink will fail with 'EEXIST'. But if the destnation + file is not a '*new_dst', just let reflink ioctl(2) fails and go ahead. */ + bool preserve_attributes = (x->preserve_ownership + && x->preserve_mode + && x->preserve_timestamps); + reflink_ok = reflink_file (src_name, dst_name, preserve_attributes, + source_desc) == 0; + if (reflink_ok) + { + *new_dst = false; + + /* Skip over the standard attributes preserve process + if reflink succeeds and they are already reflinked. */ + if (preserve_attributes) + goto close_src_desc; + } + } + /* The semantics of the following open calls are mandated by the specs for both cp and mv. */ if (! *new_dst) @@ -634,8 +706,11 @@ copy_reg (char const *src_name, char const *dst_name, goto close_src_and_dst_desc; } - if (x->reflink_mode) + if (x->reflink_mode && ! reflink_ok) { + /* It seems that the preceeding OCFS2 reflink fails, perform Btrfs + clone now, if it fails again and `cp' is invoked with '--reflink=auto', + fall back to a standard copy. */ bool clone_ok = clone_file (dest_desc, source_desc) == 0; if (clone_ok || x->reflink_mode == REFLINK_ALWAYS) { -- 1.5.4.3 Best Regards, -Jeff From debbugs-submit-bounces@debbugs.gnu.org Mon May 14 07:12:12 2012 Received: (at control) by debbugs.gnu.org; 14 May 2012 11:12:12 +0000 Received: from localhost ([127.0.0.1]:52189 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1STtCG-0002EY-7Z for submit@debbugs.gnu.org; Mon, 14 May 2012 07:12:12 -0400 Received: from mail2.vodafone.ie ([213.233.128.44]:46345) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1STtBv-0002Dk-R3 for control@debbugs.gnu.org; Mon, 14 May 2012 07:12:10 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApMBAOznsE9tTfly/2dsb2JhbAANN4V5hHmnfIQaJAoqVA0CBRYLAgsDAgECATkGAgIVCAEBhUWCL6gWkj6BMIxMgg6BGAScFI0T Received: from unknown (HELO [192.168.1.79]) ([109.77.249.114]) by mail2.vodafone.ie with ESMTP; 14 May 2012 12:11:23 +0100 Message-ID: <4FB0E85B.5000500@draigBrady.com> Date: Mon, 14 May 2012 12:11:23 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20110816 Thunderbird/6.0 MIME-Version: 1.0 To: control@debbugs.gnu.org X-Enigmail-Version: 1.3.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: unarchive 6048 forcemerge 6048 11443 [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 3.0 GOT_NO_SUBJECT No real subject -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [213.233.128.44 listed in list.dnswl.org] -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 1.8 MISSING_SUBJECT Missing Subject: header X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has identified this incoming email as possible spam. The original message has been attached to this so you can view it (if it isn't spam) or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: unarchive 6048 forcemerge 6048 11443 [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [213.233.128.44 listed in list.dnswl.org] 3.0 GOT_NO_SUBJECT No real subject -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] 1.8 MISSING_SUBJECT Missing Subject: header unarchive 6048 forcemerge 6048 11443 From unknown Wed Jun 18 23:14:02 2025 X-Loop: help-debbugs@gnu.org Subject: bug#6048: bug#11443: linux "cp" and ocfs2 reflink/clone/fastcopy/copy on write Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Mon, 14 May 2012 13:21:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 6048 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: Kai Petzke Cc: 6048@debbugs.gnu.org, Georg Schmitz Received: via spool by 6048-submit@debbugs.gnu.org id=B6048.133700161626446 (code B ref 6048); Mon, 14 May 2012 13:21:01 +0000 Received: (at 6048) by debbugs.gnu.org; 14 May 2012 13:20:16 +0000 Received: from localhost ([127.0.0.1]:52433 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1STvCB-0006sV-5k for submit@debbugs.gnu.org; Mon, 14 May 2012 09:20:15 -0400 Received: from mail2.vodafone.ie ([213.233.128.44]:3514) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1STvC7-0006sG-Mx for 6048@debbugs.gnu.org; Mon, 14 May 2012 09:20:13 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApMBAG8FsU9tTfly/2dsb2JhbAANN7cPAQEBAwEBAiRSBQsLDQsJFg8JAwIBAgEWLwYNAQcBAYgFELo3ixqGCASOd4RYgz+FBo0T Received: from unknown (HELO [192.168.1.79]) ([109.77.249.114]) by mail2.vodafone.ie with ESMTP; 14 May 2012 14:19:42 +0100 Message-ID: <4FB1066E.9040305@draigBrady.com> Date: Mon, 14 May 2012 14:19:42 +0100 From: =?UTF-8?Q?P=C3=A1draig?= Brady User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20110816 Thunderbird/6.0 MIME-Version: 1.0 References: In-Reply-To: X-Enigmail-Version: 1.3.2 Content-Type: multipart/mixed; boundary="------------090506040002030006040804" X-Spam-Score: -1.9 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.9 (-) This is a multi-part message in MIME format. --------------090506040002030006040804 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 8bit I've merged this back to bug 6048 On 05/09/2012 03:28 PM, Kai Petzke wrote: > Hello, > > > there has been work by others about adding support for the OCFS2 "reflink" ioctl() call, which is similiar to the btrfs "clone" call, and creates a copy-on-write copy of the original, thus allowing to "copy" even gigabyte sized files within a tiny fraction of a second, and without using much additional file system space. See: > http://lists.gnu.org/archive/html/coreutils/2011-08/msg00046.html > http://lists.gnu.org/archive/html/bug-coreutils/2010-04/msg00185.html > > I have updated those patches to work against coreutils 8.16, removed those bugs, that I spotted. In particular, if the destination file exists, the "reflink" ist automatically tried again after removing it, and if not all attributes are copied, it is made sure, that the following open() system call does not truncate the just created copy. > > I strongly suggest including that patch in the coreutils package, I'm less enthused about adding this as it doesn't fit very cleanly. > even though the interface to use to different system calls to achieve the same thing is awkward. > But, as laid out in the comments in the source, btrfs clone and ocfs2 reflink are semantically > quite different, so that unifying them into one on the kernel side is not likely to happen, soon, if it happens at all. That would be unfortunate. Hopefully a generic reflink() call can be sorted out. > If users don't use the --reflink option of "cp", the additional code makes no difference, so it doesn't hurt. Fair point. > And if users use "--reflink" on either of the supported file systems, they get a huge advantage out of it! I really dislike that xattrs are copied unconditionally. It might be best to auto clear xattrs after the "reflink", if possible? cheers, Pádraig. --------------090506040002030006040804 Content-Type: text/plain; name="coreutils-8.16-ocfs2-reflink.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="coreutils-8.16-ocfs2-reflink.diff" --- copy.c.orig 2012-03-24 21:26:51.000000000 +0100 +++ copy.c 2012-05-09 16:07:46.000000000 +0200 @@ -60,6 +60,12 @@ #include "areadlink.h" #include "yesno.h" +#if HAVE_SYS_VFS_H +# include +#else +# include +#endif + #if USE_XATTR # include # include @@ -218,6 +224,47 @@ return true; } +/* Perform the OCFS2 CoW reflink ioctl(2) operation if possible. + When using '-p' option, the file's default attributes(i.e. mode,timestamp, + ownership and security context if possbile) are reflinked to the destination + file as well. We will then skip over the standard preserve process for such + attributes. Also, 'xattrs' are reflinked always even if 'REFLINK_ATTR_NONE'. + Upon success, return 0, Otherwise, return -1 and set errno. */ +static inline int +reflink_file (char const *src_name, char const *dst_name, + bool preserve_attrs, int src_fd) +{ +#ifdef __linux__ +# ifndef REFLINK_ATTR_NONE +# define REFLINK_ATTR_NONE 0 +# endif +# ifndef REFLINK_ATTR_PRESERVE +# define REFLINK_ATTR_PRESERVE 1 +# endif +# ifndef OCFS2_IOC_REFLINK + struct reflink_arguments { + uint64_t old_path; + uint64_t new_path; + uint64_t preserve; + }; +# define OCFS2_IOC_REFLINK _IOW ('o', 4, struct reflink_arguments) +# endif + struct reflink_arguments args = { + .old_path = (unsigned long) src_name, + .new_path = (unsigned long) dst_name, + .preserve = preserve_attrs ? REFLINK_ATTR_PRESERVE : REFLINK_ATTR_NONE, + }; + return ioctl (src_fd, OCFS2_IOC_REFLINK, &args); +#else + (void) src_name; + (void) dst_name; + (void) preserve_attrs; + (void) src_fd; + errno = ENOTSUP; + return -1; +#endif +} + /* Perform the O(1) btrfs clone operation, if possible. Upon success, return 0. Otherwise, return -1 and set errno. */ static inline int @@ -822,11 +869,55 @@ goto close_src_desc; } + bool reflink_ok = false; + if (x->reflink_mode) + { + /* When cp is invoked with '--reflink=[WHEN]', try to do OCFS2 reflink + ioctl(2) first. If it fails, then try Btrfs clone later on. + The reason to perform those operations separately is because + the OCFS2 reflink ioctl() works on file names, while Btrfs clone + works on open file descriptors. + If OCFS2 reflink ioctl() succeeds and attribute preservation was + enabled, we are done. If OCFS2 reflink succeeds and only some of + the attributes are preserved, we still have to open the destination + file and go through the attribute copying code, but don't need + to execute the actual copy. Of course, the open() system call must + be performed without O_TRUNC set in that case. + If OCFS2 reflink fails, Btrfs clone is tried later on, after the + destination file has been opened normally. + + Note, that OCFS2 reflink ioctl() fails with errno set to EEXIST, + if the destination file already exists. If that happens, we + unlink() the destination file and try again. */ + bool preserve_attributes = (x->preserve_ownership + && x->preserve_mode + && x->preserve_timestamps); + reflink_ok = reflink_file (src_name, dst_name, preserve_attributes, + source_desc) == 0; + if (! reflink_ok && errno == EEXIST) + { + reflink_ok = unlink (dst_name) == 0 && + reflink_file (src_name, dst_name, preserve_attributes, + source_desc) == 0; + } + if (reflink_ok) + { + *new_dst = false; + data_copy_required = false; + + /* Skip over the standard attributes preserve process + if reflink succeeds and they are already reflinked. */ + if (preserve_attributes) + goto close_src_desc; + } + } + /* The semantics of the following open calls are mandated by the specs for both cp and mv. */ if (! *new_dst) { - dest_desc = open (dst_name, O_WRONLY | O_TRUNC | O_BINARY); + int open_flags = O_WRONLY | (reflink_ok ? 0 : O_TRUNC) | O_BINARY; + dest_desc = open (dst_name, open_flags); dest_errno = errno; /* When using cp --preserve=context to copy to an existing destination, @@ -955,18 +1046,19 @@ /* --attributes-only overrides --reflink. */ if (data_copy_required && x->reflink_mode) { + /* If the preceeding OCFS2 reflink failed, try Btrfs clone now. + If it fails again and `cp' is invoked with '--reflink=always', + report an error, otherwise, fall back to a standard copy. */ bool clone_ok = clone_file (dest_desc, source_desc) == 0; - if (clone_ok || x->reflink_mode == REFLINK_ALWAYS) + if (!clone_ok && x->reflink_mode == REFLINK_ALWAYS) { - if (!clone_ok) - { - error (0, errno, _("failed to clone %s from %s"), - quote_n (0, dst_name), quote_n (1, src_name)); - return_val = false; - goto close_src_and_dst_desc; - } - data_copy_required = false; - } + error (0, errno, _("failed to clone %s from %s"), + quote_n (0, dst_name), quote_n (1, src_name)); + return_val = false; + goto close_src_and_dst_desc; + } + if (clone_ok) + data_copy_required = false; } if (data_copy_required) --------------090506040002030006040804-- From unknown Wed Jun 18 23:14:02 2025 X-Loop: help-debbugs@gnu.org Subject: bug#6048: bug#11443: linux "cp" and ocfs2 reflink/clone/fastcopy/copy on write Resent-From: Assaf Gordon Original-Sender: "Debbugs-submit" Resent-CC: bug-coreutils@gnu.org Resent-Date: Wed, 10 Oct 2018 16:55:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 6048 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: 6048@debbugs.gnu.org Received: via spool by 6048-submit@debbugs.gnu.org id=B6048.153919044931806 (code B ref 6048); Wed, 10 Oct 2018 16:55:01 +0000 Received: (at 6048) by debbugs.gnu.org; 10 Oct 2018 16:54:09 +0000 Received: from localhost ([127.0.0.1]:43782 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gAHkP-0008Gw-HM for submit@debbugs.gnu.org; Wed, 10 Oct 2018 12:54:09 -0400 Received: from mail-pg1-f194.google.com ([209.85.215.194]:36960) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gAHkL-0008GR-G4 for 6048@debbugs.gnu.org; Wed, 10 Oct 2018 12:54:06 -0400 Received: by mail-pg1-f194.google.com with SMTP id c10-v6so2797385pgq.4 for <6048@debbugs.gnu.org>; Wed, 10 Oct 2018 09:54:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=9ZX72xb/LiPxUCht6JPCG++lA/0KioIdtlQY1U96Bns=; b=FhSxmLtE2YRwKy7q0gWWWezQprS1hLiQysewPixE660agHmLQyrq3G/Oq2YAUXeIup vu+qoWXriaJ3FHXLQ9EAPL/7F7x7FquE+3Y9zuW7BAPO9SJ5CrqM8M2gm/6W2rTN6MwW jzaYfVgQtekRI1aN9Q0Nm6F3jbwels0hYzJSqICqr0ItvM4oVEBij337z08eN/2NQrmM jcxysEwIlHJ9ZQnmasPYkdKOoMAxv9OMpr5tmGI/e9Y07Ejwqc03jKBywXsVO346zFZe QNYIIO7XuQpgk2UCTffH8Sqb5gy3puPwBcnDsqIbEGHHcvBQ1IO4DmxoxcslDdEDhtpr lGKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=9ZX72xb/LiPxUCht6JPCG++lA/0KioIdtlQY1U96Bns=; b=ETAYMDNCU3y5OgVblHs+dtTfG515gs4IXaTWeiOpAfHylrMZml5FaZcp2FNBWfzA5m apmaV8U5GDjTGa4SJL56JV7RYCkQh/cpVm/921qLnWzqYWFUsSsr3K3yymzChDh7UiBE nFRAlojtauxULZy4tsrNVH6Rw03VELdBpmqx7vTifO3YY7OcgyiBXstiK2vIkN+kB/VD t+SdKQMKQZJLqER4OA2BnSCGA0W5W4TSL56tcdjZrRWfg+PNxxxFiqNyVB5H1FGsfrDp YD5wK8+XECg5dngBx1u1/cDbotjondMhdnJP8axzJKmmYooufR1UvUGwyHDjJoHQNcyG OB6Q== X-Gm-Message-State: ABuFfohZALx6FcSpFG8i08Z2KqUcRh60qYQjWbjDFKUeNSlsfZG+141a 9Vtyw2NsWz++ALBNF07EcyPCO45Yu18= X-Google-Smtp-Source: ACcGV63QquQRdO8fleNqLXpg8dlejDBARGubVAFg27WNREMNdyoeqqkFyP87BOJd835hEDapkb3mhg== X-Received: by 2002:a62:564e:: with SMTP id k75-v6mr16861287pfb.33.1539190438806; Wed, 10 Oct 2018 09:53:58 -0700 (PDT) Received: from tomato.housegordon.com (moose.housegordon.com. [184.68.105.38]) by smtp.googlemail.com with ESMTPSA id r18-v6sm22392236pgv.17.2018.10.10.09.53.57 for <6048@debbugs.gnu.org> (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Oct 2018 09:53:57 -0700 (PDT) References: <4FB1066E.9040305@draigBrady.com> From: Assaf Gordon Message-ID: <3b2d09fd-95a4-78ec-d6fb-9ff2853fa27b@gmail.com> Date: Wed, 10 Oct 2018 10:53:55 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <4FB1066E.9040305@draigBrady.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) (triaging old bugs) On 14/05/12 07:19 AM, Pádraig Brady wrote: > On 05/09/2012 03:28 PM, Kai Petzke wrote: >> there has been work by others about adding support for the OCFS2 >> "reflink" ioctl() call, which is similiar to the btrfs "clone" >> call, and creates a copy-on-write copy of the original, thus >> allowing to "copy" even gigabyte sized files within a tiny fraction >> of a second, and without using much additional file system space. >> See: [...] >> even though the interface to use to different system calls to achieve the same thing is awkward. >> But, as laid out in the comments in the source, btrfs clone and ocfs2 reflink are semantically >> quite different, so that unifying them into one on the kernel side is not likely to happen, soon, if it happens at all. > > That would be unfortunate. Hopefully a generic reflink() call can be sorted out. In 2014 we had this commit: https://git.savannah.gnu.org/cgit/coreutils.git/commit/?id=b47231be6813e6cb5305266e391b4bb745f27f13 ==== commit b47231be6813e6cb5305266e391b4bb745f27f13 Author: David Sterba Date: Wed Sep 24 11:15:05 2014 +0100 mv: use reflink=auto mode by default On some filesystems (BTRFS), moving a file within the filesystem may cross subvolume boundaries and we can use a lightweight reflink copy, similar to what cp(1) can do, which is faster than a full file copy. ==== Does this 'reflink=auto' mode addresses OCFS2 as well? I believe so, but want to double-check. If there are no objections, I'll mark this as "fixed/done" in couple of days. regards, - assaf From debbugs-submit-bounces@debbugs.gnu.org Tue Oct 30 00:48:24 2018 Received: (at control) by debbugs.gnu.org; 30 Oct 2018 04:48:24 +0000 Received: from localhost ([127.0.0.1]:53036 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gHLx2-0002zK-65 for submit@debbugs.gnu.org; Tue, 30 Oct 2018 00:48:24 -0400 Received: from mail-it1-f178.google.com ([209.85.166.178]:36992) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gHLx0-0002z7-Ew for control@debbugs.gnu.org; Tue, 30 Oct 2018 00:48:22 -0400 Received: by mail-it1-f178.google.com with SMTP id e74-v6so12280427ita.2 for ; Mon, 29 Oct 2018 21:48:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:message-id:date:user-agent:mime-version:content-language :content-transfer-encoding; bh=0qE+3nbQ727HnGXR26hshIdVLmqGPV+5Btbt/PmyOWQ=; b=pt6Zd6Xo6pupoLGbKRefLMuX02K9QPJjnayg3NW7w0I4qrpjYRSbYgZYdDm+BKOVnq d3jZxM0mm4AFmt74Y7V/+mvc4Set6ciYf4hYqQ6IoIb2YH7rVxgyVNl0TeDQz7L0Styu yLXGjMJO9qt5+VkSLcUiQ9bd7ihVgxQB5ym9csFm22trFq5Ev0Q6J/xAkBISmkpmJMxY WF+CITa71uowlbaZdrcEYxsyohbgYbhpvA0f2S7SjVTrdpW08Rtj8wA+mS1j0l9NGipn 5uNyMfIdjWNRSc/ZmwkQKq7QRQByDgMY6WENOCMkSi3vdRvoYG6S8gCftvyJBIoY9afo etcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:message-id:date:user-agent:mime-version :content-language:content-transfer-encoding; bh=0qE+3nbQ727HnGXR26hshIdVLmqGPV+5Btbt/PmyOWQ=; b=VrFz6pLAnnNUxpTNT3ARwfYbRtN9CWZOowFhJ0Qmal3zsy3gzoP1qrvFiJa9+wlrPS L/NyfUjznboaT5yesj3CH+fKubw8sy+HJ1ccyz3JJzu0ebbvq2TxXhcmAwMzNG+CEy8V B4vRzdXitEyC+sAr2Kyf/pNOs6ibZiyW4TXZ8o0TDtRVgRlfbqAU8BP3M3b7om6AohGo un/cPXKfcCc4c+BvJoRl63Hb6/yv95jbZxWmezGjNctotdeJ94nWhbe/DwL4QLBoYLgp rv3tbhOLaJwYi5GSFQZRvVCINHSAC6ZMub7Syd+wGZDNZnceGhDROHJXmS0TMkWhwxbY J0hg== X-Gm-Message-State: AGRZ1gKqn/t97O3pUexoWuAORoXEhd3B/4YnwV+2cmnMw0EJ5QFxC4T7 pzu8NajN1Vp4pEu9JKuRNjjuRf5Zvug= X-Google-Smtp-Source: AJdET5c0/bqH5N12FwUNLuRasQnmA9W0Fp0S4YmRS6XWuEdAkq4bGcJ5Bb+f6CjoF1KqloCV6KMdaA== X-Received: by 2002:a24:8247:: with SMTP id t68-v6mr457747itd.68.1540874896356; Mon, 29 Oct 2018 21:48:16 -0700 (PDT) Received: from tomato.housegordon.com (moose.housegordon.com. [184.68.105.38]) by smtp.googlemail.com with ESMTPSA id w124-v6sm8299335ita.0.2018.10.29.21.48.14 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Oct 2018 21:48:15 -0700 (PDT) To: control@debbugs.gnu.org From: Assaf Gordon Message-ID: <7d17c221-0ee5-3278-365f-52dbaab7e3d4@gmail.com> Date: Mon, 29 Oct 2018 22:48:13 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: close 6048 [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (assafgordon[at]gmail.com) -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [209.85.166.178 listed in list.dnswl.org] 1.8 MISSING_SUBJECT Missing Subject: header 0.2 NO_SUBJECT Extra score for no subject X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) close 6048