From debbugs-submit-bounces@debbugs.gnu.org Fri May 07 10:15:16 2010 Received: (at submit) by debbugs.gnu.org; 7 May 2010 14:15:16 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OAOKh-0007D7-EB for submit@debbugs.gnu.org; Fri, 07 May 2010 10:15:16 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OAOKf-0007D2-2X for submit@debbugs.gnu.org; Fri, 07 May 2010 10:15:14 -0400 Received: from lists.gnu.org ([199.232.76.165]:55494) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OAOKc-0006lJ-L3 for submit@debbugs.gnu.org; Fri, 07 May 2010 10:15:10 -0400 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1OAOKb-0000OP-Od for bug-coreutils@gnu.org; Fri, 07 May 2010 10:15:09 -0400 Received: from [140.186.70.92] (port=32976 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OAOKZ-0000Lp-LP for bug-coreutils@gnu.org; Fri, 07 May 2010 10:15:09 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OAOKW-0004gI-Gw for bug-coreutils@gnu.org; Fri, 07 May 2010 10:15:07 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:46147) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OAOKW-0004es-9Q for bug-coreutils@gnu.org; Fri, 07 May 2010 10:15:04 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o47EEvrG021487 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 7 May 2010 14:14:59 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o47B7uUI026690; Fri, 7 May 2010 14:14:56 GMT Received: from abhmt008.oracle.com by acsmt355.oracle.com with ESMTP id 246476311273241618; Fri, 07 May 2010 07:13:38 -0700 Received: from [221.223.109.145] (/221.223.109.145) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 07 May 2010 07:13:35 -0700 Message-ID: <4BE41FFF.4020809@oracle.com> Date: Fri, 07 May 2010 22:13:19 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: bug-coreutils@gnu.org Subject: [PATCH]: fiemap support for efficient sparse file copy Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090201.4BE42063.019F:SCFMA922111,ss=1,pt=113351,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -3.4 (---) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , Jim Meyering , Chris Mason , Joel Becker X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.8 (----) Hello All, Add fiemap ioctl(2) feature to cp(1) for efficient sparse files copy has been discussed a few times in the past few months, thanks Jim for the review comments. I just work out a new patchsets against the latest upstream code and run some tests which I have shown before, they all works well. There is minor code change in this post in case of the ioctl(2) fails in the middle of fiemap copy process. My thought is, if this is the first time we met, go back to the standard copy as usual. Otherwise, we should abort the copy process to avoid corrupting the dest file. In order to determine if it was the first time ioctl(2) fails , make use of the variable 'i' which is the fiemap extent counter, to check if it is equal to '0', its value should be increased if the previous call ioctl(2) succeeds. Would you guys please review and consider apply the patches if there is no other issue? >From f8c78794a70f1fb45a2c61c8bbeca344087287ab Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Fri, 7 May 2010 20:48:45 +0800 Subject: [PATCH 1/3] Add fiemap.h for fiemap ioctl(2) support. It does not shipped by default, so I copy it from kernel at the moment. I have update its code style respect to GNU coding style. Signed-off-by: Jie Liu --- gnulib | 2 +- src/fiemap.h | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 103 insertions(+), 1 deletions(-) create mode 100644 src/fiemap.h diff --git a/gnulib b/gnulib index e6addf8..8df7efd 160000 --- a/gnulib +++ b/gnulib @@ -1 +1 @@ -Subproject commit e6addf84d6331d634b5d76db03f59851f3de8894 +Subproject commit 8df7efddc8ffe398cde4106d32b39848e5948df9 diff --git a/src/fiemap.h b/src/fiemap.h new file mode 100644 index 0000000..d33293b --- /dev/null +++ b/src/fiemap.h @@ -0,0 +1,102 @@ +/* FS_IOC_FIEMAP ioctl infrastructure. + Some portions copyright (C) 2007 Cluster File Systems, Inc + Authors: Mark Fasheh + Kalpak Shah + Andreas Dilger . */ + +/* Copy from kernel, modified to respect GNU code style by Jie Liu. */ + +#ifndef _LINUX_FIEMAP_H +# define _LINUX_FIEMAP_H + +# include + +struct fiemap_extent +{ + /* Logical offset in bytes for the start of the extent + from the beginning of the file. */ + uint64_t fe_logical; + + /* Physical offset in bytes for the start of the extent + from the beginning of the disk. */ + uint64_t fe_physical; + + /* Length in bytes for this extent. */ + uint64_t fe_length; + + uint64_t fe_reserved64[2]; + + /* FIEMAP_EXTENT_* flags for this extent. */ + uint32_t fe_flags; + + uint32_t fe_reserved[3]; +}; + +struct fiemap +{ + /* Logical offset(inclusive) at which to start mapping(in). */ + uint64_t fm_start; + + /* Logical length of mapping which userspace wants(in). */ + uint64_t fm_length; + + /* FIEMAP_FLAG_* flags for request(in/out). */ + uint32_t fm_flags; + + /* Number of extents that were mapped(out). */ + uint32_t fm_mapped_extents; + + /* Size of fm_extents array(in). */ + uint32_t fm_extent_count; + + uint32_t fm_reserved; + + /* Array of mapped extents(out). */ + struct fiemap_extent fm_extents[0]; +}; + +/* The maximum offset can be mapped for a file. */ +# define FIEMAP_MAX_OFFSET (~0ULL) + +/* Sync file data before map. */ +# define FIEMAP_FLAG_SYNC 0x00000001 + +/* Map extented attribute tree. */ +# define FIEMAP_FLAG_XATTR 0x00000002 + +# define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR) + +/* Last extent in file. */ +# define FIEMAP_EXTENT_LAST 0x00000001 + +/* Data location unknown. */ +# define FIEMAP_EXTENT_UNKNOWN 0x00000002 + +/* Location still pending, Sets EXTENT_UNKNOWN. */ +# define FIEMAP_EXTENT_DELALLOC 0x00000004 + +/* Data can not be read while fs is unmounted. */ +# define FIEMAP_EXTENT_ENCODED 0x00000008 + +/* Data is encrypted by fs. Sets EXTENT_NO_BYPASS. */ +# define FIEMAP_EXTENT_DATA_ENCRYPTED 0x00000080 + +/* Extent offsets may not be block aligned. */ +# define FIEMAP_EXTENT_NOT_ALIGNED 0x00000100 + +/* Data mixed with metadata. Sets EXTENT_NOT_ALIGNED. */ +# define FIEMAP_EXTENT_DATA_INLINE 0x00000200 + +/* Multiple files in block. Set EXTENT_NOT_ALIGNED. */ +# define FIEMAP_EXTENT_DATA_TAIL 0x00000400 + +/* Space allocated, but not data (i.e. zero). */ +# define FIEMAP_EXTENT_UNWRITTEN 0x00000800 + +/* File does not natively support extents. Result merged for efficiency. */ +# define FIEMAP_EXTENT_MERGED 0x00001000 + +/* Space shared with other files. */ +# define FIEMAP_EXTENT_SHARED 0x00002000 + +#endif -- 1.5.4.3 >From 12618891cf4f3aff6b65463887b689c2ad99aa8e Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Fri, 7 May 2010 21:14:05 +0800 Subject: [PATCH 2/3] Add fiemap ioctl(2) support for efficient sparse file copy. Signed-off-by: Jie Liu --- src/copy.c | 154 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 154 insertions(+), 0 deletions(-) diff --git a/src/copy.c b/src/copy.c index c16cef6..960e5fb 100644 --- a/src/copy.c +++ b/src/copy.c @@ -63,6 +63,10 @@ #include +#ifndef HAVE_FIEMAP +# include "fiemap.h" +#endif + #ifndef HAVE_FCHOWN # define HAVE_FCHOWN false # define fchown(fd, uid, gid) (-1) @@ -149,6 +153,136 @@ clone_file (int dest_fd, int src_fd) #endif } +#ifdef __linux__ +# ifndef FS_IOC_FIEMAP +# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) +# endif +/* Perform FIEMAP(available in mainline 2.6.27) copy if possible. + Call ioctl(2) with FS_IOC_FIEMAP to efficiently map file allocation + excepts holes. So the overhead to deal with holes with lseek(2) in + normal copy could be saved. This would result in much faster backups + for any kind of sparse file. */ +static bool +fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, + off_t src_total_size, char const *src_name, + char const *dst_name, bool *normal_copy_required) +{ + bool fail = false; + bool last = false; + char fiemap_buf[4096]; + struct fiemap *fiemap = (struct fiemap *)fiemap_buf; + struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; + uint32_t count = (sizeof (fiemap_buf) - sizeof (*fiemap)) / + sizeof (struct fiemap_extent); + off_t last_ext_logical = 0; + uint64_t last_ext_len = 0; + uint64_t last_read_size = 0; + unsigned int i = 0; + + do + { + fiemap->fm_start = 0ULL; + fiemap->fm_length = FIEMAP_MAX_OFFSET; + fiemap->fm_extent_count = count; + + /* When ioctl(2) fails, fall back to the normal copy only if it + is the first time we met. */ + if (ioctl (src_fd, FS_IOC_FIEMAP, (unsigned long) fiemap) < 0) + { + /* If `i > 0', then at least one ioctl(2) has been performed before. */ + if (i == 0) + *normal_copy_required = true; + return false; + } + + /* If 0 extents are returned, then more ioctls are not needed. */ + if (fiemap->fm_mapped_extents == 0) + break; + + for (i = 0; i < fiemap->fm_mapped_extents; i++) + { + assert (fm_ext[i].fe_logical <= OFF_T_MAX); + + off_t ext_logical = fm_ext[i].fe_logical; + uint64_t ext_len = fm_ext[i].fe_length; + + if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) + { + error (0, errno, _("cannot lseek %s"), quote (src_name)); + return fail; + } + + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0LL) + { + error (0, errno, _("cannot lseek %s"), quote (dst_name)); + return fail; + } + + if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST) + { + last_ext_logical = ext_logical; + last_ext_len = ext_len; + last = true; + } + + while (0 < ext_len) + { + char buf[buf_size]; + + /* Avoid reading into the holes if the left extent + length is shorter than the buffer size. */ + if (ext_len < buf_size) + buf_size = ext_len; + + ssize_t n_read = read (src_fd, buf, buf_size); + if (n_read < 0) + { +#ifdef EINTR + if (errno == EINTR) + continue; +#endif + error (0, errno, _("reading %s"), quote (src_name)); + return fail; + } + + if (n_read == 0) + { + /* Figure out how many bytes read from the last extent. */ + last_read_size = last_ext_len - ext_len; + break; + } + + if (full_write (dest_fd, buf, n_read) != n_read) + { + error (0, errno, _("writing %s"), quote (dst_name)); + return fail; + } + + ext_len -= n_read; + } + + fiemap->fm_start = (fm_ext[i-1].fe_logical + fm_ext[i-1].fe_length); + } + } while (! last); + + /* If a file ends up with holes, the sum of the last extent logical offset + and the read-returned size will be shorter than the actual size of the + file. Use ftruncate to extend the length of the destination file. */ + if (last_ext_logical + last_read_size < src_total_size) + { + if (ftruncate (dest_fd, src_total_size) < 0) + { + error (0, errno, _("extending %s"), quote (dst_name)); + return fail; + } + } + + return ! fail; +} +#else +static bool fiemap_copy_ok (ignored) { errno == ENOTSUP; return false; } +#endif + /* FIXME: describe */ /* FIXME: rewrite this to use a hash table so we avoid the quadratic performance hit that's probably noticeable only on trees deeper @@ -679,6 +813,25 @@ copy_reg (char const *src_name, char const *dst_name, #endif } + if (make_holes) + { + bool require_normal_copy = false; + /* Perform efficient FIEMAP copy for sparse files, fall back to the + standard copy only if the ioctl(2) fails. */ + if (fiemap_copy_ok (source_desc, dest_desc, buf_size, + src_open_sb.st_size, src_name, + dst_name, &require_normal_copy)) + goto preserve_metadata; + else + { + if (! require_normal_copy) + { + return_val = false; + goto close_src_and_dst_desc; + } + } + } + /* If not making a sparse file, try to use a more-efficient buffer size. */ if (! make_holes) @@ -807,6 +960,7 @@ copy_reg (char const *src_name, char const *dst_name, } } +preserve_metadata: if (x->preserve_timestamps) { struct timespec timespec[2]; -- 1.5.4.3 >From 8822b8e3f3ee70b49efb8b8aebff373792956422 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Fri, 7 May 2010 21:31:56 +0800 Subject: [PATCH 3/3] Add test script for cp(1) fiemap copy. Signed-off-by: Jie Liu --- tests/cp/sparse-fiemap | 58 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 58 insertions(+), 0 deletions(-) create mode 100755 tests/cp/sparse-fiemap diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap new file mode 100755 index 0000000..25a8fd6 --- /dev/null +++ b/tests/cp/sparse-fiemap @@ -0,0 +1,58 @@ +#!/bin/sh +# Test cp --sparse=always through fiemap copy + +# Copyright (C) 2006-2010 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +if test "$VERBOSE" = yes; then + set -x + cp --version +fi + +. $srcdir/test-lib.sh + +cp_orig=cp +cp_new="$abs_top_builddir/src/cp" + +test -d "/ext4" \ + && sparse="/ext4/sparse" \ + && normal="/ext4/sparse1" \ + && fiemap="/ext4/sparse2" \ + || skip=1 + +test $skip = 1 && skip_test_ "/ext4 does not exists" + +size=`expr 10 \* 1024` +dd if=/dev/zero bs=4k count=1 seek=$size of=$sparse > /dev/null 2>&1 || framework_failure + +# Using time(1) instead of shell built-in `time' command. +# It support "--format" option which is more convinent to calculate +# the expense time for different `cp' by combine with bc(1) for +# the performance measurement. +TIME=`which time` || skip_test_ "time(1) does not exists" + +x=$(echo "1+2" | bc) +test $x = 3 || skip_test_ "bc(1) does not exists" + +t1=$($TIME -f "%U + %S" $cp_orig --sparse=always $sparse $normal 2>&1 | bc) || fail=1 +t2=$($TIME -f "%U + %S" $cp_new --sparse=always $sparse $fiemap 2>&1 | bc) || fail=1 + +test $fail = 1 && skip_test_ "at least one sparse file copy failed" + +# Ensure that the sparse file copied through fiemap has the same size in bytes as the original. +test `stat --printf %s $sparse` -eq `stat --printf %s $fiemap` || fail=1 +echo "$t2 < $t1" | bc || fail=1 + +Exit $fail -- 1.5.4.3 Best Regards, -Jeff -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Wed May 12 04:49:05 2010 Received: (at submit) by debbugs.gnu.org; 12 May 2010 08:49:07 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OC7cn-0002Y8-CB for submit@debbugs.gnu.org; Wed, 12 May 2010 04:49:05 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OC7ck-0002Xl-UA for submit@debbugs.gnu.org; Wed, 12 May 2010 04:49:04 -0400 Received: from lists.gnu.org ([199.232.76.165]:34063) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OC7ch-0001d8-TM for submit@debbugs.gnu.org; Wed, 12 May 2010 04:49:00 -0400 Received: from [140.186.70.92] (port=48085 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OC7cg-0005WW-5s for bug-coreutils@gnu.org; Wed, 12 May 2010 04:48:59 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OC7cd-0005Da-BX for bug-coreutils@gnu.org; Wed, 12 May 2010 04:48:57 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:39691) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OC7ca-0005Cw-To for bug-coreutils@gnu.org; Wed, 12 May 2010 04:48:55 -0400 Received: from smtp1-g21.free.fr (localhost [127.0.0.1]) by smtp1-g21.free.fr (Postfix) with ESMTP id C4E5994014F; Wed, 12 May 2010 10:48:46 +0200 (CEST) Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP; Wed, 12 May 2010 10:48:45 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id DF970AF3; Wed, 12 May 2010 10:48:44 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4BE41FFF.4020809@oracle.com> (jeff liu's message of "Fri, 07 May 2010 22:13:19 +0800") References: <4BE41FFF.4020809@oracle.com> Date: Wed, 12 May 2010 10:48:44 +0200 Message-ID: <871vdho68j.fsf@meyering.net> Lines: 177 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -3.8 (---) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.1 (-----) jeff.liu wrote: > Hello All, ... > Would you guys please review and consider apply the patches if there is no other issue? Thanks for yet another iteration, and sorry it's taking so long. >>>From f8c78794a70f1fb45a2c61c8bbeca344087287ab Mon Sep 17 00:00:00 2001 > From: Jie Liu > Date: Fri, 7 May 2010 20:48:45 +0800 > Subject: [PATCH 1/3] Add fiemap.h for fiemap ioctl(2) support. > It does not shipped by default, so I copy it from kernel at the moment. > I have update its code style respect to GNU coding style. Your log message should look like this: (one-line summary, then ChangeLog-style after a blank line) add fiemap.h for fiemap ioctl(2) support * src/fiemap.h: Copied from linux's include/linux/fiemap.h, with minor formatting changes. > Signed-off-by: Jie Liu > --- > gnulib | 2 +- Please omit the gnulib part. It is irrelevant to your change. >>>From 8822b8e3f3ee70b49efb8b8aebff373792956422 Mon Sep 17 00:00:00 2001 > From: Jie Liu > Date: Fri, 7 May 2010 21:31:56 +0800 > Subject: [PATCH 3/3] Add test script for cp(1) fiemap copy. > > Signed-off-by: Jie Liu > --- > tests/cp/sparse-fiemap | 58 ++++++++++++++++++++++++++++++++++++++++++++++++ > 1 files changed, 58 insertions(+), 0 deletions(-) > create mode 100755 tests/cp/sparse-fiemap > > diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap > new file mode 100755 > index 0000000..25a8fd6 > --- /dev/null > +++ b/tests/cp/sparse-fiemap > @@ -0,0 +1,58 @@ > +#!/bin/sh > +# Test cp --sparse=always through fiemap copy > + > +# Copyright (C) 2006-2010 Free Software Foundation, Inc. > + > +# This program is free software: you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation, either version 3 of the License, or > +# (at your option) any later version. > + > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > + > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see . > + > +if test "$VERBOSE" = yes; then > + set -x > + cp --version > +fi > + > +. $srcdir/test-lib.sh > + > +cp_orig=cp > +cp_new="$abs_top_builddir/src/cp" > + > +test -d "/ext4" \ > + && sparse="/ext4/sparse" \ > + && normal="/ext4/sparse1" \ > + && fiemap="/ext4/sparse2" \ > + || skip=1 Be careful about writing to well-known files like these. That's risky if /ext4 is world-writable. Work in a temporary directory created by mktemp -d, just in case. It'd be better still to make the test work when run as root, in which case you can create and mount a file-backed ext4 partition, and then use that, rather than relying on the existence of a preexisting, fixed-name directory that may or may not be of the right FS type. But don't worry about this part if you're not inclined. We can make it do that later. > +test $skip = 1 && skip_test_ "/ext4 does not exists" s/exists/exist/ > +size=`expr 10 \* 1024` Here, just use size=10240. In general, please use $(...), not `...`. (multiple places) > +dd if=/dev/zero bs=4k count=1 seek=$size of=$sparse > /dev/null 2>&1 || framework_failure > + > +# Using time(1) instead of shell built-in `time' command. > +# It support "--format" option which is more convinent to calculate > +# the expense time for different `cp' by combine with bc(1) for > +# the performance measurement. > +TIME=`which time` || skip_test_ "time(1) does not exists" Don't use "which". It has portability problems. Instead, env time --version | grep GNU > /dev/null \ || skip_test_ "you lack the GNU time program" Another alternative: record $(date +%s.%N), before and after. Elapsed time should be adequate. Advantage: date is always available, while time may not be. > +x=$(echo "1+2" | bc) > +test $x = 3 || skip_test_ "bc(1) does not exists" > + > +t1=$($TIME -f "%U + %S" $cp_orig --sparse=always $sparse $normal 2>&1 | bc) || fail=1 > +t2=$($TIME -f "%U + %S" $cp_new --sparse=always $sparse $fiemap 2>&1 | bc) || fail=1 > + > +test $fail = 1 && skip_test_ "at least one sparse file copy failed" > + > +# Ensure that the sparse file copied through fiemap has the same size in bytes as the original. > +test `stat --printf %s $sparse` -eq `stat --printf %s $fiemap` || fail=1 s/-eq/=/ > +echo "$t2 < $t1" | bc || fail=1 At first I wrote this: Can't you require something stronger than merely that the new code is faster than the old? How do the times compare in general? Can you construct a worst-case scenario that makes the difference as large as possible, yet that still completes (with the new code) in say 5 or 10 seconds on modern equipment? Use awk rather than bc for comparison. For an example that's similar, I made parted fail a test when an efficiency-sensitive operation takes more than a minute here: http://git.debian.org/?p=parted/parted.git;a=commitdiff;h=eedc6d77dc4b3488decd4dce9cb8cafaa95755ce You can depend on some version of awk being available. Invoke it via $AWK, but for that to work in the test script, you'll have to add AWK=$(AWK) to tests/Makefile.am's TESTS_ENVIRONMENT, as I did here: http://git.debian.org/?p=parted/parted.git;a=commitdiff;h=246c953b53c1bd49b1f835f84a1ca29a6d2fbc1c Then I remembered that here we have timeout(1), so: you may ignore the above and consider this a suggestion to use timeout: But that was in Parted, where we can't guarantee that the timeout program is available. Here in coreutils, you're guaranteed to have timeout(1) (just built), so you might want to use it, too: Contrive a test that takes a very long time without FIEMAP support yet that runs in a couple seconds with it. Then run cp via timeout with a 10-second limit. If timeout's exit status is not 0, then make the test fail. That has the advantage of letting you use an example that would take far longer that we typically want to wait for a non-FIEMAP test. I.e., perform only the FIEMAP-copy and ensure that it's "quick enough". You don't have to perform a non-FIEMAP one. Another advantage: if you don't do the old/slow sparse copy, there's no need for comparison (and bc or awk) at all. From debbugs-submit-bounces@debbugs.gnu.org Thu May 13 10:27:44 2010 Received: (at submit) by debbugs.gnu.org; 13 May 2010 14:27:44 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OCZO3-0000pB-N0 for submit@debbugs.gnu.org; Thu, 13 May 2010 10:27:44 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OCZNl-0000p0-I2 for submit@debbugs.gnu.org; Thu, 13 May 2010 10:27:42 -0400 Received: from lists.gnu.org ([199.232.76.165]:44293) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OCZNe-0001nf-Fh for submit@debbugs.gnu.org; Thu, 13 May 2010 10:27:18 -0400 Received: from [140.186.70.92] (port=39618 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OCZNY-0002zX-Rz for bug-coreutils@gnu.org; Thu, 13 May 2010 10:27:17 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OCZNN-0007RB-Sz for bug-coreutils@gnu.org; Thu, 13 May 2010 10:27:05 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:56047) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OCZNN-0007LY-K8 for bug-coreutils@gnu.org; Thu, 13 May 2010 10:27:01 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4DEQOkH006891 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 13 May 2010 14:26:26 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4DAjAsp011012; Thu, 13 May 2010 14:26:22 GMT Received: from abhmt009.oracle.com by acsmt353.oracle.com with ESMTP id 238375731273760721; Thu, 13 May 2010 07:25:21 -0700 Received: from [123.119.97.139] (/123.119.97.139) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 13 May 2010 07:25:20 -0700 Message-ID: <4BEC0BCC.3080906@oracle.com> Date: Thu, 13 May 2010 22:25:16 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> In-Reply-To: <871vdho68j.fsf@meyering.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090203.4BEC0C14.0081:SCFMA922111,ss=1,pt=113351,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.8 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.0 (------) Hi Jim, Thanks for your kind advise! I'd like to adopt the timeout(1) approach for the test work. My thought is: 1. Create and mount a file-backed ext4 partition rather than relying on the HARD CODE path. 2. Create a 2gb sparse file without extent allocated for it. 3. It take nearly 30 seconds to transfer this file in normal copy, yet less than 1 second through FIEMAP-copy, is it a worst-case scenario that makes the difference as large as possible? 4. run FIEMAP-copy, use timeout(1) to limit it will complete in 1 second, I hope I understood your opinion correctly ;). The revised patches are shown as following: >From b683f930c5e70481c2b6e000a626734f975b99ac Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Thu, 13 May 2010 22:09:30 +0800 Subject: [PATCH 1/1] cp: Add FIEMAP support for efficient sparse file copy * src/fiemap.h: Add fiemap.h for fiemap ioctl(2) support. Copied from linux's include/linux/fiemap.h, with minor formatting changes. * src/copy.c (copy_reg): Now, when `cp' invoked with --sparse=[WHEN] option, we will try to do FIEMAP-copy if the underlaying file system support it, fall back to a normal copy if it fails. Signed-off-by: Jie Liu --- src/copy.c | 154 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/fiemap.h | 102 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 256 insertions(+), 0 deletions(-) create mode 100644 src/fiemap.h diff --git a/src/copy.c b/src/copy.c index c16cef6..960e5fb 100644 --- a/src/copy.c +++ b/src/copy.c @@ -63,6 +63,10 @@ #include +#ifndef HAVE_FIEMAP +# include "fiemap.h" +#endif + #ifndef HAVE_FCHOWN # define HAVE_FCHOWN false # define fchown(fd, uid, gid) (-1) @@ -149,6 +153,136 @@ clone_file (int dest_fd, int src_fd) #endif } +#ifdef __linux__ +# ifndef FS_IOC_FIEMAP +# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) +# endif +/* Perform FIEMAP(available in mainline 2.6.27) copy if possible. + Call ioctl(2) with FS_IOC_FIEMAP to efficiently map file allocation + excepts holes. So the overhead to deal with holes with lseek(2) in + normal copy could be saved. This would result in much faster backups + for any kind of sparse file. */ +static bool +fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, + off_t src_total_size, char const *src_name, + char const *dst_name, bool *normal_copy_required) +{ + bool fail = false; + bool last = false; + char fiemap_buf[4096]; + struct fiemap *fiemap = (struct fiemap *)fiemap_buf; + struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; + uint32_t count = (sizeof (fiemap_buf) - sizeof (*fiemap)) / + sizeof (struct fiemap_extent); + off_t last_ext_logical = 0; + uint64_t last_ext_len = 0; + uint64_t last_read_size = 0; + unsigned int i = 0; + + do + { + fiemap->fm_start = 0ULL; + fiemap->fm_length = FIEMAP_MAX_OFFSET; + fiemap->fm_extent_count = count; + + /* When ioctl(2) fails, fall back to the normal copy only if it + is the first time we met. */ + if (ioctl (src_fd, FS_IOC_FIEMAP, (unsigned long) fiemap) < 0) + { + /* If `i > 0', then at least one ioctl(2) has been performed before. */ + if (i == 0) + *normal_copy_required = true; + return false; + } + + /* If 0 extents are returned, then more ioctls are not needed. */ + if (fiemap->fm_mapped_extents == 0) + break; + + for (i = 0; i < fiemap->fm_mapped_extents; i++) + { + assert (fm_ext[i].fe_logical <= OFF_T_MAX); + + off_t ext_logical = fm_ext[i].fe_logical; + uint64_t ext_len = fm_ext[i].fe_length; + + if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) + { + error (0, errno, _("cannot lseek %s"), quote (src_name)); + return fail; + } + + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0LL) + { + error (0, errno, _("cannot lseek %s"), quote (dst_name)); + return fail; + } + + if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST) + { + last_ext_logical = ext_logical; + last_ext_len = ext_len; + last = true; + } + + while (0 < ext_len) + { + char buf[buf_size]; + + /* Avoid reading into the holes if the left extent + length is shorter than the buffer size. */ + if (ext_len < buf_size) + buf_size = ext_len; + + ssize_t n_read = read (src_fd, buf, buf_size); + if (n_read < 0) + { +#ifdef EINTR + if (errno == EINTR) + continue; +#endif + error (0, errno, _("reading %s"), quote (src_name)); + return fail; + } + + if (n_read == 0) + { + /* Figure out how many bytes read from the last extent. */ + last_read_size = last_ext_len - ext_len; + break; + } + + if (full_write (dest_fd, buf, n_read) != n_read) + { + error (0, errno, _("writing %s"), quote (dst_name)); + return fail; + } + + ext_len -= n_read; + } + + fiemap->fm_start = (fm_ext[i-1].fe_logical + fm_ext[i-1].fe_length); + } + } while (! last); + + /* If a file ends up with holes, the sum of the last extent logical offset + and the read-returned size will be shorter than the actual size of the + file. Use ftruncate to extend the length of the destination file. */ + if (last_ext_logical + last_read_size < src_total_size) + { + if (ftruncate (dest_fd, src_total_size) < 0) + { + error (0, errno, _("extending %s"), quote (dst_name)); + return fail; + } + } + + return ! fail; +} +#else +static bool fiemap_copy_ok (ignored) { errno == ENOTSUP; return false; } +#endif + /* FIXME: describe */ /* FIXME: rewrite this to use a hash table so we avoid the quadratic performance hit that's probably noticeable only on trees deeper @@ -679,6 +813,25 @@ copy_reg (char const *src_name, char const *dst_name, #endif } + if (make_holes) + { + bool require_normal_copy = false; + /* Perform efficient FIEMAP copy for sparse files, fall back to the + standard copy only if the ioctl(2) fails. */ + if (fiemap_copy_ok (source_desc, dest_desc, buf_size, + src_open_sb.st_size, src_name, + dst_name, &require_normal_copy)) + goto preserve_metadata; + else + { + if (! require_normal_copy) + { + return_val = false; + goto close_src_and_dst_desc; + } + } + } + /* If not making a sparse file, try to use a more-efficient buffer size. */ if (! make_holes) @@ -807,6 +960,7 @@ copy_reg (char const *src_name, char const *dst_name, } } +preserve_metadata: if (x->preserve_timestamps) { struct timespec timespec[2]; diff --git a/src/fiemap.h b/src/fiemap.h new file mode 100644 index 0000000..d33293b --- /dev/null +++ b/src/fiemap.h @@ -0,0 +1,102 @@ +/* FS_IOC_FIEMAP ioctl infrastructure. + Some portions copyright (C) 2007 Cluster File Systems, Inc + Authors: Mark Fasheh + Kalpak Shah + Andreas Dilger . */ + +/* Copy from kernel, modified to respect GNU code style by Jie Liu. */ + +#ifndef _LINUX_FIEMAP_H +# define _LINUX_FIEMAP_H + +# include + +struct fiemap_extent +{ + /* Logical offset in bytes for the start of the extent + from the beginning of the file. */ + uint64_t fe_logical; + + /* Physical offset in bytes for the start of the extent + from the beginning of the disk. */ + uint64_t fe_physical; + + /* Length in bytes for this extent. */ + uint64_t fe_length; + + uint64_t fe_reserved64[2]; + + /* FIEMAP_EXTENT_* flags for this extent. */ + uint32_t fe_flags; + + uint32_t fe_reserved[3]; +}; + +struct fiemap +{ + /* Logical offset(inclusive) at which to start mapping(in). */ + uint64_t fm_start; + + /* Logical length of mapping which userspace wants(in). */ + uint64_t fm_length; + + /* FIEMAP_FLAG_* flags for request(in/out). */ + uint32_t fm_flags; + + /* Number of extents that were mapped(out). */ + uint32_t fm_mapped_extents; + + /* Size of fm_extents array(in). */ + uint32_t fm_extent_count; + + uint32_t fm_reserved; + + /* Array of mapped extents(out). */ + struct fiemap_extent fm_extents[0]; +}; + +/* The maximum offset can be mapped for a file. */ +# define FIEMAP_MAX_OFFSET (~0ULL) + +/* Sync file data before map. */ +# define FIEMAP_FLAG_SYNC 0x00000001 + +/* Map extented attribute tree. */ +# define FIEMAP_FLAG_XATTR 0x00000002 + +# define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR) + +/* Last extent in file. */ +# define FIEMAP_EXTENT_LAST 0x00000001 + +/* Data location unknown. */ +# define FIEMAP_EXTENT_UNKNOWN 0x00000002 + +/* Location still pending, Sets EXTENT_UNKNOWN. */ +# define FIEMAP_EXTENT_DELALLOC 0x00000004 + +/* Data can not be read while fs is unmounted. */ +# define FIEMAP_EXTENT_ENCODED 0x00000008 + +/* Data is encrypted by fs. Sets EXTENT_NO_BYPASS. */ +# define FIEMAP_EXTENT_DATA_ENCRYPTED 0x00000080 + +/* Extent offsets may not be block aligned. */ +# define FIEMAP_EXTENT_NOT_ALIGNED 0x00000100 + +/* Data mixed with metadata. Sets EXTENT_NOT_ALIGNED. */ +# define FIEMAP_EXTENT_DATA_INLINE 0x00000200 + +/* Multiple files in block. Set EXTENT_NOT_ALIGNED. */ +# define FIEMAP_EXTENT_DATA_TAIL 0x00000400 + +/* Space allocated, but not data (i.e. zero). */ +# define FIEMAP_EXTENT_UNWRITTEN 0x00000800 + +/* File does not natively support extents. Result merged for efficiency. */ +# define FIEMAP_EXTENT_MERGED 0x00001000 + +/* Space shared with other files. */ +# define FIEMAP_EXTENT_SHARED 0x00002000 + +#endif -- 1.5.4.3 >From f18e1801d1dfca9fa278572b8172a5f97da2adc1 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Thu, 13 May 2010 22:17:53 +0800 Subject: [PATCH 1/1] tests: add a new test for FIEMAP-copy * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a loopbacked ext4 partition. * tests/Makefile.am (sparse-fiemap): Reference the new test. Signed-off-by: Jie Liu --- tests/Makefile.am | 2 + tests/cp/sparse-fiemap | 61 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+), 0 deletions(-) create mode 100644 tests/cp/sparse-fiemap diff --git a/tests/Makefile.am b/tests/Makefile.am index 46d388a..a76c6a7 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -25,6 +25,7 @@ root_tests = \ cp/special-bits \ cp/cp-mv-enotsup-xattr \ cp/capability \ + cp/sparse-fiemap \ dd/skip-seek-past-dev \ install/install-C-root \ ls/capability \ @@ -319,6 +320,7 @@ TESTS = \ cp/same-file \ cp/slink-2-slink \ cp/sparse \ + cp/sparse-fiemap \ cp/special-f \ cp/src-base-dot \ cp/symlink-slash \ diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap new file mode 100644 index 0000000..f9d3a94 --- /dev/null +++ b/tests/cp/sparse-fiemap @@ -0,0 +1,61 @@ +#!/bin/sh +# Test cp --sparse=always through fiemap copy + +# Copyright (C) 2006-2010 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +if test "$VERBOSE" = yes; then + set -x + cp --version +fi + +. $srcdir/test-lib.sh +require_root_ + +cwd=`pwd` +cleanup_() { cd /; umount "$cwd/mnt"; } + +# Create an ext4 loopback file system +dd if=/dev/zero of=blob bs=8192 count=1000 > /dev/null 2>&1 \ + || skip=1 +mkdir mnt +mkfs -t ext4 -F blob || + skip_test_ "failed to create ext4 file system" +mount -oloop blob mnt || skip=1 +echo test > mnt/f || skip=1 +test -s mnt/f || skip=1 + +test $skip = 1 && + skip_test_ "insufficient mount/ext4 support" + +rm -f mnt/f + +# Create a 2gb sparse file +dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=2096128 > /dev/null 2>&1 || framework_failure + +# It take more than 20 seconds to transfer the created sparse file +# through normal copy, by contrast, it take even less than 1 second +# through FIEMAP-copy. +timeout 1 cp --sparse=always mnt/sparse mnt/sparse_fiemap || fail=1 +test $? = 124 && fail=1 + +# Ensure that the sparse file copied through fiemap has the same size +# in bytes as the original. +test `stat --printf %s $sparse` = `stat --printf %s $fiemap` || fail=1 + +rm -f mnt/sparse +rm -f mnt/sparse_fiemap + +Exit $fail -- 1.5.4.3 Thanks, -Jeff > > Then I remembered that here we have timeout(1), so: > you may ignore the above and consider this a suggestion > to use timeout: > > But that was in Parted, where we can't guarantee that the timeout > program is available. Here in coreutils, you're guaranteed to > have timeout(1) (just built), so you might want to use it, too: > Contrive a test that takes a very long time without FIEMAP support > yet that runs in a couple seconds with it. Then run cp via timeout > with a 10-second limit. If timeout's exit status is not 0, > then make the test fail. > > That has the advantage of letting you use an example that would take > far longer that we typically want to wait for a non-FIEMAP test. > I.e., perform only the FIEMAP-copy and ensure that it's "quick enough". > You don't have to perform a non-FIEMAP one. > > Another advantage: if you don't do the old/slow sparse copy, > there's no need for comparison (and bc or awk) at all. > > > -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Thu May 20 15:24:19 2010 Received: (at submit) by debbugs.gnu.org; 20 May 2010 19:24:20 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFBLt-00006Y-Nu for submit@debbugs.gnu.org; Thu, 20 May 2010 15:24:18 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFBLr-00006T-HU for submit@debbugs.gnu.org; Thu, 20 May 2010 15:24:16 -0400 Received: from lists.gnu.org ([199.232.76.165]:41011) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OFBLm-00044b-Sw for submit@debbugs.gnu.org; Thu, 20 May 2010 15:24:10 -0400 Received: from [140.186.70.92] (port=58084 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OFBLj-0007tE-VW for bug-coreutils@gnu.org; Thu, 20 May 2010 15:24:10 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OFBLg-0006O7-Ag for bug-coreutils@gnu.org; Thu, 20 May 2010 15:24:07 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:42675) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFBLf-0006NW-M1 for bug-coreutils@gnu.org; Thu, 20 May 2010 15:24:04 -0400 Received: from smtp1-g21.free.fr (localhost [127.0.0.1]) by smtp1-g21.free.fr (Postfix) with ESMTP id 40C70940093; Thu, 20 May 2010 21:23:56 +0200 (CEST) Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP; Thu, 20 May 2010 21:23:55 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 30A42A26; Thu, 20 May 2010 21:23:55 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4BEC0BCC.3080906@oracle.com> (jeff liu's message of "Thu, 13 May 2010 22:25:16 +0800") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> Date: Thu, 20 May 2010 21:23:55 +0200 Message-ID: <87k4qyl6lw.fsf@meyering.net> Lines: 260 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -3.8 (---) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.1 (-----) jeff.liu wrote: > Hi Jim, > > Thanks for your kind advise! > > I'd like to adopt the timeout(1) approach for the test work. > > My thought is: > 1. Create and mount a file-backed ext4 partition rather than relying on the HARD CODE path. > 2. Create a 2gb sparse file without extent allocated for it. > 3. It take nearly 30 seconds to transfer this file in normal copy, yet less than 1 second through > FIEMAP-copy, is it a worst-case scenario that makes the difference as large as possible? > 4. run FIEMAP-copy, use timeout(1) to limit it will complete in 1 second, I hope I understood your > opinion correctly ;). > > The revised patches are shown as following: > >>>From f18e1801d1dfca9fa278572b8172a5f97da2adc1 Mon Sep 17 00:00:00 2001 > From: Jie Liu > Date: Thu, 13 May 2010 22:17:53 +0800 > Subject: [PATCH 1/1] tests: add a new test for FIEMAP-copy > > * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a > loopbacked ext4 partition. > * tests/Makefile.am (sparse-fiemap): Reference the new test. > > Signed-off-by: Jie Liu > --- > tests/Makefile.am | 2 + > tests/cp/sparse-fiemap | 61 ++++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 63 insertions(+), 0 deletions(-) > create mode 100644 tests/cp/sparse-fiemap > > diff --git a/tests/Makefile.am b/tests/Makefile.am > index 46d388a..a76c6a7 100644 > --- a/tests/Makefile.am > +++ b/tests/Makefile.am > @@ -25,6 +25,7 @@ root_tests = \ > cp/special-bits \ > cp/cp-mv-enotsup-xattr \ > cp/capability \ > + cp/sparse-fiemap \ > dd/skip-seek-past-dev \ > install/install-C-root \ > ls/capability \ > @@ -319,6 +320,7 @@ TESTS = \ > cp/same-file \ > cp/slink-2-slink \ > cp/sparse \ > + cp/sparse-fiemap \ > cp/special-f \ > cp/src-base-dot \ > cp/symlink-slash \ I've applied your patches locally and have begun adjusting them. First, I removed the addition of cp/sparse-fiemap to the TESTS list above. Adding it to the root_tests is sufficient. Then, I've made the following changes to your test script: - the original size of your test file of 2GiB was too small, in that the old (pre-fiemap) cp copied it for me in less than 1 second when the backing file was on a tmpfs file system. I've made the new size be 2TiB. The fiemap copy is still so quick that it completes in < .01 second.[*] - no point in discarding stdout/stderr, since it all goes to the log - raised timeout to 10 seconds to give more leeway on slow systems - remove those "rm -f" uses. They're not needed, since the test is run in its own temp dir, which is removed automatically when done. - remove the $? = 124 test -- the preceding test for success is sufficient [*] I tried to count syscalls with strace but got a segfault. Using valgrind I get errors, so debugged enough to get a clean run, but possibly at the expense of correctness. We'll need more tests to ensure that the non-sparse blocks in the copy all have the same offset/length as in the original. Details below. diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap old mode 100644 new mode 100755 index f9d3a94..814d537 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -28,8 +28,7 @@ cwd=`pwd` cleanup_() { cd /; umount "$cwd/mnt"; } # Create an ext4 loopback file system -dd if=/dev/zero of=blob bs=8192 count=1000 > /dev/null 2>&1 \ - || skip=1 +dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 mkdir mnt mkfs -t ext4 -F blob || skip_test_ "failed to create ext4 file system" @@ -42,20 +41,15 @@ test $skip = 1 && rm -f mnt/f -# Create a 2gb sparse file -dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=2096128 > /dev/null 2>&1 || framework_failure +# Create a 2TiB sparse file +dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=2G || framework_failure -# It take more than 20 seconds to transfer the created sparse file -# through normal copy, by contrast, it take even less than 1 second -# through FIEMAP-copy. -timeout 1 cp --sparse=always mnt/sparse mnt/sparse_fiemap || fail=1 -test $? = 124 && fail=1 +# It takes many minutes to copy this sparse file using the old method. +# By contrast, it takes far less than 1 second using FIEMAP-copy. +timeout 10 cp --sparse=always mnt/sparse mnt/sparse_fiemap || fail=1 # Ensure that the sparse file copied through fiemap has the same size # in bytes as the original. test `stat --printf %s $sparse` = `stat --printf %s $fiemap` || fail=1 -rm -f mnt/sparse -rm -f mnt/sparse_fiemap - Exit $fail ---------------------------------------- On F13, x86_64, ext4, I did this: dd if=/dev/null of=big bs=1 seek=2G valgrind ./cp --sparse=always big big2 ==4771== Conditional jump or move depends on uninitialised value(s) ==4771== at 0x40465A: fiemap_copy_ok (copy.c:205) ==4771== by 0x405B61: copy_reg (copy.c:822) ==4771== by 0x408713: copy_internal (copy.c:2163) ==4771== by 0x409237: copy (copy.c:2449) ==4771== by 0x403AC9: do_copy (cp.c:754) ==4771== by 0x4041E4: main (cp.c:1154) ==4771== ==4771== Syscall param lseek(offset) contains uninitialised byte(s) ==4771== at 0x3269CE1540: __lseek_nocancel (syscall-template.S:82) ==4771== by 0x4046D4: fiemap_copy_ok (copy.c:210) ==4771== by 0x405B61: copy_reg (copy.c:822) ==4771== by 0x408713: copy_internal (copy.c:2163) ==4771== by 0x409237: copy (copy.c:2449) ==4771== by 0x403AC9: do_copy (cp.c:754) ==4771== by 0x4041E4: main (cp.c:1154) ==4771== ==4771== Syscall param lseek(offset) contains uninitialised byte(s) ==4771== at 0x3269CE1540: __lseek_nocancel (syscall-template.S:82) ==4771== by 0x40472D: fiemap_copy_ok (copy.c:216) ==4771== by 0x405B61: copy_reg (copy.c:822) ==4771== by 0x408713: copy_internal (copy.c:2163) ==4771== by 0x409237: copy (copy.c:2449) ==4771== by 0x403AC9: do_copy (cp.c:754) ==4771== by 0x4041E4: main (cp.c:1154) ==4771== ==4771== Conditional jump or move depends on uninitialised value(s) ==4771== at 0x404792: fiemap_copy_ok (copy.c:222) ==4771== by 0x405B61: copy_reg (copy.c:822) ==4771== by 0x408713: copy_internal (copy.c:2163) ==4771== by 0x409237: copy (copy.c:2449) ==4771== by 0x403AC9: do_copy (cp.c:754) ==4771== by 0x4041E4: main (cp.c:1154) ==4771== ==4771== Conditional jump or move depends on uninitialised value(s) ==4771== at 0x40492B: fiemap_copy_ok (copy.c:229) ==4771== by 0x405B61: copy_reg (copy.c:822) ==4771== by 0x408713: copy_internal (copy.c:2163) ==4771== by 0x409237: copy (copy.c:2449) ==4771== by 0x403AC9: do_copy (cp.c:754) ==4771== by 0x4041E4: main (cp.c:1154) ==4771== ==4771== Conditional jump or move depends on uninitialised value(s) ==4771== at 0x4047FA: fiemap_copy_ok (copy.c:235) ==4771== by 0x405B61: copy_reg (copy.c:822) ==4771== by 0x408713: copy_internal (copy.c:2163) ==4771== by 0x409237: copy (copy.c:2449) ==4771== by 0x403AC9: do_copy (cp.c:754) ==4771== by 0x4041E4: main (cp.c:1154) ==4771== ==4771== Syscall param read(count) contains uninitialised byte(s) ==4771== at 0x3269CD41B0: __read_nocancel (syscall-template.S:82) ==4771== by 0x404821: fiemap_copy_ok (copy.c:238) ==4771== by 0x405B61: copy_reg (copy.c:822) ==4771== by 0x408713: copy_internal (copy.c:2163) ==4771== by 0x409237: copy (copy.c:2449) ==4771== by 0x403AC9: do_copy (cp.c:754) ==4771== by 0x4041E4: main (cp.c:1154) ==4771== ==4771== Invalid read of size 8 ==4771== at 0x404952: fiemap_copy_ok (copy.c:265) ==4771== by 0x405B61: copy_reg (copy.c:822) ==4771== by 0x408713: copy_internal (copy.c:2163) ==4771== by 0x409237: copy (copy.c:2449) ==4771== by 0x403AC9: do_copy (cp.c:754) ==4771== by 0x4041E4: main (cp.c:1154) ==4771== Address 0x3ffeffdd68 is not stack'd, malloc'd or (recently) free'd ==4771== ==4771== ==4771== Process terminating with default action of signal 11 (SIGSEGV) ==4771== Access not within mapped region at address 0x3FFEFFDD68 ==4771== at 0x404952: fiemap_copy_ok (copy.c:265) ==4771== by 0x405B61: copy_reg (copy.c:822) ==4771== by 0x408713: copy_internal (copy.c:2163) ==4771== by 0x409237: copy (copy.c:2449) ==4771== by 0x403AC9: do_copy (cp.c:754) ==4771== by 0x4041E4: main (cp.c:1154) =========================================================== The segv just above is due to hitting this line with i==0: fiemap->fm_start = (fm_ext[i-1].fe_logical + fm_ext[i-1].fe_length); the obvious fix is probably to do this instead: fiemap->fm_start = (fm_ext[i].fe_logical + fm_ext[i].fe_length); All of the used-uninitialized errors can be papered over by clearing the fiemap_buf array, like this: + memset (fiemap_buf, 0, sizeof fiemap_buf); do { fiemap->fm_start = 0ULL; However, if these are all due solely to F13's valgrind not yet knowing the semantics of the FIEMAP ioctl, then that may be adequate. Bottom line: - you may consider your test-script patch accepted, with the patch above - I'd like to see a new version of the copy.c-changing patch, including at least a fix for the fm_ext[-1] access bug. =========================================================== Solely for reference, here's the copy.c patch I used to avoid the valgrind-spotted problems: diff --git a/src/copy.c b/src/copy.c index 960e5fb..e232eaa 100644 --- a/src/copy.c +++ b/src/copy.c @@ -179,6 +179,7 @@ fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, uint64_t last_read_size = 0; unsigned int i = 0; + memset (fiemap_buf, 0, sizeof fiemap_buf); do { fiemap->fm_start = 0ULL; @@ -187,7 +188,7 @@ fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, /* When ioctl(2) fails, fall back to the normal copy only if it is the first time we met. */ - if (ioctl (src_fd, FS_IOC_FIEMAP, (unsigned long) fiemap) < 0) + if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) { /* If `i > 0', then at least one ioctl(2) has been performed before. */ if (i == 0) @@ -261,7 +262,7 @@ fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, ext_len -= n_read; } - fiemap->fm_start = (fm_ext[i-1].fe_logical + fm_ext[i-1].fe_length); + fiemap->fm_start = (fm_ext[i].fe_logical + fm_ext[i].fe_length); } } while (! last); From debbugs-submit-bounces@debbugs.gnu.org Fri May 21 08:07:28 2010 Received: (at submit) by debbugs.gnu.org; 21 May 2010 12:07:28 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFR0h-0000rw-KV for submit@debbugs.gnu.org; Fri, 21 May 2010 08:07:28 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFR0f-0000rm-HP for submit@debbugs.gnu.org; Fri, 21 May 2010 08:07:26 -0400 Received: from lists.gnu.org ([199.232.76.165]:42089) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OFR0a-0000EA-1n for submit@debbugs.gnu.org; Fri, 21 May 2010 08:07:20 -0400 Received: from [140.186.70.92] (port=32912 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OFR0P-0003SK-3F for bug-coreutils@gnu.org; Fri, 21 May 2010 08:07:19 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OFR0D-0007qI-5w for bug-coreutils@gnu.org; Fri, 21 May 2010 08:07:08 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:9428) by eggs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1OFR0C-0007p8-Ig for bug-coreutils@gnu.org; Fri, 21 May 2010 08:06:57 -0400 Received: (qmail 61295 invoked from network); 21 May 2010 12:06:54 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 21 May 2010 12:06:54 -0000 Message-ID: <4BF676CC.8040806@draigBrady.com> Date: Fri, 21 May 2010 13:04:28 +0100 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> In-Reply-To: <87k4qyl6lw.fsf@meyering.net> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9 X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.8 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Chris Mason , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.8 (----) On 13/05/10 15:25, jeff.liu wrote: > > diff --git a/src/copy.c b/src/copy.c > index c16cef6..960e5fb 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -63,6 +63,10 @@ > > #include > > +#ifndef HAVE_FIEMAP > +# include "fiemap.h" > +#endif Is HAVE_FIEMAP ever defined anywhere? In future will we use this to check for ? On 20/05/10 20:23, Jim Meyering wrote: > > diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap > old mode 100644 > new mode 100755 > index f9d3a94..814d537 > --- a/tests/cp/sparse-fiemap > +++ b/tests/cp/sparse-fiemap > @@ -28,8 +28,7 @@ cwd=`pwd` > cleanup_() { cd /; umount "$cwd/mnt"; } > > # Create an ext4 loopback file system > -dd if=/dev/zero of=blob bs=8192 count=1000 > /dev/null 2>&1 \ > - || skip=1 > +dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 > mkdir mnt > mkfs -t ext4 -F blob || > skip_test_ "failed to create ext4 file system" There is the unlikely combination of ext4 without fiemap support I think? If so then that dependency is worth a comment. > @@ -42,20 +41,15 @@ test $skip = 1 && > > rm -f mnt/f > > -# Create a 2gb sparse file > -dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=2096128 > /dev/null 2>&1 || framework_failure > +# Create a 2TiB sparse file > +dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=2G || framework_failure If we don't need any actual data in the files then one could use: truncate -s 2TB mnt/sparse For my reference, I used TB rather than TiB because on ext3 the limit is 0x1FEFF7FC000 (2194719883264) (0x1FF7FFFD000 (2196875759616) before 2.6.25) cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Fri May 21 09:00:20 2010 Received: (at submit) by debbugs.gnu.org; 21 May 2010 13:00:20 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFRpr-0001Gv-6P for submit@debbugs.gnu.org; Fri, 21 May 2010 09:00:20 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFRpl-0001Gl-GG for submit@debbugs.gnu.org; Fri, 21 May 2010 09:00:18 -0400 Received: from lists.gnu.org ([199.232.76.165]:50333) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OFRpd-0001Xg-1d for submit@debbugs.gnu.org; Fri, 21 May 2010 09:00:05 -0400 Received: from [140.186.70.92] (port=55324 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OFRpT-0005aw-4o for bug-coreutils@gnu.org; Fri, 21 May 2010 09:00:04 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OFRpJ-0006VU-RO for bug-coreutils@gnu.org; Fri, 21 May 2010 08:59:47 -0400 Received: from smtp5-g21.free.fr ([212.27.42.5]:51685) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFRpJ-0006Uz-9a for bug-coreutils@gnu.org; Fri, 21 May 2010 08:59:45 -0400 Received: from smtp5-g21.free.fr (localhost [127.0.0.1]) by smtp5-g21.free.fr (Postfix) with ESMTP id 2A450D4803E; Fri, 21 May 2010 14:59:37 +0200 (CEST) Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp5-g21.free.fr (Postfix) with ESMTP; Fri, 21 May 2010 14:59:36 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 126CFDA6; Fri, 21 May 2010 14:59:36 +0200 (CEST) From: Jim Meyering To: =?utf-8?Q?P=C3=A1draig?= Brady Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4BF676CC.8040806@draigBrady.com> (=?utf-8?Q?=22P=C3=A1draig?= Brady"'s message of "Fri, 21 May 2010 13:04:28 +0100") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF676CC.8040806@draigBrady.com> Date: Fri, 21 May 2010 14:59:36 +0200 Message-ID: <878w7dif5z.fsf@meyering.net> Lines: 67 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.1 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Chris Mason , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.1 (-----) P=C3=A1draig Brady wrote: > On 13/05/10 15:25, jeff.liu wrote: >> >> diff --git a/src/copy.c b/src/copy.c >> index c16cef6..960e5fb 100644 >> --- a/src/copy.c >> +++ b/src/copy.c >> @@ -63,6 +63,10 @@ >> >> #include >> >> +#ifndef HAVE_FIEMAP >> +# include "fiemap.h" >> +#endif > > Is HAVE_FIEMAP ever defined anywhere? > In future will we use this to check for ? I'll look. > On 20/05/10 20:23, Jim Meyering wrote: >> >> diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap >> old mode 100644 >> new mode 100755 >> index f9d3a94..814d537 >> --- a/tests/cp/sparse-fiemap >> +++ b/tests/cp/sparse-fiemap >> @@ -28,8 +28,7 @@ cwd=3D`pwd` >> cleanup_() { cd /; umount "$cwd/mnt"; } >> >> # Create an ext4 loopback file system >> -dd if=3D/dev/zero of=3Dblob bs=3D8192 count=3D1000 > /dev/null 2>&1 \ >> - || skip=3D1 >> +dd if=3D/dev/zero of=3Dblob bs=3D8192 count=3D1000 || skip=3D1 >> mkdir mnt >> mkfs -t ext4 -F blob || >> skip_test_ "failed to create ext4 file system" > > There is the unlikely combination of ext4 without fiemap support I think? > If so then that dependency is worth a comment. I don't know off hand. Is there a shell-level way to test for that? >> @@ -42,20 +41,15 @@ test $skip =3D 1 && >> >> rm -f mnt/f >> >> -# Create a 2gb sparse file >> -dd if=3D/dev/zero of=3Dmnt/sparse bs=3D1k count=3D1 seek=3D2096128 > /d= ev/null 2>&1 || framework_failure >> +# Create a 2TiB sparse file >> +dd if=3D/dev/zero of=3Dmnt/sparse bs=3D1k count=3D1 seek=3D2G || framew= ork_failure > > If we don't need any actual data in the files then one could use: > truncate -s 2TB mnt/sparse > > For my reference, I used TB rather than TiB because on ext3 > the limit is 0x1FEFF7FC000 (2194719883264) > (0x1FF7FFFD000 (2196875759616) before 2.6.25) Thanks. For now I'll limit it to 1GiB using dd. That gives a slightly less uniform input. # Create a 1TiB sparse file dd if=3D/dev/zero of=3Dmnt/sparse bs=3D1k count=3D1 seek=3D1G || framewor= k_failure From debbugs-submit-bounces@debbugs.gnu.org Fri May 21 09:02:02 2010 Received: (at submit) by debbugs.gnu.org; 21 May 2010 13:02:02 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFRrU-0001Hg-9l for submit@debbugs.gnu.org; Fri, 21 May 2010 09:02:01 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFRrS-0001Hb-6k for submit@debbugs.gnu.org; Fri, 21 May 2010 09:01:59 -0400 Received: from lists.gnu.org ([199.232.76.165]:46154) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OFRrM-0001bf-WC for submit@debbugs.gnu.org; Fri, 21 May 2010 09:01:53 -0400 Received: from [140.186.70.92] (port=55749 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OFRrJ-00060H-Fl for bug-coreutils@gnu.org; Fri, 21 May 2010 09:01:52 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OFRrE-0006oi-RW for bug-coreutils@gnu.org; Fri, 21 May 2010 09:01:49 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:44559) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFRrE-0006oI-It for bug-coreutils@gnu.org; Fri, 21 May 2010 09:01:44 -0400 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4LD1bNK013265 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 21 May 2010 13:01:38 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4L3adMl025947; Fri, 21 May 2010 13:01:33 GMT Received: from abhmt010.oracle.com by acsmt353.oracle.com with ESMTP id 257360961274446796; Fri, 21 May 2010 05:59:56 -0700 Received: from [221.223.108.50] (/221.223.108.50) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 21 May 2010 05:59:55 -0700 Message-ID: <4BF683CC.3060407@oracle.com> Date: Fri, 21 May 2010 20:59:56 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> In-Reply-To: <87k4qyl6lw.fsf@meyering.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet13.oracle.com [148.87.113.125] X-CT-RefId: str=0001.0A090204.4BF68433.00C4:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.9 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.0 (-----) Jim Meyering wrote: > jeff.liu wrote: >> Hi Jim, >> >> Thanks for your kind advise! >> >> I'd like to adopt the timeout(1) approach for the test work. >> >> My thought is: >> 1. Create and mount a file-backed ext4 partition rather than relying on the HARD CODE path. >> 2. Create a 2gb sparse file without extent allocated for it. >> 3. It take nearly 30 seconds to transfer this file in normal copy, yet less than 1 second through >> FIEMAP-copy, is it a worst-case scenario that makes the difference as large as possible? >> 4. run FIEMAP-copy, use timeout(1) to limit it will complete in 1 second, I hope I understood your >> opinion correctly ;). >> >> The revised patches are shown as following: >> >> >From f18e1801d1dfca9fa278572b8172a5f97da2adc1 Mon Sep 17 00:00:00 2001 >> From: Jie Liu >> Date: Thu, 13 May 2010 22:17:53 +0800 >> Subject: [PATCH 1/1] tests: add a new test for FIEMAP-copy >> >> * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a >> loopbacked ext4 partition. >> * tests/Makefile.am (sparse-fiemap): Reference the new test. >> >> Signed-off-by: Jie Liu >> --- >> tests/Makefile.am | 2 + >> tests/cp/sparse-fiemap | 61 ++++++++++++++++++++++++++++++++++++++++++++++++ >> 2 files changed, 63 insertions(+), 0 deletions(-) >> create mode 100644 tests/cp/sparse-fiemap >> >> diff --git a/tests/Makefile.am b/tests/Makefile.am >> index 46d388a..a76c6a7 100644 >> --- a/tests/Makefile.am >> +++ b/tests/Makefile.am >> @@ -25,6 +25,7 @@ root_tests = \ >> cp/special-bits \ >> cp/cp-mv-enotsup-xattr \ >> cp/capability \ >> + cp/sparse-fiemap \ >> dd/skip-seek-past-dev \ >> install/install-C-root \ >> ls/capability \ >> @@ -319,6 +320,7 @@ TESTS = \ >> cp/same-file \ >> cp/slink-2-slink \ >> cp/sparse \ >> + cp/sparse-fiemap \ >> cp/special-f \ >> cp/src-base-dot \ >> cp/symlink-slash \ > > I've applied your patches locally and have begun adjusting them. > First, I removed the addition of cp/sparse-fiemap to the TESTS list above. > Adding it to the root_tests is sufficient. Thank you to point it out. > Then, I've made the following changes to your test script: > - the original size of your test file of 2GiB was too small, > in that the old (pre-fiemap) cp copied it for me in less than > 1 second when the backing file was on a tmpfs file system. > I've made the new size be 2TiB. The fiemap copy is still so > quick that it completes in < .01 second.[*] > - no point in discarding stdout/stderr, since it all goes to the log > - raised timeout to 10 seconds to give more leeway on slow systems > - remove those "rm -f" uses. They're not needed, since the test is > run in its own temp dir, which is removed automatically when done. > - remove the $? = 124 test -- the preceding test for success is sufficient > > [*] I tried to count syscalls with strace but got a segfault. > Using valgrind I get errors, so debugged enough to get a clean > run, but possibly at the expense of correctness. We'll need more > tests to ensure that the non-sparse blocks in the copy all have > the same offset/length as in the original. Is it make sense if we write a utility in C through FIEMAP to show the extent info of a file? then wrap it in our current test scripts or a new test script to compare the non-sparse blocks offset and length? filefrag(8) can do such thing(http://e2fsprogs.sourceforge.net/), but maybe we can implement a compacted version focus on furture extent maping related testing only for coreutils. Details below. > > diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap > old mode 100644 > new mode 100755 > index f9d3a94..814d537 > --- a/tests/cp/sparse-fiemap > +++ b/tests/cp/sparse-fiemap > @@ -28,8 +28,7 @@ cwd=`pwd` > cleanup_() { cd /; umount "$cwd/mnt"; } > > # Create an ext4 loopback file system > -dd if=/dev/zero of=blob bs=8192 count=1000 > /dev/null 2>&1 \ > - || skip=1 > +dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 > mkdir mnt > mkfs -t ext4 -F blob || > skip_test_ "failed to create ext4 file system" > @@ -42,20 +41,15 @@ test $skip = 1 && > > rm -f mnt/f > > -# Create a 2gb sparse file > -dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=2096128 > /dev/null 2>&1 || framework_failure > +# Create a 2TiB sparse file > +dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=2G || framework_failure > > -# It take more than 20 seconds to transfer the created sparse file > -# through normal copy, by contrast, it take even less than 1 second > -# through FIEMAP-copy. > -timeout 1 cp --sparse=always mnt/sparse mnt/sparse_fiemap || fail=1 > -test $? = 124 && fail=1 > +# It takes many minutes to copy this sparse file using the old method. > +# By contrast, it takes far less than 1 second using FIEMAP-copy. > +timeout 10 cp --sparse=always mnt/sparse mnt/sparse_fiemap || fail=1 > > # Ensure that the sparse file copied through fiemap has the same size > # in bytes as the original. > test `stat --printf %s $sparse` = `stat --printf %s $fiemap` || fail=1 > > -rm -f mnt/sparse > -rm -f mnt/sparse_fiemap > - > Exit $fail > > ---------------------------------------- > On F13, x86_64, ext4, I did this: > > dd if=/dev/null of=big bs=1 seek=2G > valgrind ./cp --sparse=always big big2 > ==4771== Conditional jump or move depends on uninitialised value(s) > ==4771== at 0x40465A: fiemap_copy_ok (copy.c:205) > ==4771== by 0x405B61: copy_reg (copy.c:822) > ==4771== by 0x408713: copy_internal (copy.c:2163) > ==4771== by 0x409237: copy (copy.c:2449) > ==4771== by 0x403AC9: do_copy (cp.c:754) > ==4771== by 0x4041E4: main (cp.c:1154) > ==4771== > ==4771== Syscall param lseek(offset) contains uninitialised byte(s) > ==4771== at 0x3269CE1540: __lseek_nocancel (syscall-template.S:82) > ==4771== by 0x4046D4: fiemap_copy_ok (copy.c:210) > ==4771== by 0x405B61: copy_reg (copy.c:822) > ==4771== by 0x408713: copy_internal (copy.c:2163) > ==4771== by 0x409237: copy (copy.c:2449) > ==4771== by 0x403AC9: do_copy (cp.c:754) > ==4771== by 0x4041E4: main (cp.c:1154) > ==4771== > ==4771== Syscall param lseek(offset) contains uninitialised byte(s) > ==4771== at 0x3269CE1540: __lseek_nocancel (syscall-template.S:82) > ==4771== by 0x40472D: fiemap_copy_ok (copy.c:216) > ==4771== by 0x405B61: copy_reg (copy.c:822) > ==4771== by 0x408713: copy_internal (copy.c:2163) > ==4771== by 0x409237: copy (copy.c:2449) > ==4771== by 0x403AC9: do_copy (cp.c:754) > ==4771== by 0x4041E4: main (cp.c:1154) > ==4771== > ==4771== Conditional jump or move depends on uninitialised value(s) > ==4771== at 0x404792: fiemap_copy_ok (copy.c:222) > ==4771== by 0x405B61: copy_reg (copy.c:822) > ==4771== by 0x408713: copy_internal (copy.c:2163) > ==4771== by 0x409237: copy (copy.c:2449) > ==4771== by 0x403AC9: do_copy (cp.c:754) > ==4771== by 0x4041E4: main (cp.c:1154) > ==4771== > ==4771== Conditional jump or move depends on uninitialised value(s) > ==4771== at 0x40492B: fiemap_copy_ok (copy.c:229) > ==4771== by 0x405B61: copy_reg (copy.c:822) > ==4771== by 0x408713: copy_internal (copy.c:2163) > ==4771== by 0x409237: copy (copy.c:2449) > ==4771== by 0x403AC9: do_copy (cp.c:754) > ==4771== by 0x4041E4: main (cp.c:1154) > ==4771== > ==4771== Conditional jump or move depends on uninitialised value(s) > ==4771== at 0x4047FA: fiemap_copy_ok (copy.c:235) > ==4771== by 0x405B61: copy_reg (copy.c:822) > ==4771== by 0x408713: copy_internal (copy.c:2163) > ==4771== by 0x409237: copy (copy.c:2449) > ==4771== by 0x403AC9: do_copy (cp.c:754) > ==4771== by 0x4041E4: main (cp.c:1154) > ==4771== > ==4771== Syscall param read(count) contains uninitialised byte(s) > ==4771== at 0x3269CD41B0: __read_nocancel (syscall-template.S:82) > ==4771== by 0x404821: fiemap_copy_ok (copy.c:238) > ==4771== by 0x405B61: copy_reg (copy.c:822) > ==4771== by 0x408713: copy_internal (copy.c:2163) > ==4771== by 0x409237: copy (copy.c:2449) > ==4771== by 0x403AC9: do_copy (cp.c:754) > ==4771== by 0x4041E4: main (cp.c:1154) > ==4771== > ==4771== Invalid read of size 8 > ==4771== at 0x404952: fiemap_copy_ok (copy.c:265) > ==4771== by 0x405B61: copy_reg (copy.c:822) > ==4771== by 0x408713: copy_internal (copy.c:2163) > ==4771== by 0x409237: copy (copy.c:2449) > ==4771== by 0x403AC9: do_copy (cp.c:754) > ==4771== by 0x4041E4: main (cp.c:1154) > ==4771== Address 0x3ffeffdd68 is not stack'd, malloc'd or (recently) free'd > ==4771== > ==4771== > ==4771== Process terminating with default action of signal 11 (SIGSEGV) > ==4771== Access not within mapped region at address 0x3FFEFFDD68 > ==4771== at 0x404952: fiemap_copy_ok (copy.c:265) > ==4771== by 0x405B61: copy_reg (copy.c:822) > ==4771== by 0x408713: copy_internal (copy.c:2163) > ==4771== by 0x409237: copy (copy.c:2449) > ==4771== by 0x403AC9: do_copy (cp.c:754) > ==4771== by 0x4041E4: main (cp.c:1154) > > =========================================================== > The segv just above is due to hitting this line with i==0: > > fiemap->fm_start = (fm_ext[i-1].fe_logical + fm_ext[i-1].fe_length); strange, code should break if there is no extent allocated for a file. /* If 0 extents are returned, then more ioctls are not needed. */ if (fiemap->fm_mapped_extents == 0) break; > > the obvious fix is probably to do this instead: > > fiemap->fm_start = (fm_ext[i].fe_logical + fm_ext[i].fe_length); I just found a bug for dealing with the 'fiemap->fm_start', maybe it is the root cause of the segment fault. above line still need to write as 'fm_ext[i-1].fe_logical +....' to calculate the offset for the next ioctl(2). > > All of the used-uninitialized errors can be papered over by > clearing the fiemap_buf array, like this: > > + memset (fiemap_buf, 0, sizeof fiemap_buf); I recalled why I initialized this buf before when you ask me the reason, I was intented to initialize the 'fiemap->fm_start', so below line 'fiemap->fm_start = 0ULL' should be removed from the loop. > do > { > fiemap->fm_start = 0ULL; > > However, if these are all due solely to F13's valgrind not yet knowing the > semantics of the FIEMAP ioctl, then that may be adequate. as what I mentioned above, this line should be removed or remove out of the loop if we do not initialize the fiemap buf. > > Bottom line: > - you may consider your test-script patch accepted, with the patch above > - I'd like to see a new version of the copy.c-changing patch, > including at least a fix for the fm_ext[-1] access bug. > > =========================================================== > Solely for reference, here's the copy.c patch I used to avoid > the valgrind-spotted problems: > > diff --git a/src/copy.c b/src/copy.c > index 960e5fb..e232eaa 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -179,6 +179,7 @@ fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, > uint64_t last_read_size = 0; > unsigned int i = 0; > > + memset (fiemap_buf, 0, sizeof fiemap_buf); > do > { > fiemap->fm_start = 0ULL; > @@ -187,7 +188,7 @@ fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, > > /* When ioctl(2) fails, fall back to the normal copy only if it > is the first time we met. */ > - if (ioctl (src_fd, FS_IOC_FIEMAP, (unsigned long) fiemap) < 0) > + if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) > { > /* If `i > 0', then at least one ioctl(2) has been performed before. */ > if (i == 0) > @@ -261,7 +262,7 @@ fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, > ext_len -= n_read; > } > > - fiemap->fm_start = (fm_ext[i-1].fe_logical + fm_ext[i-1].fe_length); > + fiemap->fm_start = (fm_ext[i].fe_logical + fm_ext[i].fe_length); > } > } while (! last); > > > -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Fri May 21 10:02:10 2010 Received: (at submit) by debbugs.gnu.org; 21 May 2010 14:02:10 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFSng-0001jZ-SL for submit@debbugs.gnu.org; Fri, 21 May 2010 10:02:10 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFSne-0001jI-Jn for submit@debbugs.gnu.org; Fri, 21 May 2010 10:02:07 -0400 Received: from lists.gnu.org ([199.232.76.165]:34857) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OFSnU-000333-L8 for submit@debbugs.gnu.org; Fri, 21 May 2010 10:01:56 -0400 Received: from [140.186.70.92] (port=39083 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OFSnR-0004qz-Rp for bug-coreutils@gnu.org; Fri, 21 May 2010 10:01:56 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OFSbO-00032x-Lz for bug-coreutils@gnu.org; Fri, 21 May 2010 09:49:29 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:12937) by eggs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1OFSbO-00031u-6Y for bug-coreutils@gnu.org; Fri, 21 May 2010 09:49:26 -0400 Received: (qmail 77749 invoked from network); 21 May 2010 13:49:24 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 21 May 2010 13:49:24 -0000 Message-ID: <4BF68ED1.5080209@draigBrady.com> Date: Fri, 21 May 2010 14:46:57 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF676CC.8040806@draigBrady.com> <878w7dif5z.fsf@meyering.net> In-Reply-To: <878w7dif5z.fsf@meyering.net> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9 X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.8 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Chris Mason , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.8 (----) On 21/05/10 13:59, Jim Meyering wrote: > Pádraig Brady wrote: >> There is the unlikely combination of ext4 without fiemap support I think? >> If so then that dependency is worth a comment. > > I don't know off hand. > Is there a shell-level way to test for that? One could check if is available, but that would add the dependency on kernel-headers being installed. Alternatively one could skip the test on non linux and linux < 2.6.27 But requiring ext4 is probably good enough to limit this to systems that actually do have fiemap available. Adding a comment about linux >= 2.6.27 might help us quickly eliminate false positives that are reported. cheers. Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Fri May 21 10:28:18 2010 Received: (at submit) by debbugs.gnu.org; 21 May 2010 14:28:18 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFTCz-0001u3-SP for submit@debbugs.gnu.org; Fri, 21 May 2010 10:28:18 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFTCw-0001tw-QR for submit@debbugs.gnu.org; Fri, 21 May 2010 10:28:15 -0400 Received: from lists.gnu.org ([199.232.76.165]:38823) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OFTCr-0003lB-W3 for submit@debbugs.gnu.org; Fri, 21 May 2010 10:28:10 -0400 Received: from [140.186.70.92] (port=48004 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OFTCm-0005SC-Li for bug-coreutils@gnu.org; Fri, 21 May 2010 10:28:09 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OFTCg-00024w-Ge for bug-coreutils@gnu.org; Fri, 21 May 2010 10:28:04 -0400 Received: from smtp5-g21.free.fr ([212.27.42.5]:57322) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFTCf-000244-So for bug-coreutils@gnu.org; Fri, 21 May 2010 10:27:58 -0400 Received: from smtp5-g21.free.fr (localhost [127.0.0.1]) by smtp5-g21.free.fr (Postfix) with ESMTP id 98393D48091; Fri, 21 May 2010 16:27:51 +0200 (CEST) Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp5-g21.free.fr (Postfix) with ESMTP; Fri, 21 May 2010 16:27:49 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id B582BDA6; Fri, 21 May 2010 16:27:49 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4BF683CC.3060407@oracle.com> (jeff liu's message of "Fri, 21 May 2010 20:59:56 +0800") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> Date: Fri, 21 May 2010 16:27:49 +0200 Message-ID: <87sk5lgwii.fsf@meyering.net> Lines: 109 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.2 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.2 (-----) jeff.liu wrote: ... >> [*] I tried to count syscalls with strace but got a segfault. >> Using valgrind I get errors, so debugged enough to get a clean >> run, but possibly at the expense of correctness. We'll need more >> tests to ensure that the non-sparse blocks in the copy all have >> the same offset/length as in the original. > Is it make sense if we write a utility in C through FIEMAP to show the extent info of a file? > then wrap it in our current test scripts or a new test script to compare the non-sparse blocks > offset and length? If there were no adequate tool already available, that would be good. > filefrag(8) can do such thing(http://e2fsprogs.sourceforge.net/), but maybe we can implement a > compacted version focus on furture extent maping related testing only for coreutils. Or maybe just use filefrag, when it's available. On F13, with -v (verbose), it prints this: $ filefrag -v big Filesystem type is: ef53 File size of big is 2147483648 (524288 blocks, blocksize 4096) ext logical physical expected length flags 0 0 254527 1 big: 1 extent found >> =========================================================== >> The segv just above is due to hitting this line with i==0: >> >> fiemap->fm_start = (fm_ext[i-1].fe_logical + fm_ext[i-1].fe_length); > strange, code should break if there is no extent allocated for a file. > /* If 0 extents are returned, then more ioctls are not needed. */ > if (fiemap->fm_mapped_extents == 0) > break; There is one extent, and it is while processing it, with i == 0 that would trigger the failure when referencing fm_ext[i-1] (aka fm_ext[-1]). >> the obvious fix is probably to do this instead: >> >> fiemap->fm_start = (fm_ext[i].fe_logical + fm_ext[i].fe_length); > I just found a bug for dealing with the 'fiemap->fm_start', maybe it is the root cause of the > segment fault. above line still need to write as 'fm_ext[i-1].fe_logical +....' to calculate the > offset for the next ioctl(2). "i" can be 0 there, so it sounds like you're saying we need to reference fm_ext[-1]. If you mean that, you'll have to demonstrate how we guarantee that i > 0 there. >> All of the used-uninitialized errors can be papered over by >> clearing the fiemap_buf array, like this: >> >> + memset (fiemap_buf, 0, sizeof fiemap_buf); > I recalled why I initialized this buf before when you ask me the reason, I was intented to > initialize the 'fiemap->fm_start', so below line 'fiemap->fm_start = 0ULL' should be removed from > the loop. > >> do >> { >> fiemap->fm_start = 0ULL; >> >> However, if these are all due solely to F13's valgrind not yet knowing the >> semantics of the FIEMAP ioctl, then that may be adequate. > as what I mentioned above, this line should be removed or remove out of the loop if we do not > initialize the fiemap buf. I agree. Leaving the initialization in the loop would provoke an infinite loop, for a file with many extents. This demonstrates it: $ perl -e 'for (1..100) { sysseek(STDOUT,4096,1)' \ -e '&& syswrite(STDOUT,"."x4096) or die "$!"}' > j $ ./cp --sparse=always j j2 ^C With this statement "fiemap->fm_start = 0ULL;" in the do-while loop, the use of ./cp above would infloop. Without it, it works properly: $ env time -f %E ./cp --sparse=always j j2 0:00.01 And we can compare the extents in the two: (the awk is mainly to exclude the physical block numbers, which will always differ) $ diff -u <(filefrag -v j|awk '/^ / {print $1,$2,$NF}') \ <(filefrag -v j2|awk '/^ / {print $1,$2,$NF}') $ For reference, here's what filefrag -v output looks like, given a file with a nontrivial list of extents: $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \ -e 'for (1..5) { sysseek(*F,$n,1)' \ -e '&& syswrite *F,"."x$n or die "$!"}' > j $ filefrag -v j Filesystem type is: ef53 File size of j is 163840 (40 blocks, blocksize 4096) ext logical physical expected length flags 0 4 6258884 4 1 12 6258892 6258887 4 2 20 6258900 6258895 4 3 28 6258908 6258903 4 4 36 6258916 6258911 4 eof j: 6 extents found From debbugs-submit-bounces@debbugs.gnu.org Fri May 21 10:45:01 2010 Received: (at submit) by debbugs.gnu.org; 21 May 2010 14:45:02 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFTTA-00022Z-Rh for submit@debbugs.gnu.org; Fri, 21 May 2010 10:45:01 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFTT9-00022U-6E for submit@debbugs.gnu.org; Fri, 21 May 2010 10:44:59 -0400 Received: from lists.gnu.org ([199.232.76.165]:35678) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OFTT4-00043B-8l for submit@debbugs.gnu.org; Fri, 21 May 2010 10:44:54 -0400 Received: from [140.186.70.92] (port=46661 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OFTT2-000629-QA for bug-coreutils@gnu.org; Fri, 21 May 2010 10:44:53 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OFTSx-0004pK-I8 for bug-coreutils@gnu.org; Fri, 21 May 2010 10:44:51 -0400 Received: from smtp5-g21.free.fr ([212.27.42.5]:33419) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFTSw-0004oB-Tc for bug-coreutils@gnu.org; Fri, 21 May 2010 10:44:47 -0400 Received: from smtp5-g21.free.fr (localhost [127.0.0.1]) by smtp5-g21.free.fr (Postfix) with ESMTP id E530CD48147; Fri, 21 May 2010 16:44:40 +0200 (CEST) Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp5-g21.free.fr (Postfix) with ESMTP; Fri, 21 May 2010 16:44:39 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 116ACDA6; Fri, 21 May 2010 16:44:39 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4BF683CC.3060407@oracle.com> (jeff liu's message of "Fri, 21 May 2010 20:59:56 +0800") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> Date: Fri, 21 May 2010 16:44:39 +0200 Message-ID: <87mxvtgvqg.fsf@meyering.net> Lines: 51 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.2 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.2 (-----) jeff.liu wrote: ... >>> Subject: [PATCH 1/1] tests: add a new test for FIEMAP-copy >>> >>> * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a >>> loopbacked ext4 partition. >>> * tests/Makefile.am (sparse-fiemap): Reference the new test. BTW, I've just made this additional change to your test, diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index 6312a4c..bdc7ded 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -27,6 +27,7 @@ require_root_ cwd=`pwd` cleanup_() { cd /; umount "$cwd/mnt"; } +skip=0 # Create an ext4 loopback file system dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 mkdir mnt And will push this correction to the one you appear to have used as a model: >From c9bcbc8f9fc791c97bc85678f5f22458a76689ac Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Fri, 21 May 2010 14:55:36 +0200 Subject: [PATCH] tests: fix cp-a-selinux to skip cleanly upon mkfs failure * tests/cp/cp-a-selinux: Initialize skip, to avoid a syntax error in subsequent "test". --- tests/cp/cp-a-selinux | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/tests/cp/cp-a-selinux b/tests/cp/cp-a-selinux index b65070a..5b9ff0f 100755 --- a/tests/cp/cp-a-selinux +++ b/tests/cp/cp-a-selinux @@ -45,7 +45,7 @@ test -s err && fail=1 #there must be no stderr output for -a ls -Z e | grep $ctx || fail=1 ls -Z f | grep $ctx || fail=1 - +skip=0 # Create a file system, then mount it with the context=... option. dd if=/dev/zero of=blob bs=8192 count=200 > /dev/null 2>&1 \ || skip=1 -- 1.7.1.262.g5ef3d From debbugs-submit-bounces@debbugs.gnu.org Fri May 21 11:03:57 2010 Received: (at submit) by debbugs.gnu.org; 21 May 2010 15:03:57 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFTlV-0002Aw-7h for submit@debbugs.gnu.org; Fri, 21 May 2010 11:03:57 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFTlS-0002Aq-Op for submit@debbugs.gnu.org; Fri, 21 May 2010 11:03:55 -0400 Received: from lists.gnu.org ([199.232.76.165]:46967) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OFTlN-0004TX-Or for submit@debbugs.gnu.org; Fri, 21 May 2010 11:03:49 -0400 Received: from [140.186.70.92] (port=59736 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OFTlM-000618-Iz for bug-coreutils@gnu.org; Fri, 21 May 2010 11:03:49 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OFTlK-0007Wa-Bq for bug-coreutils@gnu.org; Fri, 21 May 2010 11:03:48 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:48229) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFTlJ-0007WM-H4 for bug-coreutils@gnu.org; Fri, 21 May 2010 11:03:46 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4LF3edp027259 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 21 May 2010 15:03:42 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4LCxOMi023972; Fri, 21 May 2010 15:03:38 GMT Received: from abhmt017.oracle.com by acsmt355.oracle.com with ESMTP id 257749291274454102; Fri, 21 May 2010 08:01:42 -0700 Received: from [221.223.108.50] (/221.223.108.50) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 21 May 2010 08:01:42 -0700 Message-ID: <4BF6A055.6020205@oracle.com> Date: Fri, 21 May 2010 23:01:41 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <87mxvtgvqg.fsf@meyering.net> In-Reply-To: <87mxvtgvqg.fsf@meyering.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090202.4BF6A0CF.00D8:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.1 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.2 (-----) Jim Meyering wrote: > jeff.liu wrote: > ... >>>> Subject: [PATCH 1/1] tests: add a new test for FIEMAP-copy >>>> >>>> * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a >>>> loopbacked ext4 partition. >>>> * tests/Makefile.am (sparse-fiemap): Reference the new test. > > BTW, I've just made this additional change to your test, > > diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap > index 6312a4c..bdc7ded 100755 > --- a/tests/cp/sparse-fiemap > +++ b/tests/cp/sparse-fiemap > @@ -27,6 +27,7 @@ require_root_ > cwd=`pwd` > cleanup_() { cd /; umount "$cwd/mnt"; } > > +skip=0 > # Create an ext4 loopback file system > dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 > mkdir mnt > > And will push this correction to the one you appear to have used as a model: > > From c9bcbc8f9fc791c97bc85678f5f22458a76689ac Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Fri, 21 May 2010 14:55:36 +0200 > Subject: [PATCH] tests: fix cp-a-selinux to skip cleanly upon mkfs failure > > * tests/cp/cp-a-selinux: Initialize skip, to avoid a syntax error > in subsequent "test". > --- > tests/cp/cp-a-selinux | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/tests/cp/cp-a-selinux b/tests/cp/cp-a-selinux > index b65070a..5b9ff0f 100755 > --- a/tests/cp/cp-a-selinux > +++ b/tests/cp/cp-a-selinux > @@ -45,7 +45,7 @@ test -s err && fail=1 #there must be no stderr output for -a > ls -Z e | grep $ctx || fail=1 > ls -Z f | grep $ctx || fail=1 > > - > +skip=0 > # Create a file system, then mount it with the context=... option. > dd if=/dev/zero of=blob bs=8192 count=200 > /dev/null 2>&1 \ > || skip=1 Hmm, here also has a '2>&1' for `dd'. :) > -- > 1.7.1.262.g5ef3d Thanks, -Jeff -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Fri May 21 11:32:55 2010 Received: (at submit) by debbugs.gnu.org; 21 May 2010 15:32:56 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFUDV-0002NM-VX for submit@debbugs.gnu.org; Fri, 21 May 2010 11:32:54 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFUDU-0002NH-5T for submit@debbugs.gnu.org; Fri, 21 May 2010 11:32:52 -0400 Received: from lists.gnu.org ([199.232.76.165]:43019) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OFUDP-00051x-8g for submit@debbugs.gnu.org; Fri, 21 May 2010 11:32:47 -0400 Received: from [140.186.70.92] (port=41881 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OFUDN-0002Ds-3j for bug-coreutils@gnu.org; Fri, 21 May 2010 11:32:46 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OFUDK-0003jX-VS for bug-coreutils@gnu.org; Fri, 21 May 2010 11:32:44 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:63725) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFUDK-0003jM-Po for bug-coreutils@gnu.org; Fri, 21 May 2010 11:32:42 -0400 Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4LFWbtN007146 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 21 May 2010 15:32:38 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4LFWZMU028919; Fri, 21 May 2010 15:32:35 GMT Received: from abhmt019.oracle.com by acsmt353.oracle.com with ESMTP id 288266851274455913; Fri, 21 May 2010 08:31:53 -0700 Received: from [221.223.108.50] (/221.223.108.50) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 21 May 2010 08:31:52 -0700 Message-ID: <4BF6A769.2050405@oracle.com> Date: Fri, 21 May 2010 23:31:53 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <87sk5lgwii.fsf@meyering.net> In-Reply-To: <87sk5lgwii.fsf@meyering.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet15.oracle.com [148.87.113.117] X-CT-RefId: str=0001.0A090204.4BF6A797.0249:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Jim Meyering wrote: > jeff.liu wrote: > ... >>> [*] I tried to count syscalls with strace but got a segfault. >>> Using valgrind I get errors, so debugged enough to get a clean >>> run, but possibly at the expense of correctness. We'll need more >>> tests to ensure that the non-sparse blocks in the copy all have >>> the same offset/length as in the original. >> Is it make sense if we write a utility in C through FIEMAP to show the extent info of a file? >> then wrap it in our current test scripts or a new test script to compare the non-sparse blocks >> offset and length? > > If there were no adequate tool already available, that would be good. > >> filefrag(8) can do such thing(http://e2fsprogs.sourceforge.net/), but maybe we can implement a >> compacted version focus on furture extent maping related testing only for coreutils. > > Or maybe just use filefrag, when it's available. > On F13, with -v (verbose), it prints this: > > $ filefrag -v big > Filesystem type is: ef53 > File size of big is 2147483648 (524288 blocks, blocksize 4096) > ext logical physical expected length flags > 0 0 254527 1 > big: 1 extent found > > >>> =========================================================== >>> The segv just above is due to hitting this line with i==0: >>> >>> fiemap->fm_start = (fm_ext[i-1].fe_logical + fm_ext[i-1].fe_length); >> strange, code should break if there is no extent allocated for a file. >> /* If 0 extents are returned, then more ioctls are not needed. */ >> if (fiemap->fm_mapped_extents == 0) >> break; > > There is one extent, and it is while processing it, with i == 0 that > would trigger the failure when referencing fm_ext[i-1] (aka fm_ext[-1]). > >>> the obvious fix is probably to do this instead: >>> >>> fiemap->fm_start = (fm_ext[i].fe_logical + fm_ext[i].fe_length); >> I just found a bug for dealing with the 'fiemap->fm_start', maybe it is the root cause of the >> segment fault. above line still need to write as 'fm_ext[i-1].fe_logical +....' to calculate the >> offset for the next ioctl(2). > > "i" can be 0 there, so it sounds like you're saying we need to > reference fm_ext[-1]. If you mean that, you'll have to demonstrate > how we guarantee that i > 0 there. Sorry for the lack of detailed info for this point, except for removing the fiemap->fm_start from the loop, I need to remove "fiemap->fm_start = (fm_ext[i-1].fe_logical + fm_ext[i-1].fe_length);" out of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)" as well. So, if there is only one extent, at least 'i == 1' when the loop finished, we'll not hit the 'fm_ext[-1]' issue. my thoughts of the fix looks like below: memset (fiemap, 0, sizeof fiemap_buf); do { ioctl (...); for (i = 0; i < fiemap->fm_mapped_extents; i++) { ... } fiemap->fm_start = (fm_ext[i-1].fe_logical + fm_ext[i-1].fe_length); } while (! last); > >>> All of the used-uninitialized errors can be papered over by >>> clearing the fiemap_buf array, like this: >>> >>> + memset (fiemap_buf, 0, sizeof fiemap_buf); >> I recalled why I initialized this buf before when you ask me the reason, I was intented to >> initialize the 'fiemap->fm_start', so below line 'fiemap->fm_start = 0ULL' should be removed from >> the loop. >> >>> do >>> { >>> fiemap->fm_start = 0ULL; >>> >>> However, if these are all due solely to F13's valgrind not yet knowing the >>> semantics of the FIEMAP ioctl, then that may be adequate. >> as what I mentioned above, this line should be removed or remove out of the loop if we do not >> initialize the fiemap buf. > > I agree. > Leaving the initialization in the loop would provoke an infinite loop, > for a file with many extents. > > This demonstrates it: > > $ perl -e 'for (1..100) { sysseek(STDOUT,4096,1)' \ > -e '&& syswrite(STDOUT,"."x4096) or die "$!"}' > j > $ ./cp --sparse=always j j2 > > ^C > > With this statement "fiemap->fm_start = 0ULL;" in the do-while loop, > the use of ./cp above would infloop. Without it, it works properly: > > $ env time -f %E ./cp --sparse=always j j2 > 0:00.01 > > And we can compare the extents in the two: > (the awk is mainly to exclude the physical block numbers, > which will always differ) > > $ diff -u <(filefrag -v j|awk '/^ / {print $1,$2,$NF}') \ > <(filefrag -v j2|awk '/^ / {print $1,$2,$NF}') > $ > > For reference, here's what filefrag -v output looks like, > given a file with a nontrivial list of extents: > > $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \ > -e 'for (1..5) { sysseek(*F,$n,1)' \ > -e '&& syswrite *F,"."x$n or die "$!"}' > j > $ filefrag -v j > Filesystem type is: ef53 > File size of j is 163840 (40 blocks, blocksize 4096) > ext logical physical expected length flags > 0 4 6258884 4 > 1 12 6258892 6258887 4 > 2 20 6258900 6258895 4 > 3 28 6258908 6258903 4 > 4 36 6258916 6258911 4 eof > j: 6 extents found Do we need another test script for this test if we choose `filefrag' to examine the extent info? I'd like to handle it. > > > Thanks, -Jeff -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Fri May 21 11:42:44 2010 Received: (at submit) by debbugs.gnu.org; 21 May 2010 15:42:44 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFUN2-0002Rr-Hg for submit@debbugs.gnu.org; Fri, 21 May 2010 11:42:44 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFUN0-0002Rl-Gl for submit@debbugs.gnu.org; Fri, 21 May 2010 11:42:43 -0400 Received: from lists.gnu.org ([199.232.76.165]:35819) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OFUMv-00059o-Ac for submit@debbugs.gnu.org; Fri, 21 May 2010 11:42:37 -0400 Received: from [140.186.70.92] (port=38035 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OFUMt-0001Xo-IK for bug-coreutils@gnu.org; Fri, 21 May 2010 11:42:36 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OFUMr-0005E4-U7 for bug-coreutils@gnu.org; Fri, 21 May 2010 11:42:35 -0400 Received: from smtp5-g21.free.fr ([212.27.42.5]:47872) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFUMr-0005Da-Bb for bug-coreutils@gnu.org; Fri, 21 May 2010 11:42:33 -0400 Received: from smtp5-g21.free.fr (localhost [127.0.0.1]) by smtp5-g21.free.fr (Postfix) with ESMTP id 1E6D7D480A9; Fri, 21 May 2010 17:42:26 +0200 (CEST) Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp5-g21.free.fr (Postfix) with ESMTP; Fri, 21 May 2010 17:42:25 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 43CE7A84; Fri, 21 May 2010 17:42:25 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4BF6A769.2050405@oracle.com> (jeff liu's message of "Fri, 21 May 2010 23:31:53 +0800") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <87sk5lgwii.fsf@meyering.net> <4BF6A769.2050405@oracle.com> Date: Fri, 21 May 2010 17:42:25 +0200 Message-ID: <87aartgt26.fsf@meyering.net> Lines: 47 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.2 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.2 (-----) jeff.liu wrote: ... > Sorry for the lack of detailed info for this point, except for removing the fiemap->fm_start from > the loop, I need to remove "fiemap->fm_start = (fm_ext[i-1].fe_logical + fm_ext[i-1].fe_length);" > out of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)" as well. > So, if there is only one extent, at least 'i == 1' when the loop finished, we'll not hit the > 'fm_ext[-1]' issue. > > my thoughts of the fix looks like below: > > memset (fiemap, 0, sizeof fiemap_buf); > do > { > ioctl (...); > > for (i = 0; i < fiemap->fm_mapped_extents; i++) > { > ... > } > fiemap->fm_start = (fm_ext[i-1].fe_logical + fm_ext[i-1].fe_length); > } while (! last); That is better. Equivalent semantics to my change, but yours avoids unnecessarily updating fiemap->fm_start for each iteration of the for loop. ... >> For reference, here's what filefrag -v output looks like, >> given a file with a nontrivial list of extents: >> >> $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \ >> -e 'for (1..5) { sysseek(*F,$n,1)' \ >> -e '&& syswrite *F,"."x$n or die "$!"}' > j >> $ filefrag -v j >> Filesystem type is: ef53 >> File size of j is 163840 (40 blocks, blocksize 4096) >> ext logical physical expected length flags >> 0 4 6258884 4 >> 1 12 6258892 6258887 4 >> 2 20 6258900 6258895 4 >> 3 28 6258908 6258903 4 >> 4 36 6258916 6258911 4 eof >> j: 6 extents found > Do we need another test script for this test if we choose `filefrag' to examine the extent info? Yes, that's why I took the time to do the above. I've already written most of it. Will post shortly. From debbugs-submit-bounces@debbugs.gnu.org Fri May 21 11:51:40 2010 Received: (at submit) by debbugs.gnu.org; 21 May 2010 15:51:40 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFUVe-0002WA-RN for submit@debbugs.gnu.org; Fri, 21 May 2010 11:51:39 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFUVb-0002W5-Qg for submit@debbugs.gnu.org; Fri, 21 May 2010 11:51:37 -0400 Received: from lists.gnu.org ([199.232.76.165]:55632) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OFUVW-0005M4-RT for submit@debbugs.gnu.org; Fri, 21 May 2010 11:51:30 -0400 Received: from [140.186.70.92] (port=51037 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OFUVU-0007tK-8D for bug-coreutils@gnu.org; Fri, 21 May 2010 11:51:30 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OFUVP-0006e2-Ok for bug-coreutils@gnu.org; Fri, 21 May 2010 11:51:27 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:57573) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OFUVP-0006du-Fk for bug-coreutils@gnu.org; Fri, 21 May 2010 11:51:23 -0400 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4LFpJAK023053 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 21 May 2010 15:51:20 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4LAKK9s017564; Fri, 21 May 2010 15:51:16 GMT Received: from abhmt017.oracle.com by acsmt355.oracle.com with ESMTP id 257934441274457066; Fri, 21 May 2010 08:51:06 -0700 Received: from [221.223.108.50] (/221.223.108.50) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 21 May 2010 08:51:05 -0700 Message-ID: <4BF6ABE7.3020600@oracle.com> Date: Fri, 21 May 2010 23:51:03 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> In-Reply-To: <4BF683CC.3060407@oracle.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet13.oracle.com [148.87.113.125] X-CT-RefId: str=0001.0A090208.4BF6ABF9.0178:SCFMA4539811,ss=1,pt=113351,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.1 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.4 (-----) Hi Jim, This is the revised version, it fixed the fiemap-start offset calculation approach to remove it out of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)' loop. I have not got a 64bits machine for the testing at the moment, at the following, the first case only run againt x86 with valgrind for the non-extent file copy, it works for me, could you help verify on x64? The second case is to test the non-sparse extents logical offset and length of the copied file are identical to the source file, `ex' is test tool I write in C to show the extents info through FIEMAP ioctl(2), it step through each extent of a file to examine and print out the logical offset/extent length/physical offset. jeff@jeff-laptop:~/opensource_dev/coreutils$ dd if=/dev/null of=/ext4/sp1 bs=1 seek=2G jeff@jeff-laptop:~/opensource_dev/coreutils$ valgrind --version valgrind-3.3.0-Debian jeff@jeff-laptop:~/opensource_dev/coreutils$ valgrind ./src/cp --sparse=always /ext4/sp1 /ext4/sp2 ==13678== Memcheck, a memory error detector. ==13678== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al. ==13678== Using LibVEX rev 1804, a library for dynamic binary translation. ==13678== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP. ==13678== Using valgrind-3.3.0-Debian, a dynamic binary instrumentation framework. ==13678== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al. ==13678== For more details, rerun with: -v ==13678== ==13678== ==13678== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 23 from 1) ==13678== malloc/free: in use at exit: 0 bytes in 0 blocks. ==13678== malloc/free: 71 allocs, 71 frees, 10,255 bytes allocated. ==13678== For counts of detected errors, rerun with: -v ==13678== All heap blocks were freed -- no leaks areFrom 2a2df00acbcc9cdaef723f23efccb65d761d9093 Mon Sep 17 00:00:00 2001 jeff@jeff-laptop:~/opensource_dev/coreutils$ ./src/cp --sparse=always /ocfs2/sparse_dir/sparse_4 /ext4/sp2 jeff@jeff-laptop:~/opensource_dev/coreutils$ ./ex /ext4/sp2 Extents in file "/ext4/sp2": 14 Extents returned: 14 Logical: ###[ 0] Ext length: ###[ 65536] Physical: ###[352321536] Logical: ###[ 98304] Ext length: ###[ 32768] Physical: ###[352419840] Logical: ###[ 229376] Ext length: ###[ 32768] Physical: ###[352550912] Logical: ###[ 458752] Ext length: ###[ 65536] Physical: ###[352780288] Logical: ###[ 950272] Ext length: ###[ 65536] Physical: ###[353271808] Logical: ###[ 1966080] Ext length: ###[ 32768] Physical: ###[354287616] Logical: ###[ 3932160] Ext length: ###[ 65536] Physical: ###[356253696] Logical: ###[ 7897088] Ext length: ###[ 65536] Physical: ###[360218624] Logical: ###[15826944] Ext length: ###[ 65536] Physical: ###[384925696] Logical: ###[31719424] Ext length: ###[ 65536] Physical: ###[1004797952] Logical: ###[63471616] Ext length: ###[ 65536] Physical: ###[1011384320] Logical: ###[126976000] Ext length: ###[ 65536] Physical: ###[1016168448] Logical: ###[254017536] Ext length: ###[ 65536] Physical: ###[1025769472] Logical: ###[508100608] Ext length: ###[ 32768] Physical: ###[1036582912] jeff@jeff-laptop:~/opensource_dev/coreutils$ ./src/cp --sparse=always /ext4/sp2 /ext4/sp2_fiemap jeff@jeff-laptop:~/opensource_dev/coreutils$ ./ex /ext4/sp2_fiemap Extents in file "/ext4/sp2_fiemap": 14 Extents returned: 14 Logical: ###[ 0] Ext length: ###[ 65536] Physical: ###[1040187392] Logical: ###[ 98304] Ext length: ###[ 32768] Physical: ###[1040285696] Logical: ###[ 229376] Ext length: ###[ 32768] Physical: ###[1040416768] Logical: ###[ 458752] Ext length: ###[ 65536] Physical: ###[1040646144] Logical: ###[ 950272] Ext length: ###[ 65536] Physical: ###[1041137664] Logical: ###[ 1966080] Ext length: ###[ 32768] Physical: ###[1042153472] Logical: ###[ 3932160] Ext length: ###[ 65536] Physical: ###[1044119552] Logical: ###[ 7897088] Ext length: ###[ 65536] Physical: ###[1048084480] Logical: ###[15826944] Ext length: ###[ 65536] Physical: ###[1056014336] Logical: ###[31719424] Ext length: ###[ 65536] Physical: ###[1063518208] Logical: ###[63471616] Ext length: ###[ 65536] Physical: ###[1070104576] Logical: ###[126976000] Ext length: ###[ 65536] Physical: ###[1125220352] Logical: ###[254017536] Ext length: ###[ 65536] Physical: ###[1134821376] Logical: ###[508100608] Ext length: ###[ 32768] Physical: ###[1145634816] >From 056bb15018466cc2b6b7ae2603fb41b6f61fa084 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Fri, 21 May 2010 22:49:03 +0800 Subject: [PATCH 1/1] cp: Add FIEMAP support for efficient sparse file copy * src/fiemap.h: Add fiemap.h for fiemap ioctl(2) support. Copied from linux's include/linux/fiemap.h, with minor formatting changes. * src/copy.c (copy_reg): Now, when `cp' invoked with --sparse=[WHEN] option, we will try to do FIEMAP-copy if the underlaying file system support it, fall back to a normal copy if it fails. Signed-off-by: Jie Liu --- src/copy.c | 153 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/fiemap.h | 102 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 255 insertions(+), 0 deletions(-) create mode 100644 src/fiemap.h diff --git a/src/copy.c b/src/copy.c index c16cef6..f32a676 100644 --- a/src/copy.c +++ b/src/copy.c @@ -63,6 +63,10 @@ #include +#ifndef HAVE_FIEMAP +# include "fiemap.h" +#endif + #ifndef HAVE_FCHOWN # define HAVE_FCHOWN false # define fchown(fd, uid, gid) (-1) @@ -149,6 +153,135 @@ clone_file (int dest_fd, int src_fd) #endif } +#ifdef __linux__ +# ifndef FS_IOC_FIEMAP +# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) +# endif +/* Perform FIEMAP(available in mainline 2.6.27) copy if possible. + Call ioctl(2) with FS_IOC_FIEMAP to efficiently map file allocation + excepts holes. So the overhead to deal with holes with lseek(2) in + normal copy could be saved. This would result in much faster backups + for any kind of sparse file. */ +static bool +fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, + off_t src_total_size, char const *src_name, + char const *dst_name, bool *normal_copy_required) +{ + bool fail = false; + bool last = false; + char fiemap_buf[4096]; + struct fiemap *fiemap = (struct fiemap *)fiemap_buf; + struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; + uint32_t count = (sizeof (fiemap_buf) - sizeof (*fiemap)) / + sizeof (struct fiemap_extent); + off_t last_ext_logical = 0; + uint64_t last_ext_len = 0; + uint64_t last_read_size = 0; + unsigned int i = 0; + + memset (fiemap, 0, sizeof fiemap_buf); + do + { + fiemap->fm_length = FIEMAP_MAX_OFFSET; + fiemap->fm_extent_count = count; + + /* When ioctl(2) fails, fall back to the normal copy only if it + is the first time we met. */ + if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) + { + /* If `i > 0', then at least one ioctl(2) has been performed before. */ + if (i == 0) + *normal_copy_required = true; + return false; + } + + /* If 0 extents are returned, then more ioctls are not needed. */ + if (fiemap->fm_mapped_extents == 0) + break; + + for (i = 0; i < fiemap->fm_mapped_extents; i++) + { + assert (fm_ext[i].fe_logical <= OFF_T_MAX); + + off_t ext_logical = fm_ext[i].fe_logical; + uint64_t ext_len = fm_ext[i].fe_length; + + if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) + { + error (0, errno, _("cannot lseek %s"), quote (src_name)); + return fail; + } + + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0LL) + { + error (0, errno, _("cannot lseek %s"), quote (dst_name)); + return fail; + } + + if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST) + { + last_ext_logical = ext_logical; + last_ext_len = ext_len; + last = true; + } + + while (0 < ext_len) + { + char buf[buf_size]; + + /* Avoid reading into the holes if the left extent + length is shorter than the buffer size. */ + if (ext_len < buf_size) + buf_size = ext_len; + + ssize_t n_read = read (src_fd, buf, buf_size); + if (n_read < 0) + { +#ifdef EINTR + if (errno == EINTR) + continue; +#endif + error (0, errno, _("reading %s"), quote (src_name)); + return fail; + } + + if (n_read == 0) + { + /* Figure out how many bytes read from the last extent. */ + last_read_size = last_ext_len - ext_len; + break; + } + + if (full_write (dest_fd, buf, n_read) != n_read) + { + error (0, errno, _("writing %s"), quote (dst_name)); + return fail; + } + + ext_len -= n_read; + } + } + fiemap->fm_start = (fm_ext[i-1].fe_logical + fm_ext[i-1].fe_length); + } while (! last); + + /* If a file ends up with holes, the sum of the last extent logical offset + and the read-returned size will be shorter than the actual size of the + file. Use ftruncate to extend the length of the destination file. */ + if (last_ext_logical + last_read_size < src_total_size) + { + if (ftruncate (dest_fd, src_total_size) < 0) + { + error (0, errno, _("extending %s"), quote (dst_name)); + return fail; + } + } + + return ! fail; +} +#else +static bool fiemap_copy_ok (ignored) { errno == ENOTSUP; return false; } +#endif + /* FIXME: describe */ /* FIXME: rewrite this to use a hash table so we avoid the quadratic performance hit that's probably noticeable only on trees deeper @@ -679,6 +812,25 @@ copy_reg (char const *src_name, char const *dst_name, #endif } + if (make_holes) + { + bool require_normal_copy = false; + /* Perform efficient FIEMAP copy for sparse files, fall back to the + standard copy only if the ioctl(2) fails. */ + if (fiemap_copy_ok (source_desc, dest_desc, buf_size, + src_open_sb.st_size, src_name, + dst_name, &require_normal_copy)) + goto preserve_metadata; + else + { + if (! require_normal_copy) + { + return_val = false; + goto close_src_and_dst_desc; + } + } + } + /* If not making a sparse file, try to use a more-efficient buffer size. */ if (! make_holes) @@ -807,6 +959,7 @@ copy_reg (char const *src_name, char const *dst_name, } } +preserve_metadata: if (x->preserve_timestamps) { struct timespec timespec[2]; diff --git a/src/fiemap.h b/src/fiemap.h new file mode 100644 index 0000000..d33293b --- /dev/null +++ b/src/fiemap.h @@ -0,0 +1,102 @@ +/* FS_IOC_FIEMAP ioctl infrastructure. + Some portions copyright (C) 2007 Cluster File Systems, Inc + Authors: Mark Fasheh + Kalpak Shah + Andreas Dilger . */ + +/* Copy from kernel, modified to respect GNU code style by Jie Liu. */ + +#ifndef _LINUX_FIEMAP_H +# define _LINUX_FIEMAP_H + +# include + +struct fiemap_extent +{ + /* Logical offset in bytes for the start of the extent + from the beginning of the file. */ + uint64_t fe_logical; + + /* Physical offset in bytes for the start of the extent + from the beginning of the disk. */ + uint64_t fe_physical; + + /* Length in bytes for this extent. */ + uint64_t fe_length; + + uint64_t fe_reserved64[2]; + + /* FIEMAP_EXTENT_* flags for this extent. */ + uint32_t fe_flags; + + uint32_t fe_reserved[3]; +}; + +struct fiemap +{ + /* Logical offset(inclusive) at which to start mapping(in). */ + uint64_t fm_start; + + /* Logical length of mapping which userspace wants(in). */ + uint64_t fm_length; + + /* FIEMAP_FLAG_* flags for request(in/out). */ + uint32_t fm_flags; + + /* Number of extents that were mapped(out). */ + uint32_t fm_mapped_extents; + + /* Size of fm_extents array(in). */ + uint32_t fm_extent_count; + + uint32_t fm_reserved; + + /* Array of mapped extents(out). */ + struct fiemap_extent fm_extents[0]; +}; + +/* The maximum offset can be mapped for a file. */ +# define FIEMAP_MAX_OFFSET (~0ULL) + +/* Sync file data before map. */ +# define FIEMAP_FLAG_SYNC 0x00000001 + +/* Map extented attribute tree. */ +# define FIEMAP_FLAG_XATTR 0x00000002 + +# define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR) + +/* Last extent in file. */ +# define FIEMAP_EXTENT_LAST 0x00000001 + +/* Data location unknown. */ +# define FIEMAP_EXTENT_UNKNOWN 0x00000002 + +/* Location still pending, Sets EXTENT_UNKNOWN. */ +# define FIEMAP_EXTENT_DELALLOC 0x00000004 + +/* Data can not be read while fs is unmounted. */ +# define FIEMAP_EXTENT_ENCODED 0x00000008 + +/* Data is encrypted by fs. Sets EXTENT_NO_BYPASS. */ +# define FIEMAP_EXTENT_DATA_ENCRYPTED 0x00000080 + +/* Extent offsets may not be block aligned. */ +# define FIEMAP_EXTENT_NOT_ALIGNED 0x00000100 + +/* Data mixed with metadata. Sets EXTENT_NOT_ALIGNED. */ +# define FIEMAP_EXTENT_DATA_INLINE 0x00000200 + +/* Multiple files in block. Set EXTENT_NOT_ALIGNED. */ +# define FIEMAP_EXTENT_DATA_TAIL 0x00000400 + +/* Space allocated, but not data (i.e. zero). */ +# define FIEMAP_EXTENT_UNWRITTEN 0x00000800 + +/* File does not natively support extents. Result merged for efficiency. */ +# define FIEMAP_EXTENT_MERGED 0x00001000 + +/* Space shared with other files. */ +# define FIEMAP_EXTENT_SHARED 0x00002000 + +#endif -- 1.5.4.3 Cheers, -Jeff -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Mon May 24 04:28:53 2010 Received: (at submit) by debbugs.gnu.org; 24 May 2010 08:28:53 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGT1o-0006ve-V8 for submit@debbugs.gnu.org; Mon, 24 May 2010 04:28:53 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGT1m-0006vY-FC for submit@debbugs.gnu.org; Mon, 24 May 2010 04:28:51 -0400 Received: from lists.gnu.org ([199.232.76.165]:33943) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OGT1j-0008Jx-JA for submit@debbugs.gnu.org; Mon, 24 May 2010 04:28:47 -0400 Received: from [140.186.70.92] (port=55159 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OGT1h-0005M4-Sz for bug-coreutils@gnu.org; Mon, 24 May 2010 04:28:46 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OGT1f-0004vK-Kv for bug-coreutils@gnu.org; Mon, 24 May 2010 04:28:45 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:60170) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGT1f-0004uf-7R for bug-coreutils@gnu.org; Mon, 24 May 2010 04:28:43 -0400 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4O8Sacs004689 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 24 May 2010 08:28:38 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4O8FmtN012974; Mon, 24 May 2010 08:28:35 GMT Received: from abhmt004.oracle.com by acsmt353.oracle.com with ESMTP id 292299311274689662; Mon, 24 May 2010 01:27:42 -0700 Received: from [10.182.121.3] (/10.182.121.3) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 24 May 2010 01:27:42 -0700 Message-ID: <4BFA3878.9080606@oracle.com> Date: Mon, 24 May 2010 16:27:36 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <87mxvtgvqg.fsf@meyering.net> <4BF6A055.6020205@oracle.com> <87ljb96920.fsf@meyering.net> In-Reply-To: <87ljb96920.fsf@meyering.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet13.oracle.com [148.87.113.125] X-CT-RefId: str=0001.0A090208.4BFA38B7.002E:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.9 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.1 (------) Jim Meyering wrote: > jeff.liu wrote: >> Jim Meyering wrote: >>> jeff.liu wrote: >>> ... >>>>>> Subject: [PATCH 1/1] tests: add a new test for FIEMAP-copy >>>>>> >>>>>> * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a >>>>>> loopbacked ext4 partition. >>>>>> * tests/Makefile.am (sparse-fiemap): Reference the new test. >>> BTW, I've just made this additional change to your test, >>> >>> diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap >>> index 6312a4c..bdc7ded 100755 >>> --- a/tests/cp/sparse-fiemap >>> +++ b/tests/cp/sparse-fiemap >>> @@ -27,6 +27,7 @@ require_root_ >>> cwd=`pwd` >>> cleanup_() { cd /; umount "$cwd/mnt"; } >>> >>> +skip=0 > > More fixes for the test: > - remove useless rm > - $sparse and $fiemap were not defined in that final test, > so cd into mnt/ and just use the file names instead. > - I prefer $(...) to `...`, and it's portable in this context > > diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap > index 32ca5fc..cec5224 100755 > --- a/tests/cp/sparse-fiemap > +++ b/tests/cp/sparse-fiemap > @@ -40,18 +40,18 @@ test -s mnt/f || skip=1 > test $skip = 1 && > skip_test_ "insufficient mount/ext4 support" > > -rm -f mnt/f > - > # Create a 1TiB sparse file > dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure > > +cd mnt || fail=1 > + > # It takes many minutes to copy this sparse file using the old method. > # By contrast, it takes far less than 1 second using FIEMAP-copy. > -timeout 10 cp --sparse=always mnt/sparse mnt/sparse_fiemap || fail=1 > +timeout 10 cp --sparse=always sparse fiemap || fail=1 > > # Ensure that the sparse file copied through fiemap has the same size > # in bytes as the original. > -test `stat --printf %s $sparse` = `stat --printf %s $fiemap` || fail=1 > +test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 > > # ================================================= > # Ensure that we exercise the FIEMAP-copying code enough > -- > 1.7.1.262.g5ef3d Thanks for the info. For the point of *- I prefer $(...) to `...`, and it's portable in this context* Could you check the tiny patch below, it does the same thing to improve the portability of 'tests/cp/sparse', I have referred to it as a model. >From 9fbffcd0f4d4706f4b88e7ac73ca13b62c789047 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Mon, 24 May 2010 16:01:06 +0800 Subject: [PATCH 1/1] cp: improve the portability of test * tests/cp/sparse: improve the portability using shell constructs. Signed-off-by: Jie Liu --- tests/cp/sparse | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/cp/sparse b/tests/cp/sparse index 73c2924..dee6de2 100755 --- a/tests/cp/sparse +++ b/tests/cp/sparse @@ -28,14 +28,14 @@ require_sparse_support_ # It has to be at least 128K in order to be sparse on some systems. # Make its size one larger than 128K, in order to tickle the # bug in coreutils-6.0. -size=`expr 128 \* 1024 + 1` +size=$((128 * 1024 + 1)) dd bs=1 seek=$size of=sparse < /dev/null 2> /dev/null || framework_failure cp --sparse=always sparse copy || fail=1 # Ensure that the copy has the same block count as the original. -test `stat --printf %b copy` -le `stat --printf %b sparse` || fail=1 +test $(stat --printf %b copy) -le $(stat --printf %b sparse) || fail=1 # Ensure that --sparse={always,never} with --reflink fail. cp --sparse=always --reflink sparse copy && fail=1 -- 1.5.4.3 Thanks, -Jeff -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Mon May 24 05:35:51 2010 Received: (at submit) by debbugs.gnu.org; 24 May 2010 09:35:51 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGU4d-0007PO-A2 for submit@debbugs.gnu.org; Mon, 24 May 2010 05:35:51 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGU4b-0007PJ-1N for submit@debbugs.gnu.org; Mon, 24 May 2010 05:35:49 -0400 Received: from lists.gnu.org ([199.232.76.165]:52493) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OGU4Y-0001vX-B5 for submit@debbugs.gnu.org; Mon, 24 May 2010 05:35:46 -0400 Received: from [140.186.70.92] (port=53183 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OGU4W-0007t0-D7 for bug-coreutils@gnu.org; Mon, 24 May 2010 05:35:45 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OGU4U-0006o4-9Z for bug-coreutils@gnu.org; Mon, 24 May 2010 05:35:44 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:60836) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGU4U-0006nm-46 for bug-coreutils@gnu.org; Mon, 24 May 2010 05:35:42 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4O9ZYt4025777 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 24 May 2010 09:35:37 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4O75Xfk005583; Mon, 24 May 2010 09:35:30 GMT Received: from abhmt006.oracle.com by acsmt354.oracle.com with ESMTP id 292441511274693646; Mon, 24 May 2010 02:34:06 -0700 Received: from [10.182.121.3] (/10.182.121.3) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 24 May 2010 02:34:06 -0700 Message-ID: <4BFA4809.7050206@oracle.com> Date: Mon, 24 May 2010 17:34:01 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <87mxvtgvqg.fsf@meyering.net> <4BF6A055.6020205@oracle.com> <87ljb96920.fsf@meyering.net> <4BFA3878.9080606@oracle.com> In-Reply-To: <4BFA3878.9080606@oracle.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090209.4BFA486B.01D7:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.2 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.2 (------) jeff.liu wrote: > Jim Meyering wrote: >> jeff.liu wrote: >>> Jim Meyering wrote: >>>> jeff.liu wrote: >>>> ... >>>>>>> Subject: [PATCH 1/1] tests: add a new test for FIEMAP-copy >>>>>>> >>>>>>> * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a >>>>>>> loopbacked ext4 partition. >>>>>>> * tests/Makefile.am (sparse-fiemap): Reference the new test. >>>> BTW, I've just made this additional change to your test, >>>> >>>> diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap >>>> index 6312a4c..bdc7ded 100755 >>>> --- a/tests/cp/sparse-fiemap >>>> +++ b/tests/cp/sparse-fiemap >>>> @@ -27,6 +27,7 @@ require_root_ >>>> cwd=`pwd` >>>> cleanup_() { cd /; umount "$cwd/mnt"; } >>>> >>>> +skip=0 >> More fixes for the test: >> - remove useless rm >> - $sparse and $fiemap were not defined in that final test, >> so cd into mnt/ and just use the file names instead. >> - I prefer $(...) to `...`, and it's portable in this context >> >> diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap >> index 32ca5fc..cec5224 100755 >> --- a/tests/cp/sparse-fiemap >> +++ b/tests/cp/sparse-fiemap >> @@ -40,18 +40,18 @@ test -s mnt/f || skip=1 >> test $skip = 1 && >> skip_test_ "insufficient mount/ext4 support" >> >> -rm -f mnt/f >> - >> # Create a 1TiB sparse file >> dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure >> >> +cd mnt || fail=1 >> + >> # It takes many minutes to copy this sparse file using the old method. >> # By contrast, it takes far less than 1 second using FIEMAP-copy. >> -timeout 10 cp --sparse=always mnt/sparse mnt/sparse_fiemap || fail=1 >> +timeout 10 cp --sparse=always sparse fiemap || fail=1 >> >> # Ensure that the sparse file copied through fiemap has the same size >> # in bytes as the original. >> -test `stat --printf %s $sparse` = `stat --printf %s $fiemap` || fail=1 >> +test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 >> >> # ================================================= >> # Ensure that we exercise the FIEMAP-copying code enough >> -- >> 1.7.1.262.g5ef3d > > Thanks for the info. > > For the point of *- I prefer $(...) to `...`, and it's portable in this context* > > Could you check the tiny patch below, it does the same thing to improve the portability of > 'tests/cp/sparse', I have referred to it as a model. > > From 9fbffcd0f4d4706f4b88e7ac73ca13b62c789047 Mon Sep 17 00:00:00 2001 > From: Jie Liu > Date: Mon, 24 May 2010 16:01:06 +0800 > Subject: [PATCH 1/1] cp: improve the portability of test > > * tests/cp/sparse: improve the portability using shell constructs. > > Signed-off-by: Jie Liu > --- > tests/cp/sparse | 4 ++-- > 1 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/tests/cp/sparse b/tests/cp/sparse > index 73c2924..dee6de2 100755 > --- a/tests/cp/sparse > +++ b/tests/cp/sparse > @@ -28,14 +28,14 @@ require_sparse_support_ > # It has to be at least 128K in order to be sparse on some systems. > # Make its size one larger than 128K, in order to tickle the > # bug in coreutils-6.0. > -size=`expr 128 \* 1024 + 1` > +size=$((128 * 1024 + 1)) > dd bs=1 seek=$size of=sparse < /dev/null 2> /dev/null || framework_failure > > > cp --sparse=always sparse copy || fail=1 > > # Ensure that the copy has the same block count as the original. > -test `stat --printf %b copy` -le `stat --printf %b sparse` || fail=1 > +test $(stat --printf %b copy) -le $(stat --printf %b sparse) || fail=1 > > # Ensure that --sparse={always,never} with --reflink fail. > cp --sparse=always --reflink sparse copy && fail=1 Please ignore above patch, I just found another issue in 'tests/cp/sparse', the new created test file also named to 'sparse', so when it running, the `cp/sparse' will be truncated to `expr 128 \* 1024 + 1`. Below patch fix it to create a sparse file 'sparse1' instead(I can not find out a better name for now), s/-le/=/ to compare the block count. >From 0669ac6d0497a3c6abfc5d53202afc6bc47d0d07 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Mon, 24 May 2010 17:29:27 +0800 Subject: [PATCH 1/1] cp: enhance the sparse file copy test * tests/cp/sparse: fix sparse file name to 'sparse1', improve the protability using shell constructs. Signed-off-by: Jie Liu --- tests/cp/sparse | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/tests/cp/sparse b/tests/cp/sparse index 73c2924..cab8b9c 100755 --- a/tests/cp/sparse +++ b/tests/cp/sparse @@ -28,17 +28,17 @@ require_sparse_support_ # It has to be at least 128K in order to be sparse on some systems. # Make its size one larger than 128K, in order to tickle the # bug in coreutils-6.0. -size=`expr 128 \* 1024 + 1` -dd bs=1 seek=$size of=sparse < /dev/null 2> /dev/null || framework_failure +size=$((128 * 1024 + 1)) +dd bs=1 seek=$size of=sparse1 < /dev/null 2> /dev/null || framework_failure -cp --sparse=always sparse copy || fail=1 +cp --sparse=always sparse1 copy || fail=1 # Ensure that the copy has the same block count as the original. -test `stat --printf %b copy` -le `stat --printf %b sparse` || fail=1 +test $(stat --printf %b copy) = $(stat --printf %b sparse1) || fail=1 # Ensure that --sparse={always,never} with --reflink fail. -cp --sparse=always --reflink sparse copy && fail=1 -cp --sparse=never --reflink sparse copy && fail=1 +cp --sparse=always --reflink sparse1 copy && fail=1 +cp --sparse=never --reflink sparse1 copy && fail=1 Exit $fail -- 1.5.4.3 -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Mon May 24 06:01:24 2010 Received: (at submit) by debbugs.gnu.org; 24 May 2010 10:01:24 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGUTK-0007a6-P9 for submit@debbugs.gnu.org; Mon, 24 May 2010 06:01:23 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGUTI-0007a1-DG for submit@debbugs.gnu.org; Mon, 24 May 2010 06:01:21 -0400 Received: from lists.gnu.org ([199.232.76.165]:42113) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OGUTF-0002T0-Oc for submit@debbugs.gnu.org; Mon, 24 May 2010 06:01:17 -0400 Received: from [140.186.70.92] (port=59043 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OGUTE-0007kK-9D for bug-coreutils@gnu.org; Mon, 24 May 2010 06:01:17 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OGUTC-0002xd-BQ for bug-coreutils@gnu.org; Mon, 24 May 2010 06:01:16 -0400 Received: from smtpfb2-g21.free.fr ([212.27.42.10]:38584) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGSPs-0007hk-Bh for bug-coreutils@gnu.org; Mon, 24 May 2010 03:49:41 -0400 Received: from smtp5-g21.free.fr (smtp5-g21.free.fr [212.27.42.5]) by smtpfb2-g21.free.fr (Postfix) with ESMTP id 99F3CD1AE13 for ; Mon, 24 May 2010 09:41:33 +0200 (CEST) Received: from smtp5-g21.free.fr (localhost [127.0.0.1]) by smtp5-g21.free.fr (Postfix) with ESMTP id 60E53D48180; Mon, 24 May 2010 09:41:28 +0200 (CEST) Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp5-g21.free.fr (Postfix) with ESMTP; Mon, 24 May 2010 09:41:27 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 8BF14DEC5; Mon, 24 May 2010 09:41:27 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4BF6A055.6020205@oracle.com> (jeff liu's message of "Fri, 21 May 2010 23:01:41 +0800") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <87mxvtgvqg.fsf@meyering.net> <4BF6A055.6020205@oracle.com> Date: Mon, 24 May 2010 09:41:27 +0200 Message-ID: <87ljb96920.fsf@meyering.net> Lines: 57 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.2 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.2 (-----) jeff.liu wrote: > Jim Meyering wrote: >> jeff.liu wrote: >> ... >>>>> Subject: [PATCH 1/1] tests: add a new test for FIEMAP-copy >>>>> >>>>> * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a >>>>> loopbacked ext4 partition. >>>>> * tests/Makefile.am (sparse-fiemap): Reference the new test. >> >> BTW, I've just made this additional change to your test, >> >> diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap >> index 6312a4c..bdc7ded 100755 >> --- a/tests/cp/sparse-fiemap >> +++ b/tests/cp/sparse-fiemap >> @@ -27,6 +27,7 @@ require_root_ >> cwd=`pwd` >> cleanup_() { cd /; umount "$cwd/mnt"; } >> >> +skip=0 More fixes for the test: - remove useless rm - $sparse and $fiemap were not defined in that final test, so cd into mnt/ and just use the file names instead. - I prefer $(...) to `...`, and it's portable in this context diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index 32ca5fc..cec5224 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -40,18 +40,18 @@ test -s mnt/f || skip=1 test $skip = 1 && skip_test_ "insufficient mount/ext4 support" -rm -f mnt/f - # Create a 1TiB sparse file dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure +cd mnt || fail=1 + # It takes many minutes to copy this sparse file using the old method. # By contrast, it takes far less than 1 second using FIEMAP-copy. -timeout 10 cp --sparse=always mnt/sparse mnt/sparse_fiemap || fail=1 +timeout 10 cp --sparse=always sparse fiemap || fail=1 # Ensure that the sparse file copied through fiemap has the same size # in bytes as the original. -test `stat --printf %s $sparse` = `stat --printf %s $fiemap` || fail=1 +test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 # ================================================= # Ensure that we exercise the FIEMAP-copying code enough -- 1.7.1.262.g5ef3d From debbugs-submit-bounces@debbugs.gnu.org Tue May 25 01:50:45 2010 Received: (at submit) by debbugs.gnu.org; 25 May 2010 05:50:45 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGn2K-000067-Tw for submit@debbugs.gnu.org; Tue, 25 May 2010 01:50:45 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGn2J-000061-B4 for submit@debbugs.gnu.org; Tue, 25 May 2010 01:50:44 -0400 Received: from lists.gnu.org ([199.232.76.165]:36480) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OGn2F-0007gt-OW for submit@debbugs.gnu.org; Tue, 25 May 2010 01:50:39 -0400 Received: from [140.186.70.92] (port=49395 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OGn2D-0006pR-RS for bug-coreutils@gnu.org; Tue, 25 May 2010 01:50:39 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OGn2C-0004aR-Ma for bug-coreutils@gnu.org; Tue, 25 May 2010 01:50:37 -0400 Received: from smtp5-g21.free.fr ([212.27.42.5]:50582) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGn2C-0004aA-3o for bug-coreutils@gnu.org; Tue, 25 May 2010 01:50:36 -0400 Received: from smtp5-g21.free.fr (localhost [127.0.0.1]) by smtp5-g21.free.fr (Postfix) with ESMTP id C683CD48050; Tue, 25 May 2010 07:50:29 +0200 (CEST) Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp5-g21.free.fr (Postfix) with ESMTP; Tue, 25 May 2010 07:50:28 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 02771F843; Tue, 25 May 2010 07:50:27 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4BFA4809.7050206@oracle.com> (jeff liu's message of "Mon, 24 May 2010 17:34:01 +0800") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <87mxvtgvqg.fsf@meyering.net> <4BF6A055.6020205@oracle.com> <87ljb96920.fsf@meyering.net> <4BFA3878.9080606@oracle.com> <4BFA4809.7050206@oracle.com> Date: Tue, 25 May 2010 07:50:27 +0200 Message-ID: <8739xgbkd8.fsf@meyering.net> Lines: 56 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.2 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.2 (-----) jeff.liu wrote: ... > Please ignore above patch, I just found another issue in 'tests/cp/sparse', the new created test > file also named to 'sparse', so when it running, the `cp/sparse' will be truncated to `expr 128 \* > 1024 + 1`. > > Below patch fix it to create a sparse file 'sparse1' instead(I can not find out a better name for > now), s/-le/=/ to compare the block count. > >>>From 0669ac6d0497a3c6abfc5d53202afc6bc47d0d07 Mon Sep 17 00:00:00 2001 > From: Jie Liu > Date: Mon, 24 May 2010 17:29:27 +0800 > Subject: [PATCH 1/1] cp: enhance the sparse file copy test > > * tests/cp/sparse: fix sparse file name to 'sparse1', improve > the protability using shell constructs. > > Signed-off-by: Jie Liu > --- > tests/cp/sparse | 12 ++++++------ > 1 files changed, 6 insertions(+), 6 deletions(-) > > diff --git a/tests/cp/sparse b/tests/cp/sparse > index 73c2924..cab8b9c 100755 > --- a/tests/cp/sparse > +++ b/tests/cp/sparse > @@ -28,17 +28,17 @@ require_sparse_support_ > # It has to be at least 128K in order to be sparse on some systems. > # Make its size one larger than 128K, in order to tickle the > # bug in coreutils-6.0. > -size=`expr 128 \* 1024 + 1` > -dd bs=1 seek=$size of=sparse < /dev/null 2> /dev/null || framework_failure > +size=$((128 * 1024 + 1)) Thank you, but I will not use this patch. First, $((...)) is *not* portable. Note that while $(...) is an improvement in coreutils tests, in most projects converting `...` to $(...) would represent a portability *regression*. It happens to be acceptable in coreutils tests (and thus preferred by me) because coreutils ensures that tests are run using a shell that is modern enough to accept $(...). Second, while I prefer $(...), it's not worth converting them one by one. There are over 500+ uses in tests/. > +dd bs=1 seek=$size of=sparse1 < /dev/null 2> /dev/null || framework_failure Are you worried about tests running in parallel, and thus this test's "sparse" file colliding with the one by the same name in the fiemap test? That's not a problem, since each is run in its own separate subdirectory, via the machinery in test-lib.sh. > -cp --sparse=always sparse copy || fail=1 > +cp --sparse=always sparse1 copy || fail=1 From debbugs-submit-bounces@debbugs.gnu.org Tue May 25 04:11:56 2010 Received: (at submit) by debbugs.gnu.org; 25 May 2010 08:11:56 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGpEw-00016g-Ka for submit@debbugs.gnu.org; Tue, 25 May 2010 04:11:56 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGpEu-00016b-PR for submit@debbugs.gnu.org; Tue, 25 May 2010 04:11:53 -0400 Received: from lists.gnu.org ([199.232.76.165]:35680) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OGpEr-0001cN-L6 for submit@debbugs.gnu.org; Tue, 25 May 2010 04:11:49 -0400 Received: from [140.186.70.92] (port=39691 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OGpEp-0008R5-Ia for bug-coreutils@gnu.org; Tue, 25 May 2010 04:11:48 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OGpEo-0008Bv-Ep for bug-coreutils@gnu.org; Tue, 25 May 2010 04:11:47 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:31377) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGpEo-0008Bf-89 for bug-coreutils@gnu.org; Tue, 25 May 2010 04:11:46 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4P8Bg5t019121 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 25 May 2010 08:11:44 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4P5visC018762; Tue, 25 May 2010 08:11:41 GMT Received: from abhmt021.oracle.com by acsmt353.oracle.com with ESMTP id 265042381274775098; Tue, 25 May 2010 01:11:38 -0700 Received: from [221.223.115.200] (/221.223.115.200) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 25 May 2010 01:11:37 -0700 Message-ID: <4BFB8635.5030805@oracle.com> Date: Tue, 25 May 2010 16:11:33 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <87mxvtgvqg.fsf@meyering.net> <4BF6A055.6020205@oracle.com> <87ljb96920.fsf@meyering.net> <4BFA3878.9080606@oracle.com> <4BFA4809.7050206@oracle.com> <8739xgbkd8.fsf@meyering.net> In-Reply-To: <8739xgbkd8.fsf@meyering.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090202.4BFB8640.0101:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.5 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.5 (-----) Jim Meyering wrote: > jeff.liu wrote: > ... >> Please ignore above patch, I just found another issue in 'tests/cp/sparse', the new created test >> file also named to 'sparse', so when it running, the `cp/sparse' will be truncated to `expr 128 \* >> 1024 + 1`. >> >> Below patch fix it to create a sparse file 'sparse1' instead(I can not find out a better name for >> now), s/-le/=/ to compare the block count. >> >> >From 0669ac6d0497a3c6abfc5d53202afc6bc47d0d07 Mon Sep 17 00:00:00 2001 >> From: Jie Liu >> Date: Mon, 24 May 2010 17:29:27 +0800 >> Subject: [PATCH 1/1] cp: enhance the sparse file copy test >> >> * tests/cp/sparse: fix sparse file name to 'sparse1', improve >> the protability using shell constructs. >> >> Signed-off-by: Jie Liu >> --- >> tests/cp/sparse | 12 ++++++------ >> 1 files changed, 6 insertions(+), 6 deletions(-) >> >> diff --git a/tests/cp/sparse b/tests/cp/sparse >> index 73c2924..cab8b9c 100755 >> --- a/tests/cp/sparse >> +++ b/tests/cp/sparse >> @@ -28,17 +28,17 @@ require_sparse_support_ >> # It has to be at least 128K in order to be sparse on some systems. >> # Make its size one larger than 128K, in order to tickle the >> # bug in coreutils-6.0. >> -size=`expr 128 \* 1024 + 1` >> -dd bs=1 seek=$size of=sparse < /dev/null 2> /dev/null || framework_failure >> +size=$((128 * 1024 + 1)) > > Thank you, but I will not use this patch. > > First, $((...)) is *not* portable. > Note that while $(...) is an improvement in coreutils tests, in most projects > converting `...` to $(...) would represent a portability *regression*. > It happens to be acceptable in coreutils tests (and thus preferred by me) > because coreutils ensures that tests are run using a shell that is > modern enough to accept $(...). Thanks for the clarification. > > Second, while I prefer $(...), it's not worth converting them > one by one. There are over 500+ uses in tests/. > >> +dd bs=1 seek=$size of=sparse1 < /dev/null 2> /dev/null || framework_failure > > Are you worried about tests running in parallel, and thus > this test's "sparse" file colliding with the one by > the same name in the fiemap test? No. At first, I took a look at the file 'cp/sparse', I consider it could be truncated by the following DD(1) operation, because the target name also named as 'sparse' which is same as 'cp/sparse', if the test run at the same physical directory(i.e. tests/cp/) ++ expr 128 '*' 1024 + 1 + size=131073 + dd bs=1 seek=131073 of=sparse + cp --sparse=always sparse copy ++ stat --printf %b copy ++ stat --printf %b sparse + test 8 -le 8 but by checking through the run log file, looks all the tests are safely run at their own sub-dirs, Sorry for my take for granted for this point. +++ /home/jeff/opensource_dev/coreutils/src/mktemp -d --tmp=/home/jeff/opensource_dev/coreutils/tests cu-sparse.XXXXXXXXXX > That's not a problem, since each is run in its own > separate subdirectory, via the machinery in test-lib.sh. > >> -cp --sparse=always sparse copy || fail=1 >> +cp --sparse=always sparse1 copy || fail=1 > > > Thanks, -Jeff -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Thu May 27 06:31:26 2010 Received: (at submit) by debbugs.gnu.org; 27 May 2010 10:31:26 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHaN3-0000J2-DU for submit@debbugs.gnu.org; Thu, 27 May 2010 06:31:26 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHaMy-0000It-Tb for submit@debbugs.gnu.org; Thu, 27 May 2010 06:31:22 -0400 Received: from lists.gnu.org ([199.232.76.165]:39963) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OHaMt-0003II-Iu for submit@debbugs.gnu.org; Thu, 27 May 2010 06:31:15 -0400 Received: from [140.186.70.92] (port=46349 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OHaMr-0005l7-5w for bug-coreutils@gnu.org; Thu, 27 May 2010 06:31:15 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OHaMl-0002p4-IA for bug-coreutils@gnu.org; Thu, 27 May 2010 06:31:13 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:51697) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHaMk-0002oQ-Mo for bug-coreutils@gnu.org; Thu, 27 May 2010 06:31:07 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id 4EA8494000A; Thu, 27 May 2010 12:30:59 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 2A856ECD; Thu, 27 May 2010 12:30:58 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4BF6ABE7.3020600@oracle.com> (jeff liu's message of "Fri, 21 May 2010 23:51:03 +0800") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> Date: Thu, 27 May 2010 12:30:58 +0200 Message-ID: <877hmpeivx.fsf@meyering.net> Lines: 780 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -3.9 (---) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.2 (-----) jeff.liu wrote: > This is the revised version, it fixed the fiemap-start offset calculation > approach to remove it out > of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)' loop. Hi Jeff, I've included below the state of my local changes. Unfortunately, with that 5-patch series, there is always a test failure on F13/ext4. Maybe someone who knows more about extents can provide an explanation? Here's a small example to demonstrate: Create a file with many extents: perl -e 'BEGIN { $n = 19 * 1024; *F = *STDOUT }' \ -e 'for (1..100) { sysseek (*F, $n, 1)' \ -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 Using the patched "cp", repeat the following 10 or 20 times: ./cp --sparse=always j1 j2; sync filefrag -v j1 | awk '/^ / {print $1,$2}' > ff1 || fail=1 filefrag -v j2 | awk '/^ / {print $1,$2}' > ff2 || fail=1 diff -u ff1 ff2 || fail=1 Usually there is no diff output, but occasionally it'll print this: [hmm... today it consistently prints these differences every other time.] $ diff -u ff1 ff2 || fail=1 --- ff1 2010-05-22 18:42:26.943501382 +0200 +++ ff2 2010-05-22 18:42:27.020876155 +0200 @@ -53,49 +53,50 @@ ext logical 51 489 52 498 53 508 -54 517 -55 527 -56 536 -57 546 -58 555 -59 565 -60 574 -61 584 -62 593 -63 603 -64 612 -65 622 -66 631 -67 641 -68 650 -69 660 -70 669 -71 679 -72 688 -73 698 -74 707 -75 717 -76 726 -77 736 -78 745 -79 755 -80 764 -81 774 -82 783 -83 793 -84 802 -85 812 -86 821 -87 831 -88 840 -89 850 -90 859 -91 869 -92 878 -93 888 -94 897 -95 907 -96 916 -97 926 -98 935 -99 945 +54 512 +55 517 +56 527 +57 536 +58 546 +59 555 +60 565 +61 574 +62 584 +63 593 +64 603 +65 612 +66 622 +67 631 +68 641 +69 650 +70 660 +71 669 +72 679 +73 688 +74 698 +75 707 +76 717 +77 726 +78 736 +79 745 +80 755 +81 764 +82 774 +83 783 +84 793 +85 802 +86 812 +87 821 +88 831 +89 840 +90 850 +91 859 +92 869 +93 878 +94 888 +95 897 +96 907 +97 916 +98 926 +99 935 +100 945 >From 2ef44bcb0bbb38cca738aa90e067caee312da939 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Thu, 13 May 2010 22:09:30 +0800 Subject: [PATCH 1/5] cp: Add FIEMAP support for efficient sparse file copy * src/fiemap.h: Add fiemap.h for fiemap ioctl(2) support. Copied from linux's include/linux/fiemap.h, with minor formatting changes. * src/copy.c (copy_reg): Now, when `cp' invoked with --sparse=[WHEN] option, we will try to do FIEMAP-copy if the underlaying file system support it, fall back to a normal copy if it fails. --- src/copy.c | 159 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/fiemap.h | 102 +++++++++++++++++++++++++++++++++++++ 2 files changed, 261 insertions(+), 0 deletions(-) create mode 100644 src/fiemap.h diff --git a/src/copy.c b/src/copy.c index c16cef6..0e54729 100644 --- a/src/copy.c +++ b/src/copy.c @@ -63,6 +63,10 @@ #include +#ifndef HAVE_FIEMAP +# include "fiemap.h" +#endif + #ifndef HAVE_FCHOWN # define HAVE_FCHOWN false # define fchown(fd, uid, gid) (-1) @@ -149,6 +153,141 @@ clone_file (int dest_fd, int src_fd) #endif } +#ifdef __linux__ +# ifndef FS_IOC_FIEMAP +# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) +# endif +/* Perform FIEMAP(available in mainline 2.6.27) copy if possible. + Call ioctl(2) with FS_IOC_FIEMAP to efficiently map file allocation + excepts holes. So the overhead to deal with holes with lseek(2) in + normal copy could be saved. This would result in much faster backups + for any kind of sparse file. */ +static bool +fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, + off_t src_total_size, char const *src_name, + char const *dst_name, bool *normal_copy_required) +{ + bool fail = false; + bool last = false; + char fiemap_buf[4096]; + struct fiemap *fiemap = (struct fiemap *)fiemap_buf; + struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; + uint32_t count = (sizeof (fiemap_buf) - sizeof (*fiemap)) / + sizeof (struct fiemap_extent); + off_t last_ext_logical = 0; + uint64_t last_ext_len = 0; + uint64_t last_read_size = 0; + unsigned int i = 0; + + /* This is required at least to initialize fiemap->fm_start, + but also serves (in May 2010) to appease valgrind, which + appears not to know the semantics of the FIEMAP ioctl. */ + memset (fiemap_buf, 0, sizeof fiemap_buf); + + do + { + fiemap->fm_length = FIEMAP_MAX_OFFSET; + fiemap->fm_extent_count = count; + + /* When ioctl(2) fails, fall back to the normal copy only if it + is the first time we met. */ + if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) + { + /* If `i > 0', then at least one ioctl(2) has been performed before. */ + if (i == 0) + *normal_copy_required = true; + return false; + } + + /* If 0 extents are returned, then more ioctls are not needed. */ + if (fiemap->fm_mapped_extents == 0) + break; + + for (i = 0; i < fiemap->fm_mapped_extents; i++) + { + assert (fm_ext[i].fe_logical <= OFF_T_MAX); + + off_t ext_logical = fm_ext[i].fe_logical; + uint64_t ext_len = fm_ext[i].fe_length; + + if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) + { + error (0, errno, _("cannot lseek %s"), quote (src_name)); + return fail; + } + + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0LL) + { + error (0, errno, _("cannot lseek %s"), quote (dst_name)); + return fail; + } + + if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST) + { + last_ext_logical = ext_logical; + last_ext_len = ext_len; + last = true; + } + + while (0 < ext_len) + { + char buf[buf_size]; + + /* Avoid reading into the holes if the left extent + length is shorter than the buffer size. */ + if (ext_len < buf_size) + buf_size = ext_len; + + ssize_t n_read = read (src_fd, buf, buf_size); + if (n_read < 0) + { +#ifdef EINTR + if (errno == EINTR) + continue; +#endif + error (0, errno, _("reading %s"), quote (src_name)); + return fail; + } + + if (n_read == 0) + { + /* Figure out how many bytes read from the last extent. */ + last_read_size = last_ext_len - ext_len; + break; + } + + if (full_write (dest_fd, buf, n_read) != n_read) + { + error (0, errno, _("writing %s"), quote (dst_name)); + return fail; + } + + ext_len -= n_read; + } + } + + fiemap->fm_start = fm_ext[i - 1].fe_logical + fm_ext[i - 1].fe_length; + + } while (! last); + + /* If a file ends up with holes, the sum of the last extent logical offset + and the read-returned size will be shorter than the actual size of the + file. Use ftruncate to extend the length of the destination file. */ + if (last_ext_logical + last_read_size < src_total_size) + { + if (ftruncate (dest_fd, src_total_size) < 0) + { + error (0, errno, _("extending %s"), quote (dst_name)); + return fail; + } + } + + return ! fail; +} +#else +static bool fiemap_copy_ok (ignored) { errno == ENOTSUP; return false; } +#endif + /* FIXME: describe */ /* FIXME: rewrite this to use a hash table so we avoid the quadratic performance hit that's probably noticeable only on trees deeper @@ -679,6 +818,25 @@ copy_reg (char const *src_name, char const *dst_name, #endif } + if (make_holes) + { + bool require_normal_copy = false; + /* Perform efficient FIEMAP copy for sparse files, fall back to the + standard copy only if the ioctl(2) fails. */ + if (fiemap_copy_ok (source_desc, dest_desc, buf_size, + src_open_sb.st_size, src_name, + dst_name, &require_normal_copy)) + goto preserve_metadata; + else + { + if (! require_normal_copy) + { + return_val = false; + goto close_src_and_dst_desc; + } + } + } + /* If not making a sparse file, try to use a more-efficient buffer size. */ if (! make_holes) @@ -807,6 +965,7 @@ copy_reg (char const *src_name, char const *dst_name, } } +preserve_metadata: if (x->preserve_timestamps) { struct timespec timespec[2]; diff --git a/src/fiemap.h b/src/fiemap.h new file mode 100644 index 0000000..d33293b --- /dev/null +++ b/src/fiemap.h @@ -0,0 +1,102 @@ +/* FS_IOC_FIEMAP ioctl infrastructure. + Some portions copyright (C) 2007 Cluster File Systems, Inc + Authors: Mark Fasheh + Kalpak Shah + Andreas Dilger . */ + +/* Copy from kernel, modified to respect GNU code style by Jie Liu. */ + +#ifndef _LINUX_FIEMAP_H +# define _LINUX_FIEMAP_H + +# include + +struct fiemap_extent +{ + /* Logical offset in bytes for the start of the extent + from the beginning of the file. */ + uint64_t fe_logical; + + /* Physical offset in bytes for the start of the extent + from the beginning of the disk. */ + uint64_t fe_physical; + + /* Length in bytes for this extent. */ + uint64_t fe_length; + + uint64_t fe_reserved64[2]; + + /* FIEMAP_EXTENT_* flags for this extent. */ + uint32_t fe_flags; + + uint32_t fe_reserved[3]; +}; + +struct fiemap +{ + /* Logical offset(inclusive) at which to start mapping(in). */ + uint64_t fm_start; + + /* Logical length of mapping which userspace wants(in). */ + uint64_t fm_length; + + /* FIEMAP_FLAG_* flags for request(in/out). */ + uint32_t fm_flags; + + /* Number of extents that were mapped(out). */ + uint32_t fm_mapped_extents; + + /* Size of fm_extents array(in). */ + uint32_t fm_extent_count; + + uint32_t fm_reserved; + + /* Array of mapped extents(out). */ + struct fiemap_extent fm_extents[0]; +}; + +/* The maximum offset can be mapped for a file. */ +# define FIEMAP_MAX_OFFSET (~0ULL) + +/* Sync file data before map. */ +# define FIEMAP_FLAG_SYNC 0x00000001 + +/* Map extented attribute tree. */ +# define FIEMAP_FLAG_XATTR 0x00000002 + +# define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR) + +/* Last extent in file. */ +# define FIEMAP_EXTENT_LAST 0x00000001 + +/* Data location unknown. */ +# define FIEMAP_EXTENT_UNKNOWN 0x00000002 + +/* Location still pending, Sets EXTENT_UNKNOWN. */ +# define FIEMAP_EXTENT_DELALLOC 0x00000004 + +/* Data can not be read while fs is unmounted. */ +# define FIEMAP_EXTENT_ENCODED 0x00000008 + +/* Data is encrypted by fs. Sets EXTENT_NO_BYPASS. */ +# define FIEMAP_EXTENT_DATA_ENCRYPTED 0x00000080 + +/* Extent offsets may not be block aligned. */ +# define FIEMAP_EXTENT_NOT_ALIGNED 0x00000100 + +/* Data mixed with metadata. Sets EXTENT_NOT_ALIGNED. */ +# define FIEMAP_EXTENT_DATA_INLINE 0x00000200 + +/* Multiple files in block. Set EXTENT_NOT_ALIGNED. */ +# define FIEMAP_EXTENT_DATA_TAIL 0x00000400 + +/* Space allocated, but not data (i.e. zero). */ +# define FIEMAP_EXTENT_UNWRITTEN 0x00000800 + +/* File does not natively support extents. Result merged for efficiency. */ +# define FIEMAP_EXTENT_MERGED 0x00001000 + +/* Space shared with other files. */ +# define FIEMAP_EXTENT_SHARED 0x00002000 + +#endif -- 1.7.1.348.gb26ba >From 260b5b89e33da2b9a5ea5bcd9dba874f503d2937 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Thu, 13 May 2010 22:17:53 +0800 Subject: [PATCH 2/5] tests: add a new test for FIEMAP-copy * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a loopbacked ext4 partition. * tests/Makefile.am (sparse-fiemap): Reference the new test. --- tests/Makefile.am | 1 + tests/cp/sparse-fiemap | 56 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 0 deletions(-) create mode 100755 tests/cp/sparse-fiemap diff --git a/tests/Makefile.am b/tests/Makefile.am index c458574..f7840c8 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -25,6 +25,7 @@ root_tests = \ cp/special-bits \ cp/cp-mv-enotsup-xattr \ cp/capability \ + cp/sparse-fiemap \ dd/skip-seek-past-dev \ install/install-C-root \ ls/capability \ diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap new file mode 100755 index 0000000..945c94b --- /dev/null +++ b/tests/cp/sparse-fiemap @@ -0,0 +1,56 @@ +#!/bin/sh +# Test cp --sparse=always through fiemap copy + +# Copyright (C) 2006-2010 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +if test "$VERBOSE" = yes; then + set -x + cp --version +fi + +. $srcdir/test-lib.sh +require_root_ + +cwd=`pwd` +cleanup_() { cd /; umount "$cwd/mnt"; } + +skip=0 +# Create an ext4 loopback file system +dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 +mkdir mnt +mkfs -t ext4 -F blob || + skip_test_ "failed to create ext4 file system" +mount -oloop blob mnt || skip=1 +echo test > mnt/f || skip=1 +test -s mnt/f || skip=1 + +test $skip = 1 && + skip_test_ "insufficient mount/ext4 support" + +# Create a 1TiB sparse file +dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure + +cd mnt || fail=1 + +# It takes many minutes to copy this sparse file using the old method. +# By contrast, it takes far less than 1 second using FIEMAP-copy. +timeout 10 cp --sparse=always sparse fiemap || fail=1 + +# Ensure that the sparse file copied through fiemap has the same size +# in bytes as the original. +test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 + +Exit $fail -- 1.7.1.348.gb26ba >From 738031aed78b6323969f799d3d5fbf19e0cfc91a Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Fri, 21 May 2010 18:28:42 +0200 Subject: [PATCH 3/5] tests: exercise more of the new FIEMAP copying code * tests/cp/sparse-fiemap: Ensure that a file with many extents (more than fit in copy.c's internal 4KiB buffer) is copied properly. --- tests/cp/sparse-fiemap | 38 ++++++++++++++++++++++++++++++++++++++ 1 files changed, 38 insertions(+), 0 deletions(-) diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index 945c94b..b1643be 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -53,4 +53,42 @@ timeout 10 cp --sparse=always sparse fiemap || fail=1 # in bytes as the original. test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 +# ================================================= +# Ensure that we exercise the FIEMAP-copying code enough +# to provoke at least two iterations of the do...while loop +# in which it calls ioctl (fd, FS_IOC_FIEMAP,... +# This also verifies that non-trivial extents are preserved. + +$PERL -e 1 || skip_test_ 'skipping part of this test; you lack perl' + +$PERL -e 'BEGIN { $n = 16 * 1024; *F = *STDOUT }' \ + -e 'for (1..100) { sysseek (*F, $n, 1)' \ + -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1 + +cp --sparse=always j1 j2 || fail=1 +cmp j1 j2 || fail=1 + +filefrag j1 | grep extent \ + || skip_test_ 'skipping part of this test; you lack filefrag' + +# Here is sample filefrag output: +# $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \ +# -e 'for (1..5) { sysseek(*F,$n,1)' \ +# -e '&& syswrite *F,"."x$n or die "$!"}' > j +# $ filefrag -v j +# Filesystem type is: ef53 +# File size of j is 163840 (40 blocks, blocksize 4096) +# ext logical physical expected length flags +# 0 4 6258884 4 +# 1 12 6258892 6258887 4 +# 2 20 6258900 6258895 4 +# 3 28 6258908 6258903 4 +# 4 36 6258916 6258911 4 eof +# j: 6 extents found + +# exclude the physical block numbers; they always differ +filefrag -v j1 | awk '/^ / {print $1,$2,$NF}' > ff1 || fail=1 +filefrag -v j2 | awk '/^ / {print $1,$2,$NF}' > ff2 || fail=1 +compare ff1 ff2 || fail=1 + Exit $fail -- 1.7.1.348.gb26ba >From e84f881cb5c2eb92f3b6d5bddaef50c3e811bc30 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sat, 22 May 2010 10:22:58 +0200 Subject: [PATCH 4/5] tests: require root access only if current partition is not ext4 * tests/cp/sparse-fiemap: Don't require root access if current partition is ext4. --- tests/cp/sparse-fiemap | 44 +++++++++++++++++++++++++------------------- 1 files changed, 25 insertions(+), 19 deletions(-) diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index b1643be..371bced 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -22,28 +22,34 @@ if test "$VERBOSE" = yes; then fi . $srcdir/test-lib.sh -require_root_ -cwd=`pwd` -cleanup_() { cd /; umount "$cwd/mnt"; } - -skip=0 -# Create an ext4 loopback file system -dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 -mkdir mnt -mkfs -t ext4 -F blob || - skip_test_ "failed to create ext4 file system" -mount -oloop blob mnt || skip=1 -echo test > mnt/f || skip=1 -test -s mnt/f || skip=1 - -test $skip = 1 && - skip_test_ "insufficient mount/ext4 support" +if df -T -t ext4 . ; then + : # Current dir is on an ext4 partition. Good! +else + # It's not; we need to create one, hence we need root access. + require_root_ + + cwd=$PWD + cleanup_() { cd /; umount "$cwd/mnt"; } + + skip=0 + # Create an ext4 loopback file system + dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 + mkdir mnt + mkfs -t ext4 -F blob || + skip_test_ "failed to create ext4 file system" + mount -oloop blob mnt || skip=1 + echo test > mnt/f || skip=1 + test -s mnt/f || skip=1 + + test $skip = 1 && + skip_test_ "insufficient mount/ext4 support" + + cd mnt || fail=1 +fi # Create a 1TiB sparse file -dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure - -cd mnt || fail=1 +dd if=/dev/zero of=sparse bs=1k count=1 seek=1G || framework_failure # It takes many minutes to copy this sparse file using the old method. # By contrast, it takes far less than 1 second using FIEMAP-copy. -- 1.7.1.348.gb26ba >From 213c9247a406b267bc5819ccbe3b729e9557c675 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sat, 22 May 2010 10:21:46 +0200 Subject: [PATCH 5/5] tests: fiemap test improvement * tests/cp/sparse-fiemap: More tests. --- tests/cp/sparse-fiemap | 69 +++++++++++++++++++++++++----------------------- 1 files changed, 36 insertions(+), 33 deletions(-) diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index 371bced..ef3742e 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -38,14 +38,14 @@ else mkdir mnt mkfs -t ext4 -F blob || skip_test_ "failed to create ext4 file system" - mount -oloop blob mnt || skip=1 - echo test > mnt/f || skip=1 - test -s mnt/f || skip=1 + mount -oloop blob mnt || skip=1 + cd mnt || skip=1 + echo test > f || skip=1 + test -s f || skip=1 test $skip = 1 && skip_test_ "insufficient mount/ext4 support" - cd mnt || fail=1 fi # Create a 1TiB sparse file @@ -67,34 +67,37 @@ test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 $PERL -e 1 || skip_test_ 'skipping part of this test; you lack perl' -$PERL -e 'BEGIN { $n = 16 * 1024; *F = *STDOUT }' \ - -e 'for (1..100) { sysseek (*F, $n, 1)' \ - -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1 - -cp --sparse=always j1 j2 || fail=1 -cmp j1 j2 || fail=1 - -filefrag j1 | grep extent \ - || skip_test_ 'skipping part of this test; you lack filefrag' - -# Here is sample filefrag output: -# $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \ -# -e 'for (1..5) { sysseek(*F,$n,1)' \ -# -e '&& syswrite *F,"."x$n or die "$!"}' > j -# $ filefrag -v j -# Filesystem type is: ef53 -# File size of j is 163840 (40 blocks, blocksize 4096) -# ext logical physical expected length flags -# 0 4 6258884 4 -# 1 12 6258892 6258887 4 -# 2 20 6258900 6258895 4 -# 3 28 6258908 6258903 4 -# 4 36 6258916 6258911 4 eof -# j: 6 extents found - -# exclude the physical block numbers; they always differ -filefrag -v j1 | awk '/^ / {print $1,$2,$NF}' > ff1 || fail=1 -filefrag -v j2 | awk '/^ / {print $1,$2,$NF}' > ff2 || fail=1 -compare ff1 ff2 || fail=1 +for i in $(seq 20); do + for j in 1 2 31 100; do + $PERL -e 'BEGIN { $n = '$i' * 1024; *F = *STDOUT }' \ + -e 'for (1..'$j') { sysseek (*F, $n, 1)' \ + -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1 + + cp --sparse=always j1 j2 || fail=1 + cmp j1 j2 || fail=1 + filefrag -v j1 | grep extent \ + || skip_test_ 'skipping part of this test; you lack filefrag' + + # Here is sample filefrag output: + # $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \ + # -e 'for (1..5) { sysseek(*F,$n,1)' \ + # -e '&& syswrite *F,"."x$n or die "$!"}' > j + # $ filefrag -v j + # Filesystem type is: ef53 + # File size of j is 163840 (40 blocks, blocksize 4096) + # ext logical physical expected length flags + # 0 4 6258884 4 + # 1 12 6258892 6258887 4 + # 2 20 6258900 6258895 4 + # 3 28 6258908 6258903 4 + # 4 36 6258916 6258911 4 eof + # j: 6 extents found + + # exclude the physical block numbers; they always differ + filefrag -v j1 | awk '/^ / {print $1,$2}' > ff1 || fail=1 + filefrag -v j2 | awk '/^ / {print $1,$2}' > ff2 || fail=1 + compare ff1 ff2 || fail=1 + done +done Exit $fail -- 1.7.1.348.gb26ba From debbugs-submit-bounces@debbugs.gnu.org Thu May 27 10:18:33 2010 Received: (at submit) by debbugs.gnu.org; 27 May 2010 14:18:34 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHduq-0002WT-FO for submit@debbugs.gnu.org; Thu, 27 May 2010 10:18:33 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHdum-0002WO-K2 for submit@debbugs.gnu.org; Thu, 27 May 2010 10:18:31 -0400 Received: from lists.gnu.org ([199.232.76.165]:39203) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OHduh-0007xk-CP for submit@debbugs.gnu.org; Thu, 27 May 2010 10:18:23 -0400 Received: from [140.186.70.92] (port=57559 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OHduc-0005PP-PZ for bug-coreutils@gnu.org; Thu, 27 May 2010 10:18:22 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OHdmQ-0006BB-J9 for bug-coreutils@gnu.org; Thu, 27 May 2010 10:09:53 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:20391) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHdmQ-0006Aj-8n for bug-coreutils@gnu.org; Thu, 27 May 2010 10:09:50 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4RE9URf012357 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 27 May 2010 14:09:32 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4RBlLGd029511; Thu, 27 May 2010 14:09:26 GMT Received: from abhmt015.oracle.com by acsmt355.oracle.com with ESMTP id 273691271274969325; Thu, 27 May 2010 07:08:45 -0700 Received: from [123.119.96.90] (/123.119.96.90) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 27 May 2010 07:08:43 -0700 Message-ID: <4BFE7CEB.4040407@oracle.com> Date: Thu, 27 May 2010 22:08:43 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> In-Reply-To: <877hmpeivx.fsf@meyering.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090201.4BFE7D24.022A:SCFMA922111,ss=1,pt=113351,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.8 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.0 (------) Jim Meyering wrote: > jeff.liu wrote: >> This is the revised version, it fixed the fiemap-start offset calculation >> approach to remove it out >> of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)' loop. > > Hi Jeff, > > I've included below the state of my local changes. > Unfortunately, with that 5-patch series, there is always a test failure > on F13/ext4. Maybe someone who knows more about extents can provide an > explanation? > > Here's a small example to demonstrate: > > Create a file with many extents: > > perl -e 'BEGIN { $n = 19 * 1024; *F = *STDOUT }' \ > -e 'for (1..100) { sysseek (*F, $n, 1)' \ > -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 > > Using the patched "cp", repeat the following 10 or 20 times: > > ./cp --sparse=always j1 j2; sync > filefrag -v j1 | awk '/^ / {print $1,$2}' > ff1 || fail=1 > filefrag -v j2 | awk '/^ / {print $1,$2}' > ff2 || fail=1 > diff -u ff1 ff2 || fail=1 > > Usually there is no diff output, but occasionally it'll print this: > [hmm... today it consistently prints these differences every other time.] Woo!!! I just run this test on btrfs/ext4/ocfs2 against mainline kernel(Linux jeff-laptop 2.6.33-rc5-00238-gb04da8b-dirty) on my laptop. Only btrfs always works well for me, Ext4 has the same issue like yours. jeff@jeff-laptop:/ext4/test$ for ((i=0; i < 100; i++)); do ./1.sh; done --- ff1 2010-05-27 22:03:25.263480260 +0800 +++ ff2 2010-05-27 22:03:25.315476210 +0800 @@ -26,77 +26,78 @@ 24 232 25 242 26 251 -27 261 -28 270 -29 280 -30 289 -31 299 -32 308 -33 318 -34 327 -35 337 -36 346 -37 356 -38 365 -39 375 -40 384 -41 394 -42 403 -43 413 -44 422 -45 432 -46 441 -47 451 -48 460 -49 470 -50 479 -51 489 -52 498 -53 508 -54 512 -55 517 -56 527 -57 536 -58 546 -59 555 -60 565 -61 574 -62 584 -63 593 -64 603 -65 612 -66 622 -67 631 -68 641 -69 650 -70 660 -71 669 -72 679 -73 688 -74 698 -75 707 -76 717 -77 726 -78 736 -79 745 -80 755 -81 764 -82 774 -83 783 -84 793 -85 802 -86 812 -87 821 -88 831 -89 840 -90 850 -91 859 -92 869 -93 878 -94 888 -95 897 -96 907 -97 916 -98 926 -99 935 -100 945 +27 256 +28 261 +29 270 +30 280 +31 289 +32 299 +33 308 +34 318 +35 327 +36 337 +37 346 +38 356 +39 365 +40 375 +41 384 +42 394 +43 403 +44 413 +45 422 +46 432 +47 441 +48 451 +49 460 +50 470 +51 479 +52 489 +53 498 +54 508 +55 512 +56 517 +57 527 +58 536 +59 546 +60 555 +61 565 +62 574 +63 584 +64 593 +65 603 +66 612 +67 622 +68 631 +69 641 +70 650 +71 660 +72 669 +73 679 +74 688 +75 698 +76 707 +77 717 +78 726 +79 736 +80 745 +81 755 +82 764 +83 774 +84 783 +85 793 +86 802 +87 812 +88 821 +89 831 +90 840 +91 850 +92 859 +93 869 +94 878 +95 888 +96 897 +97 907 +98 916 +99 926 +100 935 +101 945 OCFS2's show different for many times(repeat 100 times). But I cannot figure out this issue from the patch at the moment. :( HEROS, who can give some hints from the kernel's point of view? jeff@jeff-laptop:/ocfs2/test$ for ((i=0; i < 100; i++)); do ./1.sh; done --- ff1 2010-05-27 21:49:58.288483759 +0800 +++ ff2 2010-05-27 21:49:58.304545778 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 519 --- ff1 2010-05-27 21:50:05.260477055 +0800 +++ ff2 2010-05-27 21:50:05.276532299 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 815 --- ff1 2010-05-27 21:50:13.048484948 +0800 +++ ff2 2010-05-27 21:50:13.076486205 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 161 --- ff1 2010-05-27 21:50:19.748471818 +0800 +++ ff2 2010-05-27 21:50:19.764488230 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 457 --- ff1 2010-05-27 21:50:25.384540820 +0800 +++ ff2 2010-05-27 21:50:25.396508694 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 753 --- ff1 2010-05-27 21:50:33.504500173 +0800 +++ ff2 2010-05-27 21:50:33.536480827 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 99 --- ff1 2010-05-27 21:50:40.012512743 +0800 +++ ff2 2010-05-27 21:50:40.028511067 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 395 --- ff1 2010-05-27 21:50:46.680505132 +0800 +++ ff2 2010-05-27 21:50:46.708501290 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 691 --- ff1 2010-05-27 21:50:55.576480477 +0800 +++ ff2 2010-05-27 21:50:55.596475937 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 37 --- ff1 2010-05-27 21:51:03.491482451 +0800 +++ ff2 2010-05-27 21:51:03.515503404 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 333 --- ff1 2010-05-27 21:51:11.567497188 +0800 +++ ff2 2010-05-27 21:51:11.603479937 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 629 --- ff1 2010-05-27 21:51:20.983471486 +0800 +++ ff2 2010-05-27 21:51:21.011472813 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 925 --- ff1 2010-05-27 21:51:29.548508415 +0800 +++ ff2 2010-05-27 21:51:29.588486206 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 271 --- ff1 2010-05-27 21:51:35.927492089 +0800 +++ ff2 2010-05-27 21:51:35.959488178 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 567 --- ff1 2010-05-27 21:51:40.200480478 +0800 +++ ff2 2010-05-27 21:51:40.240471818 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 863 --- ff1 2010-05-27 21:51:45.772499475 +0800 +++ ff2 2010-05-27 21:51:45.820498986 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 209 --- ff1 2010-05-27 21:51:49.919519607 +0800 +++ ff2 2010-05-27 21:51:49.947475956 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 505 --- ff1 2010-05-27 21:51:55.195482312 +0800 +++ ff2 2010-05-27 21:51:55.231483778 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 801 --- ff1 2010-05-27 21:52:00.911479588 +0800 +++ ff2 2010-05-27 21:52:00.939473232 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 147 --- ff1 2010-05-27 21:52:06.208488091 +0800 +++ ff2 2010-05-27 21:52:06.252482433 +0800 @@ -1,2 +1,3 @@ ext logical 0 0 +1 443 .... ... > > $ diff -u ff1 ff2 || fail=1 > --- ff1 2010-05-22 18:42:26.943501382 +0200 > +++ ff2 2010-05-22 18:42:27.020876155 +0200 > @@ -53,49 +53,50 @@ ext logical > 51 489 > 52 498 > 53 508 > -54 517 > -55 527 > -56 536 > -57 546 > -58 555 > -59 565 > -60 574 > -61 584 > -62 593 > -63 603 > -64 612 > -65 622 > -66 631 > -67 641 > -68 650 > -69 660 > -70 669 > -71 679 > -72 688 > -73 698 > -74 707 > -75 717 > -76 726 > -77 736 > -78 745 > -79 755 > -80 764 > -81 774 > -82 783 > -83 793 > -84 802 > -85 812 > -86 821 > -87 831 > -88 840 > -89 850 > -90 859 > -91 869 > -92 878 > -93 888 > -94 897 > -95 907 > -96 916 > -97 926 > -98 935 > -99 945 > +54 512 > +55 517 > +56 527 > +57 536 > +58 546 > +59 555 > +60 565 > +61 574 > +62 584 > +63 593 > +64 603 > +65 612 > +66 622 > +67 631 > +68 641 > +69 650 > +70 660 > +71 669 > +72 679 > +73 688 > +74 698 > +75 707 > +76 717 > +77 726 > +78 736 > +79 745 > +80 755 > +81 764 > +82 774 > +83 783 > +84 793 > +85 802 > +86 812 > +87 821 > +88 831 > +89 840 > +90 850 > +91 859 > +92 869 > +93 878 > +94 888 > +95 897 > +96 907 > +97 916 > +98 926 > +99 935 > +100 945 > > > From 2ef44bcb0bbb38cca738aa90e067caee312da939 Mon Sep 17 00:00:00 2001 > From: Jie Liu > Date: Thu, 13 May 2010 22:09:30 +0800 > Subject: [PATCH 1/5] cp: Add FIEMAP support for efficient sparse file copy > > * src/fiemap.h: Add fiemap.h for fiemap ioctl(2) support. > Copied from linux's include/linux/fiemap.h, with minor formatting changes. > * src/copy.c (copy_reg): Now, when `cp' invoked with --sparse=[WHEN] option, we > will try to do FIEMAP-copy if the underlaying file system support it, fall back > to a normal copy if it fails. > --- > src/copy.c | 159 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > src/fiemap.h | 102 +++++++++++++++++++++++++++++++++++++ > 2 files changed, 261 insertions(+), 0 deletions(-) > create mode 100644 src/fiemap.h > > diff --git a/src/copy.c b/src/copy.c > index c16cef6..0e54729 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -63,6 +63,10 @@ > > #include > > +#ifndef HAVE_FIEMAP > +# include "fiemap.h" > +#endif > + > #ifndef HAVE_FCHOWN > # define HAVE_FCHOWN false > # define fchown(fd, uid, gid) (-1) > @@ -149,6 +153,141 @@ clone_file (int dest_fd, int src_fd) > #endif > } > > +#ifdef __linux__ > +# ifndef FS_IOC_FIEMAP > +# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) > +# endif > +/* Perform FIEMAP(available in mainline 2.6.27) copy if possible. > + Call ioctl(2) with FS_IOC_FIEMAP to efficiently map file allocation > + excepts holes. So the overhead to deal with holes with lseek(2) in > + normal copy could be saved. This would result in much faster backups > + for any kind of sparse file. */ > +static bool > +fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, > + off_t src_total_size, char const *src_name, > + char const *dst_name, bool *normal_copy_required) > +{ > + bool fail = false; > + bool last = false; > + char fiemap_buf[4096]; > + struct fiemap *fiemap = (struct fiemap *)fiemap_buf; > + struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; > + uint32_t count = (sizeof (fiemap_buf) - sizeof (*fiemap)) / > + sizeof (struct fiemap_extent); > + off_t last_ext_logical = 0; > + uint64_t last_ext_len = 0; > + uint64_t last_read_size = 0; > + unsigned int i = 0; > + > + /* This is required at least to initialize fiemap->fm_start, > + but also serves (in May 2010) to appease valgrind, which > + appears not to know the semantics of the FIEMAP ioctl. */ > + memset (fiemap_buf, 0, sizeof fiemap_buf); > + > + do > + { > + fiemap->fm_length = FIEMAP_MAX_OFFSET; > + fiemap->fm_extent_count = count; > + > + /* When ioctl(2) fails, fall back to the normal copy only if it > + is the first time we met. */ > + if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) > + { > + /* If `i > 0', then at least one ioctl(2) has been performed before. */ > + if (i == 0) > + *normal_copy_required = true; > + return false; > + } > + > + /* If 0 extents are returned, then more ioctls are not needed. */ > + if (fiemap->fm_mapped_extents == 0) > + break; > + > + for (i = 0; i < fiemap->fm_mapped_extents; i++) > + { > + assert (fm_ext[i].fe_logical <= OFF_T_MAX); > + > + off_t ext_logical = fm_ext[i].fe_logical; > + uint64_t ext_len = fm_ext[i].fe_length; > + > + if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) > + { > + error (0, errno, _("cannot lseek %s"), quote (src_name)); > + return fail; > + } > + > + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0LL) > + { > + error (0, errno, _("cannot lseek %s"), quote (dst_name)); > + return fail; > + } > + > + if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST) > + { > + last_ext_logical = ext_logical; > + last_ext_len = ext_len; > + last = true; > + } > + > + while (0 < ext_len) > + { > + char buf[buf_size]; > + > + /* Avoid reading into the holes if the left extent > + length is shorter than the buffer size. */ > + if (ext_len < buf_size) > + buf_size = ext_len; > + > + ssize_t n_read = read (src_fd, buf, buf_size); > + if (n_read < 0) > + { > +#ifdef EINTR > + if (errno == EINTR) > + continue; > +#endif > + error (0, errno, _("reading %s"), quote (src_name)); > + return fail; > + } > + > + if (n_read == 0) > + { > + /* Figure out how many bytes read from the last extent. */ > + last_read_size = last_ext_len - ext_len; > + break; > + } > + > + if (full_write (dest_fd, buf, n_read) != n_read) > + { > + error (0, errno, _("writing %s"), quote (dst_name)); > + return fail; > + } > + > + ext_len -= n_read; > + } > + } > + > + fiemap->fm_start = fm_ext[i - 1].fe_logical + fm_ext[i - 1].fe_length; > + > + } while (! last); > + > + /* If a file ends up with holes, the sum of the last extent logical offset > + and the read-returned size will be shorter than the actual size of the > + file. Use ftruncate to extend the length of the destination file. */ > + if (last_ext_logical + last_read_size < src_total_size) > + { > + if (ftruncate (dest_fd, src_total_size) < 0) > + { > + error (0, errno, _("extending %s"), quote (dst_name)); > + return fail; > + } > + } > + > + return ! fail; > +} > +#else > +static bool fiemap_copy_ok (ignored) { errno == ENOTSUP; return false; } > +#endif > + > /* FIXME: describe */ > /* FIXME: rewrite this to use a hash table so we avoid the quadratic > performance hit that's probably noticeable only on trees deeper > @@ -679,6 +818,25 @@ copy_reg (char const *src_name, char const *dst_name, > #endif > } > > + if (make_holes) > + { > + bool require_normal_copy = false; > + /* Perform efficient FIEMAP copy for sparse files, fall back to the > + standard copy only if the ioctl(2) fails. */ > + if (fiemap_copy_ok (source_desc, dest_desc, buf_size, > + src_open_sb.st_size, src_name, > + dst_name, &require_normal_copy)) > + goto preserve_metadata; > + else > + { > + if (! require_normal_copy) > + { > + return_val = false; > + goto close_src_and_dst_desc; > + } > + } > + } > + > /* If not making a sparse file, try to use a more-efficient > buffer size. */ > if (! make_holes) > @@ -807,6 +965,7 @@ copy_reg (char const *src_name, char const *dst_name, > } > } > > +preserve_metadata: > if (x->preserve_timestamps) > { > struct timespec timespec[2]; > diff --git a/src/fiemap.h b/src/fiemap.h > new file mode 100644 > index 0000000..d33293b > --- /dev/null > +++ b/src/fiemap.h > @@ -0,0 +1,102 @@ > +/* FS_IOC_FIEMAP ioctl infrastructure. > + Some portions copyright (C) 2007 Cluster File Systems, Inc > + Authors: Mark Fasheh > + Kalpak Shah > + Andreas Dilger . */ > + > +/* Copy from kernel, modified to respect GNU code style by Jie Liu. */ > + > +#ifndef _LINUX_FIEMAP_H > +# define _LINUX_FIEMAP_H > + > +# include > + > +struct fiemap_extent > +{ > + /* Logical offset in bytes for the start of the extent > + from the beginning of the file. */ > + uint64_t fe_logical; > + > + /* Physical offset in bytes for the start of the extent > + from the beginning of the disk. */ > + uint64_t fe_physical; > + > + /* Length in bytes for this extent. */ > + uint64_t fe_length; > + > + uint64_t fe_reserved64[2]; > + > + /* FIEMAP_EXTENT_* flags for this extent. */ > + uint32_t fe_flags; > + > + uint32_t fe_reserved[3]; > +}; > + > +struct fiemap > +{ > + /* Logical offset(inclusive) at which to start mapping(in). */ > + uint64_t fm_start; > + > + /* Logical length of mapping which userspace wants(in). */ > + uint64_t fm_length; > + > + /* FIEMAP_FLAG_* flags for request(in/out). */ > + uint32_t fm_flags; > + > + /* Number of extents that were mapped(out). */ > + uint32_t fm_mapped_extents; > + > + /* Size of fm_extents array(in). */ > + uint32_t fm_extent_count; > + > + uint32_t fm_reserved; > + > + /* Array of mapped extents(out). */ > + struct fiemap_extent fm_extents[0]; > +}; > + > +/* The maximum offset can be mapped for a file. */ > +# define FIEMAP_MAX_OFFSET (~0ULL) > + > +/* Sync file data before map. */ > +# define FIEMAP_FLAG_SYNC 0x00000001 > + > +/* Map extented attribute tree. */ > +# define FIEMAP_FLAG_XATTR 0x00000002 > + > +# define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR) > + > +/* Last extent in file. */ > +# define FIEMAP_EXTENT_LAST 0x00000001 > + > +/* Data location unknown. */ > +# define FIEMAP_EXTENT_UNKNOWN 0x00000002 > + > +/* Location still pending, Sets EXTENT_UNKNOWN. */ > +# define FIEMAP_EXTENT_DELALLOC 0x00000004 > + > +/* Data can not be read while fs is unmounted. */ > +# define FIEMAP_EXTENT_ENCODED 0x00000008 > + > +/* Data is encrypted by fs. Sets EXTENT_NO_BYPASS. */ > +# define FIEMAP_EXTENT_DATA_ENCRYPTED 0x00000080 > + > +/* Extent offsets may not be block aligned. */ > +# define FIEMAP_EXTENT_NOT_ALIGNED 0x00000100 > + > +/* Data mixed with metadata. Sets EXTENT_NOT_ALIGNED. */ > +# define FIEMAP_EXTENT_DATA_INLINE 0x00000200 > + > +/* Multiple files in block. Set EXTENT_NOT_ALIGNED. */ > +# define FIEMAP_EXTENT_DATA_TAIL 0x00000400 > + > +/* Space allocated, but not data (i.e. zero). */ > +# define FIEMAP_EXTENT_UNWRITTEN 0x00000800 > + > +/* File does not natively support extents. Result merged for efficiency. */ > +# define FIEMAP_EXTENT_MERGED 0x00001000 > + > +/* Space shared with other files. */ > +# define FIEMAP_EXTENT_SHARED 0x00002000 > + > +#endif > -- > 1.7.1.348.gb26ba > > > From 260b5b89e33da2b9a5ea5bcd9dba874f503d2937 Mon Sep 17 00:00:00 2001 > From: Jie Liu > Date: Thu, 13 May 2010 22:17:53 +0800 > Subject: [PATCH 2/5] tests: add a new test for FIEMAP-copy > > * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a > loopbacked ext4 partition. > * tests/Makefile.am (sparse-fiemap): Reference the new test. > --- > tests/Makefile.am | 1 + > tests/cp/sparse-fiemap | 56 ++++++++++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 57 insertions(+), 0 deletions(-) > create mode 100755 tests/cp/sparse-fiemap > > diff --git a/tests/Makefile.am b/tests/Makefile.am > index c458574..f7840c8 100644 > --- a/tests/Makefile.am > +++ b/tests/Makefile.am > @@ -25,6 +25,7 @@ root_tests = \ > cp/special-bits \ > cp/cp-mv-enotsup-xattr \ > cp/capability \ > + cp/sparse-fiemap \ > dd/skip-seek-past-dev \ > install/install-C-root \ > ls/capability \ > diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap > new file mode 100755 > index 0000000..945c94b > --- /dev/null > +++ b/tests/cp/sparse-fiemap > @@ -0,0 +1,56 @@ > +#!/bin/sh > +# Test cp --sparse=always through fiemap copy > + > +# Copyright (C) 2006-2010 Free Software Foundation, Inc. > + > +# This program is free software: you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation, either version 3 of the License, or > +# (at your option) any later version. > + > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > + > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see . > + > +if test "$VERBOSE" = yes; then > + set -x > + cp --version > +fi > + > +. $srcdir/test-lib.sh > +require_root_ > + > +cwd=`pwd` > +cleanup_() { cd /; umount "$cwd/mnt"; } > + > +skip=0 > +# Create an ext4 loopback file system > +dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 > +mkdir mnt > +mkfs -t ext4 -F blob || > + skip_test_ "failed to create ext4 file system" > +mount -oloop blob mnt || skip=1 > +echo test > mnt/f || skip=1 > +test -s mnt/f || skip=1 > + > +test $skip = 1 && > + skip_test_ "insufficient mount/ext4 support" > + > +# Create a 1TiB sparse file > +dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure > + > +cd mnt || fail=1 > + > +# It takes many minutes to copy this sparse file using the old method. > +# By contrast, it takes far less than 1 second using FIEMAP-copy. > +timeout 10 cp --sparse=always sparse fiemap || fail=1 > + > +# Ensure that the sparse file copied through fiemap has the same size > +# in bytes as the original. > +test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 > + > +Exit $fail > -- > 1.7.1.348.gb26ba > > > From 738031aed78b6323969f799d3d5fbf19e0cfc91a Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Fri, 21 May 2010 18:28:42 +0200 > Subject: [PATCH 3/5] tests: exercise more of the new FIEMAP copying code > > * tests/cp/sparse-fiemap: Ensure that a file with many extents (more > than fit in copy.c's internal 4KiB buffer) is copied properly. > --- > tests/cp/sparse-fiemap | 38 ++++++++++++++++++++++++++++++++++++++ > 1 files changed, 38 insertions(+), 0 deletions(-) > > diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap > index 945c94b..b1643be 100755 > --- a/tests/cp/sparse-fiemap > +++ b/tests/cp/sparse-fiemap > @@ -53,4 +53,42 @@ timeout 10 cp --sparse=always sparse fiemap || fail=1 > # in bytes as the original. > test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 > > +# ================================================= > +# Ensure that we exercise the FIEMAP-copying code enough > +# to provoke at least two iterations of the do...while loop > +# in which it calls ioctl (fd, FS_IOC_FIEMAP,... > +# This also verifies that non-trivial extents are preserved. > + > +$PERL -e 1 || skip_test_ 'skipping part of this test; you lack perl' > + > +$PERL -e 'BEGIN { $n = 16 * 1024; *F = *STDOUT }' \ > + -e 'for (1..100) { sysseek (*F, $n, 1)' \ > + -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1 > + > +cp --sparse=always j1 j2 || fail=1 > +cmp j1 j2 || fail=1 > + > +filefrag j1 | grep extent \ > + || skip_test_ 'skipping part of this test; you lack filefrag' > + > +# Here is sample filefrag output: > +# $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \ > +# -e 'for (1..5) { sysseek(*F,$n,1)' \ > +# -e '&& syswrite *F,"."x$n or die "$!"}' > j > +# $ filefrag -v j > +# Filesystem type is: ef53 > +# File size of j is 163840 (40 blocks, blocksize 4096) > +# ext logical physical expected length flags > +# 0 4 6258884 4 > +# 1 12 6258892 6258887 4 > +# 2 20 6258900 6258895 4 > +# 3 28 6258908 6258903 4 > +# 4 36 6258916 6258911 4 eof > +# j: 6 extents found > + > +# exclude the physical block numbers; they always differ > +filefrag -v j1 | awk '/^ / {print $1,$2,$NF}' > ff1 || fail=1 > +filefrag -v j2 | awk '/^ / {print $1,$2,$NF}' > ff2 || fail=1 > +compare ff1 ff2 || fail=1 > + > Exit $fail > -- > 1.7.1.348.gb26ba > > > From e84f881cb5c2eb92f3b6d5bddaef50c3e811bc30 Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Sat, 22 May 2010 10:22:58 +0200 > Subject: [PATCH 4/5] tests: require root access only if current partition is not ext4 > > * tests/cp/sparse-fiemap: Don't require root access if current > partition is ext4. > --- > tests/cp/sparse-fiemap | 44 +++++++++++++++++++++++++------------------- > 1 files changed, 25 insertions(+), 19 deletions(-) > > diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap > index b1643be..371bced 100755 > --- a/tests/cp/sparse-fiemap > +++ b/tests/cp/sparse-fiemap > @@ -22,28 +22,34 @@ if test "$VERBOSE" = yes; then > fi > > . $srcdir/test-lib.sh > -require_root_ > > -cwd=`pwd` > -cleanup_() { cd /; umount "$cwd/mnt"; } > - > -skip=0 > -# Create an ext4 loopback file system > -dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 > -mkdir mnt > -mkfs -t ext4 -F blob || > - skip_test_ "failed to create ext4 file system" > -mount -oloop blob mnt || skip=1 > -echo test > mnt/f || skip=1 > -test -s mnt/f || skip=1 > - > -test $skip = 1 && > - skip_test_ "insufficient mount/ext4 support" > +if df -T -t ext4 . ; then > + : # Current dir is on an ext4 partition. Good! > +else > + # It's not; we need to create one, hence we need root access. > + require_root_ > + > + cwd=$PWD > + cleanup_() { cd /; umount "$cwd/mnt"; } > + > + skip=0 > + # Create an ext4 loopback file system > + dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 > + mkdir mnt > + mkfs -t ext4 -F blob || > + skip_test_ "failed to create ext4 file system" > + mount -oloop blob mnt || skip=1 > + echo test > mnt/f || skip=1 > + test -s mnt/f || skip=1 > + > + test $skip = 1 && > + skip_test_ "insufficient mount/ext4 support" > + > + cd mnt || fail=1 > +fi > > # Create a 1TiB sparse file > -dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure > - > -cd mnt || fail=1 > +dd if=/dev/zero of=sparse bs=1k count=1 seek=1G || framework_failure > > # It takes many minutes to copy this sparse file using the old method. > # By contrast, it takes far less than 1 second using FIEMAP-copy. > -- > 1.7.1.348.gb26ba > > > From 213c9247a406b267bc5819ccbe3b729e9557c675 Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Sat, 22 May 2010 10:21:46 +0200 > Subject: [PATCH 5/5] tests: fiemap test improvement > > * tests/cp/sparse-fiemap: More tests. > --- > tests/cp/sparse-fiemap | 69 +++++++++++++++++++++++++----------------------- > 1 files changed, 36 insertions(+), 33 deletions(-) > > diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap > index 371bced..ef3742e 100755 > --- a/tests/cp/sparse-fiemap > +++ b/tests/cp/sparse-fiemap > @@ -38,14 +38,14 @@ else > mkdir mnt > mkfs -t ext4 -F blob || > skip_test_ "failed to create ext4 file system" > - mount -oloop blob mnt || skip=1 > - echo test > mnt/f || skip=1 > - test -s mnt/f || skip=1 > + mount -oloop blob mnt || skip=1 > + cd mnt || skip=1 > + echo test > f || skip=1 > + test -s f || skip=1 > > test $skip = 1 && > skip_test_ "insufficient mount/ext4 support" > > - cd mnt || fail=1 > fi > > # Create a 1TiB sparse file > @@ -67,34 +67,37 @@ test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 > > $PERL -e 1 || skip_test_ 'skipping part of this test; you lack perl' > > -$PERL -e 'BEGIN { $n = 16 * 1024; *F = *STDOUT }' \ > - -e 'for (1..100) { sysseek (*F, $n, 1)' \ > - -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1 > - > -cp --sparse=always j1 j2 || fail=1 > -cmp j1 j2 || fail=1 > - > -filefrag j1 | grep extent \ > - || skip_test_ 'skipping part of this test; you lack filefrag' > - > -# Here is sample filefrag output: > -# $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \ > -# -e 'for (1..5) { sysseek(*F,$n,1)' \ > -# -e '&& syswrite *F,"."x$n or die "$!"}' > j > -# $ filefrag -v j > -# Filesystem type is: ef53 > -# File size of j is 163840 (40 blocks, blocksize 4096) > -# ext logical physical expected length flags > -# 0 4 6258884 4 > -# 1 12 6258892 6258887 4 > -# 2 20 6258900 6258895 4 > -# 3 28 6258908 6258903 4 > -# 4 36 6258916 6258911 4 eof > -# j: 6 extents found > - > -# exclude the physical block numbers; they always differ > -filefrag -v j1 | awk '/^ / {print $1,$2,$NF}' > ff1 || fail=1 > -filefrag -v j2 | awk '/^ / {print $1,$2,$NF}' > ff2 || fail=1 > -compare ff1 ff2 || fail=1 > +for i in $(seq 20); do > + for j in 1 2 31 100; do > + $PERL -e 'BEGIN { $n = '$i' * 1024; *F = *STDOUT }' \ > + -e 'for (1..'$j') { sysseek (*F, $n, 1)' \ > + -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1 > + > + cp --sparse=always j1 j2 || fail=1 > + cmp j1 j2 || fail=1 > + filefrag -v j1 | grep extent \ > + || skip_test_ 'skipping part of this test; you lack filefrag' > + > + # Here is sample filefrag output: > + # $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \ > + # -e 'for (1..5) { sysseek(*F,$n,1)' \ > + # -e '&& syswrite *F,"."x$n or die "$!"}' > j > + # $ filefrag -v j > + # Filesystem type is: ef53 > + # File size of j is 163840 (40 blocks, blocksize 4096) > + # ext logical physical expected length flags > + # 0 4 6258884 4 > + # 1 12 6258892 6258887 4 > + # 2 20 6258900 6258895 4 > + # 3 28 6258908 6258903 4 > + # 4 36 6258916 6258911 4 eof > + # j: 6 extents found > + > + # exclude the physical block numbers; they always differ > + filefrag -v j1 | awk '/^ / {print $1,$2}' > ff1 || fail=1 > + filefrag -v j2 | awk '/^ / {print $1,$2}' > ff2 || fail=1 > + compare ff1 ff2 || fail=1 > + done > +done > > Exit $fail > -- > 1.7.1.348.gb26ba > > > -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Thu May 27 13:29:48 2010 Received: (at submit) by debbugs.gnu.org; 27 May 2010 17:29:48 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHgtv-0005Ml-HH for submit@debbugs.gnu.org; Thu, 27 May 2010 13:29:47 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHgtu-0005Mg-AZ for submit@debbugs.gnu.org; Thu, 27 May 2010 13:29:46 -0400 Received: from lists.gnu.org ([199.232.76.165]:54009) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OHgtq-0003PK-0F for submit@debbugs.gnu.org; Thu, 27 May 2010 13:29:42 -0400 Received: from [140.186.70.92] (port=57889 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OHgtn-0003d0-B7 for bug-coreutils@gnu.org; Thu, 27 May 2010 13:29:41 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OHgtm-0005bW-0E for bug-coreutils@gnu.org; Thu, 27 May 2010 13:29:39 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:52115) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHgtl-0005aB-Qt for bug-coreutils@gnu.org; Thu, 27 May 2010 13:29:37 -0400 Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4RHTTpV026829 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 27 May 2010 17:29:31 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4RGL9ag031532; Thu, 27 May 2010 17:29:16 GMT Received: from abhmt012.oracle.com by acsmt353.oracle.com with ESMTP id 303330411274981291; Thu, 27 May 2010 10:28:11 -0700 Received: from [130.35.68.118] (/130.35.68.118) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 27 May 2010 10:28:11 -0700 Message-ID: <4BFEABAA.7090507@oracle.com> Date: Thu, 27 May 2010 10:28:10 -0700 From: Sunil Mushran User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFE7CEB.4040407@oracle.com> In-Reply-To: <4BFE7CEB.4040407@oracle.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet15.oracle.com [148.87.113.117] X-CT-RefId: str=0001.0A090205.4BFEABFC.0192:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.6 (------) X-Debbugs-Envelope-To: submit Cc: Tao Ma , bug-coreutils@gnu.org, Jim Meyering , Chris Mason , Joel Becker X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) Jim Meyering wrote: >> Hi Jeff, >> >> I've included below the state of my local changes. >> Unfortunately, with that 5-patch series, there is always a test failure >> on F13/ext4. Maybe someone who knows more about extents can provide an >> explanation? >> >> Here's a small example to demonstrate: >> >> Create a file with many extents: >> >> perl -e 'BEGIN { $n = 19 * 1024; *F = *STDOUT }' \ >> -e 'for (1..100) { sysseek (*F, $n, 1)' \ >> -e '&& syswrite (*F, "."x$n) or die "$!"}'> j1 >> >> Using the patched "cp", repeat the following 10 or 20 times: >> >> ./cp --sparse=always j1 j2; sync >> filefrag -v j1 | awk '/^ / {print $1,$2}'> ff1 || fail=1 >> filefrag -v j2 | awk '/^ / {print $1,$2}'> ff2 || fail=1 >> diff -u ff1 ff2 || fail=1 >> >> Usually there is no diff output, but occasionally it'll print this: >> [hmm... today it consistently prints these differences every other time.] > The reason it does not work is because the sparse file created by cp may not be sparse (or sparse enough). And that is because cp reads is chunks of st_blocksize and skips the write only if the entire chunk is zero. The perl script creates the file in 19K chunks (alternate writes and holes). So on a 4K fs, the file created by the script will have 4 blocks as holes (avg). But when cp makes it, it could fill out those holes because the read granularity could be coarser. For example, ocfs2 fills out st_blocksize with the fs cluster size (allocation size) which could be larger than the block size. My suggestion is to not use filefrag but to use md5sum to compare the two files. In the end, the aim of this feature is to reduce the number of reads. Furthermore, the number of extents created is a function of not only cp but also the file system. (A fs does not guarantee the number of extents a file will have. It only guarantees that the data out is the same as data in.) From debbugs-submit-bounces@debbugs.gnu.org Thu May 27 14:43:59 2010 Received: (at submit) by debbugs.gnu.org; 27 May 2010 18:44:00 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHi3j-0006YZ-BA for submit@debbugs.gnu.org; Thu, 27 May 2010 14:43:59 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHi3h-0006YU-H1 for submit@debbugs.gnu.org; Thu, 27 May 2010 14:43:58 -0400 Received: from lists.gnu.org ([199.232.76.165]:49835) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OHi3c-0005by-TN for submit@debbugs.gnu.org; Thu, 27 May 2010 14:43:52 -0400 Received: from [140.186.70.92] (port=58364 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OHi3b-0005r9-4A for bug-coreutils@gnu.org; Thu, 27 May 2010 14:43:52 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OHi3Z-0002NS-Lx for bug-coreutils@gnu.org; Thu, 27 May 2010 14:43:50 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:49381) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHi3Z-0002Me-2T for bug-coreutils@gnu.org; Thu, 27 May 2010 14:43:49 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id A0413940135; Thu, 27 May 2010 20:43:41 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 7C6A9DD27; Thu, 27 May 2010 20:43:40 +0200 (CEST) From: Jim Meyering To: Sunil Mushran Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4BFEABAA.7090507@oracle.com> (Sunil Mushran's message of "Thu, 27 May 2010 10:28:10 -0700") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFE7CEB.4040407@oracle.com> <4BFEABAA.7090507@oracle.com> Date: Thu, 27 May 2010 20:43:40 +0200 Message-ID: <87k4qpb2xv.fsf@meyering.net> Lines: 55 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.2 (-----) X-Debbugs-Envelope-To: submit Cc: "jeff.liu" , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.2 (-----) Sunil Mushran wrote: > Jim Meyering wrote: >>> Hi Jeff, >>> >>> I've included below the state of my local changes. >>> Unfortunately, with that 5-patch series, there is always a test failure >>> on F13/ext4. Maybe someone who knows more about extents can provide an >>> explanation? >>> >>> Here's a small example to demonstrate: >>> >>> Create a file with many extents: >>> >>> perl -e 'BEGIN { $n = 19 * 1024; *F = *STDOUT }' \ >>> -e 'for (1..100) { sysseek (*F, $n, 1)' \ >>> -e '&& syswrite (*F, "."x$n) or die "$!"}'> j1 >>> >>> Using the patched "cp", repeat the following 10 or 20 times: >>> >>> ./cp --sparse=always j1 j2; sync >>> filefrag -v j1 | awk '/^ / {print $1,$2}'> ff1 || fail=1 >>> filefrag -v j2 | awk '/^ / {print $1,$2}'> ff2 || fail=1 >>> diff -u ff1 ff2 || fail=1 >>> >>> Usually there is no diff output, but occasionally it'll print this: >>> [hmm... today it consistently prints these differences every other time.] > > The reason it does not work is because the sparse file created by cp > may not be sparse (or sparse enough). And that is because cp reads > is chunks of st_blocksize and skips the write only if the entire chunk > is zero. The perl script creates the file in 19K chunks (alternate writes > and holes). Thanks for replying. However, your description of how GNU cp works suggests that you're looking at the pre-FIEMAP semantics. Please refer to the patches here http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/20534/focus=20630 that make it use FIEMAP. > So on a 4K fs, the file created by the script will have 4 blocks as holes > (avg). But when cp makes it, it could fill out those holes because the read > granularity could be coarser. For example, ocfs2 fills out st_blocksize > with the fs cluster size (allocation size) which could be larger than the > block size. > > My suggestion is to not use filefrag but to use md5sum to compare the two > files. That would be pointless. The goal of the test is to determine that the FIEMAP copy did indeed preserve the extents. If I do as you suggest, even if cp mistakenly filled in all holes, the test would still pass. From debbugs-submit-bounces@debbugs.gnu.org Thu May 27 14:45:43 2010 Received: (at submit) by debbugs.gnu.org; 27 May 2010 18:45:43 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHi5N-0006a7-Bn for submit@debbugs.gnu.org; Thu, 27 May 2010 14:45:43 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHi5L-0006a1-KK for submit@debbugs.gnu.org; Thu, 27 May 2010 14:45:40 -0400 Received: from lists.gnu.org ([199.232.76.165]:51006) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OHi5H-0005f9-6u for submit@debbugs.gnu.org; Thu, 27 May 2010 14:45:35 -0400 Received: from [140.186.70.92] (port=59068 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OHi5F-0006ju-6h for bug-coreutils@gnu.org; Thu, 27 May 2010 14:45:34 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OHi5D-0002lD-Tk for bug-coreutils@gnu.org; Thu, 27 May 2010 14:45:33 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:51845) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHi5D-0002kf-6a for bug-coreutils@gnu.org; Thu, 27 May 2010 14:45:31 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id 49CB29401AD; Thu, 27 May 2010 20:45:25 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 1E117DD34; Thu, 27 May 2010 20:45:24 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4BFE7CEB.4040407@oracle.com> (jeff liu's message of "Thu, 27 May 2010 22:08:43 +0800") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFE7CEB.4040407@oracle.com> Date: Thu, 27 May 2010 20:45:24 +0200 Message-ID: <87eigxb2uz.fsf@meyering.net> Lines: 38 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) jeff.liu wrote: > Jim Meyering wrote: >> jeff.liu wrote: >>> This is the revised version, it fixed the fiemap-start offset calculation >>> approach to remove it out >>> of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)' loop. >> >> Hi Jeff, >> >> I've included below the state of my local changes. >> Unfortunately, with that 5-patch series, there is always a test failure >> on F13/ext4. Maybe someone who knows more about extents can provide an >> explanation? >> >> Here's a small example to demonstrate: >> >> Create a file with many extents: >> >> perl -e 'BEGIN { $n = 19 * 1024; *F = *STDOUT }' \ >> -e 'for (1..100) { sysseek (*F, $n, 1)' \ >> -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 >> >> Using the patched "cp", repeat the following 10 or 20 times: >> >> ./cp --sparse=always j1 j2; sync >> filefrag -v j1 | awk '/^ / {print $1,$2}' > ff1 || fail=1 >> filefrag -v j2 | awk '/^ / {print $1,$2}' > ff2 || fail=1 >> diff -u ff1 ff2 || fail=1 >> >> Usually there is no diff output, but occasionally it'll print this: >> [hmm... today it consistently prints these differences every other time.] > Woo!!! > I just run this test on btrfs/ext4/ocfs2 against mainline kernel(Linux jeff-laptop > 2.6.33-rc5-00238-gb04da8b-dirty) on my laptop. > Only btrfs always works well for me, Ext4 has the same issue like yours. One more point of reference: the tests all passed on an XFS file system. From debbugs-submit-bounces@debbugs.gnu.org Thu May 27 15:04:35 2010 Received: (at submit) by debbugs.gnu.org; 27 May 2010 19:04:35 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHiNf-0006mH-FO for submit@debbugs.gnu.org; Thu, 27 May 2010 15:04:35 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHiNc-0006mC-HD for submit@debbugs.gnu.org; Thu, 27 May 2010 15:04:33 -0400 Received: from lists.gnu.org ([199.232.76.165]:43046) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OHiNW-00067E-Rs for submit@debbugs.gnu.org; Thu, 27 May 2010 15:04:26 -0400 Received: from [140.186.70.92] (port=57953 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OHiNU-0000o3-QC for bug-coreutils@gnu.org; Thu, 27 May 2010 15:04:26 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OHiNI-0005fs-0a for bug-coreutils@gnu.org; Thu, 27 May 2010 15:04:13 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:45339) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHiNH-0005fi-RY for bug-coreutils@gnu.org; Thu, 27 May 2010 15:04:11 -0400 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4RJ44Vx028813 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 27 May 2010 19:04:05 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4RJ3ANi017100; Thu, 27 May 2010 19:04:00 GMT Received: from abhmt008.oracle.com by acsmt354.oracle.com with ESMTP id 303587131274986990; Thu, 27 May 2010 12:03:10 -0700 Received: from [130.35.68.118] (/130.35.68.118) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 27 May 2010 12:03:10 -0700 Message-ID: <4BFEC1ED.40308@oracle.com> Date: Thu, 27 May 2010 12:03:09 -0700 From: Sunil Mushran User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFE7CEB.4040407@oracle.com> <4BFEABAA.7090507@oracle.com> <87k4qpb2xv.fsf@meyering.net> In-Reply-To: <87k4qpb2xv.fsf@meyering.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet13.oracle.com [148.87.113.125] X-CT-RefId: str=0001.0A090206.4BFEC225.0179:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.6 (------) X-Debbugs-Envelope-To: submit Cc: "jeff.liu" , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) On 05/27/2010 11:43 AM, Jim Meyering wrote: > Sunil Mushran wrote: > >> Jim Meyering wrote: >> >>>> Hi Jeff, >>>> >>>> I've included below the state of my local changes. >>>> Unfortunately, with that 5-patch series, there is always a test failure >>>> on F13/ext4. Maybe someone who knows more about extents can provide an >>>> explanation? >>>> >>>> Here's a small example to demonstrate: >>>> >>>> Create a file with many extents: >>>> >>>> perl -e 'BEGIN { $n = 19 * 1024; *F = *STDOUT }' \ >>>> -e 'for (1..100) { sysseek (*F, $n, 1)' \ >>>> -e '&& syswrite (*F, "."x$n) or die "$!"}'> j1 >>>> >>>> Using the patched "cp", repeat the following 10 or 20 times: >>>> >>>> ./cp --sparse=always j1 j2; sync >>>> filefrag -v j1 | awk '/^ / {print $1,$2}'> ff1 || fail=1 >>>> filefrag -v j2 | awk '/^ / {print $1,$2}'> ff2 || fail=1 >>>> diff -u ff1 ff2 || fail=1 >>>> >>>> Usually there is no diff output, but occasionally it'll print this: >>>> [hmm... today it consistently prints these differences every other time.] >>>> >> The reason it does not work is because the sparse file created by cp >> may not be sparse (or sparse enough). And that is because cp reads >> is chunks of st_blocksize and skips the write only if the entire chunk >> is zero. The perl script creates the file in 19K chunks (alternate writes >> and holes). >> > Thanks for replying. > > However, your description of how GNU cp works suggests that you're > looking at the pre-FIEMAP semantics. Please refer to the patches here > > http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/20534/focus=20630 > > that make it use FIEMAP. > > >> So on a 4K fs, the file created by the script will have 4 blocks as holes >> (avg). But when cp makes it, it could fill out those holes because the read >> granularity could be coarser. For example, ocfs2 fills out st_blocksize >> with the fs cluster size (allocation size) which could be larger than the >> block size. >> >> My suggestion is to not use filefrag but to use md5sum to compare the two >> files. >> > That would be pointless. > The goal of the test is to determine that the FIEMAP copy > did indeed preserve the extents. If I do as you suggest, even if > cp mistakenly filled in all holes, the test would still pass. > I wouldn't call it pointless. More efficient read is still a big win. My problem with the filefrag compare is that it is expecting a behavior that no filesystem guarantees. But having said that, I am intrigued as to why it is not working as expected. Do you have a strace for me to look at? strace of the fiemap enabled cp. Also, output of "stat -f j1; stat j1 j2;filefrag -v j1; filefrag -v j2;" Thanks Sunil From debbugs-submit-bounces@debbugs.gnu.org Thu May 27 16:27:08 2010 Received: (at submit) by debbugs.gnu.org; 27 May 2010 20:27:09 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHjfY-0007NO-8i for submit@debbugs.gnu.org; Thu, 27 May 2010 16:27:08 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHjfW-0007Mz-Aj for submit@debbugs.gnu.org; Thu, 27 May 2010 16:27:07 -0400 Received: from lists.gnu.org ([199.232.76.165]:46114) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OHjfR-0008GI-U6 for submit@debbugs.gnu.org; Thu, 27 May 2010 16:27:02 -0400 Received: from [140.186.70.92] (port=34211 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OHjfQ-0001qS-GV for bug-coreutils@gnu.org; Thu, 27 May 2010 16:27:01 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OHjfP-0002E1-By for bug-coreutils@gnu.org; Thu, 27 May 2010 16:27:00 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:43979) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHjfO-0002D5-Qr for bug-coreutils@gnu.org; Thu, 27 May 2010 16:26:59 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id 72E74940193; Thu, 27 May 2010 22:26:51 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 10D92BC3; Thu, 27 May 2010 22:26:50 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <87eigxb2uz.fsf@meyering.net> (Jim Meyering's message of "Thu, 27 May 2010 20:45:24 +0200") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFE7CEB.4040407@oracle.com> <87eigxb2uz.fsf@meyering.net> Date: Thu, 27 May 2010 22:26:50 +0200 Message-ID: <8739xday5x.fsf@meyering.net> Lines: 84 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Tao Ma , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Jim Meyering wrote: > jeff.liu wrote: >> Jim Meyering wrote: >>> jeff.liu wrote: >>>> This is the revised version, it fixed the fiemap-start offset calculation >>>> approach to remove it out >>>> of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)' loop. >>> >>> Hi Jeff, >>> >>> I've included below the state of my local changes. >>> Unfortunately, with that 5-patch series, there is always a test failure >>> on F13/ext4. Maybe someone who knows more about extents can provide an >>> explanation? >>> >>> Here's a small example to demonstrate: >>> >>> Create a file with many extents: >>> >>> perl -e 'BEGIN { $n = 19 * 1024; *F = *STDOUT }' \ >>> -e 'for (1..100) { sysseek (*F, $n, 1)' \ >>> -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 >>> >>> Using the patched "cp", repeat the following 10 or 20 times: >>> >>> ./cp --sparse=always j1 j2; sync >>> filefrag -v j1 | awk '/^ / {print $1,$2}' > ff1 || fail=1 >>> filefrag -v j2 | awk '/^ / {print $1,$2}' > ff2 || fail=1 >>> diff -u ff1 ff2 || fail=1 >>> >>> Usually there is no diff output, but occasionally it'll print this: >>> [hmm... today it consistently prints these differences every other time.] >> Woo!!! >> I just run this test on btrfs/ext4/ocfs2 against mainline kernel(Linux jeff-laptop >> 2.6.33-rc5-00238-gb04da8b-dirty) on my laptop. >> Only btrfs always works well for me, Ext4 has the same issue like yours. > > One more point of reference: > the tests all passed on an XFS file system. Since XFS is relatively stable, I'll use it instead of ext4. This incremental will permit use of either XFS or btrfs for this test, as non-root, and otherwise, create an XFS loopback partition when run as root: diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index ef3742e..763c3b7 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -23,8 +23,8 @@ fi . $srcdir/test-lib.sh -if df -T -t ext4 . ; then - : # Current dir is on an ext4 partition. Good! +if df -T -t btrfs -t xfs . ; then + : # Current dir is on a partition with working extents. Good! else # It's not; we need to create one, hence we need root access. require_root_ @@ -33,18 +33,18 @@ else cleanup_() { cd /; umount "$cwd/mnt"; } skip=0 - # Create an ext4 loopback file system - dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 + # Create an XFS loopback file system + dd if=/dev/zero of=blob bs=32k count=1000 || skip=1 mkdir mnt - mkfs -t ext4 -F blob || - skip_test_ "failed to create ext4 file system" + mkfs -t xfs blob || + skip_test_ "failed to create XFS file system" mount -oloop blob mnt || skip=1 cd mnt || skip=1 echo test > f || skip=1 test -s f || skip=1 test $skip = 1 && - skip_test_ "insufficient mount/ext4 support" + skip_test_ "insufficient mount/XFS support" fi From debbugs-submit-bounces@debbugs.gnu.org Fri May 28 04:08:31 2010 Received: (at submit) by debbugs.gnu.org; 28 May 2010 08:08:31 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHucH-0003uy-5c for submit@debbugs.gnu.org; Fri, 28 May 2010 04:08:31 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHuOh-0003nK-Dn for submit@debbugs.gnu.org; Fri, 28 May 2010 03:54:39 -0400 Received: from lists.gnu.org ([199.232.76.165]:56301) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OHuOb-0002TK-Go for submit@debbugs.gnu.org; Fri, 28 May 2010 03:54:21 -0400 Received: from [140.186.70.92] (port=56008 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OHuOW-0000z5-I3 for bug-coreutils@gnu.org; Fri, 28 May 2010 03:54:20 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OHuOR-0008Py-CT for bug-coreutils@gnu.org; Fri, 28 May 2010 03:54:12 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:59717) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OHuOR-0008PT-5A for bug-coreutils@gnu.org; Fri, 28 May 2010 03:54:11 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4S7s5JW022073 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 28 May 2010 07:54:06 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4S77fZj016223; Fri, 28 May 2010 07:54:04 GMT Received: from abhmt016.oracle.com by acsmt353.oracle.com with ESMTP id 276332531275033158; Fri, 28 May 2010 00:52:38 -0700 Received: from [10.182.120.191] (/10.182.120.191) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 28 May 2010 00:52:37 -0700 Message-ID: <4BFF763A.9010205@oracle.com> Date: Fri, 28 May 2010 15:52:26 +0800 From: Tao Ma User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> In-Reply-To: <877hmpeivx.fsf@meyering.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090209.4BFF769F.0005:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Fri, 28 May 2010 04:08:28 -0400 Cc: Sunil Mushran , "jeff.liu" , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.4 (-----) Hi Jim On 05/27/2010 06:30 PM, Jim Meyering wrote: > jeff.liu wrote: >> This is the revised version, it fixed the fiemap-start offset calculation >> approach to remove it out >> of the 'for (i = 0; i< fiemap->fm_mapped_extents; i++)' loop. > > Hi Jeff, > > I've included below the state of my local changes. > Unfortunately, with that 5-patch series, there is always a test failure > on F13/ext4. Maybe someone who knows more about extents can provide an > explanation? Just want to clarify why ocfs2 didn't work here. I guess the reason also works for ext4 since both ext4 and ocfs2 use block group to organize their blocks in the volume. I checked the perl test script to create sparse src file, it will create contiguous bytes(around 20-24k) at an interval of around 40k.So in general, these 20-24k should be contiguous. But that does exist some scenario that they could be separately into 2 extents. Consider one block group is used to allocate blocks to this file, when the block group only has 10K left while you are requiring 20K, it will use the left 10K in this group and allocate 10K from another block group. That would become 2 extents since they can't be contiguous. So I guess the right step is to check the holes by using filefrag if you prefer this tool and want to make sure cp doesn't copy holes(I get this point from another e-mail written by you). How to find holes with filefrag? I guess it is quite simple since it also use fiemap and we can calculating holes easily by comparing the 2 consecutive records. I guess we can get what you want with ext4 after this update. Regards, Tao From debbugs-submit-bounces@debbugs.gnu.org Fri May 28 12:44:52 2010 Received: (at submit) by debbugs.gnu.org; 28 May 2010 16:44:53 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OI2fz-000894-TK for submit@debbugs.gnu.org; Fri, 28 May 2010 12:44:52 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OI2fy-00088z-6m for submit@debbugs.gnu.org; Fri, 28 May 2010 12:44:50 -0400 Received: from lists.gnu.org ([199.232.76.165]:57978) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OI2ft-0005Ct-Ca for submit@debbugs.gnu.org; Fri, 28 May 2010 12:44:45 -0400 Received: from [140.186.70.92] (port=58711 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OI2fr-0006Dg-JX for bug-coreutils@gnu.org; Fri, 28 May 2010 12:44:44 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OI2fq-0007vi-Cj for bug-coreutils@gnu.org; Fri, 28 May 2010 12:44:43 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:49773) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OI2fp-0007v4-S0 for bug-coreutils@gnu.org; Fri, 28 May 2010 12:44:42 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id D954694014C; Fri, 28 May 2010 18:44:34 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id B557DDD14; Fri, 28 May 2010 18:44:33 +0200 (CEST) From: Jim Meyering To: Tao Ma Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4BFF763A.9010205@oracle.com> (Tao Ma's message of "Fri, 28 May 2010 15:52:26 +0800") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> Date: Fri, 28 May 2010 18:44:33 +0200 Message-ID: <8763286kni.fsf@meyering.net> Lines: 51 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , "jeff.liu" , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Tao Ma wrote: > Hi Jim > > On 05/27/2010 06:30 PM, Jim Meyering wrote: >> jeff.liu wrote: >>> This is the revised version, it fixed the fiemap-start offset calculation >>> approach to remove it out >>> of the 'for (i = 0; i< fiemap->fm_mapped_extents; i++)' loop. >> >> Hi Jeff, >> >> I've included below the state of my local changes. >> Unfortunately, with that 5-patch series, there is always a test failure >> on F13/ext4. Maybe someone who knows more about extents can provide an >> explanation? > Just want to clarify why ocfs2 didn't work here. I guess the reason > also works for ext4 since both ext4 and ocfs2 use block group to > organize their blocks in the volume. Hi Tao, Thank you for the explanation. I'm glad to hear that there is no underlying problem. > I checked the perl test script to create sparse src file, it will > create contiguous bytes(around 20-24k) at an interval of around 40k.So > in general, these 20-24k should be contiguous. But that does exist > some scenario that they could be separately into 2 extents. Consider > one block group is used to allocate blocks to this file, when the > block group only has 10K left while you are requiring 20K, it will use > the left 10K in this group and allocate 10K from another block > group. That would become 2 extents since they can't be contiguous. > So I guess the right step is to check the holes by using filefrag if > you prefer this tool and want to make sure cp doesn't copy holes(I get Do you know of a tool other than filefrag that I could use? It looks like a small script could filter filefrag -v output, detect split extents and rewrite to make the output match what's expected. Probably not worth it, though, since this is already a very fragile test. It would be nice to be able to perform the test in non-root mode on any of ext4, ocfs2, btrfs, xfs file systems, but for now, due to this difference, I can use only the latter two. > this point from another e-mail written by you). How to find holes with > filefrag? I guess it is quite simple since it also use fiemap and we > can calculating holes easily by comparing the 2 consecutive records. I > guess we can get what you want with ext4 after this update. From debbugs-submit-bounces@debbugs.gnu.org Sun May 30 10:28:40 2010 Received: (at submit) by debbugs.gnu.org; 30 May 2010 14:28:40 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OIjVG-00013b-RW for submit@debbugs.gnu.org; Sun, 30 May 2010 10:28:39 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OIjVF-00013U-4R for submit@debbugs.gnu.org; Sun, 30 May 2010 10:28:37 -0400 Received: from lists.gnu.org ([199.232.76.165]:37305) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OIjVA-0004FE-8Z for submit@debbugs.gnu.org; Sun, 30 May 2010 10:28:32 -0400 Received: from [140.186.70.92] (port=43953 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OIjV8-0007bC-PY for bug-coreutils@gnu.org; Sun, 30 May 2010 10:28:31 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OIjV7-0001tj-IP for bug-coreutils@gnu.org; Sun, 30 May 2010 10:28:30 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:26914) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OIjV7-0001tX-CH for bug-coreutils@gnu.org; Sun, 30 May 2010 10:28:29 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4UESNQ7011043 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sun, 30 May 2010 14:28:25 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4UAK2bC006579; Sun, 30 May 2010 14:28:23 GMT Received: from abhmt015.oracle.com by acsmt353.oracle.com with ESMTP id 308991431275229701; Sun, 30 May 2010 07:28:21 -0700 Received: from [192.168.1.2] (/221.217.40.250) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sun, 30 May 2010 07:28:20 -0700 Message-ID: <4C0275FA.1030901@oracle.com> Date: Sun, 30 May 2010 22:28:10 +0800 From: Tao Ma User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> In-Reply-To: <8763286kni.fsf@meyering.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090207.4C027609.01B0:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , "jeff.liu" , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.9 (-----) Hi Jim, On 05/29/2010 12:44 AM, Jim Meyering wrote: > Tao Ma wrote: > >> Hi Jim >> >> On 05/27/2010 06:30 PM, Jim Meyering wrote: >>> jeff.liu wrote: >>>> This is the revised version, it fixed the fiemap-start offset calculation >>>> approach to remove it out >>>> of the 'for (i = 0; i< fiemap->fm_mapped_extents; i++)' loop. >>> >>> Hi Jeff, >>> >>> I've included below the state of my local changes. >>> Unfortunately, with that 5-patch series, there is always a test failure >>> on F13/ext4. Maybe someone who knows more about extents can provide an >>> explanation? >> Just want to clarify why ocfs2 didn't work here. I guess the reason >> also works for ext4 since both ext4 and ocfs2 use block group to >> organize their blocks in the volume. > > Hi Tao, > > Thank you for the explanation. > I'm glad to hear that there is no underlying problem. > >> I checked the perl test script to create sparse src file, it will >> create contiguous bytes(around 20-24k) at an interval of around 40k.So >> in general, these 20-24k should be contiguous. But that does exist >> some scenario that they could be separately into 2 extents. Consider >> one block group is used to allocate blocks to this file, when the >> block group only has 10K left while you are requiring 20K, it will use >> the left 10K in this group and allocate 10K from another block >> group. That would become 2 extents since they can't be contiguous. > >> So I guess the right step is to check the holes by using filefrag if >> you prefer this tool and want to make sure cp doesn't copy holes(I get > > Do you know of a tool other than filefrag that I could use? nope. > > It looks like a small script could filter filefrag -v output, detect > split extents and rewrite to make the output match what's expected. > Probably not worth it, though, since this is already a very fragile test. yeah, actually I guess what we want is not the extents number but the result of whether extents match. So if we can find all the pairs(v_cpos, len) both in the src and the target, and there is no extra pair in the target, that is good enough. The extents number is really a tricky thing and depends on what the internal file system does. > > It would be nice to be able to perform the test in non-root > mode on any of ext4, ocfs2, btrfs, xfs file systems, but for > now, due to this difference, I can use only the latter two. I think with the above method we should test ext4 and ocfs2 also. So jeff, could you please write a small script to test whether the above method work? Regards, Tao From debbugs-submit-bounces@debbugs.gnu.org Sun May 30 16:12:37 2010 Received: (at submit) by debbugs.gnu.org; 30 May 2010 20:12:37 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OIos9-0003Ry-3O for submit@debbugs.gnu.org; Sun, 30 May 2010 16:12:37 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OIos7-0003Rt-9b for submit@debbugs.gnu.org; Sun, 30 May 2010 16:12:36 -0400 Received: from lists.gnu.org ([199.232.76.165]:35663) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OIos2-0000Os-Tv for submit@debbugs.gnu.org; Sun, 30 May 2010 16:12:30 -0400 Received: from [140.186.70.92] (port=51233 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OIos1-0007ww-5n for bug-coreutils@gnu.org; Sun, 30 May 2010 16:12:30 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OIorz-0003eB-Vr for bug-coreutils@gnu.org; Sun, 30 May 2010 16:12:29 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:54337) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OIorz-0003dQ-C4 for bug-coreutils@gnu.org; Sun, 30 May 2010 16:12:27 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id 461DF9400C6; Sun, 30 May 2010 22:12:19 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id DC78CDC1; Sun, 30 May 2010 22:12:17 +0200 (CEST) From: Jim Meyering To: Tao Ma Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4C0275FA.1030901@oracle.com> (Tao Ma's message of "Sun, 30 May 2010 22:28:10 +0800") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> Date: Sun, 30 May 2010 22:12:17 +0200 Message-ID: <874ohpywri.fsf@meyering.net> Lines: 108 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , "jeff.liu" , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Tao Ma wrote: > Hi Jim, > On 05/29/2010 12:44 AM, Jim Meyering wrote: >> Tao Ma wrote: >> >>> Hi Jim >>> >>> On 05/27/2010 06:30 PM, Jim Meyering wrote: >>>> jeff.liu wrote: >>>>> This is the revised version, it fixed the fiemap-start offset calculation >>>>> approach to remove it out >>>>> of the 'for (i = 0; i< fiemap->fm_mapped_extents; i++)' loop. >>>> >>>> Hi Jeff, >>>> >>>> I've included below the state of my local changes. >>>> Unfortunately, with that 5-patch series, there is always a test failure >>>> on F13/ext4. Maybe someone who knows more about extents can provide an >>>> explanation? >>> Just want to clarify why ocfs2 didn't work here. I guess the reason >>> also works for ext4 since both ext4 and ocfs2 use block group to >>> organize their blocks in the volume. >> >> Hi Tao, >> >> Thank you for the explanation. >> I'm glad to hear that there is no underlying problem. >> >>> I checked the perl test script to create sparse src file, it will >>> create contiguous bytes(around 20-24k) at an interval of around 40k.So >>> in general, these 20-24k should be contiguous. But that does exist >>> some scenario that they could be separately into 2 extents. Consider >>> one block group is used to allocate blocks to this file, when the >>> block group only has 10K left while you are requiring 20K, it will use >>> the left 10K in this group and allocate 10K from another block >>> group. That would become 2 extents since they can't be contiguous. >> >>> So I guess the right step is to check the holes by using filefrag if >>> you prefer this tool and want to make sure cp doesn't copy holes(I get >> >> Do you know of a tool other than filefrag that I could use? > nope. >> >> It looks like a small script could filter filefrag -v output, detect >> split extents and rewrite to make the output match what's expected. >> Probably not worth it, though, since this is already a very fragile test. I went ahead and did it, after all. Here's the script, filefrag-extent-compare. With it, this test should pass when run on any of those four file system types. eval '(exit $?0)' && eval 'exec perl -wS "$0" ${1+"$@"}' & eval 'exec perl -wS "$0" $argv:q' if 0; # Determine whether two files have the same extents by comparing # the logical block numbers and lengths from filefrag -v for each. # Invoke like this: # This helper function, f, extracts logical block number and lengths. # f() { awk '/^ *[0-9]/ {printf "%d %d ",$2,NF<5?$NF:$5} END {print ""}'; } # { filefrag -v j1 | f; filefrag -v j2 | f; } | ./filefrag-extent-compare use warnings; use strict; (my $ME = $0) =~ s|.*/||; my @line = <>; my $n_lines = @line; $n_lines == 2 or die "$ME: expected exactly two input lines; got $n_lines\n"; my @A = split ' ', $line[0]; my @B = split ' ', $line[1]; @A % 2 || @B % 2 and die "$ME: unexpected input: odd number of numbers; expected even\n"; my @a; my @b; foreach my $i (0..@A/2-1) { $a[$i] = { L_BLK => $A[2*$i], LEN => $A[2*$i+1] } }; foreach my $i (0..@B/2-1) { $b[$i] = { L_BLK => $B[2*$i], LEN => $B[2*$i+1] } }; my $i = 0; my $j = 0; while (1) { !defined $a[$i] && !defined $b[$j] and exit 0; defined $a[$i] && defined $b[$j] or die "\@a and \@b have different lengths, even after adjustment\n"; ($a[$i]->{L_BLK} == $b[$j]->{L_BLK} && $a[$i]->{LEN} == $b[$j]->{LEN}) and next; ($a[$i]->{LEN} < $b[$j]->{LEN} && exists $a[$i+1] && $a[$i]->{LEN} + $a[$i+1]->{LEN} == $b[$j]->{LEN}) and ++$i, next; exists $b[$j+1] && $a[$i]->{LEN} == $b[$i]->{LEN} + $b[$i+1]->{LEN} and ++$j, next; die "differing extent:\n" . " [$i]=$a[$i]->{L_BLK} $a[$i]->{LEN}\n" . " [$j]=$b[$j]->{L_BLK} $b[$j]->{LEN}\n" } continue { ++$i; ++$j; } From debbugs-submit-bounces@debbugs.gnu.org Sun May 30 22:53:04 2010 Received: (at submit) by debbugs.gnu.org; 31 May 2010 02:53:05 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OIv7g-00062x-Gz for submit@debbugs.gnu.org; Sun, 30 May 2010 22:53:04 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OIv7e-00062a-ER for submit@debbugs.gnu.org; Sun, 30 May 2010 22:53:03 -0400 Received: from lists.gnu.org ([199.232.76.165]:50377) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OIv7a-0005aP-RY for submit@debbugs.gnu.org; Sun, 30 May 2010 22:52:58 -0400 Received: from [140.186.70.92] (port=51087 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OIv7Z-0006gW-0V for bug-coreutils@gnu.org; Sun, 30 May 2010 22:52:58 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OIv7X-0007Hb-SV for bug-coreutils@gnu.org; Sun, 30 May 2010 22:52:56 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:64925) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OIv7X-0007HG-MS for bug-coreutils@gnu.org; Sun, 30 May 2010 22:52:55 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4V2qnNf023886 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 31 May 2010 02:52:51 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o4V2qml9018967; Mon, 31 May 2010 02:52:48 GMT Received: from abhmt013.oracle.com by acsmt354.oracle.com with ESMTP id 281543461275274309; Sun, 30 May 2010 19:51:49 -0700 Received: from [10.182.121.3] (/10.182.121.3) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sun, 30 May 2010 19:51:48 -0700 Message-ID: <4C032447.3040301@oracle.com> Date: Mon, 31 May 2010 10:51:51 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering , Tao Ma Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> In-Reply-To: <874ohpywri.fsf@meyering.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090208.4C032483.00F0:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.2 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.2 (------) Jim Meyering wrote: > Tao Ma wrote: >> Hi Jim, >> On 05/29/2010 12:44 AM, Jim Meyering wrote: >>> Tao Ma wrote: >>> >>>> Hi Jim >>>> >>>> On 05/27/2010 06:30 PM, Jim Meyering wrote: >>>>> jeff.liu wrote: >>>>>> This is the revised version, it fixed the fiemap-start offset calculation >>>>>> approach to remove it out >>>>>> of the 'for (i = 0; i< fiemap->fm_mapped_extents; i++)' loop. >>>>> Hi Jeff, >>>>> >>>>> I've included below the state of my local changes. >>>>> Unfortunately, with that 5-patch series, there is always a test failure >>>>> on F13/ext4. Maybe someone who knows more about extents can provide an >>>>> explanation? >>>> Just want to clarify why ocfs2 didn't work here. I guess the reason >>>> also works for ext4 since both ext4 and ocfs2 use block group to >>>> organize their blocks in the volume. >>> Hi Tao, >>> >>> Thank you for the explanation. >>> I'm glad to hear that there is no underlying problem. >>> >>>> I checked the perl test script to create sparse src file, it will >>>> create contiguous bytes(around 20-24k) at an interval of around 40k.So >>>> in general, these 20-24k should be contiguous. But that does exist >>>> some scenario that they could be separately into 2 extents. Consider >>>> one block group is used to allocate blocks to this file, when the >>>> block group only has 10K left while you are requiring 20K, it will use >>>> the left 10K in this group and allocate 10K from another block >>>> group. That would become 2 extents since they can't be contiguous. >>>> So I guess the right step is to check the holes by using filefrag if >>>> you prefer this tool and want to make sure cp doesn't copy holes(I get >>> Do you know of a tool other than filefrag that I could use? >> nope. >>> It looks like a small script could filter filefrag -v output, detect >>> split extents and rewrite to make the output match what's expected. >>> Probably not worth it, though, since this is already a very fragile test. > > I went ahead and did it, after all. > Here's the script, filefrag-extent-compare. > With it, this test should pass when run on any of those four > file system types. > > > eval '(exit $?0)' && eval 'exec perl -wS "$0" ${1+"$@"}' > & eval 'exec perl -wS "$0" $argv:q' > if 0; > # Determine whether two files have the same extents by comparing > # the logical block numbers and lengths from filefrag -v for each. > > # Invoke like this: > # This helper function, f, extracts logical block number and lengths. > # f() { awk '/^ *[0-9]/ {printf "%d %d ",$2,NF<5?$NF:$5} END {print ""}'; } > # { filefrag -v j1 | f; filefrag -v j2 | f; } | ./filefrag-extent-compare > > use warnings; > use strict; > (my $ME = $0) =~ s|.*/||; > > my @line = <>; > my $n_lines = @line; > $n_lines == 2 > or die "$ME: expected exactly two input lines; got $n_lines\n"; > > my @A = split ' ', $line[0]; > my @B = split ' ', $line[1]; > @A % 2 || @B % 2 > and die "$ME: unexpected input: odd number of numbers; expected even\n"; > > my @a; > my @b; > foreach my $i (0..@A/2-1) { $a[$i] = { L_BLK => $A[2*$i], LEN => $A[2*$i+1] } }; > foreach my $i (0..@B/2-1) { $b[$i] = { L_BLK => $B[2*$i], LEN => $B[2*$i+1] } }; > > my $i = 0; > my $j = 0; > while (1) > { > !defined $a[$i] && !defined $b[$j] > and exit 0; > defined $a[$i] && defined $b[$j] > or die "\@a and \@b have different lengths, even after adjustment\n"; > ($a[$i]->{L_BLK} == $b[$j]->{L_BLK} > && $a[$i]->{LEN} == $b[$j]->{LEN}) > and next; > ($a[$i]->{LEN} < $b[$j]->{LEN} > && exists $a[$i+1] && $a[$i]->{LEN} + $a[$i+1]->{LEN} == $b[$j]->{LEN}) > and ++$i, next; > exists $b[$j+1] && $a[$i]->{LEN} == $b[$i]->{LEN} + $b[$i+1]->{LEN} > and ++$j, next; > die "differing extent:\n" > . " [$i]=$a[$i]->{L_BLK} $a[$i]->{LEN}\n" > . " [$j]=$b[$j]->{L_BLK} $b[$j]->{LEN}\n" > } > continue > { > ++$i; > ++$j; > } > > > Thanks Tao and Jim for helping this out! -Jeff -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 02 03:28:17 2010 Received: (at submit) by debbugs.gnu.org; 2 Jun 2010 07:28:17 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OJiN7-0007RP-8l for submit@debbugs.gnu.org; Wed, 02 Jun 2010 03:28:17 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OJiN4-0007RI-Ci for submit@debbugs.gnu.org; Wed, 02 Jun 2010 03:28:15 -0400 Received: from lists.gnu.org ([199.232.76.165]:47482) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OJiN1-0001Nv-0I for submit@debbugs.gnu.org; Wed, 02 Jun 2010 03:28:11 -0400 Received: from [140.186.70.92] (port=58587 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OJiMu-0007Tn-Gy for bug-coreutils@gnu.org; Wed, 02 Jun 2010 03:28:10 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OJiMt-0003zQ-9o for bug-coreutils@gnu.org; Wed, 02 Jun 2010 03:28:04 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:43086) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OJiMs-0003yN-Nr for bug-coreutils@gnu.org; Wed, 02 Jun 2010 03:28:03 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id 2A5E69400AD; Wed, 2 Jun 2010 09:27:54 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id BD750E003; Wed, 2 Jun 2010 09:27:53 +0200 (CEST) From: Jim Meyering To: Tao Ma Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <874ohpywri.fsf@meyering.net> (Jim Meyering's message of "Sun, 30 May 2010 22:12:17 +0200") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> Date: Wed, 02 Jun 2010 09:27:53 +0200 Message-ID: <87y6exuc5i.fsf@meyering.net> Lines: 52 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , "jeff.liu" , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Jim Meyering wrote: ... >>> Do you know of a tool other than filefrag that I could use? >> nope. >>> >>> It looks like a small script could filter filefrag -v output, detect >>> split extents and rewrite to make the output match what's expected. >>> Probably not worth it, though, since this is already a very fragile test. > > I went ahead and did it, after all. > Here's the script, filefrag-extent-compare. > With it, this test should pass when run on any of those four > file system types. Not quite. Parsing filefrag -v output is part of what is fragile. Here are two examples: ==> ff1 <== Filesystem type is: ef53 File size of j1 is 49152 (12 blocks, blocksize 4096) ext logical physical expected length flags 0 3 10256380 3 1 9 10740733 10256382 3 eof ==> ff2 <== Filesystem type is: ef53 File size of j2 is 49152 (12 blocks, blocksize 4096) ext logical physical expected length flags 0 3 10283520 3 1 9 10283523 3 eof ^^^^^^ Note the missing "expected" number. That doesn't happen often, but it's enough to cause occasional false-positive failures, since the awk filter counts fields. I don't dare try to count columns, because those physical block numbers are likely to have width greater than 8 some of the time. Instead, I adjusted the filter to remove the "eof" and let the existing awk code handle the rest: # Extract logical block number and length pairs from filefrag -v output. # The initial sed is to remove the "eof" from the normally-empty "flags" field. # That is required when that final extent has no number in the "expected" field. f() { sed 's/ eof$//' $@ \ | awk '/^ *[0-9]/ {printf "%d %d ", $2 ,NF < 5 ? $NF : $5 } END {print ""}' } I'll post a new patch soon. From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 08 07:34:27 2010 Received: (at submit) by debbugs.gnu.org; 8 Jun 2010 11:34:27 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OLx4Z-0003Tg-JM for submit@debbugs.gnu.org; Tue, 08 Jun 2010 07:34:26 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OLx4V-0003TZ-0p for submit@debbugs.gnu.org; Tue, 08 Jun 2010 07:34:21 -0400 Received: from lists.gnu.org ([199.232.76.165]:55854) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OLx4O-0003i3-BE for submit@debbugs.gnu.org; Tue, 08 Jun 2010 07:34:12 -0400 Received: from [140.186.70.92] (port=42982 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OLx4G-0007Sf-Ju for bug-coreutils@gnu.org; Tue, 08 Jun 2010 07:34:11 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OLx49-00080S-5x for bug-coreutils@gnu.org; Tue, 08 Jun 2010 07:34:03 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:49398) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OLx48-0007zw-B7 for bug-coreutils@gnu.org; Tue, 08 Jun 2010 07:33:57 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id C927694007C; Tue, 8 Jun 2010 13:33:48 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id A526FBD2; Tue, 8 Jun 2010 13:33:47 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <87y6exuc5i.fsf@meyering.net> (Jim Meyering's message of "Wed, 02 Jun 2010 09:27:53 +0200") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> Date: Tue, 08 Jun 2010 13:33:47 +0200 Message-ID: <874ohdu5b8.fsf@meyering.net> Lines: 1122 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Chris Mason , bug-coreutils@gnu.org, Joel Becker , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Jim Meyering wrote: > Jim Meyering wrote: > ... >>>> Do you know of a tool other than filefrag that I could use? >>> nope. >>>> >>>> It looks like a small script could filter filefrag -v output, detect >>>> split extents and rewrite to make the output match what's expected. >>>> Probably not worth it, though, since this is already a very fragile test. >> >> I went ahead and did it, after all. >> Here's the script, filefrag-extent-compare. >> With it, this test should pass when run on any of those four >> file system types. > > Not quite. > Parsing filefrag -v output is part of what is fragile. > > Here are two examples: > > ==> ff1 <== > Filesystem type is: ef53 > File size of j1 is 49152 (12 blocks, blocksize 4096) > ext logical physical expected length flags > 0 3 10256380 3 > 1 9 10740733 10256382 3 eof > > ==> ff2 <== > Filesystem type is: ef53 > File size of j2 is 49152 (12 blocks, blocksize 4096) > ext logical physical expected length flags > 0 3 10283520 3 > 1 9 10283523 3 eof > ^^^^^^ > Note the missing "expected" number. > That doesn't happen often, but it's enough to cause occasional > false-positive failures, since the awk filter counts fields. > I don't dare try to count columns, because those physical block > numbers are likely to have width greater than 8 some of the time. > > Instead, I adjusted the filter to remove the "eof" > and let the existing awk code handle the rest: > > # Extract logical block number and length pairs from filefrag -v output. > # The initial sed is to remove the "eof" from the normally-empty "flags" field. > # That is required when that final extent has no number in the "expected" field. > f() > { > sed 's/ eof$//' $@ \ > | awk '/^ *[0-9]/ {printf "%d %d ", $2 ,NF < 5 ? $NF : $5 } END {print ""}' > } > > I'll post a new patch soon. Slowly but surely... I'll squash the two most recent changes (at the end, below) into yours, Jeff. >From 9f7a9882944455bfc2b6aec9c9f5431ac429b88b Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Thu, 13 May 2010 22:09:30 +0800 Subject: [PATCH 01/10] cp: Add FIEMAP support for efficient sparse file copy * src/fiemap.h: Add fiemap.h for fiemap ioctl(2) support. Copied from linux's include/linux/fiemap.h, with minor formatting changes. * src/copy.c (copy_reg): Now, when `cp' invoked with --sparse=[WHEN] option, we will try to do FIEMAP-copy if the underlaying file system support it, fall back to a normal copy if it fails. --- src/copy.c | 159 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ src/fiemap.h | 102 +++++++++++++++++++++++++++++++++++++ 2 files changed, 261 insertions(+), 0 deletions(-) create mode 100644 src/fiemap.h diff --git a/src/copy.c b/src/copy.c index 171499c..2c15ca0 100644 --- a/src/copy.c +++ b/src/copy.c @@ -63,6 +63,10 @@ #include +#ifndef HAVE_FIEMAP +# include "fiemap.h" +#endif + #ifndef HAVE_FCHOWN # define HAVE_FCHOWN false # define fchown(fd, uid, gid) (-1) @@ -149,6 +153,141 @@ clone_file (int dest_fd, int src_fd) #endif } +#ifdef __linux__ +# ifndef FS_IOC_FIEMAP +# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) +# endif +/* Perform FIEMAP(available in mainline 2.6.27) copy if possible. + Call ioctl(2) with FS_IOC_FIEMAP to efficiently map file allocation + excepts holes. So the overhead to deal with holes with lseek(2) in + normal copy could be saved. This would result in much faster backups + for any kind of sparse file. */ +static bool +fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, + off_t src_total_size, char const *src_name, + char const *dst_name, bool *normal_copy_required) +{ + bool fail = false; + bool last = false; + char fiemap_buf[4096]; + struct fiemap *fiemap = (struct fiemap *)fiemap_buf; + struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; + uint32_t count = (sizeof (fiemap_buf) - sizeof (*fiemap)) / + sizeof (struct fiemap_extent); + off_t last_ext_logical = 0; + uint64_t last_ext_len = 0; + uint64_t last_read_size = 0; + unsigned int i = 0; + + /* This is required at least to initialize fiemap->fm_start, + but also serves (in May 2010) to appease valgrind, which + appears not to know the semantics of the FIEMAP ioctl. */ + memset (fiemap_buf, 0, sizeof fiemap_buf); + + do + { + fiemap->fm_length = FIEMAP_MAX_OFFSET; + fiemap->fm_extent_count = count; + + /* When ioctl(2) fails, fall back to the normal copy only if it + is the first time we met. */ + if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) + { + /* If `i > 0', then at least one ioctl(2) has been performed before. */ + if (i == 0) + *normal_copy_required = true; + return false; + } + + /* If 0 extents are returned, then more ioctls are not needed. */ + if (fiemap->fm_mapped_extents == 0) + break; + + for (i = 0; i < fiemap->fm_mapped_extents; i++) + { + assert (fm_ext[i].fe_logical <= OFF_T_MAX); + + off_t ext_logical = fm_ext[i].fe_logical; + uint64_t ext_len = fm_ext[i].fe_length; + + if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) + { + error (0, errno, _("cannot lseek %s"), quote (src_name)); + return fail; + } + + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0LL) + { + error (0, errno, _("cannot lseek %s"), quote (dst_name)); + return fail; + } + + if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST) + { + last_ext_logical = ext_logical; + last_ext_len = ext_len; + last = true; + } + + while (0 < ext_len) + { + char buf[buf_size]; + + /* Avoid reading into the holes if the left extent + length is shorter than the buffer size. */ + if (ext_len < buf_size) + buf_size = ext_len; + + ssize_t n_read = read (src_fd, buf, buf_size); + if (n_read < 0) + { +#ifdef EINTR + if (errno == EINTR) + continue; +#endif + error (0, errno, _("reading %s"), quote (src_name)); + return fail; + } + + if (n_read == 0) + { + /* Figure out how many bytes read from the last extent. */ + last_read_size = last_ext_len - ext_len; + break; + } + + if (full_write (dest_fd, buf, n_read) != n_read) + { + error (0, errno, _("writing %s"), quote (dst_name)); + return fail; + } + + ext_len -= n_read; + } + } + + fiemap->fm_start = fm_ext[i - 1].fe_logical + fm_ext[i - 1].fe_length; + + } while (! last); + + /* If a file ends up with holes, the sum of the last extent logical offset + and the read-returned size will be shorter than the actual size of the + file. Use ftruncate to extend the length of the destination file. */ + if (last_ext_logical + last_read_size < src_total_size) + { + if (ftruncate (dest_fd, src_total_size) < 0) + { + error (0, errno, _("extending %s"), quote (dst_name)); + return fail; + } + } + + return ! fail; +} +#else +static bool fiemap_copy_ok (ignored) { errno == ENOTSUP; return false; } +#endif + /* FIXME: describe */ /* FIXME: rewrite this to use a hash table so we avoid the quadratic performance hit that's probably noticeable only on trees deeper @@ -679,6 +818,25 @@ copy_reg (char const *src_name, char const *dst_name, #endif } + if (make_holes) + { + bool require_normal_copy = false; + /* Perform efficient FIEMAP copy for sparse files, fall back to the + standard copy only if the ioctl(2) fails. */ + if (fiemap_copy_ok (source_desc, dest_desc, buf_size, + src_open_sb.st_size, src_name, + dst_name, &require_normal_copy)) + goto preserve_metadata; + else + { + if (! require_normal_copy) + { + return_val = false; + goto close_src_and_dst_desc; + } + } + } + /* If not making a sparse file, try to use a more-efficient buffer size. */ if (! make_holes) @@ -807,6 +965,7 @@ copy_reg (char const *src_name, char const *dst_name, } } +preserve_metadata: if (x->preserve_timestamps) { struct timespec timespec[2]; diff --git a/src/fiemap.h b/src/fiemap.h new file mode 100644 index 0000000..d33293b --- /dev/null +++ b/src/fiemap.h @@ -0,0 +1,102 @@ +/* FS_IOC_FIEMAP ioctl infrastructure. + Some portions copyright (C) 2007 Cluster File Systems, Inc + Authors: Mark Fasheh + Kalpak Shah + Andreas Dilger . */ + +/* Copy from kernel, modified to respect GNU code style by Jie Liu. */ + +#ifndef _LINUX_FIEMAP_H +# define _LINUX_FIEMAP_H + +# include + +struct fiemap_extent +{ + /* Logical offset in bytes for the start of the extent + from the beginning of the file. */ + uint64_t fe_logical; + + /* Physical offset in bytes for the start of the extent + from the beginning of the disk. */ + uint64_t fe_physical; + + /* Length in bytes for this extent. */ + uint64_t fe_length; + + uint64_t fe_reserved64[2]; + + /* FIEMAP_EXTENT_* flags for this extent. */ + uint32_t fe_flags; + + uint32_t fe_reserved[3]; +}; + +struct fiemap +{ + /* Logical offset(inclusive) at which to start mapping(in). */ + uint64_t fm_start; + + /* Logical length of mapping which userspace wants(in). */ + uint64_t fm_length; + + /* FIEMAP_FLAG_* flags for request(in/out). */ + uint32_t fm_flags; + + /* Number of extents that were mapped(out). */ + uint32_t fm_mapped_extents; + + /* Size of fm_extents array(in). */ + uint32_t fm_extent_count; + + uint32_t fm_reserved; + + /* Array of mapped extents(out). */ + struct fiemap_extent fm_extents[0]; +}; + +/* The maximum offset can be mapped for a file. */ +# define FIEMAP_MAX_OFFSET (~0ULL) + +/* Sync file data before map. */ +# define FIEMAP_FLAG_SYNC 0x00000001 + +/* Map extented attribute tree. */ +# define FIEMAP_FLAG_XATTR 0x00000002 + +# define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR) + +/* Last extent in file. */ +# define FIEMAP_EXTENT_LAST 0x00000001 + +/* Data location unknown. */ +# define FIEMAP_EXTENT_UNKNOWN 0x00000002 + +/* Location still pending, Sets EXTENT_UNKNOWN. */ +# define FIEMAP_EXTENT_DELALLOC 0x00000004 + +/* Data can not be read while fs is unmounted. */ +# define FIEMAP_EXTENT_ENCODED 0x00000008 + +/* Data is encrypted by fs. Sets EXTENT_NO_BYPASS. */ +# define FIEMAP_EXTENT_DATA_ENCRYPTED 0x00000080 + +/* Extent offsets may not be block aligned. */ +# define FIEMAP_EXTENT_NOT_ALIGNED 0x00000100 + +/* Data mixed with metadata. Sets EXTENT_NOT_ALIGNED. */ +# define FIEMAP_EXTENT_DATA_INLINE 0x00000200 + +/* Multiple files in block. Set EXTENT_NOT_ALIGNED. */ +# define FIEMAP_EXTENT_DATA_TAIL 0x00000400 + +/* Space allocated, but not data (i.e. zero). */ +# define FIEMAP_EXTENT_UNWRITTEN 0x00000800 + +/* File does not natively support extents. Result merged for efficiency. */ +# define FIEMAP_EXTENT_MERGED 0x00001000 + +/* Space shared with other files. */ +# define FIEMAP_EXTENT_SHARED 0x00002000 + +#endif -- 1.7.1.501.g23b46 >From dc98ac611dbb141275613fe30f5a8d2486817a3b Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Thu, 13 May 2010 22:17:53 +0800 Subject: [PATCH 02/10] tests: add a new test for FIEMAP-copy * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a loopbacked ext4 partition. * tests/Makefile.am (sparse-fiemap): Reference the new test. --- tests/Makefile.am | 1 + tests/cp/sparse-fiemap | 56 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+), 0 deletions(-) create mode 100755 tests/cp/sparse-fiemap diff --git a/tests/Makefile.am b/tests/Makefile.am index c458574..f7840c8 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -25,6 +25,7 @@ root_tests = \ cp/special-bits \ cp/cp-mv-enotsup-xattr \ cp/capability \ + cp/sparse-fiemap \ dd/skip-seek-past-dev \ install/install-C-root \ ls/capability \ diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap new file mode 100755 index 0000000..945c94b --- /dev/null +++ b/tests/cp/sparse-fiemap @@ -0,0 +1,56 @@ +#!/bin/sh +# Test cp --sparse=always through fiemap copy + +# Copyright (C) 2006-2010 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +if test "$VERBOSE" = yes; then + set -x + cp --version +fi + +. $srcdir/test-lib.sh +require_root_ + +cwd=`pwd` +cleanup_() { cd /; umount "$cwd/mnt"; } + +skip=0 +# Create an ext4 loopback file system +dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 +mkdir mnt +mkfs -t ext4 -F blob || + skip_test_ "failed to create ext4 file system" +mount -oloop blob mnt || skip=1 +echo test > mnt/f || skip=1 +test -s mnt/f || skip=1 + +test $skip = 1 && + skip_test_ "insufficient mount/ext4 support" + +# Create a 1TiB sparse file +dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure + +cd mnt || fail=1 + +# It takes many minutes to copy this sparse file using the old method. +# By contrast, it takes far less than 1 second using FIEMAP-copy. +timeout 10 cp --sparse=always sparse fiemap || fail=1 + +# Ensure that the sparse file copied through fiemap has the same size +# in bytes as the original. +test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 + +Exit $fail -- 1.7.1.501.g23b46 >From af8047bc22c93e72c38cfd9cd7e1fdd455ce81e0 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Fri, 28 May 2010 09:24:15 +0200 Subject: [PATCH 03/10] tests: sparse-fiemap: factor out some set-up * tests/cp/sparse-fiemap: Cd into test directory sooner. --- tests/cp/sparse-fiemap | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index 945c94b..21b02ac 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -33,9 +33,10 @@ dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 mkdir mnt mkfs -t ext4 -F blob || skip_test_ "failed to create ext4 file system" -mount -oloop blob mnt || skip=1 -echo test > mnt/f || skip=1 -test -s mnt/f || skip=1 +mount -oloop blob mnt || skip=1 +cd mnt || skip=1 +echo test > f || skip=1 +test -s f || skip=1 test $skip = 1 && skip_test_ "insufficient mount/ext4 support" @@ -43,7 +44,6 @@ test $skip = 1 && # Create a 1TiB sparse file dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure -cd mnt || fail=1 # It takes many minutes to copy this sparse file using the old method. # By contrast, it takes far less than 1 second using FIEMAP-copy. -- 1.7.1.501.g23b46 >From 0a6df6c17b2043515f7719491b48eb15d2812456 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Fri, 21 May 2010 18:28:42 +0200 Subject: [PATCH 04/10] tests: exercise more of the new FIEMAP copying code * tests/cp/sparse-fiemap: Ensure that a file with many extents (more than fit in copy.c's internal 4KiB buffer) is copied properly. --- tests/cp/sparse-fiemap | 38 ++++++++++++++++++++++++++++++++++++++ 1 files changed, 38 insertions(+), 0 deletions(-) diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index 21b02ac..3608db3 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -53,4 +53,42 @@ timeout 10 cp --sparse=always sparse fiemap || fail=1 # in bytes as the original. test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 +# ================================================= +# Ensure that we exercise the FIEMAP-copying code enough +# to provoke at least two iterations of the do...while loop +# in which it calls ioctl (fd, FS_IOC_FIEMAP,... +# This also verifies that non-trivial extents are preserved. + +$PERL -e 1 || skip_test_ 'skipping part of this test; you lack perl' + +$PERL -e 'BEGIN { $n = 16 * 1024; *F = *STDOUT }' \ + -e 'for (1..100) { sysseek (*F, $n, 1)' \ + -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1 + +cp --sparse=always j1 j2 || fail=1 +cmp j1 j2 || fail=1 + +filefrag j1 | grep extent \ + || skip_test_ 'skipping part of this test; you lack filefrag' + +# Here is sample filefrag output: +# $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \ +# -e 'for (1..5) { sysseek(*F,$n,1)' \ +# -e '&& syswrite *F,"."x$n or die "$!"}' > j +# $ filefrag -v j +# File system type is: ef53 +# File size of j is 163840 (40 blocks, blocksize 4096) +# ext logical physical expected length flags +# 0 4 6258884 4 +# 1 12 6258892 6258887 4 +# 2 20 6258900 6258895 4 +# 3 28 6258908 6258903 4 +# 4 36 6258916 6258911 4 eof +# j: 6 extents found + +# exclude the physical block numbers; they always differ +filefrag -v j1 | awk '/^ / {print $1,$2,$NF}' > ff1 || fail=1 +filefrag -v j2 | awk '/^ / {print $1,$2,$NF}' > ff2 || fail=1 +compare ff1 ff2 || fail=1 + Exit $fail -- 1.7.1.501.g23b46 >From b257d65c6d923567af5d4b757810dd62e1be91bb Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sat, 22 May 2010 10:22:58 +0200 Subject: [PATCH 05/10] tests: require root only if current partition is neither btrfs nor xfs * tests/cp/sparse-fiemap: Don't require root access if current partition is btrfs or xfs. --- tests/cp/sparse-fiemap | 43 ++++++++++++++++++++++++------------------- 1 files changed, 24 insertions(+), 19 deletions(-) diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index 3608db3..17ed4f9 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -22,28 +22,33 @@ if test "$VERBOSE" = yes; then fi . $srcdir/test-lib.sh -require_root_ -cwd=`pwd` -cleanup_() { cd /; umount "$cwd/mnt"; } - -skip=0 -# Create an ext4 loopback file system -dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1 -mkdir mnt -mkfs -t ext4 -F blob || - skip_test_ "failed to create ext4 file system" -mount -oloop blob mnt || skip=1 -cd mnt || skip=1 -echo test > f || skip=1 -test -s f || skip=1 - -test $skip = 1 && - skip_test_ "insufficient mount/ext4 support" +if df -T -t btrfs -t xfs . ; then + : # Current dir is on a partition with working extents. Good! +else + # It's not; we need to create one, hence we need root access. + require_root_ + + cwd=$PWD + cleanup_() { cd /; umount "$cwd/mnt"; } + + skip=0 + # Create an XFS loopback file system + dd if=/dev/zero of=blob bs=32k count=1000 || skip=1 + mkdir mnt + mkfs -t xfs blob || + skip_test_ "failed to create XFS file system" + mount -oloop blob mnt || skip=1 + cd mnt || skip=1 + echo test > f || skip=1 + test -s f || skip=1 + + test $skip = 1 && + skip_test_ "insufficient mount/XFS support" +fi # Create a 1TiB sparse file -dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure - +dd if=/dev/zero of=sparse bs=1k count=1 seek=1G || framework_failure # It takes many minutes to copy this sparse file using the old method. # By contrast, it takes far less than 1 second using FIEMAP-copy. -- 1.7.1.501.g23b46 >From 71b4c7df63795345781bc2012f79695fc6933c7d Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sat, 22 May 2010 10:21:46 +0200 Subject: [PATCH 06/10] tests: test fiemap-enabled cp more thoroughly * tests/cp/sparse-fiemap: More tests. --- tests/cp/sparse-fiemap | 61 +++++++++++++++++++++++++---------------------- 1 files changed, 32 insertions(+), 29 deletions(-) diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index 17ed4f9..6d0bd83 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -66,34 +66,37 @@ test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 $PERL -e 1 || skip_test_ 'skipping part of this test; you lack perl' -$PERL -e 'BEGIN { $n = 16 * 1024; *F = *STDOUT }' \ - -e 'for (1..100) { sysseek (*F, $n, 1)' \ - -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1 - -cp --sparse=always j1 j2 || fail=1 -cmp j1 j2 || fail=1 - -filefrag j1 | grep extent \ - || skip_test_ 'skipping part of this test; you lack filefrag' - -# Here is sample filefrag output: -# $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \ -# -e 'for (1..5) { sysseek(*F,$n,1)' \ -# -e '&& syswrite *F,"."x$n or die "$!"}' > j -# $ filefrag -v j -# File system type is: ef53 -# File size of j is 163840 (40 blocks, blocksize 4096) -# ext logical physical expected length flags -# 0 4 6258884 4 -# 1 12 6258892 6258887 4 -# 2 20 6258900 6258895 4 -# 3 28 6258908 6258903 4 -# 4 36 6258916 6258911 4 eof -# j: 6 extents found - -# exclude the physical block numbers; they always differ -filefrag -v j1 | awk '/^ / {print $1,$2,$NF}' > ff1 || fail=1 -filefrag -v j2 | awk '/^ / {print $1,$2,$NF}' > ff2 || fail=1 -compare ff1 ff2 || fail=1 +for i in $(seq 20); do + for j in 1 2 31 100; do + $PERL -e 'BEGIN { $n = '$i' * 1024; *F = *STDOUT }' \ + -e 'for (1..'$j') { sysseek (*F, $n, 1)' \ + -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1 + + cp --sparse=always j1 j2 || fail=1 + cmp j1 j2 || fail=1 + filefrag -v j1 | grep extent \ + || skip_test_ 'skipping part of this test; you lack filefrag' + + # Here is sample filefrag output: + # $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \ + # -e 'for (1..5) { sysseek(*F,$n,1)' \ + # -e '&& syswrite *F,"."x$n or die "$!"}' > j + # $ filefrag -v j + # File system type is: ef53 + # File size of j is 163840 (40 blocks, blocksize 4096) + # ext logical physical expected length flags + # 0 4 6258884 4 + # 1 12 6258892 6258887 4 + # 2 20 6258900 6258895 4 + # 3 28 6258908 6258903 4 + # 4 36 6258916 6258911 4 eof + # j: 6 extents found + + # exclude the physical block numbers; they always differ + filefrag -v j1 | awk '/^ / {print $1,$2}' > ff1 || fail=1 + filefrag -v j2 | awk '/^ / {print $1,$2}' > ff2 || fail=1 + compare ff1 ff2 || fail=1 + done +done Exit $fail -- 1.7.1.501.g23b46 >From f674f49bfd315143ffb4ab376e30546fd021f719 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sat, 29 May 2010 21:22:40 +0200 Subject: [PATCH 07/10] tests: relax the root-tests cross-check * cfg.mk (sc_root_tests): Allow spaces before "require_root_", now that tests/cp/sparse-fiemap has a conditional use. --- cfg.mk | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/cfg.mk b/cfg.mk index dff5de5..17267cf 100644 --- a/cfg.mk +++ b/cfg.mk @@ -80,7 +80,7 @@ sc_root_tests: @if test -d tests \ && grep check-root tests/Makefile.am>/dev/null 2>&1; then \ t1=sc-root.expected; t2=sc-root.actual; \ - grep -nl '^require_root_$$' \ + grep -nl '^ *require_root_$$' \ $$($(VC_LIST) tests) |sed s,tests/,, |sort > $$t1; \ sed -n '/^root_tests =[ ]*\\$$/,/[^\]$$/p' \ $(srcdir)/tests/Makefile.am \ -- 1.7.1.501.g23b46 >From 034c56d9a9a408ce336d039dba072c4ec97d7d04 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sun, 30 May 2010 21:20:30 +0200 Subject: [PATCH 08/10] tests: improve fiemap test to work with 4 FS types; fall back on ext4 * tests/cp/sparse-fiemap: Improve. * tests/filefrag-extent-compare: New file. --- tests/cp/sparse-fiemap | 38 ++++++++++++++++------ tests/filefrag-extent-compare | 68 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 11 deletions(-) create mode 100644 tests/filefrag-extent-compare diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index 6d0bd83..3e7c11f 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -23,7 +23,7 @@ fi . $srcdir/test-lib.sh -if df -T -t btrfs -t xfs . ; then +if df -T -t btrfs -t xfs -t ext4 -t ocfs2 . ; then : # Current dir is on a partition with working extents. Good! else # It's not; we need to create one, hence we need root access. @@ -33,18 +33,18 @@ else cleanup_() { cd /; umount "$cwd/mnt"; } skip=0 - # Create an XFS loopback file system + # Create an ext4 loopback file system dd if=/dev/zero of=blob bs=32k count=1000 || skip=1 mkdir mnt - mkfs -t xfs blob || - skip_test_ "failed to create XFS file system" + mkfs -t ext4 -F blob || + skip_test_ "failed to create ext4 file system" mount -oloop blob mnt || skip=1 cd mnt || skip=1 echo test > f || skip=1 test -s f || skip=1 test $skip = 1 && - skip_test_ "insufficient mount/XFS support" + skip_test_ "insufficient mount/ext4 support" fi # Create a 1TiB sparse file @@ -66,13 +66,26 @@ test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 $PERL -e 1 || skip_test_ 'skipping part of this test; you lack perl' -for i in $(seq 20); do +# Extract logical block number and length pairs from filefrag -v output. +# The initial sed is to remove the "eof" from the normally-empty "flags" field. +# That is required when that final extent has no number in the "expected" field. +f() +{ + sed 's/ eof$//' $@ \ + | awk '/^ *[0-9]/ {printf "%d %d ", $2 ,NF < 5 ? $NF : $5 } END {print ""}' +} + +for i in $(seq 1 2 21); do for j in 1 2 31 100; do $PERL -e 'BEGIN { $n = '$i' * 1024; *F = *STDOUT }' \ -e 'for (1..'$j') { sysseek (*F, $n, 1)' \ - -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1 - + -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1 || fail=1 + # sync cp --sparse=always j1 j2 || fail=1 + # sync + # Technically we may need the 'sync' uses above, but + # uncommenting them makes this test take much longer. + cmp j1 j2 || fail=1 filefrag -v j1 | grep extent \ || skip_test_ 'skipping part of this test; you lack filefrag' @@ -93,10 +106,13 @@ for i in $(seq 20); do # j: 6 extents found # exclude the physical block numbers; they always differ - filefrag -v j1 | awk '/^ / {print $1,$2}' > ff1 || fail=1 - filefrag -v j2 | awk '/^ / {print $1,$2}' > ff2 || fail=1 - compare ff1 ff2 || fail=1 + filefrag -v j1 > ff1 || fail=1 + filefrag -v j2 > ff2 || fail=1 + { f ff1; f ff2; } \ + | $PERL $abs_top_srcdir/tests/filefrag-extent-compare \ + || { fail=1; break; } done + test $fail = 1 && break done Exit $fail diff --git a/tests/filefrag-extent-compare b/tests/filefrag-extent-compare new file mode 100644 index 0000000..3c095d5 --- /dev/null +++ b/tests/filefrag-extent-compare @@ -0,0 +1,68 @@ +eval '(exit $?0)' && eval 'exec perl -wS "$0" ${1+"$@"}' + & eval 'exec perl -wS "$0" $argv:q' + if 0; +# Determine whether two files have the same extents by comparing +# the logical block numbers and lengths from filefrag -v for each. + +# Invoke like this: +# This helper function, f, extracts logical block number and lengths. +# f() { awk '/^ *[0-9]/ {printf "%d %d ",$2,NF<5?$NF:$5} END {print ""}'; } +# { filefrag -v j1 | f; filefrag -v j2 | f; } | ./filefrag-extent-compare + +use warnings; +use strict; +(my $ME = $0) =~ s|.*/||; + +my @line = <>; +my $n_lines = @line; +$n_lines == 2 + or die "$ME: expected exactly two input lines; got $n_lines\n"; + +my @A = split ' ', $line[0]; +my @B = split ' ', $line[1]; +@A % 2 || @B % 2 + and die "$ME: unexpected input: odd number of numbers; expected even\n"; + +my @a; +my @b; +foreach my $i (0..@A/2-1) { $a[$i] = { L_BLK => $A[2*$i], LEN => $A[2*$i+1] } }; +foreach my $i (0..@B/2-1) { $b[$i] = { L_BLK => $B[2*$i], LEN => $B[2*$i+1] } }; + +my $i = 0; +my $j = 0; +while (1) + { + !defined $a[$i] && !defined $b[$j] + and exit 0; + defined $a[$i] && defined $b[$j] + or die "\@a and \@b have different lengths, even after adjustment\n"; + ($a[$i]->{L_BLK} == $b[$j]->{L_BLK} + && $a[$i]->{LEN} == $b[$j]->{LEN}) + and next; + ($a[$i]->{LEN} < $b[$j]->{LEN} + && exists $a[$i+1] && $a[$i]->{LEN} + $a[$i+1]->{LEN} == $b[$j]->{LEN}) + and ++$i, next; + exists $b[$j+1] && $a[$i]->{LEN} == $b[$i]->{LEN} + $b[$i+1]->{LEN} + and ++$j, next; + die "differing extent:\n" + . " [$i]=$a[$i]->{L_BLK} $a[$i]->{LEN}\n" + . " [$j]=$b[$j]->{L_BLK} $b[$j]->{LEN}\n" + } +continue + { + ++$i; + ++$j; + } + +### Setup "GNU" style for perl-mode and cperl-mode. +## Local Variables: +## mode: perl +## perl-indent-level: 2 +## perl-continued-statement-offset: 2 +## perl-continued-brace-offset: 0 +## perl-brace-offset: 0 +## perl-brace-imaginary-offset: 0 +## perl-label-offset: -2 +## perl-extra-newline-before-brace: t +## perl-merge-trailing-else: nil +## End: -- 1.7.1.501.g23b46 >From a7a188a4fd4f58404b825f463f3670193232a037 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sat, 5 Jun 2010 10:17:48 +0200 Subject: [PATCH 09/10] cleanup * src/copy.c (fiemap_copy): Rename from fiemap_copy_ok. Add/improve comments. Remove local, "fail". --- src/copy.c | 42 +++++++++++++++++++++--------------------- 1 files changed, 21 insertions(+), 21 deletions(-) diff --git a/src/copy.c b/src/copy.c index 2c15ca0..6a4631a 100644 --- a/src/copy.c +++ b/src/copy.c @@ -157,17 +157,17 @@ clone_file (int dest_fd, int src_fd) # ifndef FS_IOC_FIEMAP # define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) # endif -/* Perform FIEMAP(available in mainline 2.6.27) copy if possible. - Call ioctl(2) with FS_IOC_FIEMAP to efficiently map file allocation - excepts holes. So the overhead to deal with holes with lseek(2) in - normal copy could be saved. This would result in much faster backups - for any kind of sparse file. */ +/* Perform a FIEMAP copy, if possible. + Call ioctl(2) with FS_IOC_FIEMAP (available in linux 2.6.27) to + obtain a map of file extents excluding holes. This avoids the + overhead of detecting holes in a hole-introducing/preserving copy, + and thus makes copying sparse files much more efficient. + Upon a successful copy, return true. Otherwise, return false. */ static bool -fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, - off_t src_total_size, char const *src_name, - char const *dst_name, bool *normal_copy_required) +fiemap_copy (int src_fd, int dest_fd, size_t buf_size, + off_t src_total_size, char const *src_name, + char const *dst_name, bool *normal_copy_required) { - bool fail = false; bool last = false; char fiemap_buf[4096]; struct fiemap *fiemap = (struct fiemap *)fiemap_buf; @@ -180,7 +180,7 @@ fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, unsigned int i = 0; /* This is required at least to initialize fiemap->fm_start, - but also serves (in May 2010) to appease valgrind, which + but also serves (in mid 2010) to appease valgrind, which appears not to know the semantics of the FIEMAP ioctl. */ memset (fiemap_buf, 0, sizeof fiemap_buf); @@ -213,13 +213,13 @@ fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) { error (0, errno, _("cannot lseek %s"), quote (src_name)); - return fail; + return false; } if (lseek (dest_fd, ext_logical, SEEK_SET) < 0LL) { error (0, errno, _("cannot lseek %s"), quote (dst_name)); - return fail; + return false; } if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST) @@ -246,7 +246,7 @@ fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, continue; #endif error (0, errno, _("reading %s"), quote (src_name)); - return fail; + return false; } if (n_read == 0) @@ -259,7 +259,7 @@ fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, if (full_write (dest_fd, buf, n_read) != n_read) { error (0, errno, _("writing %s"), quote (dst_name)); - return fail; + return false; } ext_len -= n_read; @@ -277,15 +277,15 @@ fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size, { if (ftruncate (dest_fd, src_total_size) < 0) { - error (0, errno, _("extending %s"), quote (dst_name)); - return fail; + error (0, errno, _("failed to extend %s"), quote (dst_name)); + return false; } } - return ! fail; + return true; } #else -static bool fiemap_copy_ok (ignored) { errno == ENOTSUP; return false; } +static bool fiemap_copy (ignored) { errno == ENOTSUP; return false; } #endif /* FIXME: describe */ @@ -823,9 +823,9 @@ copy_reg (char const *src_name, char const *dst_name, bool require_normal_copy = false; /* Perform efficient FIEMAP copy for sparse files, fall back to the standard copy only if the ioctl(2) fails. */ - if (fiemap_copy_ok (source_desc, dest_desc, buf_size, - src_open_sb.st_size, src_name, - dst_name, &require_normal_copy)) + if (fiemap_copy (source_desc, dest_desc, buf_size, + src_open_sb.st_size, src_name, + dst_name, &require_normal_copy)) goto preserve_metadata; else { -- 1.7.1.501.g23b46 >From 7d10de7a2ba4161bc2b910f30ff4841f8c91512d Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sat, 5 Jun 2010 10:56:36 +0200 Subject: [PATCH 10/10] adjust normal_copy_required semantics * src/copy.c (fiemap_copy): Do not require caller to set "normal_copy_required" before calling fiemap_copy. --- src/copy.c | 18 +++++++++++------- 1 files changed, 11 insertions(+), 7 deletions(-) diff --git a/src/copy.c b/src/copy.c index 6a4631a..eb67700 100644 --- a/src/copy.c +++ b/src/copy.c @@ -161,8 +161,10 @@ clone_file (int dest_fd, int src_fd) Call ioctl(2) with FS_IOC_FIEMAP (available in linux 2.6.27) to obtain a map of file extents excluding holes. This avoids the overhead of detecting holes in a hole-introducing/preserving copy, - and thus makes copying sparse files much more efficient. - Upon a successful copy, return true. Otherwise, return false. */ + and thus makes copying sparse files much more efficient. Upon a + successful copy, return true. If the initial ioctl fails, set + *NORMAL_COPY_REQUIRED to true and return false. Upon any other + failure, set *NORMAL_COPY_REQUIRED to false and return false. */ static bool fiemap_copy (int src_fd, int dest_fd, size_t buf_size, off_t src_total_size, char const *src_name, @@ -170,14 +172,15 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, { bool last = false; char fiemap_buf[4096]; - struct fiemap *fiemap = (struct fiemap *)fiemap_buf; + struct fiemap *fiemap = (struct fiemap *) fiemap_buf; struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; - uint32_t count = (sizeof (fiemap_buf) - sizeof (*fiemap)) / - sizeof (struct fiemap_extent); + uint32_t count = ((sizeof fiemap_buf - sizeof (*fiemap)) + / sizeof (struct fiemap_extent)); off_t last_ext_logical = 0; uint64_t last_ext_len = 0; uint64_t last_read_size = 0; unsigned int i = 0; + *normal_copy_required = false; /* This is required at least to initialize fiemap->fm_start, but also serves (in mid 2010) to appease valgrind, which @@ -193,7 +196,8 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, is the first time we met. */ if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) { - /* If `i > 0', then at least one ioctl(2) has been performed before. */ + /* If the first ioctl fails, tell the caller that it is + ok to proceed with a normal copy. */ if (i == 0) *normal_copy_required = true; return false; @@ -820,7 +824,7 @@ copy_reg (char const *src_name, char const *dst_name, if (make_holes) { - bool require_normal_copy = false; + bool require_normal_copy; /* Perform efficient FIEMAP copy for sparse files, fall back to the standard copy only if the ioctl(2) fails. */ if (fiemap_copy (source_desc, dest_desc, buf_size, -- 1.7.1.501.g23b46 From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 08 08:11:53 2010 Received: (at submit) by debbugs.gnu.org; 8 Jun 2010 12:11:53 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OLxeq-0003kK-SQ for submit@debbugs.gnu.org; Tue, 08 Jun 2010 08:11:53 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OLxeo-0003kF-4X for submit@debbugs.gnu.org; Tue, 08 Jun 2010 08:11:51 -0400 Received: from lists.gnu.org ([199.232.76.165]:49175) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OLxei-0004oF-PJ for submit@debbugs.gnu.org; Tue, 08 Jun 2010 08:11:44 -0400 Received: from [140.186.70.92] (port=41363 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OLxeY-0002Fl-Ui for bug-coreutils@gnu.org; Tue, 08 Jun 2010 08:11:44 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OLxeL-0007BD-97 for bug-coreutils@gnu.org; Tue, 08 Jun 2010 08:11:34 -0400 Received: from mx1.redhat.com ([209.132.183.28]:39999) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OLxeL-0007Az-0a for bug-coreutils@gnu.org; Tue, 08 Jun 2010 08:11:21 -0400 Received: from int-mx05.intmail.prod.int.phx2.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.18]) by mx1.redhat.com (8.13.8/8.13.8) with ESMTP id o58CBJo3011109 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 8 Jun 2010 08:11:19 -0400 Received: from [10.11.11.241] (vpn-11-241.rdu.redhat.com [10.11.11.241]) by int-mx05.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id o58CBI6C016753; Tue, 8 Jun 2010 08:11:18 -0400 Message-ID: <4C0E333C.1010605@redhat.com> Date: Tue, 08 Jun 2010 06:10:36 -0600 From: Eric Blake Organization: Red Hat User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.9) Gecko/20100430 Fedora/3.0.4-3.fc13 Lightning/1.0b2pre Mnenhy/0.8.2 Thunderbird/3.0.4 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> In-Reply-To: <874ohdu5b8.fsf@meyering.net> X-Enigmail-Version: 1.0.1 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig0B029EEB3A5E718E4E9F9F27" X-Scanned-By: MIMEDefang 2.67 on 10.5.11.18 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -7.8 (-------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -7.8 (-------) This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig0B029EEB3A5E718E4E9F9F27 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 06/08/2010 05:33 AM, Jim Meyering wrote: > new file mode 100644 > index 0000000..d33293b > --- /dev/null > +++ b/src/fiemap.h > @@ -0,0 +1,102 @@ > +/* FS_IOC_FIEMAP ioctl infrastructure. > + Some portions copyright (C) 2007 Cluster File Systems, Inc > + Authors: Mark Fasheh > + Kalpak Shah > + Andreas Dilger . */ Shouldn't we also add an FSF Copyright 2010 to this new file, to cover our changes? > +++ b/tests/cp/sparse-fiemap > @@ -0,0 +1,56 @@ > +#!/bin/sh > +# Test cp --sparse=3Dalways through fiemap copy > + > +# Copyright (C) 2006-2010 Free Software Foundation, Inc. How much of this content comes from other files from 2006, vs. new content needing only 2010? --=20 Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org --------------enig0B029EEB3A5E718E4E9F9F27 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (GNU/Linux) Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ iQEcBAEBCAAGBQJMDjM8AAoJEKeha0olJ0NqxG4IAIdZr5SGNrl2fbJKFWMYf0i7 KzUp6d4MkcSZpUBDnnBCA+N3p69skCA8iPdIvhESvVpxkKAEEWLt+BnZhgHFiiHN tH6tztdy8/0oTxMBFia4kh2YVl1RcRRmtGwdPSHSHJsckAvIUgdW7ZEBKWOHW4Tj 4Iy8wNlMDQm+haTYWbB9eYv8wEPTWMWvq+Dv5IlZbCuCxZfjM9SsTAERFk/WH3+W F9aF+I7g0nMDgs9uxdbSNnatcYg2tE5yU+GnLas4mZyFqFlw0qZk1lEf5yfJbhL+ mCM9GKdwgBOGhpog5zx7PTjtVFTiXFwC2TAwj361WuXoLqSVxKnqFolD5Xmz3b0= =tEW/ -----END PGP SIGNATURE----- --------------enig0B029EEB3A5E718E4E9F9F27-- From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 08 12:13:11 2010 Received: (at submit) by debbugs.gnu.org; 8 Jun 2010 16:13:11 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OM1QM-0005W6-Jf for submit@debbugs.gnu.org; Tue, 08 Jun 2010 12:13:11 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OM1QJ-0005Vh-0H for submit@debbugs.gnu.org; Tue, 08 Jun 2010 12:13:08 -0400 Received: from lists.gnu.org ([199.232.76.165]:49494) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OM1QC-0001o9-VF for submit@debbugs.gnu.org; Tue, 08 Jun 2010 12:13:01 -0400 Received: from [140.186.70.92] (port=47543 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OM1Q8-0004AL-PI for bug-coreutils@gnu.org; Tue, 08 Jun 2010 12:13:00 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OM1Q4-0003ug-5j for bug-coreutils@gnu.org; Tue, 08 Jun 2010 12:12:55 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:55775) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OM1Q3-0003uJ-VE for bug-coreutils@gnu.org; Tue, 08 Jun 2010 12:12:52 -0400 Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o58GCkL2032052 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 8 Jun 2010 16:12:47 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o58EgUSI007504; Tue, 8 Jun 2010 16:12:45 GMT Received: from abhmt012.oracle.com by acsmt354.oracle.com with ESMTP id 330273711276013510; Tue, 08 Jun 2010 09:11:50 -0700 Received: from [130.35.68.118] (/130.35.68.118) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 08 Jun 2010 09:11:48 -0700 Message-ID: <4C0E6BC2.2040407@oracle.com> Date: Tue, 08 Jun 2010 09:11:46 -0700 From: Sunil Mushran User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: Eric Blake Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <4C0E333C.1010605@redhat.com> In-Reply-To: <4C0E333C.1010605@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet15.oracle.com [148.87.113.117] X-CT-RefId: str=0001.0A090209.4C0E6C00.0093:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.6 (------) X-Debbugs-Envelope-To: submit Cc: bug-coreutils@gnu.org, Jim Meyering , Joel Becker , "jeff.liu" , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) On 06/08/2010 05:10 AM, Eric Blake wrote: >> new file mode 100644 >> index 0000000..d33293b >> --- /dev/null >> +++ b/src/fiemap.h >> @@ -0,0 +1,102 @@ >> +/* FS_IOC_FIEMAP ioctl infrastructure. >> + Some portions copyright (C) 2007 Cluster File Systems, Inc >> + Authors: Mark Fasheh >> + Kalpak Shah >> + Andreas Dilger. */ >> > Shouldn't we also add an FSF Copyright 2010 to this new file, to cover > our changes? > +/* Copy from kernel, modified to respect GNU code style by Jie Liu. */ The difference is only in style. From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 08 16:33:06 2010 Received: (at submit) by debbugs.gnu.org; 8 Jun 2010 20:33:07 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OM5Tu-0007Vi-IP for submit@debbugs.gnu.org; Tue, 08 Jun 2010 16:33:06 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OM5To-0007VG-1L for submit@debbugs.gnu.org; Tue, 08 Jun 2010 16:33:05 -0400 Received: from lists.gnu.org ([199.232.76.165]:42195) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OM5Tc-0007IQ-4G for submit@debbugs.gnu.org; Tue, 08 Jun 2010 16:32:48 -0400 Received: from [140.186.70.92] (port=51064 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OM5TU-0002F1-Tm for bug-coreutils@gnu.org; Tue, 08 Jun 2010 16:32:47 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OM5TH-0005KJ-IL for bug-coreutils@gnu.org; Tue, 08 Jun 2010 16:32:32 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:57953) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OM5TG-0005Jx-SU for bug-coreutils@gnu.org; Tue, 08 Jun 2010 16:32:27 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id CE155940182; Tue, 8 Jun 2010 22:32:19 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 6CD15BC4; Tue, 8 Jun 2010 22:32:18 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <874ohdu5b8.fsf@meyering.net> (Jim Meyering's message of "Tue, 08 Jun 2010 13:33:47 +0200") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> Date: Tue, 08 Jun 2010 22:32:18 +0200 Message-ID: <87eighs1t9.fsf@meyering.net> Lines: 89 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Chris Mason , bug-coreutils@gnu.org, Joel Becker , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Jim Meyering wrote: > Subject: [PATCH 01/10] cp: Add FIEMAP support for efficient sparse file copy FYI, using those patches, I ran a test for the first time in a few days: check -C tests TESTS=cp/sparse-fiemap VERBOSE=yes It failed like this on an ext4 partition using F13: + timeout 10 cp --sparse=always sparse fiemap + fail=1 ++ stat --printf %s sparse ++ stat --printf %s fiemap + test 1099511628800 = 0 + fail=1 That is very odd. No diagnostic from cp, yet it failed after creating a zero-length file. Here's the corresponding piece of the script: # It takes many minutes to copy this sparse file using the old method. # By contrast, it takes far less than 1 second using FIEMAP-copy. timeout 10 cp --sparse=always sparse fiemap || fail=1 # Ensure that the sparse file copied through fiemap has the same size # in bytes as the original. test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 However, so far I've been unable to reproduce the failure, running hundreds of iterations: for i in $(seq 300); do printf .; make check -C tests \ TESTS=cp/sparse-fiemap VERBOSE=yes >& makerr-$i || break; done Have any of you heard of a problem whereby a cold cache can cause such a thing? "echo 3 > /proc/sys/vm/drop_caches" didn't help. I suspect that having so many extents is unusual, so maybe this is a rarely exercised corner case. =============================== As I wrote the above, I realized I probably had enough information to deduce where things were going wrong, even if so far I've been unable to reproduce it. And sure enough. There is a way to provoke exactly that failure. If the *second* (or later) FIEMAP ioctl fails: do { fiemap->fm_length = FIEMAP_MAX_OFFSET; fiemap->fm_extent_count = count; /* When ioctl(2) fails, fall back to the normal copy only if it is the first time we met. */ if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) { /* If the first ioctl fails, tell the caller that it is ok to proceed with a normal copy. */ if (i == 0) *normal_copy_required = true; return false; } In that case, fiemap_copy returns false (with no diagnostic) and cp fails silently. Obviously I will now add code to diagnose the failure, but do any of you know off hand how to reproduce this or what the failure might have been? Here's the patch I plan to merge: diff --git a/src/copy.c b/src/copy.c index eb67700..07d605e 100644 --- a/src/copy.c +++ b/src/copy.c @@ -200,6 +200,12 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, ok to proceed with a normal copy. */ if (i == 0) *normal_copy_required = true; + else + { + /* If the second or subsequent ioctl fails, diagnose it, + since it ends up causing the entire copy/cp to fail. */ + error (0, errno, _("%s: FIEMAP ioctl failed"), quote (src_name)); + } return false; } From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 08 17:09:06 2010 Received: (at submit) by debbugs.gnu.org; 8 Jun 2010 21:09:07 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OM62k-0007mZ-EV for submit@debbugs.gnu.org; Tue, 08 Jun 2010 17:09:06 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OM622-0007m8-V6 for submit@debbugs.gnu.org; Tue, 08 Jun 2010 17:09:05 -0400 Received: from lists.gnu.org ([199.232.76.165]:45137) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OM61q-0007by-Ba for submit@debbugs.gnu.org; Tue, 08 Jun 2010 17:08:10 -0400 Received: from [140.186.70.92] (port=49308 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OM61j-0004Ph-LK for bug-coreutils@gnu.org; Tue, 08 Jun 2010 17:08:09 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OM5ry-0001Zj-Pd for bug-coreutils@gnu.org; Tue, 08 Jun 2010 16:58:01 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:51230) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OM5ry-0001ZO-6c for bug-coreutils@gnu.org; Tue, 08 Jun 2010 16:57:58 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id 3B690940166; Tue, 8 Jun 2010 22:57:52 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 110CEDD5; Tue, 8 Jun 2010 22:57:51 +0200 (CEST) From: Jim Meyering To: Eric Blake Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4C0E333C.1010605@redhat.com> (Eric Blake's message of "Tue, 08 Jun 2010 06:10:36 -0600") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <4C0E333C.1010605@redhat.com> Date: Tue, 08 Jun 2010 22:57:51 +0200 Message-ID: <8739wxs0mo.fsf@meyering.net> Lines: 51 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Eric Blake wrote: >> +++ b/tests/cp/sparse-fiemap >> @@ -0,0 +1,56 @@ >> +#!/bin/sh >> +# Test cp --sparse=always through fiemap copy >> + >> +# Copyright (C) 2006-2010 Free Software Foundation, Inc. > > How much of this content comes from other files from 2006, vs. new > content needing only 2010? Good point. This started as a largely copied test, but now it's mostly new. I'm taking this opportunity to make it use init.sh, too. I'll squash this. Think of init.sh as the next-generation test-lib.sh. Thanks. >From c7f9d3d0ff23d72cadd435ceef8d44b7eab7f072 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Tue, 8 Jun 2010 22:53:51 +0200 Subject: [PATCH] test-use init.sh --- tests/cp/sparse-fiemap | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index 3e7c11f..dc0cf60 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -1,7 +1,7 @@ #!/bin/sh # Test cp --sparse=always through fiemap copy -# Copyright (C) 2006-2010 Free Software Foundation, Inc. +# Copyright (C) 2010 Free Software Foundation, Inc. # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -21,7 +21,7 @@ if test "$VERBOSE" = yes; then cp --version fi -. $srcdir/test-lib.sh +. "${srcdir=.}/init.sh"; path_prepend_ ../src if df -T -t btrfs -t xfs -t ext4 -t ocfs2 . ; then : # Current dir is on a partition with working extents. Good! -- 1.7.1.501.g23b46 From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 08 20:44:37 2010 Received: (at submit) by debbugs.gnu.org; 9 Jun 2010 00:44:38 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OM9PJ-0001Mn-5n for submit@debbugs.gnu.org; Tue, 08 Jun 2010 20:44:37 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OM9PH-0001Mi-CI for submit@debbugs.gnu.org; Tue, 08 Jun 2010 20:44:36 -0400 Received: from lists.gnu.org ([199.232.76.165]:33149) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OM9PD-00033E-JG for submit@debbugs.gnu.org; Tue, 08 Jun 2010 20:44:31 -0400 Received: from [140.186.70.92] (port=33655 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OM9PC-0005Yz-8R for bug-coreutils@gnu.org; Tue, 08 Jun 2010 20:44:31 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OM9PB-0004ci-66 for bug-coreutils@gnu.org; Tue, 08 Jun 2010 20:44:30 -0400 Received: from kiwi.cs.ucla.edu ([131.179.128.19]:41771) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OM9PA-0004cV-NT for bug-coreutils@gnu.org; Tue, 08 Jun 2010 20:44:29 -0400 Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200]) by kiwi.cs.ucla.edu (8.13.8+Sun/8.13.8/UCLACS-6.0) with ESMTP id o590iPZp028646; Tue, 8 Jun 2010 17:44:25 -0700 (PDT) Message-ID: <4C0EE3E9.8010802@cs.ucla.edu> Date: Tue, 08 Jun 2010 17:44:25 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> In-Reply-To: <874ohdu5b8.fsf@meyering.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.1 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.2 (----) Jeff Liu and Jim Meyering wrote: > diff --git a/src/fiemap.h b/src/fiemap.h > new file mode 100644 > index 0000000..d33293b > --- /dev/null > +++ b/src/fiemap.h Why is this file necessary? fiemap.h is included only if it exists, right? Shouldn't we just use the kernel's fiemap.h rather than copying it here and assuming kernel internals? > if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) For this sort of thing, please just use "0" rather than "0LL". "0" is easier to read and it has the same effect here. > char buf[buf_size]; This assumes C99, since buf_size is not known at compile time. Also, isn't there a potential segmentation-violation problem if buf_size is sufficiently large? More generally, since the caller is already allocating a buffer of the appropiate size, shouldn't we just reuse that buffer, rather than allocating a new one? That would avoid the problems of assuming C99 and possible segmentation violations. > char fiemap_buf[4096]; > struct fiemap *fiemap = (struct fiemap *) fiemap_buf; > struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; > uint32_t count = ((sizeof fiemap_buf - sizeof (*fiemap)) > / sizeof (struct fiemap_extent)); This code isn't portable, since fiemap_buf is only char-aligned, and struct fiemap may well require stricter alignment. The code will work on the x86 despite the alignment problem, but that's just a happy coincidence. A lesser point: the code assumes that 'struct fiemap' is sufficiently small (considerably less than 4096 bytes in size); I expect that this is universally true but we might as well check this assumption, since it's easy to do so without any runtime overhead. So I propose something like this instead: union { struct fiemap f; char c[4096]; } fiemap_buf; struct fiemap *fiemap = &fiemap_buf.f; struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; verify (count != 0); From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 09 02:45:03 2010 Received: (at submit) by debbugs.gnu.org; 9 Jun 2010 06:45:03 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMF26-0004JP-N7 for submit@debbugs.gnu.org; Wed, 09 Jun 2010 02:45:03 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMF24-0004Iw-PY for submit@debbugs.gnu.org; Wed, 09 Jun 2010 02:45:01 -0400 Received: from lists.gnu.org ([199.232.76.165]:34278) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OMF21-0007Il-9i for submit@debbugs.gnu.org; Wed, 09 Jun 2010 02:44:57 -0400 Received: from [140.186.70.92] (port=39573 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OMF1y-0002XC-IB for bug-coreutils@gnu.org; Wed, 09 Jun 2010 02:44:56 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OMElN-0008OF-SS for bug-coreutils@gnu.org; Wed, 09 Jun 2010 02:27:47 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:32800) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMElN-0008NN-4k for bug-coreutils@gnu.org; Wed, 09 Jun 2010 02:27:45 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id A72BA940099; Wed, 9 Jun 2010 08:27:37 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id C40DADD44; Wed, 9 Jun 2010 08:27:35 +0200 (CEST) From: Jim Meyering To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4C0EE3E9.8010802@cs.ucla.edu> (Paul Eggert's message of "Tue, 08 Jun 2010 17:44:25 -0700") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <4C0EE3E9.8010802@cs.ucla.edu> Date: Wed, 09 Jun 2010 08:27:35 +0200 Message-ID: <87typcra94.fsf@meyering.net> Lines: 171 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Paul Eggert wrote: > Jeff Liu and Jim Meyering wrote: >> diff --git a/src/fiemap.h b/src/fiemap.h >> new file mode 100644 >> index 0000000..d33293b >> --- /dev/null >> +++ b/src/fiemap.h Hi Paul, Thanks for the review. > Why is this file necessary? fiemap.h is included only if it exists, > right? Shouldn't we just use the kernel's fiemap.h rather than > copying it here and assuming kernel internals? The ioctl is available/usable in 2.6.27 and newer that do not publish this file. For example, it's in F13's (2.6.33's) /usr/include/linux/fiemap.h, as well as the one in debian unstable's 2.6.32, but probably not in much older kernels. Hmm.. I see that it's available even in F11's 2.6.30.9-x It would be better to include when present, else our copy of that header. Then, eventually, the else clause will become unused. Note that even on newer kernels, the linux/* headers are optional. Eventually we'll have a hard requirement on kernel headers -- at least when building against a linux kernel. >> if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) > > For this sort of thing, please just use "0" rather than "0LL". > "0" is easier to read and it has the same effect here. Included in the patch below. >> char buf[buf_size]; > > This assumes C99, since buf_size is not known at compile time. > Also, isn't there a potential segmentation-violation problem > if buf_size is sufficiently large? > > More generally, since the caller is already allocating a buffer of the > appropiate size, shouldn't we just reuse that buffer, rather than > allocating a new one? That would avoid the problems of assuming > C99 and possible segmentation violations. Good point. Thanks. We can definitely avoid that allocation. Do you feel like writing the patch? I've just pushed this series to a branch, "fiemap-copy", so others can follow along and contribute more easily. >> char fiemap_buf[4096]; >> struct fiemap *fiemap = (struct fiemap *) fiemap_buf; >> struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; >> uint32_t count = ((sizeof fiemap_buf - sizeof (*fiemap)) >> / sizeof (struct fiemap_extent)); > > This code isn't portable, since fiemap_buf is only char-aligned, and > struct fiemap may well require stricter alignment. The code will work > on the x86 despite the alignment problem, but that's just a happy > coincidence. > > A lesser point: the code assumes that 'struct fiemap' is sufficiently > small (considerably less than 4096 bytes in size); I expect that this > is universally true but we might as well check this assumption, since > it's easy to do so without any runtime overhead. > > So I propose something like this instead: > > union { struct fiemap f; char c[4096]; } fiemap_buf; > struct fiemap *fiemap = &fiemap_buf.f; > struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; > enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; > verify (count != 0); I've done this in your name: >From fffa2e8661a27978927fcc8afb6873631a753292 Mon Sep 17 00:00:00 2001 From: Paul Eggert Date: Wed, 9 Jun 2010 08:15:07 +0200 Subject: [PATCH] copy.c: ensure proper alignment of fiemap buffer * src/copy.c (fiemap_copy): Ensure that our fiemap buffer is large enough and well-aligned. Replace "0LL" with equivalent "0" as 3rd argument to lseek. --- src/copy.c | 15 ++++++++------- 1 files changed, 8 insertions(+), 7 deletions(-) diff --git a/src/copy.c b/src/copy.c index f629771..27e083a 100644 --- a/src/copy.c +++ b/src/copy.c @@ -171,11 +171,12 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, char const *dst_name, bool *normal_copy_required) { bool last = false; - char fiemap_buf[4096]; - struct fiemap *fiemap = (struct fiemap *) fiemap_buf; + union { struct fiemap f; char c[4096]; } fiemap_buf; + struct fiemap *fiemap = &fiemap_buf.f; struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; - uint32_t count = ((sizeof fiemap_buf - sizeof (*fiemap)) - / sizeof (struct fiemap_extent)); + enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; + verify (count != 0); + off_t last_ext_logical = 0; uint64_t last_ext_len = 0; uint64_t last_read_size = 0; @@ -185,7 +186,7 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, /* This is required at least to initialize fiemap->fm_start, but also serves (in mid 2010) to appease valgrind, which appears not to know the semantics of the FIEMAP ioctl. */ - memset (fiemap_buf, 0, sizeof fiemap_buf); + memset (&fiemap_buf, 0, sizeof fiemap_buf); do { @@ -220,13 +221,13 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, off_t ext_logical = fm_ext[i].fe_logical; uint64_t ext_len = fm_ext[i].fe_length; - if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) + if (lseek (src_fd, ext_logical, SEEK_SET) < 0) { error (0, errno, _("cannot lseek %s"), quote (src_name)); return false; } - if (lseek (dest_fd, ext_logical, SEEK_SET) < 0LL) + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0) { error (0, errno, _("cannot lseek %s"), quote (dst_name)); return false; -- 1.7.1.501.g23b46 Also, I've squashed this clean-up patch onto Jeff's original, since ext_len is unsigned (of type uint64_t). >From bad13e737c683757a2ed05404564d8863c5da30e Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Wed, 9 Jun 2010 08:24:39 +0200 Subject: [PATCH] remove 0 < --- src/copy.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/src/copy.c b/src/copy.c index 27e083a..f149be4 100644 --- a/src/copy.c +++ b/src/copy.c @@ -240,7 +240,7 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, last = true; } - while (0 < ext_len) + while (ext_len) { char buf[buf_size]; -- 1.7.1.501.g23b46 From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 09 03:35:00 2010 Received: (at submit) by debbugs.gnu.org; 9 Jun 2010 07:35:00 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMFoR-0004gB-32 for submit@debbugs.gnu.org; Wed, 09 Jun 2010 03:34:59 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMFoO-0004g6-Vi for submit@debbugs.gnu.org; Wed, 09 Jun 2010 03:34:58 -0400 Received: from lists.gnu.org ([199.232.76.165]:39028) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OMFoL-0008Cq-PC for submit@debbugs.gnu.org; Wed, 09 Jun 2010 03:34:53 -0400 Received: from [140.186.70.92] (port=56281 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OMFoF-0006og-5a for bug-coreutils@gnu.org; Wed, 09 Jun 2010 03:34:53 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OMFo9-0003F2-2G for bug-coreutils@gnu.org; Wed, 09 Jun 2010 03:34:46 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:29646) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMFo8-0003Ea-Pl for bug-coreutils@gnu.org; Wed, 09 Jun 2010 03:34:40 -0400 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o597YUCb008674 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 9 Jun 2010 07:34:33 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5966XY7032361; Wed, 9 Jun 2010 07:34:28 GMT Received: from abhmt006.oracle.com by acsmt355.oracle.com with ESMTP id 332095881276068800; Wed, 09 Jun 2010 00:33:20 -0700 Received: from [141.144.144.192] (/141.144.144.192) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 09 Jun 2010 00:33:19 -0700 Message-ID: <4C0F43BE.1030709@oracle.com> Date: Wed, 09 Jun 2010 15:33:18 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering , Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <4C0EE3E9.8010802@cs.ucla.edu> <87typcra94.fsf@meyering.net> In-Reply-To: <87typcra94.fsf@meyering.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet13.oracle.com [148.87.113.125] X-CT-RefId: str=0001.0A090205.4C0F440A.0192:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.6 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Chris Mason , bug-coreutils@gnu.org, Joel Becker , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) Jim Meyering wrote: > Paul Eggert wrote: >> Jeff Liu and Jim Meyering wrote: >>> diff --git a/src/fiemap.h b/src/fiemap.h >>> new file mode 100644 >>> index 0000000..d33293b >>> --- /dev/null >>> +++ b/src/fiemap.h > > Hi Paul, > > Thanks for the review. > >> Why is this file necessary? fiemap.h is included only if it exists, >> right? Shouldn't we just use the kernel's fiemap.h rather than >> copying it here and assuming kernel internals? > > The ioctl is available/usable in 2.6.27 and newer that do not publish > this file. For example, it's in F13's (2.6.33's) /usr/include/linux/fiemap.h, > as well as the one in debian unstable's 2.6.32, but probably > not in much older kernels. > > Hmm.. I see that it's available even in F11's 2.6.30.9-x > > It would be better to include when present, > else our copy of that header. Then, eventually, the else > clause will become unused. Note that even on newer kernels, > the linux/* headers are optional. > > Eventually we'll have a hard requirement on kernel headers -- > at least when building against a linux kernel. > >>> if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) >> For this sort of thing, please just use "0" rather than "0LL". >> "0" is easier to read and it has the same effect here. > > Included in the patch below. > >>> char buf[buf_size]; >> This assumes C99, since buf_size is not known at compile time. >> Also, isn't there a potential segmentation-violation problem >> if buf_size is sufficiently large? >> >> More generally, since the caller is already allocating a buffer of the >> appropiate size, shouldn't we just reuse that buffer, rather than >> allocating a new one? That would avoid the problems of assuming >> C99 and possible segmentation violations. > > Good point. Thanks. > We can definitely avoid that allocation. > Do you feel like writing the patch? > > I've just pushed this series to a branch, "fiemap-copy", > so others can follow along and contribute more easily. > >>> char fiemap_buf[4096]; >>> struct fiemap *fiemap = (struct fiemap *) fiemap_buf; >>> struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; >>> uint32_t count = ((sizeof fiemap_buf - sizeof (*fiemap)) >>> / sizeof (struct fiemap_extent)); >> This code isn't portable, since fiemap_buf is only char-aligned, and >> struct fiemap may well require stricter alignment. The code will work >> on the x86 despite the alignment problem, but that's just a happy >> coincidence. >> >> A lesser point: the code assumes that 'struct fiemap' is sufficiently >> small (considerably less than 4096 bytes in size); I expect that this >> is universally true but we might as well check this assumption, since >> it's easy to do so without any runtime overhead. >> >> So I propose something like this instead: >> >> union { struct fiemap f; char c[4096]; } fiemap_buf; >> struct fiemap *fiemap = &fiemap_buf.f; >> struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; >> enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; >> verify (count != 0); Cooool!! Thanks Paul and Jim for the sharing. Regards, -Jeff > > I've done this in your name: > > From fffa2e8661a27978927fcc8afb6873631a753292 Mon Sep 17 00:00:00 2001 > From: Paul Eggert > Date: Wed, 9 Jun 2010 08:15:07 +0200 > Subject: [PATCH] copy.c: ensure proper alignment of fiemap buffer > > * src/copy.c (fiemap_copy): Ensure that our fiemap buffer > is large enough and well-aligned. > Replace "0LL" with equivalent "0" as 3rd argument to lseek. > --- > src/copy.c | 15 ++++++++------- > 1 files changed, 8 insertions(+), 7 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index f629771..27e083a 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -171,11 +171,12 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, > char const *dst_name, bool *normal_copy_required) > { > bool last = false; > - char fiemap_buf[4096]; > - struct fiemap *fiemap = (struct fiemap *) fiemap_buf; > + union { struct fiemap f; char c[4096]; } fiemap_buf; > + struct fiemap *fiemap = &fiemap_buf.f; > struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; > - uint32_t count = ((sizeof fiemap_buf - sizeof (*fiemap)) > - / sizeof (struct fiemap_extent)); > + enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; > + verify (count != 0); > + > off_t last_ext_logical = 0; > uint64_t last_ext_len = 0; > uint64_t last_read_size = 0; > @@ -185,7 +186,7 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, > /* This is required at least to initialize fiemap->fm_start, > but also serves (in mid 2010) to appease valgrind, which > appears not to know the semantics of the FIEMAP ioctl. */ > - memset (fiemap_buf, 0, sizeof fiemap_buf); > + memset (&fiemap_buf, 0, sizeof fiemap_buf); > > do > { > @@ -220,13 +221,13 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, > off_t ext_logical = fm_ext[i].fe_logical; > uint64_t ext_len = fm_ext[i].fe_length; > > - if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) > + if (lseek (src_fd, ext_logical, SEEK_SET) < 0) > { > error (0, errno, _("cannot lseek %s"), quote (src_name)); > return false; > } > > - if (lseek (dest_fd, ext_logical, SEEK_SET) < 0LL) > + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0) > { > error (0, errno, _("cannot lseek %s"), quote (dst_name)); > return false; > -- > 1.7.1.501.g23b46 > > > Also, I've squashed this clean-up patch onto Jeff's original, > since ext_len is unsigned (of type uint64_t). > > From bad13e737c683757a2ed05404564d8863c5da30e Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Wed, 9 Jun 2010 08:24:39 +0200 > Subject: [PATCH] remove 0 < > > --- > src/copy.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index 27e083a..f149be4 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -240,7 +240,7 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, > last = true; > } > > - while (0 < ext_len) > + while (ext_len) > { > char buf[buf_size]; > > -- > 1.7.1.501.g23b46 > > > -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 09 05:09:23 2010 Received: (at submit) by debbugs.gnu.org; 9 Jun 2010 09:09:23 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMHHm-0005Lo-MM for submit@debbugs.gnu.org; Wed, 09 Jun 2010 05:09:23 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMHHj-0005Lj-It for submit@debbugs.gnu.org; Wed, 09 Jun 2010 05:09:21 -0400 Received: from lists.gnu.org ([199.232.76.165]:55935) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OMHHg-0001jc-KF for submit@debbugs.gnu.org; Wed, 09 Jun 2010 05:09:16 -0400 Received: from [140.186.70.92] (port=36041 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OMHHe-0004uO-Pu for bug-coreutils@gnu.org; Wed, 09 Jun 2010 05:09:16 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OMHHZ-0002ov-94 for bug-coreutils@gnu.org; Wed, 09 Jun 2010 05:09:14 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:44566) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMHHY-0002of-Vl for bug-coreutils@gnu.org; Wed, 09 Jun 2010 05:09:09 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5998xpP010709 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 9 Jun 2010 09:09:02 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5998v9L023702; Wed, 9 Jun 2010 09:08:58 GMT Received: from abhmt001.oracle.com by acsmt355.oracle.com with ESMTP id 309311771276074455; Wed, 09 Jun 2010 02:07:35 -0700 Received: from [141.144.144.192] (/141.144.144.192) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 09 Jun 2010 02:07:34 -0700 Message-ID: <4C0F59D5.2010305@oracle.com> Date: Wed, 09 Jun 2010 17:07:33 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Paul Eggert , Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <4C0EE3E9.8010802@cs.ucla.edu> <87typcra94.fsf@meyering.net> In-Reply-To: <87typcra94.fsf@meyering.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090201.4C0F5A2F.0189:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.6 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Chris Mason , bug-coreutils@gnu.org, Joel Becker , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) Jim Meyering wrote: > Paul Eggert wrote: >> Jeff Liu and Jim Meyering wrote: >>> diff --git a/src/fiemap.h b/src/fiemap.h >>> new file mode 100644 >>> index 0000000..d33293b >>> --- /dev/null >>> +++ b/src/fiemap.h > > Hi Paul, > > Thanks for the review. > >> Why is this file necessary? fiemap.h is included only if it exists, >> right? Shouldn't we just use the kernel's fiemap.h rather than >> copying it here and assuming kernel internals? > > The ioctl is available/usable in 2.6.27 and newer that do not publish > this file. For example, it's in F13's (2.6.33's) /usr/include/linux/fiemap.h, > as well as the one in debian unstable's 2.6.32, but probably > not in much older kernels. > > Hmm.. I see that it's available even in F11's 2.6.30.9-x > > It would be better to include when present, > else our copy of that header. Then, eventually, the else > clause will become unused. Note that even on newer kernels, > the linux/* headers are optional. > > Eventually we'll have a hard requirement on kernel headers -- > at least when building against a linux kernel. > >>> if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) >> For this sort of thing, please just use "0" rather than "0LL". >> "0" is easier to read and it has the same effect here. > > Included in the patch below. > >>> char buf[buf_size]; >> This assumes C99, since buf_size is not known at compile time. >> Also, isn't there a potential segmentation-violation problem >> if buf_size is sufficiently large? >> >> More generally, since the caller is already allocating a buffer of the >> appropiate size, shouldn't we just reuse that buffer, rather than >> allocating a new one? That would avoid the problems of assuming >> C99 and possible segmentation violations. > > Good point. Thanks. > We can definitely avoid that allocation. > Do you feel like writing the patch? Thanks for pointing this out! Hi Paul, I am appreciated if you have time for writing the patch. Or else, I will follow up sometime in the next few days since I have an urgent task need to handle over at present. Regards, -Jeff > > I've just pushed this series to a branch, "fiemap-copy", > so others can follow along and contribute more easily. > >>> char fiemap_buf[4096]; >>> struct fiemap *fiemap = (struct fiemap *) fiemap_buf; >>> struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; >>> uint32_t count = ((sizeof fiemap_buf - sizeof (*fiemap)) >>> / sizeof (struct fiemap_extent)); >> This code isn't portable, since fiemap_buf is only char-aligned, and >> struct fiemap may well require stricter alignment. The code will work >> on the x86 despite the alignment problem, but that's just a happy >> coincidence. >> >> A lesser point: the code assumes that 'struct fiemap' is sufficiently >> small (considerably less than 4096 bytes in size); I expect that this >> is universally true but we might as well check this assumption, since >> it's easy to do so without any runtime overhead. >> >> So I propose something like this instead: >> >> union { struct fiemap f; char c[4096]; } fiemap_buf; >> struct fiemap *fiemap = &fiemap_buf.f; >> struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; >> enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; >> verify (count != 0); > > I've done this in your name: > > From fffa2e8661a27978927fcc8afb6873631a753292 Mon Sep 17 00:00:00 2001 > From: Paul Eggert > Date: Wed, 9 Jun 2010 08:15:07 +0200 > Subject: [PATCH] copy.c: ensure proper alignment of fiemap buffer > > * src/copy.c (fiemap_copy): Ensure that our fiemap buffer > is large enough and well-aligned. > Replace "0LL" with equivalent "0" as 3rd argument to lseek. > --- > src/copy.c | 15 ++++++++------- > 1 files changed, 8 insertions(+), 7 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index f629771..27e083a 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -171,11 +171,12 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, > char const *dst_name, bool *normal_copy_required) > { > bool last = false; > - char fiemap_buf[4096]; > - struct fiemap *fiemap = (struct fiemap *) fiemap_buf; > + union { struct fiemap f; char c[4096]; } fiemap_buf; > + struct fiemap *fiemap = &fiemap_buf.f; > struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; > - uint32_t count = ((sizeof fiemap_buf - sizeof (*fiemap)) > - / sizeof (struct fiemap_extent)); > + enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; > + verify (count != 0); > + > off_t last_ext_logical = 0; > uint64_t last_ext_len = 0; > uint64_t last_read_size = 0; > @@ -185,7 +186,7 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, > /* This is required at least to initialize fiemap->fm_start, > but also serves (in mid 2010) to appease valgrind, which > appears not to know the semantics of the FIEMAP ioctl. */ > - memset (fiemap_buf, 0, sizeof fiemap_buf); > + memset (&fiemap_buf, 0, sizeof fiemap_buf); > > do > { > @@ -220,13 +221,13 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, > off_t ext_logical = fm_ext[i].fe_logical; > uint64_t ext_len = fm_ext[i].fe_length; > > - if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL) > + if (lseek (src_fd, ext_logical, SEEK_SET) < 0) > { > error (0, errno, _("cannot lseek %s"), quote (src_name)); > return false; > } > > - if (lseek (dest_fd, ext_logical, SEEK_SET) < 0LL) > + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0) > { > error (0, errno, _("cannot lseek %s"), quote (dst_name)); > return false; > -- > 1.7.1.501.g23b46 > > > Also, I've squashed this clean-up patch onto Jeff's original, > since ext_len is unsigned (of type uint64_t). > > From bad13e737c683757a2ed05404564d8863c5da30e Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Wed, 9 Jun 2010 08:24:39 +0200 > Subject: [PATCH] remove 0 < > > --- > src/copy.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index 27e083a..f149be4 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -240,7 +240,7 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, > last = true; > } > > - while (0 < ext_len) > + while (ext_len) > { > char buf[buf_size]; > > -- > 1.7.1.501.g23b46 > > > -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 09 10:48:10 2010 Received: (at submit) by debbugs.gnu.org; 9 Jun 2010 14:48:10 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMMZd-0008MJ-UR for submit@debbugs.gnu.org; Wed, 09 Jun 2010 10:48:10 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMMZb-0008Lz-9a for submit@debbugs.gnu.org; Wed, 09 Jun 2010 10:48:08 -0400 Received: from lists.gnu.org ([199.232.76.165]:49429) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OMMZW-0000lp-7v for submit@debbugs.gnu.org; Wed, 09 Jun 2010 10:48:02 -0400 Received: from [140.186.70.92] (port=53858 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OMMZU-0006iW-A4 for bug-coreutils@gnu.org; Wed, 09 Jun 2010 10:48:01 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OMMZS-00013E-HV for bug-coreutils@gnu.org; Wed, 09 Jun 2010 10:47:59 -0400 Received: from acsinet11.oracle.com ([141.146.126.233]:35118) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMMZS-00012n-Aq for bug-coreutils@gnu.org; Wed, 09 Jun 2010 10:47:58 -0400 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by acsinet11.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o59ElrJm001206 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 9 Jun 2010 14:47:55 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o59ElpO8015679; Wed, 9 Jun 2010 14:47:52 GMT Received: from abhmt009.oracle.com by acsmt354.oracle.com with ESMTP id 333156101276094787; Wed, 09 Jun 2010 07:46:27 -0700 Received: from [123.119.99.202] (/123.119.99.202) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 09 Jun 2010 07:46:25 -0700 Message-ID: <4C0FA93C.6020205@oracle.com> Date: Wed, 09 Jun 2010 22:46:20 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> In-Reply-To: <87eighs1t9.fsf@meyering.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090207.4C0FA998.01E5:SCFMA4539814,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.0 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Chris Mason , bug-coreutils@gnu.org, Joel Becker , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.1 (------) Jim Meyering wrote: > Jim Meyering wrote: >> Subject: [PATCH 01/10] cp: Add FIEMAP support for efficient sparse file copy > > FYI, using those patches, I ran a test for the first time in a few days: > > check -C tests TESTS=cp/sparse-fiemap VERBOSE=yes > > It failed like this on an ext4 partition using F13: > > + timeout 10 cp --sparse=always sparse fiemap > + fail=1 > ++ stat --printf %s sparse > ++ stat --printf %s fiemap > + test 1099511628800 = 0 > + fail=1 > > That is very odd. No diagnostic from cp, yet it failed > after creating a zero-length file. > > Here's the corresponding piece of the script: > > # It takes many minutes to copy this sparse file using the old method. > # By contrast, it takes far less than 1 second using FIEMAP-copy. > timeout 10 cp --sparse=always sparse fiemap || fail=1 > > # Ensure that the sparse file copied through fiemap has the same size > # in bytes as the original. > test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 > > However, so far I've been unable to reproduce the failure, > running hundreds of iterations: > > for i in $(seq 300); do printf .; make check -C tests \ > TESTS=cp/sparse-fiemap VERBOSE=yes >& makerr-$i || break; done > > Have any of you heard of a problem whereby a cold cache can cause > such a thing? "echo 3 > /proc/sys/vm/drop_caches" didn't help. Hi Jim, Have you run `sync' before clean the buffer and caches? Actually, even run `sync' first, sometimes, maybe the dirty objects still can not be freed in some cases. :( I can reproduce this issue on ext4 and btrfs(physical mounted partition) or just run the sparse-fiemap test script, ocfs2 always works fine in this case. I guess this issue might caused by the 'cold cache' as your above mentioned. According to my tryout, after clean out the caches, cp via filemap always works in my test environment, otherwise, it failed from time to time. My kernel version: Linux jeff-laptop 2.6.33-rc5-00238-gb04da8b-dirty #11 SMP Sat Dec 19 22:02:01 CST 2009 i686 GNU/Linux jeff@jeff-laptop:/ext4$ dd if=/dev/zero of=sparse bs=1k count=1 seek=1G 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000156654 s, 6.5 MB/s jeff@jeff-laptop:/ext4$ ls -l sparse -rw-r--r-- 1 jeff jeff 1099511628800 Jun 9 22:21 sparse jeff@jeff-laptop:/ext4$ filefrag sparse sparse: 0 extents found jeff@jeff-laptop:/ext4$ filefrag -v sparse Filesystem type is: ef53 File size of sparse is 1099511628800 (268435457 blocks, blocksize 4096) ext logical physical expected length flags sparse: 1 extent found To free the buffer cache: ========================= jeff@jeff-laptop:/ext4$ free total used free shared buffers cached Mem: 1980300 719972 1260328 0 2836 94104 -/+ buffers/cache: 623032 1357268 Swap: 0 0 0 jeff@jeff-laptop:/ext4$ sync In another root console, run 'echo 3 > /proc/sys/vm/drop_caches' jeff@jeff-laptop:/ext4$ free total used free shared buffers cached Mem: 1980300 716780 1263520 0 1184 88592 <<<<<-----freed -/+ buffers/cache: 627004 1353296 Swap: 0 0 0 jeff@jeff-laptop:/ext4$ filefrag -v sparse Filesystem type is: ef53 File size of sparse is 1099511628800 (268435457 blocks, blocksize 4096) ext logical physical expected length flags 0 268435456 32999 1 eof sparse: 2 extents found jeff@jeff-laptop:/ext4$ ./cp --sparse=always sparse f1 last_ext_logical 1099511627776 last_read_size 1024 src_total_size 1099511628800 jeff@jeff-laptop:/ext4$ filefrag -v f1 Filesystem type is: ef53 File size of f1 is 1099511628800 (268435457 blocks, blocksize 4096) ext logical physical expected length flags 0 268435456 296960 1 eof f1: 2 extents found jeff@jeff-laptop:/ext4$ ./cp --sparse=always sparse f2 last_ext_logical 1099511627776 last_read_size 1024 src_total_size 1099511628800 jeff@jeff-laptop:/ext4$ filefrag -v f2 Filesystem type is: ef53 File size of f2 is 1099511628800 (268435457 blocks, blocksize 4096) ext logical physical expected length flags f2: 1 extent found jeff@jeff-laptop:/ext4$ sync and 'clean memory via /proc on another root console' jeff@jeff-laptop:/ext4$ filefrag -v f2 Filesystem type is: ef53 File size of f2 is 1099511628800 (268435457 blocks, blocksize 4096) ext logical physical expected length flags 0 268435456 33379 1 eof f2: 2 extents found I will do a double check for my original patch to ensure this is not a code bug for that issue once get through an urgent task on hand. Thanks, -Jeff > I suspect that having so many extents is unusual, so maybe > this is a rarely exercised corner case. > > =============================== > As I wrote the above, I realized I probably had enough > information to deduce where things were going wrong, even > if so far I've been unable to reproduce it. > > And sure enough. There is a way to provoke exactly > that failure. If the *second* (or later) FIEMAP ioctl fails: > > do > { > fiemap->fm_length = FIEMAP_MAX_OFFSET; > fiemap->fm_extent_count = count; > > /* When ioctl(2) fails, fall back to the normal copy only if it > is the first time we met. */ > if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) > { > /* If the first ioctl fails, tell the caller that it is > ok to proceed with a normal copy. */ > if (i == 0) > *normal_copy_required = true; > return false; > } > > In that case, fiemap_copy returns false (with no diagnostic) > and cp fails silently. > > Obviously I will now add code to diagnose the failure, > but do any of you know off hand how to reproduce this > or what the failure might have been? > > Here's the patch I plan to merge: > > diff --git a/src/copy.c b/src/copy.c > index eb67700..07d605e 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -200,6 +200,12 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, > ok to proceed with a normal copy. */ > if (i == 0) > *normal_copy_required = true; > + else > + { > + /* If the second or subsequent ioctl fails, diagnose it, > + since it ends up causing the entire copy/cp to fail. */ > + error (0, errno, _("%s: FIEMAP ioctl failed"), quote (src_name)); > + } > return false; > } -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 09 11:12:28 2010 Received: (at submit) by debbugs.gnu.org; 9 Jun 2010 15:12:28 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMMx9-0008W0-HY for submit@debbugs.gnu.org; Wed, 09 Jun 2010 11:12:27 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMMx8-0008Vv-Eg for submit@debbugs.gnu.org; Wed, 09 Jun 2010 11:12:27 -0400 Received: from lists.gnu.org ([199.232.76.165]:49649) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OMMx3-0001Gz-HQ for submit@debbugs.gnu.org; Wed, 09 Jun 2010 11:12:21 -0400 Received: from [140.186.70.92] (port=46176 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OMMx1-0003lB-Re for bug-coreutils@gnu.org; Wed, 09 Jun 2010 11:12:20 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OMMwy-0005iE-LA for bug-coreutils@gnu.org; Wed, 09 Jun 2010 11:12:17 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:46103) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMMwy-0005hP-2p for bug-coreutils@gnu.org; Wed, 09 Jun 2010 11:12:16 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id BCEB5940155; Wed, 9 Jun 2010 17:12:08 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 5FDE9DA59; Wed, 9 Jun 2010 17:12:07 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4C0FA93C.6020205@oracle.com> (jeff liu's message of "Wed, 09 Jun 2010 22:46:20 +0800") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> Date: Wed, 09 Jun 2010 17:12:07 +0200 Message-ID: <87iq5sp7eg.fsf@meyering.net> Lines: 69 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Chris Mason , bug-coreutils@gnu.org, Joel Becker , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) jeff.liu wrote: > Jim Meyering wrote: >> Jim Meyering wrote: >>> Subject: [PATCH 01/10] cp: Add FIEMAP support for efficient sparse file copy >> >> FYI, using those patches, I ran a test for the first time in a few days: >> >> check -C tests TESTS=cp/sparse-fiemap VERBOSE=yes >> >> It failed like this on an ext4 partition using F13: >> >> + timeout 10 cp --sparse=always sparse fiemap >> + fail=1 >> ++ stat --printf %s sparse >> ++ stat --printf %s fiemap >> + test 1099511628800 = 0 >> + fail=1 >> >> That is very odd. No diagnostic from cp, yet it failed >> after creating a zero-length file. >> >> Here's the corresponding piece of the script: >> >> # It takes many minutes to copy this sparse file using the old method. >> # By contrast, it takes far less than 1 second using FIEMAP-copy. >> timeout 10 cp --sparse=always sparse fiemap || fail=1 >> >> # Ensure that the sparse file copied through fiemap has the same size >> # in bytes as the original. >> test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 >> >> However, so far I've been unable to reproduce the failure, >> running hundreds of iterations: >> >> for i in $(seq 300); do printf .; make check -C tests \ >> TESTS=cp/sparse-fiemap VERBOSE=yes >& makerr-$i || break; done >> >> Have any of you heard of a problem whereby a cold cache can cause >> such a thing? "echo 3 > /proc/sys/vm/drop_caches" didn't help. > Hi Jim, > > Have you run `sync' before clean the buffer and caches? Actually, even run `sync' first, sometimes, > maybe the dirty objects still can not be freed in some cases. :( Hi Jeff, Thanks for the feedback. The test script I ran above does not invoke sync between the dd invocation and the cp --sparse. If running sync before cp is required in order to avoid a failure, then I consider that a bug in cp. > I can reproduce this issue on ext4 and btrfs(physical mounted partition) or just run the If you can reproduce the cp failure, then please use my latest patch that diagnoses 2nd and subsequent FIEMAP ioctl failure and tell me what cp prints when it fails. I would really like to know how the first ioctl can succeed, yet the second one fails. Your demonstrations show that *after* the cp, one may have to sync in order to see precisely if/how the newly-created destination file is fragmented, but that is nothing new. Note the commented-out "sync" uses in the test script. As far as I can see, your examples do not show cp failing like it did for me. From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 10 02:57:17 2010 Received: (at submit) by debbugs.gnu.org; 10 Jun 2010 06:57:17 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMbhT-0008N9-Pb for submit@debbugs.gnu.org; Thu, 10 Jun 2010 02:57:17 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMbhR-0008N3-GP for submit@debbugs.gnu.org; Thu, 10 Jun 2010 02:57:14 -0400 Received: from lists.gnu.org ([199.232.76.165]:53483) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OMbhO-00017D-9s for submit@debbugs.gnu.org; Thu, 10 Jun 2010 02:57:10 -0400 Received: from [140.186.70.92] (port=56557 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OMbhM-0003nq-Q2 for bug-coreutils@gnu.org; Thu, 10 Jun 2010 02:57:09 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OMbhK-0004XQ-Tc for bug-coreutils@gnu.org; Thu, 10 Jun 2010 02:57:08 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:57592) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMbhK-0004XI-ND for bug-coreutils@gnu.org; Thu, 10 Jun 2010 02:57:06 -0400 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5A6v15q028058 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 10 Jun 2010 06:57:03 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5A11YbX020519; Thu, 10 Jun 2010 06:57:00 GMT Received: from abhmt021.oracle.com by acsmt353.oracle.com with ESMTP id 312660611276153018; Wed, 09 Jun 2010 23:56:58 -0700 Received: from [10.182.121.3] (/10.182.121.3) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 09 Jun 2010 23:56:57 -0700 Message-ID: <4C108CB9.8010805@oracle.com> Date: Thu, 10 Jun 2010 14:56:57 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> In-Reply-To: <87iq5sp7eg.fsf@meyering.net> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet13.oracle.com [148.87.113.125] X-CT-RefId: str=0001.0A090205.4C108CC0.00A0:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.2 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Chris Mason , bug-coreutils@gnu.org, Joel Becker , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) Jim Meyering wrote: > jeff.liu wrote: > >> Jim Meyering wrote: >>> Jim Meyering wrote: >>>> Subject: [PATCH 01/10] cp: Add FIEMAP support for efficient sparse file copy >>> FYI, using those patches, I ran a test for the first time in a few days: >>> >>> check -C tests TESTS=cp/sparse-fiemap VERBOSE=yes >>> >>> It failed like this on an ext4 partition using F13: >>> >>> + timeout 10 cp --sparse=always sparse fiemap >>> + fail=1 >>> ++ stat --printf %s sparse >>> ++ stat --printf %s fiemap >>> + test 1099511628800 = 0 >>> + fail=1 >>> >>> That is very odd. No diagnostic from cp, yet it failed >>> after creating a zero-length file. >>> >>> Here's the corresponding piece of the script: >>> >>> # It takes many minutes to copy this sparse file using the old method. >>> # By contrast, it takes far less than 1 second using FIEMAP-copy. >>> timeout 10 cp --sparse=always sparse fiemap || fail=1 >>> >>> # Ensure that the sparse file copied through fiemap has the same size >>> # in bytes as the original. >>> test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1 >>> >>> However, so far I've been unable to reproduce the failure, >>> running hundreds of iterations: >>> >>> for i in $(seq 300); do printf .; make check -C tests \ >>> TESTS=cp/sparse-fiemap VERBOSE=yes >& makerr-$i || break; done >>> >>> Have any of you heard of a problem whereby a cold cache can cause >>> such a thing? "echo 3 > /proc/sys/vm/drop_caches" didn't help. >> Hi Jim, >> >> Have you run `sync' before clean the buffer and caches? Actually, even run `sync' first, sometimes, >> maybe the dirty objects still can not be freed in some cases. :( > > Hi Jeff, > > Thanks for the feedback. > > The test script I ran above does not invoke sync between > the dd invocation and the cp --sparse. > If running sync before cp is required in order to avoid a failure, > then I consider that a bug in cp. > >> I can reproduce this issue on ext4 and btrfs(physical mounted partition) or just run the > > If you can reproduce the cp failure, then please use my latest > patch that diagnoses 2nd and subsequent FIEMAP ioctl failure > and tell me what cp prints when it fails. Strange!! Today I can not reproduce this issue on my laptop even repeat the test script up to 300 times. Yesterday I also run test against your fiemap-copy git-tree with the latest diagnose patch, but it did not show me the 2nd ioctl failure info because that 1TB sparse file only have one extent allocated on ext4, so the ioctl(2) only hit once. > > I would really like to know how the first ioctl can succeed, > yet the second one fails. > > Your demonstrations show that *after* the cp, one may have > to sync in order to see precisely if/how the newly-created > destination file is fragmented, but that is nothing new. > Note the commented-out "sync" uses in the test script. > As far as I can see, your examples do not show cp failing > like it did for me. Yeah, I just realized that the behaviour I observed is caused by the delay allocation mechanism of the particular FS. Thanks, -Jeff > > > -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 10 19:47:39 2010 Received: (at submit) by debbugs.gnu.org; 10 Jun 2010 23:47:39 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMrTG-0007t0-UH for submit@debbugs.gnu.org; Thu, 10 Jun 2010 19:47:39 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMrTE-0007su-Hp for submit@debbugs.gnu.org; Thu, 10 Jun 2010 19:47:37 -0400 Received: from lists.gnu.org ([199.232.76.165]:43508) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OMrTA-0004V2-MQ for submit@debbugs.gnu.org; Thu, 10 Jun 2010 19:47:32 -0400 Received: from [140.186.70.92] (port=39459 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OMrT8-0008QP-U6 for bug-coreutils@gnu.org; Thu, 10 Jun 2010 19:47:31 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OMrT7-0004b2-Ms for bug-coreutils@gnu.org; Thu, 10 Jun 2010 19:47:30 -0400 Received: from kiwi.cs.ucla.edu ([131.179.128.19]:55690) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMrT7-0004aK-9U for bug-coreutils@gnu.org; Thu, 10 Jun 2010 19:47:29 -0400 Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200]) by kiwi.cs.ucla.edu (8.13.8+Sun/8.13.8/UCLACS-6.0) with ESMTP id o5ANlOpw027577; Thu, 10 Jun 2010 16:47:24 -0700 (PDT) Message-ID: <4C11798B.6090500@cs.ucla.edu> Date: Thu, 10 Jun 2010 16:47:23 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> In-Reply-To: <4C108CB9.8010805@oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -3.3 (---) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.6 (----) On 06/09/2010 11:56 PM, jeff.liu wrote: > Yeah, I just realized that the behaviour I observed is caused by the delay allocation mechanism of > the particular FS. If the file system is using delayed allocation, then can the fiemap ioctl tell us that a file contains a hole (because nothing has been allocated there), but read() would tell us that the file contains nonzero data at the same location (because it's sitting in a buffer somewhere)? If so, we'd need to do something like invoke fdatasync() on the file before issuing the fiemap ioctl, to force allocation; or perhaps there's another ioctl that will do the allocation without having to actually do a sync. There's also the issue of copying from a file at the same time that some other process is writing to it, but that is allowed to produce ill-defined behavior. I'm more worried about the case where some other process writes to the source file just before 'cp' starts. (Sorry, I haven't had time yet to dive into the proposed change; I'm still trying to understand the environment.) One other thing: Solaris 10 supports lseek with the SEEK_HOLE and SEEK_DATA options, which are easier to use and which (as far as I can tell from the manual) shouldn't require anything fdatasync-ish. Any objection if I propose support for that too? It is supposed to work with ZFS, something I can test here. From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 10 20:30:17 2010 Received: (at submit) by debbugs.gnu.org; 11 Jun 2010 00:30:17 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMs8W-0008Aq-Cx for submit@debbugs.gnu.org; Thu, 10 Jun 2010 20:30:17 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMs8V-0008Ak-AP for submit@debbugs.gnu.org; Thu, 10 Jun 2010 20:30:15 -0400 Received: from lists.gnu.org ([199.232.76.165]:38061) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OMs8R-00059j-I5 for submit@debbugs.gnu.org; Thu, 10 Jun 2010 20:30:11 -0400 Received: from [140.186.70.92] (port=59063 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OMs8P-0001xR-Sy for bug-coreutils@gnu.org; Thu, 10 Jun 2010 20:30:10 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OMs8O-00037t-Oc for bug-coreutils@gnu.org; Thu, 10 Jun 2010 20:30:09 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:54072) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMs8O-00037M-Jg for bug-coreutils@gnu.org; Thu, 10 Jun 2010 20:30:08 -0400 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5B0U2fb025133 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 11 Jun 2010 00:30:03 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5B0U1Mo022252; Fri, 11 Jun 2010 00:30:01 GMT Received: from ca-server1.us.oracle.com by acsmt354.oracle.com with ESMTP id 315771691276216116; Thu, 10 Jun 2010 17:28:36 -0700 Received: from jlbec by ca-server1.us.oracle.com with local (Exim 4.69) (envelope-from ) id 1OMs6u-0000Wg-2V; Thu, 10 Jun 2010 17:28:36 -0700 Date: Thu, 10 Jun 2010 17:28:35 -0700 From: Joel Becker To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy Message-ID: <20100611002835.GH5934@mail.oracle.com> References: <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C11798B.6090500@cs.ucla.edu> X-Burt-Line: Trees are cool. X-Red-Smith: Ninety feet between bases is perhaps as close as man has ever come to perfection. User-Agent: Mutt/1.5.20 (2009-06-14) X-Auth-Type: Internal IP X-Source-IP: rcsinet13.oracle.com [148.87.113.125] X-CT-RefId: str=0001.0A090207.4C11838C.001E:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , "jeff.liu" , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.4 (-----) On Thu, Jun 10, 2010 at 04:47:23PM -0700, Paul Eggert wrote: > On 06/09/2010 11:56 PM, jeff.liu wrote: > > > Yeah, I just realized that the behaviour I observed is caused by the delay allocation mechanism of > > the particular FS. > > If the file system is using delayed allocation, then can > the fiemap ioctl tell us that a file contains a hole (because nothing has been > allocated there), but read() would tell us that the file contains nonzero data at the same location > (because it's sitting in a buffer somewhere)? If so, we'd need to do something like invoke > fdatasync() on the file before issuing the fiemap ioctl, to force allocation; or perhaps > there's another ioctl that will do the allocation without having to actually do a sync. I can think of a couple things, like returning an extent record with a zero block number: "I have an extent here, but I don't have physical storage." Or having a filesystem's fiemap call force the allocation before returning. What say we copy linux-ext4 here? Joel -- "If you took all of the grains of sand in the world, and lined them up end to end in a row, you'd be working for the government!" - Mr. Interesting Joel Becker Principal Software Developer Oracle E-mail: joel.becker@oracle.com Phone: (650) 506-8127 From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 10 20:32:24 2010 Received: (at submit) by debbugs.gnu.org; 11 Jun 2010 00:32:24 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMsAZ-0008Bw-LQ for submit@debbugs.gnu.org; Thu, 10 Jun 2010 20:32:24 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMsAX-0008Bp-FG for submit@debbugs.gnu.org; Thu, 10 Jun 2010 20:32:21 -0400 Received: from lists.gnu.org ([199.232.76.165]:53342) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OMsAT-0005B1-Qq for submit@debbugs.gnu.org; Thu, 10 Jun 2010 20:32:17 -0400 Received: from [140.186.70.92] (port=37729 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OMsAS-0003Xy-A2 for bug-coreutils@gnu.org; Thu, 10 Jun 2010 20:32:17 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OMsAR-0003PW-7I for bug-coreutils@gnu.org; Thu, 10 Jun 2010 20:32:16 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:63421) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMsAR-0003P7-1I for bug-coreutils@gnu.org; Thu, 10 Jun 2010 20:32:15 -0400 Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5B0WA52027473 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 11 Jun 2010 00:32:11 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5B0W6WW024003; Fri, 11 Jun 2010 00:32:06 GMT Received: from abhmt019.oracle.com by acsmt353.oracle.com with ESMTP id 315776561276216318; Thu, 10 Jun 2010 17:31:58 -0700 Received: from [130.35.68.118] (/130.35.68.118) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 10 Jun 2010 17:31:58 -0700 Message-ID: <4C1183FD.9010502@oracle.com> Date: Thu, 10 Jun 2010 17:31:57 -0700 From: Sunil Mushran User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> In-Reply-To: <4C11798B.6090500@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet15.oracle.com [148.87.113.117] X-CT-RefId: str=0001.0A090209.4C11840B.01BD:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.6 (------) X-Debbugs-Envelope-To: submit Cc: bug-coreutils@gnu.org, Jim Meyering , Joel Becker , "jeff.liu" , Tao Ma , "linux-ext4@vger.kernel.org" , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) On 06/10/2010 04:47 PM, Paul Eggert wrote: > On 06/09/2010 11:56 PM, jeff.liu wrote: > >> Yeah, I just realized that the behaviour I observed is caused by the delay allocation mechanism of >> the particular FS. >> > If the file system is using delayed allocation, then can > the fiemap ioctl tell us that a file contains a hole (because nothing has been > allocated there), but read() would tell us that the file contains nonzero data at the same location > (because it's sitting in a buffer somewhere)? If so, we'd need to do something like invoke > fdatasync() on the file before issuing the fiemap ioctl, to force allocation; or perhaps > there's another ioctl that will do the allocation without having to actually do a sync. > I guess we'll have to use FIEMAP_FLAG_SYNC. > There's also the issue of copying from a file at the same time that some other process > is writing to it, but that is allowed to produce ill-defined behavior. I'm more worried > about the case where some other process writes to the source file just before 'cp' starts. > cp's behavior with active files is undefined. But we know it reads from offset 0 to MAX. With fiemap it will continue to do the same with the exception that it will skip reads (and thus writes) depending on the extent map it gets at the very beginning. > (Sorry, I haven't had time yet to dive into the proposed change; I'm still trying to understand > the environment.) > > One other thing: Solaris 10 supports lseek with the SEEK_HOLE and SEEK_DATA options, which > are easier to use and which (as far as I can tell from the manual) shouldn't require anything > fdatasync-ish. Any objection if I propose support for that too? It is supposed to work > with ZFS, something I can test here. > There is no plan to implement SEEK_HOLE/SEEK_DATA in the kernel. At most glibc will use fiemap to extend lseek(). BTW, SEEK_HOLE/DATA also have the same problem with active files. ccing linux-ext4. From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 10 20:50:58 2010 Received: (at submit) by debbugs.gnu.org; 11 Jun 2010 00:50:58 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMsSW-0008KU-Tv for submit@debbugs.gnu.org; Thu, 10 Jun 2010 20:50:57 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMsSQ-0008KO-Mr for submit@debbugs.gnu.org; Thu, 10 Jun 2010 20:50:56 -0400 Received: from lists.gnu.org ([199.232.76.165]:57950) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OMsS7-0005OD-IO for submit@debbugs.gnu.org; Thu, 10 Jun 2010 20:50:31 -0400 Received: from [140.186.70.92] (port=39671 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OMsRq-0001EF-E7 for bug-coreutils@gnu.org; Thu, 10 Jun 2010 20:50:30 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OMsEU-0003wz-6W for bug-coreutils@gnu.org; Thu, 10 Jun 2010 20:36:27 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:24672) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMsEU-0003wP-1b for bug-coreutils@gnu.org; Thu, 10 Jun 2010 20:36:26 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5B0aJG5000640 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 11 Jun 2010 00:36:21 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5AKcBro032579; Fri, 11 Jun 2010 00:36:18 GMT Received: from abhmt020.oracle.com by acsmt353.oracle.com with ESMTP id 315779831276216492; Thu, 10 Jun 2010 17:34:52 -0700 Received: from tma-home.cn.oracle.com (/141.144.148.161) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 10 Jun 2010 17:34:51 -0700 Message-ID: <4C1184B4.2050309@oracle.com> Date: Fri, 11 Jun 2010 08:35:00 +0800 From: Tao Ma User-Agent: Mozilla-Thunderbird 2.0.0.22 (X11/20090706) MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> In-Reply-To: <4C11798B.6090500@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090204.4C118506.0047:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.1 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , "jeff.liu" , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.2 (------) Paul Eggert wrote: > On 06/09/2010 11:56 PM, jeff.liu wrote: > > >> Yeah, I just realized that the behaviour I observed is caused by the delay allocation mechanism of >> the particular FS. >> > > If the file system is using delayed allocation, then can > the fiemap ioctl tell us that a file contains a hole (because nothing has been > allocated there), but read() would tell us that the file contains nonzero data at the same location > (because it's sitting in a buffer somewhere)? If so, we'd need to do something like invoke > fdatasync() on the file before issuing the fiemap ioctl, to force allocation; or perhaps > there's another ioctl that will do the allocation without having to actually do a sync. > actually there is a flag FIEMAP_EXTENT_DELALLOC in fiemap, and in this case, the file system should return us a fiemap extent with this flag. I just did a simple test with ext4, and it looks that it has a problem with sparse file. For a file like this: dd if=/dev/zero of=/mnt/ext4/a bs=1M count=1 We can get a fiemap extent with DEALLOC successfully. while with dd if=/dev/zero of=/mnt/ext4/a bs=1M count=1 seek=1 the ext4 can't return a valid fiemap extent. Don't have time to dove into it yet. > There's also the issue of copying from a file at the same time that some other process > is writing to it, but that is allowed to produce ill-defined behavior. I'm more worried > about the case where some other process writes to the source file just before 'cp' starts. > If the file system can return the right DEALLOC fiemap, there should be no problem for it I guess. > (Sorry, I haven't had time yet to dive into the proposed change; I'm still trying to understand > the environment.) > > One other thing: Solaris 10 supports lseek with the SEEK_HOLE and SEEK_DATA options, which > are easier to use and which (as far as I can tell from the manual) shouldn't require anything > fdatasync-ish. Any objection if I propose support for that too? It is supposed to work > with ZFS, something I can test here. > I am afraid not. There are some discussions about this long time ago. See this article: http://lwn.net/Articles/260795/ So it looks that SEEK_HOLE and SEEK_DATA are not welcomed by the kernel. Regards, Tao From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 11 04:33:17 2010 Received: (at submit) by debbugs.gnu.org; 11 Jun 2010 08:33:18 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMzfw-0004VG-VC for submit@debbugs.gnu.org; Fri, 11 Jun 2010 04:33:17 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMzfv-0004V9-0C for submit@debbugs.gnu.org; Fri, 11 Jun 2010 04:33:16 -0400 Received: from lists.gnu.org ([199.232.76.165]:36934) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OMzfr-0004sr-T7 for submit@debbugs.gnu.org; Fri, 11 Jun 2010 04:33:11 -0400 Received: from [140.186.70.92] (port=37262 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OMzfp-00089F-F5 for bug-coreutils@gnu.org; Fri, 11 Jun 2010 04:33:11 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OMzfo-0003RM-4G for bug-coreutils@gnu.org; Fri, 11 Jun 2010 04:33:09 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:17101) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OMzfn-0003Qr-UB for bug-coreutils@gnu.org; Fri, 11 Jun 2010 04:33:08 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5B8Wwsu019886 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 11 Jun 2010 08:32:59 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5B2xkUm020215; Fri, 11 Jun 2010 08:32:57 GMT Received: from abhmt018.oracle.com by acsmt353.oracle.com with ESMTP id 338519061276245077; Fri, 11 Jun 2010 01:31:17 -0700 Received: from [10.182.121.3] (/10.182.121.3) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 11 Jun 2010 01:31:16 -0700 Message-ID: <4C11F451.8040304@oracle.com> Date: Fri, 11 Jun 2010 16:31:13 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Sunil Mushran , Paul Eggert , Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1183FD.9010502@oracle.com> In-Reply-To: <4C1183FD.9010502@oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090208.4C11F4BC.00FF:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: submit Cc: Tao Ma , "linux-ext4@vger.kernel.org" , bug-coreutils@gnu.org, Joel Becker , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) Sunil Mushran wrote: > On 06/10/2010 04:47 PM, Paul Eggert wrote: >> On 06/09/2010 11:56 PM, jeff.liu wrote: >> >>> Yeah, I just realized that the behaviour I observed is caused by the >>> delay allocation mechanism of >>> the particular FS. >>> >> If the file system is using delayed allocation, then can >> the fiemap ioctl tell us that a file contains a hole (because nothing >> has been >> allocated there), but read() would tell us that the file contains >> nonzero data at the same location >> (because it's sitting in a buffer somewhere)? If so, we'd need to do >> something like invoke >> fdatasync() on the file before issuing the fiemap ioctl, to force >> allocation; or perhaps >> there's another ioctl that will do the allocation without having to >> actually do a sync. >> > > I guess we'll have to use FIEMAP_FLAG_SYNC. Hi Sunil, Thanks for the comments. So we can ensure the source file synced before mapping in this way. Hi Jim and Paul, How about the tiny patch below? >From d6d619a169ff68a9a310a69d8089b9fbf83b5f91 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Fri, 11 Jun 2010 16:29:02 +0800 Subject: [PATCH 1/1] copy.c: add FIEMAP_FLAG_SYNC to fiemap ioctl * src/copy.c (fiemap_copy): Force kernel to sync the source file before mapping. Signed-off-by: Jie Liu --- src/copy.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/src/copy.c b/src/copy.c index f149be4..f48c74d 100644 --- a/src/copy.c +++ b/src/copy.c @@ -191,6 +191,7 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, do { fiemap->fm_length = FIEMAP_MAX_OFFSET; + fiemap->fm_flags = FIEMAP_FLAG_SYNC; fiemap->fm_extent_count = count; /* When ioctl(2) fails, fall back to the normal copy only if it -- 1.5.4.3 Thanks, -Jeff > >> There's also the issue of copying from a file at the same time that >> some other process >> is writing to it, but that is allowed to produce ill-defined >> behavior. I'm more worried >> about the case where some other process writes to the source file just >> before 'cp' starts. >> > > cp's behavior with active files is undefined. But we know it reads from > offset 0 to MAX. With fiemap it will continue to do the same with the > exception that it will skip reads (and thus writes) depending on the extent > map it gets at the very beginning. > >> (Sorry, I haven't had time yet to dive into the proposed change; I'm >> still trying to understand >> the environment.) >> >> One other thing: Solaris 10 supports lseek with the SEEK_HOLE and >> SEEK_DATA options, which >> are easier to use and which (as far as I can tell from the manual) >> shouldn't require anything >> fdatasync-ish. Any objection if I propose support for that too? It >> is supposed to work >> with ZFS, something I can test here. >> > > There is no plan to implement SEEK_HOLE/SEEK_DATA in the kernel. > At most glibc will use fiemap to extend lseek(). BTW, SEEK_HOLE/DATA > also have the same problem with active files. > > ccing linux-ext4. > > > -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 11 08:43:58 2010 Received: (at submit) by debbugs.gnu.org; 11 Jun 2010 12:43:58 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ON3aY-000732-G8 for submit@debbugs.gnu.org; Fri, 11 Jun 2010 08:43:58 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ON3aV-00072x-Od for submit@debbugs.gnu.org; Fri, 11 Jun 2010 08:43:56 -0400 Received: from lists.gnu.org ([199.232.76.165]:49107) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1ON3aQ-0001WX-7I for submit@debbugs.gnu.org; Fri, 11 Jun 2010 08:43:50 -0400 Received: from [140.186.70.92] (port=39800 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1ON3aO-0004vr-KA for bug-coreutils@gnu.org; Fri, 11 Jun 2010 08:43:49 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1ON3V4-0005On-JN for bug-coreutils@gnu.org; Fri, 11 Jun 2010 08:38:19 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:58473) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ON3V4-0005Nu-2H for bug-coreutils@gnu.org; Fri, 11 Jun 2010 08:38:18 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id 832F5940122; Fri, 11 Jun 2010 14:38:09 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 1384D58B; Fri, 11 Jun 2010 14:38:08 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4C11F451.8040304@oracle.com> (jeff liu's message of "Fri, 11 Jun 2010 16:31:13 +0800") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1183FD.9010502@oracle.com> <4C11F451.8040304@oracle.com> Date: Fri, 11 Jun 2010 14:38:08 +0200 Message-ID: <87typ9iw27.fsf@meyering.net> Lines: 80 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , "linux-ext4@vger.kernel.org" , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) jeff.liu wrote: > Sunil Mushran wrote: ... >> I guess we'll have to use FIEMAP_FLAG_SYNC. > Hi Sunil, > > Thanks for the comments. > So we can ensure the source file synced before mapping in this way. > > Hi Jim and Paul, > > How about the tiny patch below? ... > Subject: [PATCH 1/1] copy.c: add FIEMAP_FLAG_SYNC to fiemap ioctl > > * src/copy.c (fiemap_copy): Force kernel to sync the source > file before mapping. > > Signed-off-by: Jie Liu > --- > src/copy.c | 1 + > 1 files changed, 1 insertions(+), 0 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index f149be4..f48c74d 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -191,6 +191,7 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, > do > { > fiemap->fm_length = FIEMAP_MAX_OFFSET; > + fiemap->fm_flags = FIEMAP_FLAG_SYNC; > fiemap->fm_extent_count = count; Thanks to both of you. That patch looks fine, Jeff. However, at least with ext4 and fedora 13, I require this change to pass "make check". Note the new entry in the "flags" column: ==> ff2 <== Filesystem type is: ef53 File size of j2 is 2048 (1 block, blocksize 4096) ext logical physical expected length flags 0 0 0 1 unknown,delalloc,eof j2: 1 extent found So I will apply the test-fixing patch first, then your change, Jeff, and then rebase to the latest on master. >From 484903dc41246cb3c774f178f695725561b105a0 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Fri, 11 Jun 2010 14:34:03 +0200 Subject: [PATCH 1/2] tests: accommodate varying filefrag -v "flags" output * tests/cp/sparse-fiemap: Accommodate values other than "eof" in the "flags" column of filefrag -v output --- tests/cp/sparse-fiemap | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index dc0cf60..b6b1103 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -68,10 +68,11 @@ $PERL -e 1 || skip_test_ 'skipping part of this test; you lack perl' # Extract logical block number and length pairs from filefrag -v output. # The initial sed is to remove the "eof" from the normally-empty "flags" field. +# Similarly, remove flags values like "unknown,delalloc,eof". # That is required when that final extent has no number in the "expected" field. f() { - sed 's/ eof$//' $@ \ + sed 's/ [a-z,][a-z,]*$//' $@ \ | awk '/^ *[0-9]/ {printf "%d %d ", $2 ,NF < 5 ? $NF : $5 } END {print ""}' } -- 1.7.1.501.g23b46 From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 11 10:53:01 2010 Received: (at submit) by debbugs.gnu.org; 11 Jun 2010 14:53:01 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ON5bR-00080h-4Y for submit@debbugs.gnu.org; Fri, 11 Jun 2010 10:53:01 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ON5bO-00080b-LU for submit@debbugs.gnu.org; Fri, 11 Jun 2010 10:52:59 -0400 Received: from lists.gnu.org ([199.232.76.165]:37647) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1ON5bJ-0004Ai-Mb for submit@debbugs.gnu.org; Fri, 11 Jun 2010 10:52:53 -0400 Received: from [140.186.70.92] (port=60551 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1ON5bI-0003E4-6E for bug-coreutils@gnu.org; Fri, 11 Jun 2010 10:52:53 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1ON5bG-0001lE-2k for bug-coreutils@gnu.org; Fri, 11 Jun 2010 10:52:51 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:44310) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ON5bF-0001kq-HI for bug-coreutils@gnu.org; Fri, 11 Jun 2010 10:52:50 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id A32ED940034; Fri, 11 Jun 2010 16:52:41 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 7A4139DE; Fri, 11 Jun 2010 16:52:40 +0200 (CEST) From: Jim Meyering To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4C11798B.6090500@cs.ucla.edu> (Paul Eggert's message of "Thu, 10 Jun 2010 16:47:23 -0700") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> Date: Fri, 11 Jun 2010 16:52:40 +0200 Message-ID: <878w6liptz.fsf@meyering.net> Lines: 27 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Paul Eggert wrote: > On 06/09/2010 11:56 PM, jeff.liu wrote: > >> Yeah, I just realized that the behaviour I observed is caused by the delay allocation mechanism of >> the particular FS. > > If the file system is using delayed allocation, then can > the fiemap ioctl tell us that a file contains a hole (because nothing has been > allocated there), but read() would tell us that the file contains nonzero data at the same location > (because it's sitting in a buffer somewhere)? If so, we'd need to do something like invoke > fdatasync() on the file before issuing the fiemap ioctl, to force allocation; or perhaps > there's another ioctl that will do the allocation without having to actually do a sync. > > There's also the issue of copying from a file at the same time that some other process > is writing to it, but that is allowed to produce ill-defined behavior. I'm more worried > about the case where some other process writes to the source file just before 'cp' starts. > > (Sorry, I haven't had time yet to dive into the proposed change; I'm still trying to understand > the environment.) > > One other thing: Solaris 10 supports lseek with the SEEK_HOLE and SEEK_DATA options, which > are easier to use and which (as far as I can tell from the manual) shouldn't require anything > fdatasync-ish. Any objection if I propose support for that too? It is supposed to work > with ZFS, something I can test here. Sure, having an implementation that also works on ZFS would be welcome. I updated (non-fast-forward) the fiemap-copy branch an hour or so ago. From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 11 11:55:44 2010 Received: (at submit) by debbugs.gnu.org; 11 Jun 2010 15:55:44 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ON6a7-0008Ue-4x for submit@debbugs.gnu.org; Fri, 11 Jun 2010 11:55:44 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ON4pz-0007ep-GR for submit@debbugs.gnu.org; Fri, 11 Jun 2010 10:04:04 -0400 Received: from lists.gnu.org ([199.232.76.165]:34705) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1ON4po-0003Lu-Kl for submit@debbugs.gnu.org; Fri, 11 Jun 2010 10:03:48 -0400 Received: from [140.186.70.92] (port=58210 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1ON4pj-0001TA-Cj for bug-coreutils@gnu.org; Fri, 11 Jun 2010 10:03:47 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1ON4pc-0000Zi-4e for bug-coreutils@gnu.org; Fri, 11 Jun 2010 10:03:41 -0400 Received: from mx1.redhat.com ([209.132.183.28]:46163) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ON4pb-0000ZJ-S6 for bug-coreutils@gnu.org; Fri, 11 Jun 2010 10:03:36 -0400 Received: from int-mx08.intmail.prod.int.phx2.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by mx1.redhat.com (8.13.8/8.13.8) with ESMTP id o5BE3XVT015286 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 11 Jun 2010 10:03:33 -0400 Received: from liberator.sandeen.net (ovpn01.gateway.prod.ext.phx2.redhat.com [10.5.9.1]) by int-mx08.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id o5BE3TTD023015 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 11 Jun 2010 10:03:31 -0400 Message-ID: <4C124231.7020304@redhat.com> Date: Fri, 11 Jun 2010 09:03:29 -0500 From: Eric Sandeen User-Agent: Thunderbird 2.0.0.24 (Macintosh/20100228) MIME-Version: 1.0 To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1183FD.9010502@oracle.com> <4C11F451.8040304@oracle.com> In-Reply-To: <4C11F451.8040304@oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.67 on 10.5.11.21 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.9 (-----) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Fri, 11 Jun 2010 11:55:41 -0400 Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , Tao Ma , "linux-ext4@vger.kernel.org" , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.9 (-----) jeff.liu wrote: > Sunil Mushran wrote: >> On 06/10/2010 04:47 PM, Paul Eggert wrote: >>> On 06/09/2010 11:56 PM, jeff.liu wrote: >>> >>>> Yeah, I just realized that the behaviour I observed is caused by the >>>> delay allocation mechanism of >>>> the particular FS. >>>> >>> If the file system is using delayed allocation, then can >>> the fiemap ioctl tell us that a file contains a hole (because nothing >>> has been >>> allocated there), but read() would tell us that the file contains >>> nonzero data at the same location >>> (because it's sitting in a buffer somewhere)? If so, we'd need to do >>> something like invoke >>> fdatasync() on the file before issuing the fiemap ioctl, to force >>> allocation; or perhaps >>> there's another ioctl that will do the allocation without having to >>> actually do a sync. >>> >> I guess we'll have to use FIEMAP_FLAG_SYNC. > Hi Sunil, > > Thanks for the comments. > So we can ensure the source file synced before mapping in this way. > > Hi Jim and Paul, > > How about the tiny patch below? I agree that this is needed, thanks. -Eric > From d6d619a169ff68a9a310a69d8089b9fbf83b5f91 Mon Sep 17 00:00:00 2001 > From: Jie Liu > Date: Fri, 11 Jun 2010 16:29:02 +0800 > Subject: [PATCH 1/1] copy.c: add FIEMAP_FLAG_SYNC to fiemap ioctl > > * src/copy.c (fiemap_copy): Force kernel to sync the source > file before mapping. > > Signed-off-by: Jie Liu > --- > src/copy.c | 1 + > 1 files changed, 1 insertions(+), 0 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index f149be4..f48c74d 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -191,6 +191,7 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, > do > { > fiemap->fm_length = FIEMAP_MAX_OFFSET; > + fiemap->fm_flags = FIEMAP_FLAG_SYNC; > fiemap->fm_extent_count = count; > > /* When ioctl(2) fails, fall back to the normal copy only if it From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 12 23:38:45 2010 Received: (at submit) by debbugs.gnu.org; 13 Jun 2010 03:38:45 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ONe21-0008GD-0o for submit@debbugs.gnu.org; Sat, 12 Jun 2010 23:38:45 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ONe1z-0008G8-Dh for submit@debbugs.gnu.org; Sat, 12 Jun 2010 23:38:43 -0400 Received: from lists.gnu.org ([199.232.76.165]:35855) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1ONe1v-0004lL-Ub for submit@debbugs.gnu.org; Sat, 12 Jun 2010 23:38:40 -0400 Received: from [140.186.70.92] (port=55901 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1ONe1u-0002f8-CX for bug-coreutils@gnu.org; Sat, 12 Jun 2010 23:38:39 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1ONe1t-0002Nv-3J for bug-coreutils@gnu.org; Sat, 12 Jun 2010 23:38:38 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:20307) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ONe1s-0002Nk-Tr for bug-coreutils@gnu.org; Sat, 12 Jun 2010 23:38:37 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5D3cSB2032168 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sun, 13 Jun 2010 03:38:30 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5D3cRZY018868; Sun, 13 Jun 2010 03:38:27 GMT Received: from abhmt008.oracle.com by acsmt355.oracle.com with ESMTP id 320382051276400246; Sat, 12 Jun 2010 20:37:26 -0700 Received: from [221.223.96.56] (/221.223.96.56) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sat, 12 Jun 2010 20:37:26 -0700 Message-ID: <4C14526E.9030103@oracle.com> Date: Sun, 13 Jun 2010 11:37:18 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Eric Sandeen Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1183FD.9010502@oracle.com> <4C11F451.8040304@oracle.com> <4C124231.7020304@redhat.com> In-Reply-To: <4C124231.7020304@redhat.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090201.4C1452B7.014C:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.6 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , Tao Ma , "linux-ext4@vger.kernel.org" , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.6 (-----) Eric Sandeen wrote: > jeff.liu wrote: >> Sunil Mushran wrote: >>> On 06/10/2010 04:47 PM, Paul Eggert wrote: >>>> On 06/09/2010 11:56 PM, jeff.liu wrote: >>>> >>>>> Yeah, I just realized that the behaviour I observed is caused by the >>>>> delay allocation mechanism of >>>>> the particular FS. >>>>> >>>> If the file system is using delayed allocation, then can >>>> the fiemap ioctl tell us that a file contains a hole (because nothing >>>> has been >>>> allocated there), but read() would tell us that the file contains >>>> nonzero data at the same location >>>> (because it's sitting in a buffer somewhere)? If so, we'd need to do >>>> something like invoke >>>> fdatasync() on the file before issuing the fiemap ioctl, to force >>>> allocation; or perhaps >>>> there's another ioctl that will do the allocation without having to >>>> actually do a sync. >>>> >>> I guess we'll have to use FIEMAP_FLAG_SYNC. >> Hi Sunil, >> >> Thanks for the comments. >> So we can ensure the source file synced before mapping in this way. >> >> Hi Jim and Paul, >> >> How about the tiny patch below? > > I agree that this is needed, thanks. Thanks for your confirming response. Regards, -Jeff > > -Eric > >> From d6d619a169ff68a9a310a69d8089b9fbf83b5f91 Mon Sep 17 00:00:00 2001 >> From: Jie Liu >> Date: Fri, 11 Jun 2010 16:29:02 +0800 >> Subject: [PATCH 1/1] copy.c: add FIEMAP_FLAG_SYNC to fiemap ioctl >> >> * src/copy.c (fiemap_copy): Force kernel to sync the source >> file before mapping. >> >> Signed-off-by: Jie Liu >> --- >> src/copy.c | 1 + >> 1 files changed, 1 insertions(+), 0 deletions(-) >> >> diff --git a/src/copy.c b/src/copy.c >> index f149be4..f48c74d 100644 >> --- a/src/copy.c >> +++ b/src/copy.c >> @@ -191,6 +191,7 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, >> do >> { >> fiemap->fm_length = FIEMAP_MAX_OFFSET; >> + fiemap->fm_flags = FIEMAP_FLAG_SYNC; >> fiemap->fm_extent_count = count; >> >> /* When ioctl(2) fails, fall back to the normal copy only if it > > > > -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 13 10:54:23 2010 Received: (at submit) by debbugs.gnu.org; 13 Jun 2010 14:54:23 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ONoZr-0005EY-7j for submit@debbugs.gnu.org; Sun, 13 Jun 2010 10:54:23 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ONoZo-0005ET-KL for submit@debbugs.gnu.org; Sun, 13 Jun 2010 10:54:21 -0400 Received: from lists.gnu.org ([199.232.76.165]:46198) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1ONoZj-0003uX-Ov for submit@debbugs.gnu.org; Sun, 13 Jun 2010 10:54:15 -0400 Received: from [140.186.70.92] (port=56766 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1ONoZi-0007N1-8L for bug-coreutils@gnu.org; Sun, 13 Jun 2010 10:54:15 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1ONoZg-00084X-Jz for bug-coreutils@gnu.org; Sun, 13 Jun 2010 10:54:14 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:46987) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ONoZg-00083s-1U for bug-coreutils@gnu.org; Sun, 13 Jun 2010 10:54:12 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id 8910B9400E5; Sun, 13 Jun 2010 16:54:03 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id E6C5A109DA; Sun, 13 Jun 2010 16:53:56 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <87typ9iw27.fsf@meyering.net> (Jim Meyering's message of "Fri, 11 Jun 2010 14:38:08 +0200") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1183FD.9010502@oracle.com> <4C11F451.8040304@oracle.com> <87typ9iw27.fsf@meyering.net> Date: Sun, 13 Jun 2010 16:53:56 +0200 Message-ID: <87mxuzkmpn.fsf@meyering.net> Lines: 58 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Chris Mason , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Jim Meyering wrote: ... > So I will apply the test-fixing patch first, then your change, > Jeff, and then rebase to the latest on master. FYI, I did that. In addition, I needed two more changes in order to avoid "make distcheck" failure: >From be5548445e90a36ab5018cac0fb19f2498d0521c Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sun, 13 Jun 2010 16:19:29 +0200 Subject: [PATCH 1/2] build: distribute new file, fiemap.h * src/Makefile.am (noinst_HEADERS): Add fiemap.h. --- src/Makefile.am | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/src/Makefile.am b/src/Makefile.am index 0630a06..7d56312 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -145,6 +145,7 @@ noinst_HEADERS = \ copy.h \ cp-hash.h \ dircolors.h \ + fiemap.h \ fs.h \ group-list.h \ ls.h \ -- 1.7.1.511.g2f531 >From f25181d32c40f82ee26dea6de6b7f4b385352a14 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sun, 13 Jun 2010 16:34:42 +0200 Subject: [PATCH 2/2] build: distribute new test script, filefrag-extent-compare * tests/Makefile.am (EXTRA_DIST): Add filefrag-extent-compare. --- tests/Makefile.am | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/tests/Makefile.am b/tests/Makefile.am index f7840c8..61ccf01 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -10,6 +10,7 @@ EXTRA_DIST = \ CuTmpdir.pm \ check.mk \ envvar-check \ + filefrag-extent-compare \ init.sh \ lang-default \ other-fs-tmpdir \ -- 1.7.1.511.g2f531 From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 15 17:09:35 2010 Received: (at submit) by debbugs.gnu.org; 15 Jun 2010 21:09:35 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdO2-0005xg-T8 for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:09:35 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdO0-0005xb-F3 for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:09:33 -0400 Received: from lists.gnu.org ([199.232.76.165]:36830) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOdNv-0006jt-W1 for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:09:28 -0400 Received: from [140.186.70.92] (port=34569 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOdNu-0000jA-Mp for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:09:27 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOdNt-0007BN-F7 for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:09:26 -0400 Received: from kiwi.cs.ucla.edu ([131.179.128.19]:51894) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdNt-0007Ax-3m for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:09:25 -0400 Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200]) by kiwi.cs.ucla.edu (8.13.8+Sun/8.13.8/UCLACS-6.0) with ESMTP id o5FL9M36004936; Tue, 15 Jun 2010 14:09:22 -0700 (PDT) Message-ID: <4C17EC02.7040505@cs.ucla.edu> Date: Tue, 15 Jun 2010 14:09:22 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1183FD.9010502@oracle.com> <4C11F451.8040304@oracle.com> In-Reply-To: <4C11F451.8040304@oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.7 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , Tao Ma , "linux-ext4@vger.kernel.org" , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.8 (----) On 06/11/2010 01:31 AM, jeff.liu wrote: > + fiemap->fm_flags = FIEMAP_FLAG_SYNC; If I'm reading the Linux source code correctly, this forces all the dirty blocks in the input file to disk, and forces the kernel to wait until all those blocks actually hit the disk. We don't need that; there shouldn't be a need to force any blocks to hit the disk. All that 'cp' needs to know is: please tell me the next block which might contain nonzero data. Is there some way that we can get that information without forcing the input file's blocks to disk? From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 15 17:11:12 2010 Received: (at submit) by debbugs.gnu.org; 15 Jun 2010 21:11:12 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdPc-0005ym-AD for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:11:12 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdPa-0005yh-GX for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:11:10 -0400 Received: from lists.gnu.org ([199.232.76.165]:36851) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOdPW-0006lZ-1h for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:11:06 -0400 Received: from [140.186.70.92] (port=55092 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOdPT-0000lM-VV for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:11:05 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOdPS-0007Yr-Ho for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:11:03 -0400 Received: from kiwi.cs.ucla.edu ([131.179.128.19]:51918) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdPS-0007Yj-9G for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:11:02 -0400 Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200]) by kiwi.cs.ucla.edu (8.13.8+Sun/8.13.8/UCLACS-6.0) with ESMTP id o5FLB189004972; Tue, 15 Jun 2010 14:11:01 -0700 (PDT) Message-ID: <4C17EC65.6040406@cs.ucla.edu> Date: Tue, 15 Jun 2010 14:11:01 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: Sunil Mushran Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1183FD.9010502@oracle.com> In-Reply-To: <4C1183FD.9010502@oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.9 (----) X-Debbugs-Envelope-To: submit Cc: bug-coreutils@gnu.org, Jim Meyering , Joel Becker , "jeff.liu" , Tao Ma , "linux-ext4@vger.kernel.org" , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.9 (----) Sunil Mushran wrote: > SEEK_HOLE/DATA also have the same problem with active files. Yes, that's true if 'cp' is copying a file while someone else is writing to it. But the case we're worried about is when 'cp' starts copying a file immediately after someone else has finished writing to it but data has not been sent to disk; in that case SEEK_HOLE/DATA should work just as well as the fiemap ioctl. From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 15 17:12:02 2010 Received: (at submit) by debbugs.gnu.org; 15 Jun 2010 21:12:04 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdQQ-0005zG-Ir for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:12:02 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdQP-0005z1-1R for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:12:01 -0400 Received: from lists.gnu.org ([199.232.76.165]:36875) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOdQL-0006mA-2P for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:11:57 -0400 Received: from [140.186.70.92] (port=55189 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOdQJ-0000m9-JL for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:11:56 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOdQI-0007fD-LA for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:11:55 -0400 Received: from kiwi.cs.ucla.edu ([131.179.128.19]:51927) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdQI-0007f6-Bk for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:11:54 -0400 Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200]) by kiwi.cs.ucla.edu (8.13.8+Sun/8.13.8/UCLACS-6.0) with ESMTP id o5FLBrlx004982; Tue, 15 Jun 2010 14:11:53 -0700 (PDT) Message-ID: <4C17EC99.4020908@cs.ucla.edu> Date: Tue, 15 Jun 2010 14:11:53 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: Tao Ma Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> In-Reply-To: <4C1184B4.2050309@oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.9 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , "jeff.liu" , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.0 (-----) On 06/10/2010 05:35 PM, Tao Ma wrote: > there is a flag FIEMAP_EXTENT_DELALLOC in fiemap ... > with dd if=/dev/zero of=/mnt/ext4/a bs=1M count=1 seek=1 > the ext4 can't return a valid fiemap extent. Hmm, this sounds like a fairly serious bug, in that it would prevent this part of cp from working. What does the fiemap ioctl return in this buggy case? Is there some way that cp can detect the bug, and report it to the user, or work around it? Or should we just assume that the bug will get fixed and that cp shouldn't worry about it? From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 15 17:28:55 2010 Received: (at submit) by debbugs.gnu.org; 15 Jun 2010 21:28:56 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdgk-00066R-Qb for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:28:55 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdgi-00066L-Ih for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:28:53 -0400 Received: from lists.gnu.org ([199.232.76.165]:60084) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOdge-00074g-FT for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:28:48 -0400 Received: from [140.186.70.92] (port=50205 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOdgc-0005qw-UM for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:28:47 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOdgb-0001jU-QH for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:28:46 -0400 Received: from kiwi.cs.ucla.edu ([131.179.128.19]:52016) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdgb-0001jD-Es for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:28:45 -0400 Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200]) by kiwi.cs.ucla.edu (8.13.8+Sun/8.13.8/UCLACS-6.0) with ESMTP id o5FLShuB005143; Tue, 15 Jun 2010 14:28:43 -0700 (PDT) Message-ID: <4C17F08B.4020203@cs.ucla.edu> Date: Tue, 15 Jun 2010 14:28:43 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: Tao Ma Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> In-Reply-To: <4C1184B4.2050309@oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , "jeff.liu" , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.0 (-----) I looked at the proposed fiemap patch for cp and see another issue with it, which needs to be thought through more carefully. The proposed fiemap code tries to copy the _physical_ holes in the source file. But currently cp copies the _logical_ holes: that is, it skips over writing (and thus attempts to create physical holes) whenever the source file contains a block of zeros, regardless of whether that block of zeros is physically stored in the file system. The current cp behavior is often the desired behavior (particularly for cp --sparse=always) and the fiemap code shouldn't alter that. For example, if a fiemap_extent has the FIEMAP_EXTENT_UNWRITTEN flag set, cp should treat that as a hole, because the extent is all zeros. (This will greatly help performance in some cases.) Also, if an input extent is read and a block of it is found to be zeros, cp should skip over that block when writing. If the proposed fiemap code was intended to copy the physical holes, then the code wasn't successful in implementing the intent, as there's no guarantee that the extents of the destination are the same as the extents of the source. If it is useful to have 'cp' copy the physical holes, this could be supported via a new cp option or options, but surely the default should continue to be to copy logical holes. From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 15 17:43:54 2010 Received: (at submit) by debbugs.gnu.org; 15 Jun 2010 21:43:54 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdvG-0006D1-3K for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:43:54 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdvD-0006Cv-IM for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:43:52 -0400 Received: from lists.gnu.org ([199.232.76.165]:49234) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOdv9-0007JC-CI for submit@debbugs.gnu.org; Tue, 15 Jun 2010 17:43:47 -0400 Received: from [140.186.70.92] (port=40675 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOdv7-0002C7-AE for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:43:46 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOdv3-0006N2-67 for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:43:45 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:49527) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOdv2-0006IY-JK for bug-coreutils@gnu.org; Tue, 15 Jun 2010 17:43:41 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id DE03494012E; Tue, 15 Jun 2010 23:43:31 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id B55AAD897; Tue, 15 Jun 2010 23:43:25 +0200 (CEST) From: Jim Meyering To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4C17F08B.4020203@cs.ucla.edu> (Paul Eggert's message of "Tue, 15 Jun 2010 14:28:43 -0700") References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> Date: Tue, 15 Jun 2010 23:43:25 +0200 Message-ID: <87sk4odlaa.fsf@meyering.net> Lines: 32 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Paul Eggert wrote: > I looked at the proposed fiemap patch for cp and see another issue > with it, which needs to be thought through more carefully. > > The proposed fiemap code tries to copy the _physical_ holes in the > source file. But currently cp copies the _logical_ holes: that is, it > skips over writing (and thus attempts to create physical holes) > whenever the source file contains a block of zeros, regardless of > whether that block of zeros is physically stored in the file system. > The current cp behavior is often the desired behavior (particularly > for cp --sparse=always) and the fiemap code shouldn't alter that. > > For example, if a fiemap_extent has the FIEMAP_EXTENT_UNWRITTEN flag > set, cp should treat that as a hole, because the extent is all zeros. > (This will greatly help performance in some cases.) Also, if an input > extent is read and a block of it is found to be zeros, cp should skip > over that block when writing. > > If the proposed fiemap code was intended to copy the physical holes, > then the code wasn't successful in implementing the intent, as there's > no guarantee that the extents of the destination are the same as the > extents of the source. If it is useful to have 'cp' copy the physical > holes, this could be supported via a new cp option or options, but > surely the default should continue to be to copy logical holes. Good point! I think that copying physical holes via FIEMAP should be the default, when possible. One problem is that the current code on the fiemap-copy branch does not honor --sparse=WHEN when in fiemap-copying mode. The solution would seem to be to change the regular-file-copying loop in the fiemap_copy function to use the same hole-preserving code that is used in copy_reg. From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 15 18:12:31 2010 Received: (at submit) by debbugs.gnu.org; 15 Jun 2010 22:12:31 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOeMx-0006P6-Ao for submit@debbugs.gnu.org; Tue, 15 Jun 2010 18:12:31 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOeMv-0006P1-P4 for submit@debbugs.gnu.org; Tue, 15 Jun 2010 18:12:30 -0400 Received: from lists.gnu.org ([199.232.76.165]:47895) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOeMr-0007bc-04 for submit@debbugs.gnu.org; Tue, 15 Jun 2010 18:12:25 -0400 Received: from [140.186.70.92] (port=35739 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOeMo-0003rn-HB for bug-coreutils@gnu.org; Tue, 15 Jun 2010 18:12:23 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOeMN-0003u5-CE for bug-coreutils@gnu.org; Tue, 15 Jun 2010 18:11:56 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:51302) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOeMN-0003sY-7B for bug-coreutils@gnu.org; Tue, 15 Jun 2010 18:11:55 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5FMBlpu015825 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 15 Jun 2010 22:11:48 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5FMBk0l014226; Tue, 15 Jun 2010 22:11:46 GMT Received: from ca-server1.us.oracle.com by acsmt353.oracle.com with ESMTP id 348095041276639821; Tue, 15 Jun 2010 15:10:21 -0700 Received: from jlbec by ca-server1.us.oracle.com with local (Exim 4.69) (envelope-from ) id 1OOeKq-0003Fl-Mc; Tue, 15 Jun 2010 15:10:20 -0700 Date: Tue, 15 Jun 2010 15:10:19 -0700 From: Joel Becker To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy Message-ID: <20100615221019.GC30497@mail.oracle.com> References: <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C17F08B.4020203@cs.ucla.edu> X-Burt-Line: Trees are cool. X-Red-Smith: Ninety feet between bases is perhaps as close as man has ever come to perfection. User-Agent: Mutt/1.5.20 (2009-06-14) X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090209.4C17FAA5.02DE:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.7 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , "jeff.liu" , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.8 (-----) On Tue, Jun 15, 2010 at 02:28:43PM -0700, Paul Eggert wrote: > I looked at the proposed fiemap patch for cp and see another issue > with it, which needs to be thought through more carefully. > > The proposed fiemap code tries to copy the _physical_ holes in the > source file. But currently cp copies the _logical_ holes: that is, it > skips over writing (and thus attempts to create physical holes) > whenever the source file contains a block of zeros, regardless of > whether that block of zeros is physically stored in the file system. > The current cp behavior is often the desired behavior (particularly > for cp --sparse=always) and the fiemap code shouldn't alter that. > > For example, if a fiemap_extent has the FIEMAP_EXTENT_UNWRITTEN flag > set, cp should treat that as a hole, because the extent is all zeros. > (This will greatly help performance in some cases.) Also, if an input > extent is read and a block of it is found to be zeros, cp should skip > over that block when writing. I pretty much agree with everything you wrote above. Fiemap is introduced into cp(1) to optimize the existing --sparse behavior. So we should be holding to that behavior. In fact, I'll go one further. While I haven't looked at the actual cp code in a little bit, I do not think that fiemap is used for the non-sparse copy. I contend that fiemap should be used for all copies if the capability exists. Even if cp is doing a non-sparse copy, there's no need for it to call read(2) on a 1GB hole in the file. It can write zeros just fine on its own. Basically, these capabilities (fiemap, SEEK_HOLE) are ways to avoid reading zeros we can determine without reading and scanning. We should always take advantage of that. > If the proposed fiemap code was intended to copy the physical holes, > then the code wasn't successful in implementing the intent, as there's > no guarantee that the extents of the destination are the same as the > extents of the source. If it is useful to have 'cp' copy the physical > holes, this could be supported via a new cp option or options, but > surely the default should continue to be to copy logical holes. The goal was not physical holes, it was logical ones as you surmise. I can't see any way to ensure the file has identical extents. I don't know any filesystem that guarantees such allocations on write(2). I do think there might be an interesting performance optimization to be had, but it is independent of fiemap. Above, you describe the usual "cp --sparse" behavior of reading allocated data and checking for zeroes. If you're optimizing for space usage -- what "cp --sparse" is defined to do -- you definitely want to skip writing blocks that are all zero. But it actually creates a crappier allocation and I/O pattern for some files. Imagine you have a data extent that is alternating blocks of zeroes and ones: hexdump -C /tmp/blk 00000000 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 |................| * 00001000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00002000 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 01 |................| * 00003000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00004000 ... (continued for 4MB in this example) 00400000 Classic "cp --sparse" will spend CPU time scanning this extent, which can be megabytes in size. It will then write out every other block, which is terribly slow, and it leavs a horribly fragmented destination file. The result is even more disastrous if those holes ever get filled in. I'm, of course, exaggerating a worst-case here. But folks looking at I/O performance would love detection like "here's a linear MB, sure it has some zero blocks, but it's faster to write the entire MB than to chop it up." Joel -- Life's Little Instruction Book #456 "Send your loved one flowers. Think of a reason later." Joel Becker Principal Software Developer Oracle E-mail: joel.becker@oracle.com Phone: (650) 506-8127 From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 15 18:19:27 2010 Received: (at submit) by debbugs.gnu.org; 15 Jun 2010 22:19:27 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOeTe-0006T3-O2 for submit@debbugs.gnu.org; Tue, 15 Jun 2010 18:19:26 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOeTc-0006Sw-D1 for submit@debbugs.gnu.org; Tue, 15 Jun 2010 18:19:25 -0400 Received: from lists.gnu.org ([199.232.76.165]:37743) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOeTY-0007xl-6S for submit@debbugs.gnu.org; Tue, 15 Jun 2010 18:19:20 -0400 Received: from [140.186.70.92] (port=45030 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOeTV-0007NI-Ae for bug-coreutils@gnu.org; Tue, 15 Jun 2010 18:19:19 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOeTU-0007Pv-2S for bug-coreutils@gnu.org; Tue, 15 Jun 2010 18:19:17 -0400 Received: from kiwi.cs.ucla.edu ([131.179.128.19]:52348) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOeTT-0007PU-Ot for bug-coreutils@gnu.org; Tue, 15 Jun 2010 18:19:16 -0400 Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200]) by kiwi.cs.ucla.edu (8.13.8+Sun/8.13.8/UCLACS-6.0) with ESMTP id o5FMJDJl005755; Tue, 15 Jun 2010 15:19:13 -0700 (PDT) Message-ID: <4C17FC61.2020401@cs.ucla.edu> Date: Tue, 15 Jun 2010 15:19:13 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> In-Reply-To: <87sk4odlaa.fsf@meyering.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.1 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.1 (-----) On 06/15/2010 02:43 PM, Jim Meyering wrote: > I think that copying physical holes via FIEMAP should be the default, when > possible. One problem is that the current code on the fiemap-copy branch > does not honor --sparse=WHEN when in fiemap-copying mode. The solution > would seem to be to change the regular-file-copying loop in the fiemap_copy > function to use the same hole-preserving code that is used in copy_reg. I assume that the solution would be used only with cp --sparse=always? (Otherwise, it would amount to copying logical holes by default.) If so, under this proposal, --sparse=always would copy logical holes, --sparse=never would never copy holes, and --sparse=auto (the default) would copy physical holes if supported, else it would copy logical holes if the file seems to have enough physical holes, and otherwise it would copy no holes. (Whew!) But this raises another issue. Under this proposal, the default behavior copies physical _holes_, but it doesn't copy physical _extents_. For example, suppose the input file has a single 128 MiB extent. The proposal could create an output file with two 64-MiB extents that are logically contiguous. This is because the code does not advise the output file about the extents that the input file had, and the operating system might assign a smaller output extent at first, discovering only too late that more space was needed. If the intent is to copy the physical extents, then the proposed code needs to be fixed so that it uses fallocate() to attempt to create the same extents in the destination that existed in the source. Obviously this could fail for any number of reasons, for example, the destination file system might be different from the source; but the goal would be to preserve extents if the underlying OS is "nice" about implementing fallocate(). Which raises another question: should cp attempt to create _better_ extents in the destination than in the source? For example, if the source contains two logically-adjacent 64-MiB extents, should the destination contain just one 128-MiB extent? I can see where that might be worthwile. This all is getting a bit complicated, I'm afraid. Perhaps it'd be better to try this stuff out with a new option, "cp --sparse=extents" say, and keep the default as-is for a while? From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 15 18:30:45 2010 Received: (at submit) by debbugs.gnu.org; 15 Jun 2010 22:30:46 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOeea-0006Z5-UR for submit@debbugs.gnu.org; Tue, 15 Jun 2010 18:30:45 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOeeZ-0006Yz-3f for submit@debbugs.gnu.org; Tue, 15 Jun 2010 18:30:43 -0400 Received: from lists.gnu.org ([199.232.76.165]:43448) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOeeV-0008Fd-0G for submit@debbugs.gnu.org; Tue, 15 Jun 2010 18:30:39 -0400 Received: from [140.186.70.92] (port=54627 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOeeT-0003sl-1l for bug-coreutils@gnu.org; Tue, 15 Jun 2010 18:30:38 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOeeO-0002mK-5s for bug-coreutils@gnu.org; Tue, 15 Jun 2010 18:30:36 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:50578) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOeeO-0002l6-0w for bug-coreutils@gnu.org; Tue, 15 Jun 2010 18:30:32 -0400 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5FMUPV5012741 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 15 Jun 2010 22:30:26 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5FJ4xpU025340; Tue, 15 Jun 2010 22:30:24 GMT Received: from abhmt007.oracle.com by acsmt355.oracle.com with ESMTP id 348126311276641017; Tue, 15 Jun 2010 15:30:17 -0700 Received: from [130.35.68.118] (/130.35.68.118) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 15 Jun 2010 15:30:16 -0700 Message-ID: <4C17FEF7.2030700@oracle.com> Date: Tue, 15 Jun 2010 15:30:15 -0700 From: Sunil Mushran User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> In-Reply-To: <4C17FC61.2020401@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet13.oracle.com [148.87.113.125] X-CT-RefId: str=0001.0A090208.4C17FF02.0217:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.6 (------) X-Debbugs-Envelope-To: submit Cc: bug-coreutils@gnu.org, Jim Meyering , Joel Becker , "jeff.liu" , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) On 06/15/2010 03:19 PM, Paul Eggert wrote: > On 06/15/2010 02:43 PM, Jim Meyering wrote: > > >> I think that copying physical holes via FIEMAP should be the default, when >> possible. One problem is that the current code on the fiemap-copy branch >> does not honor --sparse=WHEN when in fiemap-copying mode. The solution >> would seem to be to change the regular-file-copying loop in the fiemap_copy >> function to use the same hole-preserving code that is used in copy_reg. >> > I assume that the solution would be used only with cp --sparse=always? > (Otherwise, it would amount to copying logical holes by default.) > > If so, under this proposal, --sparse=always would copy logical holes, > --sparse=never would never copy holes, and --sparse=auto (the default) > would copy physical holes if supported, else it would copy logical > holes if the file seems to have enough physical holes, and otherwise > it would copy no holes. (Whew!) > > But this raises another issue. Under this proposal, the default behavior > copies physical _holes_, but it doesn't copy physical _extents_. > For example, suppose the input file has a single 128 MiB extent. The > proposal could create an output file with two 64-MiB extents > that are logically contiguous. This is because the code does not advise the > output file about the extents that the input file had, and the operating > system might assign a smaller output extent at first, discovering only too late > that more space was needed. > > If the intent is to copy the physical extents, then the proposed code > needs to be fixed so that it uses fallocate() to attempt to create the > same extents in the destination that existed in the source. Obviously > this could fail for any number of reasons, for example, the destination > file system might be different from the source; but the goal would be > to preserve extents if the underlying OS is "nice" about implementing > fallocate(). > > Which raises another question: should cp attempt to create _better_ > extents in the destination than in the source? For example, if > the source contains two logically-adjacent 64-MiB extents, should the > destination contain just one 128-MiB extent? I can see where that > might be worthwile. > > This all is getting a bit complicated, I'm afraid. Perhaps it'd be better > to try this stuff out with a new option, "cp --sparse=extents" say, and > keep the default as-is for a while? > Expecting to control the number of extents in a file in any file system is expecting too much of the fs. No file system guarantees that. fallocate() only allocates the space. It does not give any guarantees for the number of extents. From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 15 18:42:21 2010 Received: (at submit) by debbugs.gnu.org; 15 Jun 2010 22:42:21 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOepp-0006f2-7Y for submit@debbugs.gnu.org; Tue, 15 Jun 2010 18:42:21 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOepm-0006et-Bt for submit@debbugs.gnu.org; Tue, 15 Jun 2010 18:42:19 -0400 Received: from lists.gnu.org ([199.232.76.165]:36730) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOepi-0008OM-By for submit@debbugs.gnu.org; Tue, 15 Jun 2010 18:42:14 -0400 Received: from [140.186.70.92] (port=57713 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOepg-0001Um-Dx for bug-coreutils@gnu.org; Tue, 15 Jun 2010 18:42:13 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOepf-0006Sq-8U for bug-coreutils@gnu.org; Tue, 15 Jun 2010 18:42:12 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:35088) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOepf-0006SW-2h for bug-coreutils@gnu.org; Tue, 15 Jun 2010 18:42:11 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5FMg5th029302 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 15 Jun 2010 22:42:07 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5FM7aUP003739; Tue, 15 Jun 2010 22:42:04 GMT Received: from ca-server1.us.oracle.com by acsmt354.oracle.com with ESMTP id 328402911276641671; Tue, 15 Jun 2010 15:41:11 -0700 Received: from jlbec by ca-server1.us.oracle.com with local (Exim 4.69) (envelope-from ) id 1OOeog-0003JT-SA; Tue, 15 Jun 2010 15:41:10 -0700 Date: Tue, 15 Jun 2010 15:41:10 -0700 From: Joel Becker To: Sunil Mushran Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy Message-ID: <20100615224110.GD30497@mail.oracle.com> References: <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <4C17FEF7.2030700@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C17FEF7.2030700@oracle.com> X-Burt-Line: Trees are cool. X-Red-Smith: Ninety feet between bases is perhaps as close as man has ever come to perfection. User-Agent: Mutt/1.5.20 (2009-06-14) X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090201.4C1801BF.022C:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.0 (------) X-Debbugs-Envelope-To: submit Cc: Paul Eggert , bug-coreutils@gnu.org, Jim Meyering , "jeff.liu" , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.1 (------) On Tue, Jun 15, 2010 at 03:30:15PM -0700, Sunil Mushran wrote: > On 06/15/2010 03:19 PM, Paul Eggert wrote: > >This all is getting a bit complicated, I'm afraid. Perhaps it'd be better > >to try this stuff out with a new option, "cp --sparse=extents" say, and > >keep the default as-is for a while? > > Expecting to control the number of extents in a file in any file system > is expecting too much of the fs. No file system guarantees that. > fallocate() only allocates the space. It does not give any guarantees > for the number of extents. I don't think it's ever interesting to try and mimic the source file's extent pattern. Really, what's the point? I have done a lot of administration in my time, and I've worked with a lot of people. I've never needed nor heard anyone ask for this capability. What people want is "--sparse=always, but not slow". Either they know their input file has 100MB holes, and they want their destination to have 100MB holes or they have a 100MB file that has few zeros, and they wouldn't mind those zeros being unallocated. What they really don't want is to wait while the CPU churns figuring this out. When people choose "--sparse=auto", they don't mean they want non-sparse files; they just want to avoid lots of work for a file that isn't very sparse. With fiemap, cp can give them the best of both worlds. Joel -- "You must remember this: A kiss is just a kiss, A sigh is just a sigh. The fundamental rules apply As time goes by." Joel Becker Principal Software Developer Oracle E-mail: joel.becker@oracle.com Phone: (650) 506-8127 From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 15 19:24:54 2010 Received: (at submit) by debbugs.gnu.org; 15 Jun 2010 23:24:54 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOfUz-0006x3-Rb for submit@debbugs.gnu.org; Tue, 15 Jun 2010 19:24:54 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOfUv-0006ww-Mr for submit@debbugs.gnu.org; Tue, 15 Jun 2010 19:24:53 -0400 Received: from lists.gnu.org ([199.232.76.165]:55811) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOfUX-0000cH-0R for submit@debbugs.gnu.org; Tue, 15 Jun 2010 19:24:25 -0400 Received: from [140.186.70.92] (port=47101 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOfTh-0007HE-PG for bug-coreutils@gnu.org; Tue, 15 Jun 2010 19:24:24 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOfTa-00074J-GO for bug-coreutils@gnu.org; Tue, 15 Jun 2010 19:23:28 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:8387) by eggs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1OOfTa-000745-5z for bug-coreutils@gnu.org; Tue, 15 Jun 2010 19:23:26 -0400 Received: (qmail 22914 invoked from network); 15 Jun 2010 23:23:24 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 15 Jun 2010 23:23:24 -0000 Message-ID: <4C180B21.4050704@draigBrady.com> Date: Wed, 16 Jun 2010 00:22:09 +0100 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> In-Reply-To: <4C17FC61.2020401@cs.ucla.edu> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9 X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.7 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , "jeff.liu" , Chris Mason , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.8 (----) On 15/06/10 23:19, Paul Eggert wrote: > On 06/15/2010 02:43 PM, Jim Meyering wrote: > >> I think that copying physical holes via FIEMAP should be the default, when >> possible. One problem is that the current code on the fiemap-copy branch >> does not honor --sparse=WHEN when in fiemap-copying mode. The solution >> would seem to be to change the regular-file-copying loop in the fiemap_copy >> function to use the same hole-preserving code that is used in copy_reg. > > I assume that the solution would be used only with cp --sparse=always? > (Otherwise, it would amount to copying logical holes by default.) > > If so, under this proposal, --sparse=always would copy logical holes, > --sparse=never would never copy holes, and --sparse=auto (the default) > would copy physical holes if supported, else it would copy logical > holes if the file seems to have enough physical holes, and otherwise > it would copy no holes. (Whew!) > > But this raises another issue. Under this proposal, the default behavior > copies physical _holes_, but it doesn't copy physical _extents_. > For example, suppose the input file has a single 128 MiB extent. The > proposal could create an output file with two 64-MiB extents > that are logically contiguous. This is because the code does not advise the > output file about the extents that the input file had, and the operating > system might assign a smaller output extent at first, discovering only too late > that more space was needed. > > If the intent is to copy the physical extents, then the proposed code > needs to be fixed so that it uses fallocate() to attempt to create the > same extents in the destination that existed in the source. Obviously > this could fail for any number of reasons, for example, the destination > file system might be different from the source; but the goal would be > to preserve extents if the underlying OS is "nice" about implementing > fallocate(). > > Which raises another question: should cp attempt to create _better_ > extents in the destination than in the source? For example, if > the source contains two logically-adjacent 64-MiB extents, should the > destination contain just one 128-MiB extent? I can see where that > might be worthwile. > > This all is getting a bit complicated, I'm afraid. Perhaps it'd be better > to try this stuff out with a new option, "cp --sparse=extents" say, and > keep the default as-is for a while? Well I previously had a patch to call fallocate() to alloc the largest extents possible, which I think is sensible to do by default. I didn't apply it as I was waiting for a change or even a comment on the weird fallocate() interface (which we can forget about this stage). http://lists.gnu.org/archive/html/bug-coreutils/2009-05/msg00260.html cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 16 01:46:08 2010 Received: (at submit) by debbugs.gnu.org; 16 Jun 2010 05:46:09 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOlRw-0001QF-30 for submit@debbugs.gnu.org; Wed, 16 Jun 2010 01:46:08 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOlRu-0001Ps-9Y for submit@debbugs.gnu.org; Wed, 16 Jun 2010 01:46:07 -0400 Received: from lists.gnu.org ([199.232.76.165]:42963) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOlRq-0005vb-9y for submit@debbugs.gnu.org; Wed, 16 Jun 2010 01:46:02 -0400 Received: from [140.186.70.92] (port=60837 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOlRo-0007xd-2m for bug-coreutils@gnu.org; Wed, 16 Jun 2010 01:46:01 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOlRm-0007RL-9T for bug-coreutils@gnu.org; Wed, 16 Jun 2010 01:45:59 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:24054) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOlRm-0007RB-4X for bug-coreutils@gnu.org; Wed, 16 Jun 2010 01:45:58 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5G5jmHE009938 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 16 Jun 2010 05:45:50 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5G287Gl002696; Wed, 16 Jun 2010 05:45:47 GMT Received: from abhmt002.oracle.com by acsmt353.oracle.com with ESMTP id 329143051276667146; Tue, 15 Jun 2010 22:45:46 -0700 Received: from [192.168.1.2] (/114.245.225.225) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 15 Jun 2010 22:45:46 -0700 Message-ID: <4C186505.70804@oracle.com> Date: Wed, 16 Jun 2010 13:45:41 +0800 From: Tao Ma User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17EC99.4020908@cs.ucla.edu> In-Reply-To: <4C17EC99.4020908@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090201.4C186510.003B:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , "jeff.liu" , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) Hi Paul, On 06/16/2010 05:11 AM, Paul Eggert wrote: > On 06/10/2010 05:35 PM, Tao Ma wrote: > >> there is a flag FIEMAP_EXTENT_DELALLOC in fiemap ... >> with dd if=/dev/zero of=/mnt/ext4/a bs=1M count=1 seek=1 >> the ext4 can't return a valid fiemap extent. > > Hmm, this sounds like a fairly serious bug, in that it would prevent this > part of cp from working. What does the fiemap ioctl return in this buggy case? > Is there some way that cp can detect the bug, and report it to the user, > or work around it? Or should we just assume that the bug will get fixed > and that cp shouldn't worry about it? I have prompted this to the linux ext4 community(http://lkml.org/lkml/2010/6/10/412), they are working on it now I guess. Regards, Tao From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 16 02:45:40 2010 Received: (at submit) by debbugs.gnu.org; 16 Jun 2010 06:45:40 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOmNX-0001qi-0l for submit@debbugs.gnu.org; Wed, 16 Jun 2010 02:45:39 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOmNT-0001qd-Jf for submit@debbugs.gnu.org; Wed, 16 Jun 2010 02:45:37 -0400 Received: from lists.gnu.org ([199.232.76.165]:40795) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOmNL-0006rG-Nb for submit@debbugs.gnu.org; Wed, 16 Jun 2010 02:45:27 -0400 Received: from [140.186.70.92] (port=45227 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOmNJ-0002sQ-A4 for bug-coreutils@gnu.org; Wed, 16 Jun 2010 02:45:27 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOmNC-0008Bn-7y for bug-coreutils@gnu.org; Wed, 16 Jun 2010 02:45:19 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:37148) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOmNB-0008BQ-Lp for bug-coreutils@gnu.org; Wed, 16 Jun 2010 02:45:18 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id 3D9EF940045; Wed, 16 Jun 2010 08:45:09 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id E20C4D8B6; Wed, 16 Jun 2010 08:45:08 +0200 (CEST) From: Jim Meyering To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4C17FC61.2020401@cs.ucla.edu> (Paul Eggert's message of "Tue, 15 Jun 2010 15:19:13 -0700") References: <4BE41FFF.4020809@oracle.com> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> Date: Wed, 16 Jun 2010 08:45:08 +0200 Message-ID: <87zkyvcw7f.fsf@meyering.net> Lines: 87 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Paul Eggert wrote: > On 06/15/2010 02:43 PM, Jim Meyering wrote: >> I think that copying physical holes via FIEMAP should be the default, when >> possible. One problem is that the current code on the fiemap-copy branch >> does not honor --sparse=WHEN when in fiemap-copying mode. The solution >> would seem to be to change the regular-file-copying loop in the fiemap_copy >> function to use the same hole-preserving code that is used in copy_reg. > > I assume that the solution would be used only with cp --sparse=always? > (Otherwise, it would amount to copying logical holes by default.) Right. I.e., use the same code that honors (or not) the "make_holes" variable. > If so, under this proposal, --sparse=always would copy logical holes, > --sparse=never would never copy holes, and --sparse=auto (the default) > would copy physical holes if supported, else it would copy logical > holes if the file seems to have enough physical holes, and otherwise > it would copy no holes. (Whew!) Right. With --sparse=always and fiemap support, it would take advantage of existing extents to minimize copying time, and for the nontrivial extents, it would detect/induce new holes when possible. Perhaps we need a new logical option that would make a difference only when there are nontrivial fiemap extents: - read nontrivial extents, as is done now - read them, *and* search-for/induce-holes as we do in legacy copy for --sparse=always. Or, putting it another way, perhaps we need a new command line option to control whether we even attempt a fiemap copy. IMHO the default should be to enable it, once all of the underlying bits are deemed to be stable enough. --fiemap --no-fiemap Then, --fiemap --sparse=never would do what the existing fiemap_copy function does, and --fiemap --sparse=always would work once the internal-to-fiemap_copy copying code is adjusted to use the hole-preserving code in copy_reg. Then, to guarantee the legacy --sparse=never behavior, one would have to specify --no-fiemap --sparse=never > But this raises another issue. Under this proposal, the default behavior > copies physical _holes_, but it doesn't copy physical _extents_. > For example, suppose the input file has a single 128 MiB extent. Yes, I saw that already, in testing, even with relatively simply structured inputs. That's why the test script had to be adjusted to accommodate differences in extent layout, effectively merging split extents when comparing src and dest filefrag -v output. > The > proposal could create an output file with two 64-MiB extents > that are logically contiguous. This is because the code does not advise the > output file about the extents that the input file had, and the operating > system might assign a smaller output extent at first, discovering only too late > that more space was needed. > > If the intent is to copy the physical extents, then the proposed code That appears to be infeasible. > needs to be fixed so that it uses fallocate() to attempt to create the > same extents in the destination that existed in the source. Obviously > this could fail for any number of reasons, for example, the destination > file system might be different from the source; but the goal would be > to preserve extents if the underlying OS is "nice" about implementing > fallocate(). > > Which raises another question: should cp attempt to create _better_ > extents in the destination than in the source? For example, if It's already great that most of this functionality appears to be usable on four(!) file system types. I don't want to push our luck. > This all is getting a bit complicated, I'm afraid. Perhaps it'd be better > to try this stuff out with a new option, "cp --sparse=extents" say, and > keep the default as-is for a while? I think we can keep it conceptually simple enough that it will be safe to make fiemap copying the default. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 16 02:59:54 2010 Received: (at submit) by debbugs.gnu.org; 16 Jun 2010 06:59:54 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOmbK-0001wd-BB for submit@debbugs.gnu.org; Wed, 16 Jun 2010 02:59:54 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOmZi-0001va-FG for submit@debbugs.gnu.org; Wed, 16 Jun 2010 02:59:53 -0400 Received: from lists.gnu.org ([199.232.76.165]:47221) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOmZU-00071m-9H for submit@debbugs.gnu.org; Wed, 16 Jun 2010 02:58:00 -0400 Received: from [140.186.70.92] (port=45938 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOmZR-0002vf-NH for bug-coreutils@gnu.org; Wed, 16 Jun 2010 02:57:59 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOmYy-0004Tm-Md for bug-coreutils@gnu.org; Wed, 16 Jun 2010 02:57:29 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:36054) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOmYy-0004Sx-H0 for bug-coreutils@gnu.org; Wed, 16 Jun 2010 02:57:28 -0400 Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5G6vMGR000961 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 16 Jun 2010 06:57:23 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5G6gaal025548; Wed, 16 Jun 2010 06:57:21 GMT Received: from abhmt018.oracle.com by acsmt354.oracle.com with ESMTP id 329300251276671425; Tue, 15 Jun 2010 23:57:05 -0700 Received: from [221.223.104.244] (/221.223.104.244) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 15 Jun 2010 23:57:05 -0700 Message-ID: <4C1875BD.5030506@oracle.com> Date: Wed, 16 Jun 2010 14:57:01 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> In-Reply-To: <4C17F08B.4020203@cs.ucla.edu> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet15.oracle.com [148.87.113.117] X-CT-RefId: str=0001.0A090208.4C1875D3.022B:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.6 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.6 (-----) Paul Eggert wrote: > I looked at the proposed fiemap patch for cp and see another issue > with it, which needs to be thought through more carefully. > > The proposed fiemap code tries to copy the _physical_ holes in the > source file. But currently cp copies the _logical_ holes: that is, it > skips over writing (and thus attempts to create physical holes) > whenever the source file contains a block of zeros, regardless of > whether that block of zeros is physically stored in the file system. > The current cp behavior is often the desired behavior (particularly > for cp --sparse=always) and the fiemap code shouldn't alter that. > > For example, if a fiemap_extent has the FIEMAP_EXTENT_UNWRITTEN flag > set, cp should treat that as a hole, because the extent is all zeros. > (This will greatly help performance in some cases.) Also, if an input > extent is read and a block of it is found to be zeros, cp should skip > over that block when writing. If FIEMAP_EXTENT_UNWRITTEN flag is set, we can call fallocate(2) against the dest file directly for the performance boost. Thanks, -Jeff > > If the proposed fiemap code was intended to copy the physical holes, > then the code wasn't successful in implementing the intent, as there's > no guarantee that the extents of the destination are the same as the > extents of the source. If it is useful to have 'cp' copy the physical > holes, this could be supported via a new cp option or options, but > surely the default should continue to be to copy logical holes. > -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 16 03:23:35 2010 Received: (at submit) by debbugs.gnu.org; 16 Jun 2010 07:23:35 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOmyE-00026y-LD for submit@debbugs.gnu.org; Wed, 16 Jun 2010 03:23:34 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOmyD-00026r-21 for submit@debbugs.gnu.org; Wed, 16 Jun 2010 03:23:33 -0400 Received: from lists.gnu.org ([199.232.76.165]:53343) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOmyA-0007bp-0B for submit@debbugs.gnu.org; Wed, 16 Jun 2010 03:23:30 -0400 Received: from [140.186.70.92] (port=54512 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOmy7-0007Lr-VP for bug-coreutils@gnu.org; Wed, 16 Jun 2010 03:23:28 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOmy6-00016u-Ji for bug-coreutils@gnu.org; Wed, 16 Jun 2010 03:23:27 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:52174) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOmy6-00016Z-Bp for bug-coreutils@gnu.org; Wed, 16 Jun 2010 03:23:26 -0400 Received: from rcsinet15.oracle.com (rcsinet15.oracle.com [148.87.113.117]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5G7NMoj002530 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 16 Jun 2010 07:23:23 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5G4LWKs019468; Wed, 16 Jun 2010 07:23:20 GMT Received: from abhmt013.oracle.com by acsmt353.oracle.com with ESMTP id 348903131276672932; Wed, 16 Jun 2010 00:22:12 -0700 Received: from [221.223.104.244] (/221.223.104.244) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 16 Jun 2010 00:22:11 -0700 Message-ID: <4C187BA0.3030605@oracle.com> Date: Wed, 16 Jun 2010 15:22:08 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <871vdho68j.fsf@meyering.net> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17EC99.4020908@cs.ucla.edu> <4C186505.70804@oracle.com> In-Reply-To: <4C186505.70804@oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet15.oracle.com [148.87.113.117] X-CT-RefId: str=0001.0A090208.4C187BEB.01E5:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.7 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.7 (-----) Tao Ma wrote: > Hi Paul, > > On 06/16/2010 05:11 AM, Paul Eggert wrote: >> On 06/10/2010 05:35 PM, Tao Ma wrote: >> >>> there is a flag FIEMAP_EXTENT_DELALLOC in fiemap ... >>> with dd if=/dev/zero of=/mnt/ext4/a bs=1M count=1 seek=1 >>> the ext4 can't return a valid fiemap extent. >> >> Hmm, this sounds like a fairly serious bug, in that it would prevent this >> part of cp from working. What does the fiemap ioctl return in this >> buggy case? In this case, fiemap ioctl returns successfully, but 'fm_mapped_extents' will set to ZERO to indicate there is no extents allocated for the file. >> Is there some way that cp can detect the bug, and report it to the user, >> or work around it? work around is add FIEMAP_FLAG_SYNC to fiemap ioctl at the moment. Thanks, -Jeff Or should we just assume that the bug will get fixed >> and that cp shouldn't worry about it? > I have prompted this to the linux ext4 > community(http://lkml.org/lkml/2010/6/10/412), they are working on it > now I guess. > > Regards, > Tao -- With Windows 7, Microsoft is asserting legal control over your computer and is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 16 04:32:59 2010 Received: (at submit) by debbugs.gnu.org; 16 Jun 2010 08:32:59 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOo3P-0003Aq-3z for submit@debbugs.gnu.org; Wed, 16 Jun 2010 04:32:59 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOo3N-0003Al-FM for submit@debbugs.gnu.org; Wed, 16 Jun 2010 04:32:57 -0400 Received: from lists.gnu.org ([199.232.76.165]:45093) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOo3K-0000dh-AF for submit@debbugs.gnu.org; Wed, 16 Jun 2010 04:32:54 -0400 Received: from [140.186.70.92] (port=58500 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOo3I-0001v0-Pw for bug-coreutils@gnu.org; Wed, 16 Jun 2010 04:32:53 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOo3H-000578-FU for bug-coreutils@gnu.org; Wed, 16 Jun 2010 04:32:52 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:19217) by eggs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1OOo3H-00056n-4e for bug-coreutils@gnu.org; Wed, 16 Jun 2010 04:32:51 -0400 Received: (qmail 86816 invoked from network); 16 Jun 2010 08:32:48 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 16 Jun 2010 08:32:48 -0000 Message-ID: <4C188BE3.7080103@draigBrady.com> Date: Wed, 16 Jun 2010 09:31:31 +0100 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <4BEC0BCC.3080906@oracle.com> <87k4qyl6lw.fsf@meyering.net> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> In-Reply-To: <87zkyvcw7f.fsf@meyering.net> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9 X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.8 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Chris Mason , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.8 (----) On 16/06/10 07:45, Jim Meyering wrote: > Paul Eggert wrote: >> On 06/15/2010 02:43 PM, Jim Meyering wrote: >>> I think that copying physical holes via FIEMAP should be the default, when >>> possible. One problem is that the current code on the fiemap-copy branch >>> does not honor --sparse=WHEN when in fiemap-copying mode. The solution >>> would seem to be to change the regular-file-copying loop in the fiemap_copy >>> function to use the same hole-preserving code that is used in copy_reg. >> >> I assume that the solution would be used only with cp --sparse=always? >> (Otherwise, it would amount to copying logical holes by default.) > > Right. > I.e., use the same code that honors (or not) the "make_holes" variable. > >> If so, under this proposal, --sparse=always would copy logical holes, >> --sparse=never would never copy holes, and --sparse=auto (the default) >> would copy physical holes if supported, else it would copy logical >> holes if the file seems to have enough physical holes, and otherwise >> it would copy no holes. (Whew!) > > Right. With --sparse=always and fiemap support, it would take advantage > of existing extents to minimize copying time, and for the nontrivial > extents, it would detect/induce new holes when possible. > > Perhaps we need a new logical option that would make a difference > only when there are nontrivial fiemap extents: > - read nontrivial extents, as is done now > - read them, *and* search-for/induce-holes as we do in legacy > copy for --sparse=always. > > Or, putting it another way, perhaps we need a new command line > option to control whether we even attempt a fiemap copy. > IMHO the default should be to enable it, once all of the > underlying bits are deemed to be stable enough. > > --fiemap > --no-fiemap > > Then, --fiemap --sparse=never would do what the existing fiemap_copy > function does, and --fiemap --sparse=always would work once the > internal-to-fiemap_copy copying code is adjusted to use the > hole-preserving code in copy_reg. > > Then, to guarantee the legacy --sparse=never behavior, > one would have to specify --no-fiemap --sparse=never I would suggest not to add options like this, and only do what's possible without new options as it would push too many implementation details to the docs/users IMHO. cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 16 04:49:27 2010 Received: (at submit) by debbugs.gnu.org; 16 Jun 2010 08:49:28 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOoJK-0003JK-QK for submit@debbugs.gnu.org; Wed, 16 Jun 2010 04:49:27 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOoJI-0003JE-Vi for submit@debbugs.gnu.org; Wed, 16 Jun 2010 04:49:25 -0400 Received: from lists.gnu.org ([199.232.76.165]:40655) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOoJF-0000wF-SM for submit@debbugs.gnu.org; Wed, 16 Jun 2010 04:49:21 -0400 Received: from [140.186.70.92] (port=44089 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOoJE-0003Xl-0c for bug-coreutils@gnu.org; Wed, 16 Jun 2010 04:49:21 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOoJC-00087I-Ct for bug-coreutils@gnu.org; Wed, 16 Jun 2010 04:49:19 -0400 Received: from smtp1-g21.free.fr ([212.27.42.1]:39577) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOoJB-00086N-Q6 for bug-coreutils@gnu.org; Wed, 16 Jun 2010 04:49:18 -0400 Received: from mx.meyering.net (mx.meyering.net [82.230.74.64]) by smtp1-g21.free.fr (Postfix) with ESMTP id D70B594015C; Wed, 16 Jun 2010 10:49:09 +0200 (CEST) Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id B6B1CD8A1; Wed, 16 Jun 2010 10:49:08 +0200 (CEST) From: Jim Meyering To: =?utf-8?Q?P=C3=A1draig?= Brady Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4C188BE3.7080103@draigBrady.com> (=?utf-8?Q?=22P=C3=A1draig?= Brady"'s message of "Wed, 16 Jun 2010 09:31:31 +0100") References: <4BE41FFF.4020809@oracle.com> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> <4C188BE3.7080103@draigBrady.com> Date: Wed, 16 Jun 2010 10:49:08 +0200 Message-ID: <87fx0ncqgr.fsf@meyering.net> Lines: 64 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Chris Mason , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.4 (-----) P=C3=A1draig Brady wrote: > On 16/06/10 07:45, Jim Meyering wrote: >> Paul Eggert wrote: >>> On 06/15/2010 02:43 PM, Jim Meyering wrote: >>>> I think that copying physical holes via FIEMAP should be the default, = when >>>> possible. One problem is that the current code on the fiemap-copy bra= nch >>>> does not honor --sparse=3DWHEN when in fiemap-copying mode. The solut= ion >>>> would seem to be to change the regular-file-copying loop in the fiemap= _copy >>>> function to use the same hole-preserving code that is used in copy_reg. >>> >>> I assume that the solution would be used only with cp --sparse=3Dalways? >>> (Otherwise, it would amount to copying logical holes by default.) >> >> Right. >> I.e., use the same code that honors (or not) the "make_holes" variable. >> >>> If so, under this proposal, --sparse=3Dalways would copy logical holes, >>> --sparse=3Dnever would never copy holes, and --sparse=3Dauto (the defau= lt) >>> would copy physical holes if supported, else it would copy logical >>> holes if the file seems to have enough physical holes, and otherwise >>> it would copy no holes. (Whew!) >> >> Right. With --sparse=3Dalways and fiemap support, it would take advanta= ge >> of existing extents to minimize copying time, and for the nontrivial >> extents, it would detect/induce new holes when possible. >> >> Perhaps we need a new logical option that would make a difference >> only when there are nontrivial fiemap extents: >> - read nontrivial extents, as is done now >> - read them, *and* search-for/induce-holes as we do in legacy >> copy for --sparse=3Dalways. >> >> Or, putting it another way, perhaps we need a new command line >> option to control whether we even attempt a fiemap copy. >> IMHO the default should be to enable it, once all of the >> underlying bits are deemed to be stable enough. >> >> --fiemap >> --no-fiemap >> >> Then, --fiemap --sparse=3Dnever would do what the existing fiemap_copy >> function does, and --fiemap --sparse=3Dalways would work once the >> internal-to-fiemap_copy copying code is adjusted to use the >> hole-preserving code in copy_reg. >> >> Then, to guarantee the legacy --sparse=3Dnever behavior, >> one would have to specify --no-fiemap --sparse=3Dnever > > I would suggest not to add options like this, > and only do what's possible without new options > as it would push too many implementation details > to the docs/users IMHO. Good point. Smaller/simpler would be better. It'd be great if we can eliminate as never-useful one of the new combinations. Currently on the fiemap-copy branch, once we get a single successful ioctl, the code will not bother with the usual hole-detecting/introducing technique implied by --sparse=3Dalways. Should it do that ever? Always? Does this need an option? From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 16 05:04:24 2010 Received: (at submit) by debbugs.gnu.org; 16 Jun 2010 09:04:25 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOoXo-0003QX-6a for submit@debbugs.gnu.org; Wed, 16 Jun 2010 05:04:24 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOoXl-0003QM-IG for submit@debbugs.gnu.org; Wed, 16 Jun 2010 05:04:22 -0400 Received: from lists.gnu.org ([199.232.76.165]:34598) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOoXi-00019C-Vu for submit@debbugs.gnu.org; Wed, 16 Jun 2010 05:04:19 -0400 Received: from [140.186.70.92] (port=60927 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOoXb-0001CB-VV for bug-coreutils@gnu.org; Wed, 16 Jun 2010 05:04:17 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOoXX-00025B-I3 for bug-coreutils@gnu.org; Wed, 16 Jun 2010 05:04:09 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:31427) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOoXX-00024u-BB for bug-coreutils@gnu.org; Wed, 16 Jun 2010 05:04:07 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5G93mP3007215 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 16 Jun 2010 09:03:51 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5G5fRWw013290; Wed, 16 Jun 2010 09:03:45 GMT Received: from ca-server1.us.oracle.com by acsmt355.oracle.com with ESMTP id 329639511276679024; Wed, 16 Jun 2010 02:03:44 -0700 Received: from jlbec by ca-server1.us.oracle.com with local (Exim 4.69) (envelope-from ) id 1OOoX9-0001KW-PV; Wed, 16 Jun 2010 02:03:43 -0700 Date: Wed, 16 Jun 2010 02:03:42 -0700 From: Joel Becker To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy Message-ID: <20100616090341.GB14109@mail.oracle.com> References: <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <4C1875BD.5030506@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C1875BD.5030506@oracle.com> X-Burt-Line: Trees are cool. X-Red-Smith: Ninety feet between bases is perhaps as close as man has ever come to perfection. User-Agent: Mutt/1.5.20 (2009-06-14) X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090207.4C189385.0115:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.2 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Jim Meyering , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) On Wed, Jun 16, 2010 at 02:57:01PM +0800, jeff.liu wrote: > Paul Eggert wrote: > > For example, if a fiemap_extent has the FIEMAP_EXTENT_UNWRITTEN flag > > set, cp should treat that as a hole, because the extent is all zeros. > > (This will greatly help performance in some cases.) Also, if an input > > extent is read and a block of it is found to be zeros, cp should skip > > over that block when writing. > If FIEMAP_EXTENT_UNWRITTEN flag is set, we can call fallocate(2) against the dest file directly for > the performance boost. Nope. An UNWRITTEN extent is a hole, and the user asked for holes. If cp sees an UNWRITTEN extent it is skipped. Joel -- "Friends may come and go, but enemies accumulate." - Thomas Jones Joel Becker Principal Software Developer Oracle E-mail: joel.becker@oracle.com Phone: (650) 506-8127 From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 16 05:04:25 2010 Received: (at submit) by debbugs.gnu.org; 16 Jun 2010 09:04:25 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOoXo-0003QZ-Gh for submit@debbugs.gnu.org; Wed, 16 Jun 2010 05:04:24 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOoXl-0003QN-II for submit@debbugs.gnu.org; Wed, 16 Jun 2010 05:04:23 -0400 Received: from lists.gnu.org ([199.232.76.165]:34600) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOoXj-00019E-2k for submit@debbugs.gnu.org; Wed, 16 Jun 2010 05:04:19 -0400 Received: from [140.186.70.92] (port=60926 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOoXb-0001Bh-VT for bug-coreutils@gnu.org; Wed, 16 Jun 2010 05:04:18 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOoXX-000256-Hs for bug-coreutils@gnu.org; Wed, 16 Jun 2010 05:04:09 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:31425) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOoXX-00024t-BA for bug-coreutils@gnu.org; Wed, 16 Jun 2010 05:04:07 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5G93h5i006983 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 16 Jun 2010 09:03:45 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5G5fRWu013290; Wed, 16 Jun 2010 09:03:41 GMT Received: from ca-server1.us.oracle.com by acsmt353.oracle.com with ESMTP id 329634141276678949; Wed, 16 Jun 2010 02:02:29 -0700 Received: from jlbec by ca-server1.us.oracle.com with local (Exim 4.69) (envelope-from ) id 1OOoVx-0007au-09; Wed, 16 Jun 2010 02:02:29 -0700 Date: Wed, 16 Jun 2010 02:02:26 -0700 From: Joel Becker To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy Message-ID: <20100616090226.GA14109@mail.oracle.com> References: <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87fx0ncqgr.fsf@meyering.net> X-Burt-Line: Trees are cool. X-Red-Smith: Ninety feet between bases is perhaps as close as man has ever come to perfection. User-Agent: Mutt/1.5.20 (2009-06-14) X-Auth-Type: Internal IP X-Source-IP: acsinet15.oracle.com [141.146.126.227] X-CT-RefId: str=0001.0A090206.4C18937F.00BC:SCFMA922111,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.2 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, "jeff.liu" , Chris Mason , =?iso-8859-1?Q?P=E1draig?= Brady , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) On Wed, Jun 16, 2010 at 10:49:08AM +0200, Jim Meyering wrote: > > I would suggest not to add options like this, > > and only do what's possible without new options > > as it would push too many implementation details > > to the docs/users IMHO. > > Currently on the fiemap-copy branch, once we get a single > successful ioctl, the code will not bother with the usual > hole-detecting/introducing technique implied by --sparse=always. > > Should it do that ever? Always? > Does this need an option? There should be no new or modified options. I think the code should, at all times and in all modes, attempt to get an fiemap of the source file. If cp is in --sparse=never mode, it can use the mapping to plan its source file reads. If it is in --sparse=auto, it can always skip holes and unwritten extents, while using the original --sparse=auto code to determine whether to scan allocated extents for zeros. With this map, cp will never read holes or unwritten extents; --sparse=never mode can write out the zeros without reading them. Calling the ioctl will never hurt the cp operation. If the ioctl fails, cp can use all of its original heuristics and copy code. Joel -- "Copy from one, it's plagiarism; copy from two, it's research." - Wilson Mizner Joel Becker Principal Software Developer Oracle E-mail: joel.becker@oracle.com Phone: (650) 506-8127 From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 16 05:28:08 2010 Received: (at submit) by debbugs.gnu.org; 16 Jun 2010 09:28:08 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOoul-0003bk-Qt for submit@debbugs.gnu.org; Wed, 16 Jun 2010 05:28:08 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOouk-0003bN-9p for submit@debbugs.gnu.org; Wed, 16 Jun 2010 05:28:07 -0400 Received: from lists.gnu.org ([199.232.76.165]:35868) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOouh-0001Ya-P9 for submit@debbugs.gnu.org; Wed, 16 Jun 2010 05:28:03 -0400 Received: from [140.186.70.92] (port=55134 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOouf-0004ma-JV for bug-coreutils@gnu.org; Wed, 16 Jun 2010 05:28:02 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOoue-0005Wx-5L for bug-coreutils@gnu.org; Wed, 16 Jun 2010 05:28:01 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:9845) by eggs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1OOoud-0005Wh-Ru for bug-coreutils@gnu.org; Wed, 16 Jun 2010 05:28:00 -0400 Received: (qmail 97071 invoked from network); 16 Jun 2010 09:27:58 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 16 Jun 2010 09:27:58 -0000 Message-ID: <4C1898D0.3000204@draigBrady.com> Date: Wed, 16 Jun 2010 10:26:40 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> In-Reply-To: <87fx0ncqgr.fsf@meyering.net> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9 X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.8 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , "jeff.liu" , Chris Mason , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.8 (----) On 16/06/10 09:49, Jim Meyering wrote: > Pádraig Brady wrote: > >> On 16/06/10 07:45, Jim Meyering wrote: >>> Paul Eggert wrote: >>>> On 06/15/2010 02:43 PM, Jim Meyering wrote: >>>>> I think that copying physical holes via FIEMAP should be the default, when >>>>> possible. One problem is that the current code on the fiemap-copy branch >>>>> does not honor --sparse=WHEN when in fiemap-copying mode. The solution >>>>> would seem to be to change the regular-file-copying loop in the fiemap_copy >>>>> function to use the same hole-preserving code that is used in copy_reg. >>>> >>>> I assume that the solution would be used only with cp --sparse=always? >>>> (Otherwise, it would amount to copying logical holes by default.) >>> >>> Right. >>> I.e., use the same code that honors (or not) the "make_holes" variable. >>> >>>> If so, under this proposal, --sparse=always would copy logical holes, >>>> --sparse=never would never copy holes, and --sparse=auto (the default) >>>> would copy physical holes if supported, else it would copy logical >>>> holes if the file seems to have enough physical holes, and otherwise >>>> it would copy no holes. (Whew!) >>> >>> Right. With --sparse=always and fiemap support, it would take advantage >>> of existing extents to minimize copying time, and for the nontrivial >>> extents, it would detect/induce new holes when possible. >>> >>> Perhaps we need a new logical option that would make a difference >>> only when there are nontrivial fiemap extents: >>> - read nontrivial extents, as is done now >>> - read them, *and* search-for/induce-holes as we do in legacy >>> copy for --sparse=always. >>> >>> Or, putting it another way, perhaps we need a new command line >>> option to control whether we even attempt a fiemap copy. >>> IMHO the default should be to enable it, once all of the >>> underlying bits are deemed to be stable enough. >>> >>> --fiemap >>> --no-fiemap >>> >>> Then, --fiemap --sparse=never would do what the existing fiemap_copy >>> function does, and --fiemap --sparse=always would work once the >>> internal-to-fiemap_copy copying code is adjusted to use the >>> hole-preserving code in copy_reg. >>> >>> Then, to guarantee the legacy --sparse=never behavior, >>> one would have to specify --no-fiemap --sparse=never >> >> I would suggest not to add options like this, >> and only do what's possible without new options >> as it would push too many implementation details >> to the docs/users IMHO. > > Good point. Smaller/simpler would be better. > It'd be great if we can eliminate as never-useful one of the > new combinations. > > Currently on the fiemap-copy branch, once we get a single > successful ioctl, the code will not bother with the usual > hole-detecting/introducing technique implied by --sparse=always. > > Should it do that ever? Always? > Does this need an option? I don't see the need for new options TBH I see fiemap just as a way to efficiently detect/read holes, and should have no bearing on the destination. cp --sparse=auto (this is currently what cp does by default) recreate the original fiemap holes or resort to existing heuristic if fiemap not available cp --sparse=never write all data, but use fiemap if available to efficiently read cp --sparse=always recreate original holes and perhaps extend add to them for other runs of zero bytes. Without having looked at the code I see this as a little tricky to mix with fiemap. Now since fiemap is only an optimization we can skip it completely for this uncommon case if too tricky (just add a FIXME for now). cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 16 10:00:42 2010 Received: (at submit) by debbugs.gnu.org; 16 Jun 2010 14:00:42 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOtAY-0006OW-23 for submit@debbugs.gnu.org; Wed, 16 Jun 2010 10:00:42 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOtAX-0006OP-1J for submit@debbugs.gnu.org; Wed, 16 Jun 2010 10:00:41 -0400 Received: from lists.gnu.org ([199.232.76.165]:58216) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OOtAR-0006uV-UA for submit@debbugs.gnu.org; Wed, 16 Jun 2010 10:00:36 -0400 Received: from [140.186.70.92] (port=57587 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OOtAO-0000W2-Pq for bug-coreutils@gnu.org; Wed, 16 Jun 2010 10:00:35 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OOtAN-00046P-B0 for bug-coreutils@gnu.org; Wed, 16 Jun 2010 10:00:32 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:40806) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OOtAN-00046D-4o for bug-coreutils@gnu.org; Wed, 16 Jun 2010 10:00:31 -0400 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5GDxrkX001775 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 16 Jun 2010 13:59:54 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o5GDxp4N019997; Wed, 16 Jun 2010 13:59:51 GMT Received: from abhmt007.oracle.com by acsmt355.oracle.com with ESMTP id 349722821276696692; Wed, 16 Jun 2010 06:58:12 -0700 Received: from [192.168.1.2] (/221.217.40.44) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 16 Jun 2010 06:58:10 -0700 Message-ID: <4C18D86A.2060404@oracle.com> Date: Wed, 16 Jun 2010 21:58:02 +0800 From: Tao Ma User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Joel Becker Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <4C1875BD.5030506@oracle.com> <20100616090341.GB14109@mail.oracle.com> In-Reply-To: <20100616090341.GB14109@mail.oracle.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Auth-Type: Internal IP X-Source-IP: rcsinet13.oracle.com [148.87.113.125] X-CT-RefId: str=0001.0A090201.4C18D8E2.01A3:SCFMA4539811,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.4 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Jim Meyering , "jeff.liu" , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.4 (------) On 06/16/2010 05:03 PM, Joel Becker wrote: > On Wed, Jun 16, 2010 at 02:57:01PM +0800, jeff.liu wrote: >> Paul Eggert wrote: >>> For example, if a fiemap_extent has the FIEMAP_EXTENT_UNWRITTEN flag >>> set, cp should treat that as a hole, because the extent is all zeros. >>> (This will greatly help performance in some cases.) Also, if an input >>> extent is read and a block of it is found to be zeros, cp should skip >>> over that block when writing. >> If FIEMAP_EXTENT_UNWRITTEN flag is set, we can call fallocate(2) against the dest file directly for >> the performance boost. > > Nope. An UNWRITTEN extent is a hole, and the user asked for > holes. If cp sees an UNWRITTEN extent it is skipped. yeah, I agree with this. The unwritten extent is allocated by the user for the source file. Since there is no user say to us that the target also need extra allocation, we'd better just leave a hole there. ;) Regards, Tao From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 14 02:59:43 2010 Received: (at submit) by debbugs.gnu.org; 14 Jul 2010 06:59:43 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OYvwU-00088O-JS for submit@debbugs.gnu.org; Wed, 14 Jul 2010 02:59:43 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OYvwS-00087t-BK for submit@debbugs.gnu.org; Wed, 14 Jul 2010 02:59:41 -0400 Received: from lists.gnu.org ([199.232.76.165]:38432) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OYvwZ-00052f-N6 for submit@debbugs.gnu.org; Wed, 14 Jul 2010 02:59:47 -0400 Received: from [140.186.70.92] (port=37215 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OYvwY-0005zx-0p for bug-coreutils@gnu.org; Wed, 14 Jul 2010 02:59:47 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OYvwW-0002Hb-Ho for bug-coreutils@gnu.org; Wed, 14 Jul 2010 02:59:45 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:18974) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OYvwW-0002H3-CM for bug-coreutils@gnu.org; Wed, 14 Jul 2010 02:59:44 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o6E6xY4l012307 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 14 Jul 2010 06:59:35 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o6D71PKC029401; Wed, 14 Jul 2010 06:59:32 GMT Received: from abhmt001.oracle.com by acsmt354.oracle.com with ESMTP id 424503751279090679; Tue, 13 Jul 2010 23:57:59 -0700 Received: from [10.182.121.3] (/10.182.121.3) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 13 Jul 2010 23:57:59 -0700 Message-ID: <4C3D6008.90302@oracle.com> Date: Wed, 14 Jul 2010 14:58:16 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> <4C1898D0.3000204@draigBrady.com> In-Reply-To: <4C1898D0.3000204@draigBrady.com> Content-Type: text/plain; charset=UTF-8 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090201.4C3D6055.0234:SCFMA4539814,ss=1,fgs=0 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by rcsinet10.oracle.com id o6E6xY4l012307 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) P=C3=A1draig Brady wrote: > On 16/06/10 09:49, Jim Meyering wrote: >> P=C3=A1draig Brady wrote: >> >>> On 16/06/10 07:45, Jim Meyering wrote: >>>> Paul Eggert wrote: >>>>> On 06/15/2010 02:43 PM, Jim Meyering wrote: >>>>>> I think that copying physical holes via FIEMAP should be the defau= lt, when >>>>>> possible. One problem is that the current code on the fiemap-copy= branch >>>>>> does not honor --sparse=3DWHEN when in fiemap-copying mode. The s= olution >>>>>> would seem to be to change the regular-file-copying loop in the fi= emap_copy >>>>>> function to use the same hole-preserving code that is used in copy= _reg. >>>>> I assume that the solution would be used only with cp --sparse=3Dal= ways? >>>>> (Otherwise, it would amount to copying logical holes by default.) >>>> Right. >>>> I.e., use the same code that honors (or not) the "make_holes" variab= le. >>>> >>>>> If so, under this proposal, --sparse=3Dalways would copy logical ho= les, >>>>> --sparse=3Dnever would never copy holes, and --sparse=3Dauto (the d= efault) >>>>> would copy physical holes if supported, else it would copy logical >>>>> holes if the file seems to have enough physical holes, and otherwis= e >>>>> it would copy no holes. (Whew!) >>>> Right. With --sparse=3Dalways and fiemap support, it would take adv= antage >>>> of existing extents to minimize copying time, and for the nontrivial >>>> extents, it would detect/induce new holes when possible. >>>> >>>> Perhaps we need a new logical option that would make a difference >>>> only when there are nontrivial fiemap extents: >>>> - read nontrivial extents, as is done now >>>> - read them, *and* search-for/induce-holes as we do in legacy >>>> copy for --sparse=3Dalways. >>>> >>>> Or, putting it another way, perhaps we need a new command line >>>> option to control whether we even attempt a fiemap copy. >>>> IMHO the default should be to enable it, once all of the >>>> underlying bits are deemed to be stable enough. >>>> >>>> --fiemap >>>> --no-fiemap >>>> >>>> Then, --fiemap --sparse=3Dnever would do what the existing fiemap_co= py >>>> function does, and --fiemap --sparse=3Dalways would work once the >>>> internal-to-fiemap_copy copying code is adjusted to use the >>>> hole-preserving code in copy_reg. >>>> >>>> Then, to guarantee the legacy --sparse=3Dnever behavior, >>>> one would have to specify --no-fiemap --sparse=3Dnever >>> I would suggest not to add options like this, >>> and only do what's possible without new options >>> as it would push too many implementation details >>> to the docs/users IMHO. >> Good point. Smaller/simpler would be better. >> It'd be great if we can eliminate as never-useful one of the >> new combinations. >> >> Currently on the fiemap-copy branch, once we get a single >> successful ioctl, the code will not bother with the usual >> hole-detecting/introducing technique implied by --sparse=3Dalways. >> >> Should it do that ever? Always? >> Does this need an option? Hello All, I am very sorry for the late response! I have an urgent task need to deli= ver in the past few weeks. Thanks for all your suggestions, I would like to improve the fiemap-copy = accordingly, so does the following sentence is the final decision? > I don't see the need for new options TBH > I see fiemap just as a way to efficiently detect/read holes, > and should have no bearing on the destination. >=20 > cp --sparse=3Dauto (this is currently what cp does by default) > recreate the original fiemap holes or resort to existing > heuristic if fiemap not available > cp --sparse=3Dnever > write all data, but use fiemap if available to efficiently read > cp --sparse=3Dalways > recreate original holes and perhaps extend add to them for > other runs of zero bytes. Without having looked at the code > I see this as a little tricky to mix with fiemap. > Now since fiemap is only an optimization we can skip it > completely for this uncommon case if too tricky (just add a FIXME for= now). Current code handle 'cp --sparse=3Dauto' and 'cp --sparse=3Dalways' in sa= me way since these two options all setup 'make_holes' to True, though '--sparse=3Dauto' use a heuristic = to determine whether SRC_NAME contains any sparse blocks. For 'cp --sparse=3Dnever', when detected holes from SRC file, do not lsee= k(2) against DST file, instead, write ZEROs to DST file, Am I right? If so, IMHO, we can pass 'sparse_mode' to fiemap_copy(), then decide how = to do operation against DST file according to its mode. In addition to this, I'd like to fix another issue in current code as Pau= l suggested, 1. Do not allocate buffer in fiemap_copy() for data pass between SRC and = DST files, instead, take advantage of the buffer allocated outside. 2. Performance optimization, invoke fallocate(2) if an extent flag is UNW= RITTEN For this case, maybe we have to wait until fallocate interface become sta= ble just as P=C3=A1draig methoned before: http://lists.gnu.org/archive/html/bug-coreutils/2009-05/msg00260.html > cheers, > P=C3=A1draig. Thanks, -Jeff --=20 With Windows 7, Microsoft is asserting legal control over your computer a= nd is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 14 04:40:11 2010 Received: (at submit) by debbugs.gnu.org; 14 Jul 2010 08:40:11 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OYxVi-0002W6-UM for submit@debbugs.gnu.org; Wed, 14 Jul 2010 04:40:11 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OYxVf-0002Vh-6G for submit@debbugs.gnu.org; Wed, 14 Jul 2010 04:40:08 -0400 Received: from lists.gnu.org ([199.232.76.165]:58105) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OYxVd-0006aF-Rx for submit@debbugs.gnu.org; Wed, 14 Jul 2010 04:40:05 -0400 Received: from [140.186.70.92] (port=60516 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OYxVa-0007k3-MP for bug-coreutils@gnu.org; Wed, 14 Jul 2010 04:40:04 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OYxVT-0000I8-RX for bug-coreutils@gnu.org; Wed, 14 Jul 2010 04:39:57 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:15393) by eggs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1OYxVT-0000Hq-IA for bug-coreutils@gnu.org; Wed, 14 Jul 2010 04:39:55 -0400 Received: (qmail 5858 invoked from network); 14 Jul 2010 08:39:53 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 14 Jul 2010 08:39:53 -0000 Message-ID: <4C3D778C.6000401@draigBrady.com> Date: Wed, 14 Jul 2010 09:38:36 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> In-Reply-To: <4C3D6008.90302@oracle.com> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9 X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.8 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , Chris Mason , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.8 (----) On 14/07/10 07:58, jeff.liu wrote: > Pádraig Brady wrote: > > Hello All, > > I am very sorry for the late response! I have an urgent task need to deliver in the past few weeks. > > Thanks for all your suggestions, I would like to improve the fiemap-copy accordingly, so does the > following sentence is the final decision? > >> I don't see the need for new options TBH >> I see fiemap just as a way to efficiently detect/read holes, >> and should have no bearing on the destination. >> >> cp --sparse=auto (this is currently what cp does by default) >> recreate the original fiemap holes or resort to existing >> heuristic if fiemap not available >> cp --sparse=never >> write all data, but use fiemap if available to efficiently read >> cp --sparse=always >> recreate original holes and perhaps extend add to them for >> other runs of zero bytes. Without having looked at the code >> I see this as a little tricky to mix with fiemap. >> Now since fiemap is only an optimization we can skip it >> completely for this uncommon case if too tricky (just add a FIXME for now). Joel, overlapped with my response and concurred. > Current code handle 'cp --sparse=auto' and 'cp --sparse=always' in same way since these two options > all setup 'make_holes' to True, though '--sparse=auto' use a heuristic to determine whether SRC_NAME > contains any sparse blocks. > > For 'cp --sparse=never', when detected holes from SRC file, do not lseek(2) against DST file, > instead, write ZEROs to DST file, Am I right? right > > If so, IMHO, we can pass 'sparse_mode' to fiemap_copy(), then decide how to do operation against DST > file according to its mode. > > In addition to this, I'd like to fix another issue in current code as Paul suggested, > 1. Do not allocate buffer in fiemap_copy() for data pass between SRC and DST files, instead, take > advantage of the buffer allocated outside. good > 2. Performance optimization, invoke fallocate(2) if an extent flag is UNWRITTEN > For this case, maybe we have to wait until fallocate interface become stable just as Pádraig > methoned before: > http://lists.gnu.org/archive/html/bug-coreutils/2009-05/msg00260.html If you decide to do that, then please do it as a separate patch. cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 14 13:45:32 2010 Received: (at submit) by debbugs.gnu.org; 14 Jul 2010 17:45:32 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZ61T-0001Iz-Q0 for submit@debbugs.gnu.org; Wed, 14 Jul 2010 13:45:32 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZ61Q-0001It-Bh for submit@debbugs.gnu.org; Wed, 14 Jul 2010 13:45:29 -0400 Received: from lists.gnu.org ([199.232.76.165]:37129) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OZ61Z-0008E5-1H for submit@debbugs.gnu.org; Wed, 14 Jul 2010 13:45:37 -0400 Received: from [140.186.70.92] (port=51263 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OZ61X-00054C-Mh for bug-coreutils@gnu.org; Wed, 14 Jul 2010 13:45:36 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OZ61W-0000eH-BU for bug-coreutils@gnu.org; Wed, 14 Jul 2010 13:45:35 -0400 Received: from kiwi.cs.ucla.edu ([131.179.128.19]:45167) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZ61W-0000dl-2f for bug-coreutils@gnu.org; Wed, 14 Jul 2010 13:45:34 -0400 Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200]) by kiwi.cs.ucla.edu (8.13.8+Sun/8.13.8/UCLACS-6.0) with ESMTP id o6EHjNYw025899; Wed, 14 Jul 2010 10:45:23 -0700 (PDT) Message-ID: <4C3DF7B3.4050104@cs.ucla.edu> Date: Wed, 14 Jul 2010 10:45:23 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.10) Gecko/20100527 Thunderbird/3.0.5 MIME-Version: 1.0 To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> In-Reply-To: <4C3D6008.90302@oracle.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , Chris Mason , =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.0 (-----) >> I see fiemap just as a way to efficiently detect/read holes, >> and should have no bearing on the destination. Hmm, but the proposal quoted below would mean that fiemap does have a bearing on the destination, in the --sparse=auto case. I guess this is OK, but it should be documented. >> cp --sparse=auto (this is currently what cp does by default) >> recreate the original fiemap holes or resort to existing >> heuristic if fiemap not available It's not just fiemap. It's also the Solaris interface with SEEK_HOLE and SEEK_DATA. The change should involve a module that isolates these low-level details from copy.c. copy.c should ask the new module for the locations of the holes (or the non-holes: that could be more convenient). On traditional hosts without fiemap or SEEK_DATA, the module should report that it doesn't know where the holes are; this can let copy.c resort to the existing heuristic of looking at the size and the disk usage and using the --sparse=always approach if the file "smells" like it's sparse. >> cp --sparse=never >> write all data, but use fiemap if available to efficiently read Surely there's no need to write all the data if fallocate works. >> cp --sparse=always >> recreate original holes and perhaps extend add to them for >> other runs of zero bytes. Without having looked at the code >> I see this as a little tricky to mix with fiemap. >> Now since fiemap is only an optimization we can skip it >> completely for this uncommon case if too tricky (just add a FIXME for now). Yes, that makes sense. --sparse=always should never invoke fallocate. > For 'cp --sparse=never', when detected holes from SRC file, do not lseek(2) against DST file, > instead, write ZEROs to DST file, Am I right? Only if fallocate doesn't work. If fallocate works, there's no need to write zeros to the destination. > 2. Performance optimization, invoke fallocate(2) if an extent flag is UNWRITTEN This doesn't sound right. A FIEMAP_EXTENT_UNWRITTEN extent is all zeros, and so it should act as if it were a hole. The goal is not to copy the exact fiemap structure of the source (that's impossible): the goal is to use as little time and space as possible. > If you decide to do that, then please do it as a separate patch. It's not clear to me that the fiemap stuff can be cleanly separated from the fallocate stuff. To some extent they're the same issue. If they can easily be separated, that's better of course. From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 14 19:52:54 2010 Received: (at submit) by debbugs.gnu.org; 14 Jul 2010 23:52:55 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZBl0-0003v8-JQ for submit@debbugs.gnu.org; Wed, 14 Jul 2010 19:52:54 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZBky-0003v2-FE for submit@debbugs.gnu.org; Wed, 14 Jul 2010 19:52:53 -0400 Received: from lists.gnu.org ([199.232.76.165]:50383) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OZBl7-0006U1-RY for submit@debbugs.gnu.org; Wed, 14 Jul 2010 19:53:01 -0400 Received: from [140.186.70.92] (port=50948 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OZBl6-0005HQ-GC for bug-coreutils@gnu.org; Wed, 14 Jul 2010 19:53:01 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OZBl4-0002qZ-U0 for bug-coreutils@gnu.org; Wed, 14 Jul 2010 19:53:00 -0400 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:11051) by eggs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1OZBl4-0002qN-J2 for bug-coreutils@gnu.org; Wed, 14 Jul 2010 19:52:58 -0400 Received: (qmail 74985 invoked from network); 14 Jul 2010 23:52:56 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 14 Jul 2010 23:52:56 -0000 Message-ID: <4C3E4D88.3040305@draigBrady.com> Date: Thu, 15 Jul 2010 00:51:36 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> In-Reply-To: <4C3DF7B3.4050104@cs.ucla.edu> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9 X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.7 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , "jeff.liu" , Chris Mason , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.7 (----) On 14/07/10 18:45, Paul Eggert wrote: >>> I see fiemap just as a way to efficiently detect/read holes, >>> and should have no bearing on the destination. > > Hmm, but the proposal quoted below would mean that fiemap does have a > bearing on the destination, in the --sparse=auto case. > I guess this is OK, but it should be documented. > >>> cp --sparse=auto (this is currently what cp does by default) >>> recreate the original fiemap holes or resort to existing >>> heuristic if fiemap not available > > It's not just fiemap. It's also the Solaris interface with SEEK_HOLE > and SEEK_DATA. The change should involve a module that isolates these > low-level details from copy.c. copy.c should ask the new module for the > locations of the holes (or the non-holes: that could be more convenient). > On traditional hosts without fiemap or SEEK_DATA, the module should report > that it doesn't know where the holes are; this can let copy.c resort to > the existing heuristic of looking at the size and the disk usage and > using the --sparse=always approach if the file "smells" like it's sparse. > >>> cp --sparse=never >>> write all data, but use fiemap if available to efficiently read > > Surely there's no need to write all the data if fallocate works. > >>> cp --sparse=always >>> recreate original holes and perhaps extend add to them for >>> other runs of zero bytes. Without having looked at the code >>> I see this as a little tricky to mix with fiemap. >>> Now since fiemap is only an optimization we can skip it >>> completely for this uncommon case if too tricky (just add a FIXME for now). > > Yes, that makes sense. --sparse=always should never invoke fallocate. > >> For 'cp --sparse=never', when detected holes from SRC file, do not lseek(2) against DST file, >> instead, write ZEROs to DST file, Am I right? > > Only if fallocate doesn't work. If fallocate works, there's no need > to write zeros to the destination. What you're describing here is posix_fallocate() which uses fallocate() if available or falls back to an implementation that writes a single 0 byte to each block. > >> 2. Performance optimization, invoke fallocate(2) if an extent flag is UNWRITTEN > > This doesn't sound right. A FIEMAP_EXTENT_UNWRITTEN extent is all zeros, and > so it should act as if it were a hole. The goal is not to copy the exact > fiemap structure of the source (that's impossible): the goal is to use as > little time and space as possible. > >> If you decide to do that, then please do it as a separate patch. > > It's not clear to me that the fiemap stuff can be cleanly separated > from the fallocate stuff. To some extent they're the same issue. > If they can easily be separated, that's better of course. I see fiemap as optimizing reads, posix_fallocate() as optimizing writing zeros and fallocate() as optimizing allocation. So not having thought much about implementation details, it seems like they could be logically separated. I.E. we could optimize the writing zeros and allocation later when we have the fallocate and posix_fallocate gnulib modules in place. In saying that, doing both now is better when these details are in everyone's minds. I'll not get to resubmitting my fallocate gnulib patch, or doing a posix_fallocate module, this week at least I think. cheers, Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 14 21:50:05 2010 Received: (at submit) by debbugs.gnu.org; 15 Jul 2010 01:50:05 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZDaO-0004et-Hv for submit@debbugs.gnu.org; Wed, 14 Jul 2010 21:50:04 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZDaL-0004eU-U5 for submit@debbugs.gnu.org; Wed, 14 Jul 2010 21:50:02 -0400 Received: from lists.gnu.org ([199.232.76.165]:33994) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OZDaV-0007fE-9S for submit@debbugs.gnu.org; Wed, 14 Jul 2010 21:50:11 -0400 Received: from [140.186.70.92] (port=34627 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OZDaT-0006Bd-Ti for bug-coreutils@gnu.org; Wed, 14 Jul 2010 21:50:11 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OZDaS-0007YI-Bw for bug-coreutils@gnu.org; Wed, 14 Jul 2010 21:50:09 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:22162) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZDaS-0007Y3-6s for bug-coreutils@gnu.org; Wed, 14 Jul 2010 21:50:08 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o6F1nxY6024857 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 15 Jul 2010 01:50:01 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o6F1IvAv031128; Thu, 15 Jul 2010 01:49:58 GMT Received: from abhmt001.oracle.com by acsmt354.oracle.com with ESMTP id 427457021279158492; Wed, 14 Jul 2010 18:48:12 -0700 Received: from [221.223.110.230] (/221.223.110.230) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 14 Jul 2010 18:48:11 -0700 Message-ID: <4C3E68EC.4000408@oracle.com> Date: Thu, 15 Jul 2010 09:48:28 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> In-Reply-To: <4C3E4D88.3040305@draigBrady.com> Content-Type: text/plain; charset=UTF-8 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090202.4C3E6947.01A5:SCFMA4539814,ss=1,fgs=0 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by rcsinet10.oracle.com id o6F1nxY6024857 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.7 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , Tao Ma , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.8 (-----) Hi P=C3=A1draig and Paul, Thanks for your quick response. P=C3=A1draig Brady wrote: > On 14/07/10 18:45, Paul Eggert wrote: >>>> I see fiemap just as a way to efficiently detect/read holes, >>>> and should have no bearing on the destination. >> Hmm, but the proposal quoted below would mean that fiemap does have a >> bearing on the destination, in the --sparse=3Dauto case. >> I guess this is OK, but it should be documented. >> >>>> cp --sparse=3Dauto (this is currently what cp does by default) >>>> recreate the original fiemap holes or resort to existing >>>> heuristic if fiemap not available >> It's not just fiemap. It's also the Solaris interface with SEEK_HOLE >> and SEEK_DATA. The change should involve a module that isolates these >> low-level details from copy.c. copy.c should ask the new module for t= he >> locations of the holes (or the non-holes: that could be more convenien= t). Consider the expansibility, its better to add a new file involves fiemap = and Solaris interface(I'll implement the fiemap at the moment). just like 'copy.c' shared the functions between cp(1) and mv(1). Maybe it could be used for other utilities to add new features related to= them. >> On traditional hosts without fiemap or SEEK_DATA, the module should re= port >> that it doesn't know where the holes are; this can let copy.c resort t= o >> the existing heuristic of looking at the size and the disk usage and >> using the --sparse=3Dalways approach if the file "smells" like it's sp= arse. >> >>>> cp --sparse=3Dnever >>>> write all data, but use fiemap if available to efficiently read >> Surely there's no need to write all the data if fallocate works. >> >>>> cp --sparse=3Dalways >>>> recreate original holes and perhaps extend add to them for >>>> other runs of zero bytes. Without having looked at the code >>>> I see this as a little tricky to mix with fiemap. >>>> Now since fiemap is only an optimization we can skip it >>>> completely for this uncommon case if too tricky (just add a FIXME = for now). >> Yes, that makes sense. --sparse=3Dalways should never invoke fallocat= e. >> >>> For 'cp --sparse=3Dnever', when detected holes from SRC file, do not = lseek(2) against DST file, >>> instead, write ZEROs to DST file, Am I right? >> Only if fallocate doesn't work. If fallocate works, there's no need >> to write zeros to the destination. >=20 > What you're describing here is posix_fallocate() > which uses fallocate() if available or falls back > to an implementation that writes a single 0 byte > to each block. >=20 >>> 2. Performance optimization, invoke fallocate(2) if an extent flag is= UNWRITTEN >> This doesn't sound right. A FIEMAP_EXTENT_UNWRITTEN extent is all zer= os, and >> so it should act as if it were a hole. The goal is not to copy the ex= act >> fiemap structure of the source (that's impossible): the goal is to use= as >> little time and space as possible. A FIEMAP_EXTENT_UNWRITTEN extent is marked to allocated although read it = will return ZEROs through the filesystem. So why not using fallocate(2) to deal with it? IMHO, it= meet the goal to use little time and space as possible, Am I miss something? >>=20 >>> If you decide to do that, then please do it as a separate patch. >> It's not clear to me that the fiemap stuff can be cleanly separated >> from the fallocate stuff. To some extent they're the same issue. >> If they can easily be separated, that's better of course. >=20 > I see fiemap as optimizing reads, > posix_fallocate() as optimizing writing zeros > and fallocate() as optimizing allocation. >=20 > So not having thought much about implementation details, > it seems like they could be logically separated. > I.E. we could optimize the writing zeros and allocation > later when we have the fallocate and posix_fallocate > gnulib modules in place. I think so, its better to wait until the changes done in gnulib, for now,= we can add FIXME for both cases. >=20 > In saying that, doing both now is better > when these details are in everyone's minds. > I'll not get to resubmitting my fallocate gnulib patch, > or doing a posix_fallocate module, this week at least I think. >=20 > cheers, > P=C3=A1draig. Thanks, -Jeff --=20 With Windows 7, Microsoft is asserting legal control over your computer a= nd is using this power to abuse computer users. From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 15 18:13:05 2010 Received: (at submit) by debbugs.gnu.org; 15 Jul 2010 22:13:05 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZWfx-0005v2-FG for submit@debbugs.gnu.org; Thu, 15 Jul 2010 18:13:05 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZWfw-0005ug-2O for submit@debbugs.gnu.org; Thu, 15 Jul 2010 18:13:04 -0400 Received: from lists.gnu.org ([199.232.76.165]:44847) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OZWg7-0003Me-NS for submit@debbugs.gnu.org; Thu, 15 Jul 2010 18:13:15 -0400 Received: from [140.186.70.92] (port=36980 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OZWg6-0007TM-0N for bug-coreutils@gnu.org; Thu, 15 Jul 2010 18:13:15 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OZWg4-0005MY-W1 for bug-coreutils@gnu.org; Thu, 15 Jul 2010 18:13:14 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:34723) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZWg4-0005JB-Qz for bug-coreutils@gnu.org; Thu, 15 Jul 2010 18:13:12 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o6FMD4Kr020937 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 15 Jul 2010 22:13:05 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o6FJDLZE008896; Thu, 15 Jul 2010 22:13:03 GMT Received: from ca-server1.us.oracle.com by acsmt355.oracle.com with ESMTP id 409575821279231980; Thu, 15 Jul 2010 15:13:00 -0700 Received: from jlbec by ca-server1.us.oracle.com with local (Exim 4.69) (envelope-from ) id 1OZWfs-0008LO-68; Thu, 15 Jul 2010 15:13:00 -0700 Date: Thu, 15 Jul 2010 15:12:58 -0700 From: Joel Becker To: =?iso-8859-1?Q?P=E1draig?= Brady Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy Message-ID: <20100715221258.GD29283@mail.oracle.com> References: <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <4C3E4D88.3040305@draigBrady.com> X-Burt-Line: Trees are cool. X-Red-Smith: Ninety feet between bases is perhaps as close as man has ever come to perfection. User-Agent: Mutt/1.5.20 (2009-06-14) X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090207.4C3F87EF.0231:SCFMA4539814,ss=1,fgs=0 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by rcsinet10.oracle.com id o6FMD4Kr020937 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Jim Meyering , "jeff.liu" , Chris Mason , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) On Thu, Jul 15, 2010 at 12:51:36AM +0100, P=E1draig Brady wrote: > On 14/07/10 18:45, Paul Eggert wrote: First and foremost, I re-concur with the broad strokes of the --sparse=3D{always,never,auto} conversation. I think you all knew that, though ;-) > > It's not just fiemap. It's also the Solaris interface with SEEK_HOLE > > and SEEK_DATA. The change should involve a module that isolates thes= e > > low-level details from copy.c. copy.c should ask the new module for = the > > locations of the holes (or the non-holes: that could be more convenie= nt). > > On traditional hosts without fiemap or SEEK_DATA, the module should r= eport > > that it doesn't know where the holes are; this can let copy.c resort = to > > the existing heuristic of looking at the size and the disk usage and > > using the --sparse=3Dalways approach if the file "smells" like it's s= parse. While I think the final result wants to support both fiemap and SEEK_HOLE, I think baby steps are in order. If we just implement fiemap right now, we can later turn that into init_extent_detection() and=20 get_next_extent(). > >> 2. Performance optimization, invoke fallocate(2) if an extent flag i= s UNWRITTEN > >=20 > > This doesn't sound right. A FIEMAP_EXTENT_UNWRITTEN extent is all ze= ros, and > > so it should act as if it were a hole. The goal is not to copy the e= xact > > fiemap structure of the source (that's impossible): the goal is to us= e as > > little time and space as possible. What he said. If you find an FIEMAP_EXTENT_UNWRITTEN extent, you just skip it. It is a hole for the purposes of copying. If someone really wants to clone the extent layout, they can use reflink(8). > > It's not clear to me that the fiemap stuff can be cleanly separated > > from the fallocate stuff. To some extent they're the same issue. > > If they can easily be separated, that's better of course. >=20 > I see fiemap as optimizing reads, > posix_fallocate() as optimizing writing zeros > and fallocate() as optimizing allocation. >=20 > So not having thought much about implementation details, > it seems like they could be logically separated. I think they should absolutely be separated. The fiemap patch doesn't have to do anything with fallocate()/posix_fallocate() on the write side. Let's get a happy fiemap patch. Then a happy [posix]_fallocate() patch. Then a happy SEEK_HOLE patch. Joel --=20 "For every complex problem there exists a solution that is brief, concise, and totally wrong." -Unknown Joel Becker Consulting Software Developer Oracle E-mail: joel.becker@oracle.com Phone: (650) 506-8127 From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 15 18:31:58 2010 Received: (at submit) by debbugs.gnu.org; 15 Jul 2010 22:31:59 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZWyE-00062Q-JD for submit@debbugs.gnu.org; Thu, 15 Jul 2010 18:31:58 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZWyB-00062L-V6 for submit@debbugs.gnu.org; Thu, 15 Jul 2010 18:31:56 -0400 Received: from lists.gnu.org ([199.232.76.165]:47844) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OZWyN-0003qd-Tm for submit@debbugs.gnu.org; Thu, 15 Jul 2010 18:32:07 -0400 Received: from [140.186.70.92] (port=50005 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OZWyM-0003WG-6e for bug-coreutils@gnu.org; Thu, 15 Jul 2010 18:32:07 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OZWyL-00006p-7P for bug-coreutils@gnu.org; Thu, 15 Jul 2010 18:32:06 -0400 Received: from kiwi.cs.ucla.edu ([131.179.128.19]:53100) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZWyK-00006V-Ur for bug-coreutils@gnu.org; Thu, 15 Jul 2010 18:32:05 -0400 Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200]) by kiwi.cs.ucla.edu (8.13.8+Sun/8.13.8/UCLACS-6.0) with ESMTP id o6FMVrPe011635; Thu, 15 Jul 2010 15:31:53 -0700 (PDT) Message-ID: <4C3F8C58.2020706@cs.ucla.edu> Date: Thu, 15 Jul 2010 15:31:52 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.10) Gecko/20100527 Thunderbird/3.0.5 MIME-Version: 1.0 To: Joel Becker Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <20100715221258.GD29283@mail.oracle.com> In-Reply-To: <20100715221258.GD29283@mail.oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , "jeff.liu" , Chris Mason , =?ISO-8859-1?Q?P=E1draig_Brady?= , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.0 (-----) >>> This doesn't sound right. A FIEMAP_EXTENT_UNWRITTEN extent is all zeros, and >>> so it should act as if it were a hole. The goal is not to copy the exact >>> fiemap structure of the source (that's impossible): the goal is to use as >>> little time and space as possible. > A FIEMAP_EXTENT_UNWRITTEN extent is marked to allocated although > read it will return ZEROs through the filesystem. So why not using > fallocate(2) to deal with it? IMHO, it meet the goal to use little > time and space as possible, Am I miss something? It's faster to simply skip around that extent while reading it, and to skip around it when writing it, than to call fallocate when writing it. Logically, a FIEMAP_EXTENT_UNWRITTEN extent is a hole, and should be optimized when reading, just like any hole. >> I see fiemap as optimizing reads, >> posix_fallocate() as optimizing writing zeros >> and fallocate() as optimizing allocation. It may not be quite that simple. Some platforms won't have fallocate and so posix_fallocate will have to do double duty as optimizing allocation too. Also, lseek is part of the process of optimizing reads, and of optimizing writing zeros. And fallocate also optimizes writing zeros, I expect. I'm not objecting to breaking these improvements into two or three pieces, if someone wants to do that. However, it shouldn't be required to break them up; it's OK if someone wants to do it all at once. (This stuff is not that hard, after all.) I was planning to give it a shot at some point but obviously have not done so yet. From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 15 18:51:42 2010 Received: (at submit) by debbugs.gnu.org; 15 Jul 2010 22:51:42 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZXHK-0006BO-BD for submit@debbugs.gnu.org; Thu, 15 Jul 2010 18:51:42 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZXHH-0006BJ-Od for submit@debbugs.gnu.org; Thu, 15 Jul 2010 18:51:40 -0400 Received: from lists.gnu.org ([199.232.76.165]:40032) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OZXHT-00048g-J3 for submit@debbugs.gnu.org; Thu, 15 Jul 2010 18:51:51 -0400 Received: from [140.186.70.92] (port=52397 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OZXHR-0007WL-UL for bug-coreutils@gnu.org; Thu, 15 Jul 2010 18:51:50 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OZXHQ-0002xR-Rb for bug-coreutils@gnu.org; Thu, 15 Jul 2010 18:51:50 -0400 Received: from kiwi.cs.ucla.edu ([131.179.128.19]:53205) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZXHQ-0002xJ-GN for bug-coreutils@gnu.org; Thu, 15 Jul 2010 18:51:48 -0400 Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200]) by kiwi.cs.ucla.edu (8.13.8+Sun/8.13.8/UCLACS-6.0) with ESMTP id o6FMpfhV011804; Thu, 15 Jul 2010 15:51:41 -0700 (PDT) Message-ID: <4C3F90FC.5060907@cs.ucla.edu> Date: Thu, 15 Jul 2010 15:51:40 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.10) Gecko/20100527 Thunderbird/3.0.5 MIME-Version: 1.0 To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> In-Reply-To: <4C3E68EC.4000408@oracle.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , Chris Mason , =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.0 (-----) >>> This doesn't sound right. A FIEMAP_EXTENT_UNWRITTEN extent is all zeros, and >>> so it should act as if it were a hole. The goal is not to copy the exact >>> fiemap structure of the source (that's impossible): the goal is to use as >>> little time and space as possible. > A FIEMAP_EXTENT_UNWRITTEN extent is marked to allocated although > read it will return ZEROs through the filesystem. So why not using > fallocate(2) to deal with it? IMHO, it meet the goal to use little > time and space as possible, Am I miss something? It's faster to simply skip around that extent while reading it, and to skip around it when writing it, than to allocate it with fallocate when writing it. Logically, a FIEMAP_EXTENT_UNWRITTEN extent is a hole, and should be optimized when reading and writing, just like any hole. >> I see fiemap as optimizing reads, >> posix_fallocate() as optimizing writing zeros >> and fallocate() as optimizing allocation. It may not be quite that simple. Some platforms won't have fallocate and so posix_fallocate will have to do double duty as optimizing allocation too. Also, lseek is part of the process of optimizing reads, and of optimizing writing zeros. Most important, the heuristics for optimizing the writes should use info derived from optimizing the reads. I'm not objecting to breaking these improvements into two or three pieces, if someone wants to do that. However, it shouldn't be required to break them up; it's OK if someone wants to do it all at once. (This stuff is not that hard, after all.) I was planning to give it a shot at some point but obviously have not done so yet. From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 16 10:50:42 2010 Received: (at submit) by debbugs.gnu.org; 16 Jul 2010 14:50:43 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZmFO-0005Y9-06 for submit@debbugs.gnu.org; Fri, 16 Jul 2010 10:50:42 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZmFL-0005Y4-3Y for submit@debbugs.gnu.org; Fri, 16 Jul 2010 10:50:41 -0400 Received: from lists.gnu.org ([199.232.76.165]:48782) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OZmFX-00029N-B4 for submit@debbugs.gnu.org; Fri, 16 Jul 2010 10:50:51 -0400 Received: from [140.186.70.92] (port=50267 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OZmFR-0002eC-GW for bug-coreutils@gnu.org; Fri, 16 Jul 2010 10:50:50 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OZmFL-0003Q7-V6 for bug-coreutils@gnu.org; Fri, 16 Jul 2010 10:50:44 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:28624) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZmFL-0003Pb-Lu for bug-coreutils@gnu.org; Fri, 16 Jul 2010 10:50:39 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o6GEoSTT000837 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 16 Jul 2010 14:50:30 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o6GBdvOI016152; Fri, 16 Jul 2010 14:50:27 GMT Received: from abhmt016.oracle.com by acsmt355.oracle.com with ESMTP id 432666491279291787; Fri, 16 Jul 2010 07:49:47 -0700 Received: from [123.119.97.177] (/123.119.97.177) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 16 Jul 2010 07:49:45 -0700 Message-ID: <4C407197.4050908@oracle.com> Date: Fri, 16 Jul 2010 22:49:59 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> In-Reply-To: <4C3F90FC.5060907@cs.ucla.edu> Content-Type: text/plain; charset=UTF-8 X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090204.4C4071B4.013E:SCFMA4539814,ss=1,pt=113351,fgs=0 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by rcsinet10.oracle.com id o6GEoSTT000837 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.4 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , Chris Mason , =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.1 (------) Paul Eggert wrote: >>>> This doesn't sound right. A FIEMAP_EXTENT_UNWRITTEN extent is all z= eros, and >>>> so it should act as if it were a hole. The goal is not to copy the = exact >>>> fiemap structure of the source (that's impossible): the goal is to u= se as >>>> little time and space as possible. >=20 >> A FIEMAP_EXTENT_UNWRITTEN extent is marked to allocated although >> read it will return ZEROs through the filesystem. So why not using >> fallocate(2) to deal with it? IMHO, it meet the goal to use little >> time and space as possible, Am I miss something? >=20 > It's faster to simply skip around that extent while reading it, and to > skip around it when writing it, than to allocate it with fallocate > when writing it. Logically, a FIEMAP_EXTENT_UNWRITTEN extent is a > hole, and should be optimized when reading and writing, just like any > hole. >=20 >>> I see fiemap as optimizing reads, >>> posix_fallocate() as optimizing writing zeros >>> and fallocate() as optimizing allocation. >=20 > It may not be quite that simple. Some platforms won't have fallocate > and so posix_fallocate will have to do double duty as optimizing > allocation too. Also, lseek is part of the process of optimizing > reads, and of optimizing writing zeros. Most important, the > heuristics for optimizing the writes should use info derived from > optimizing the reads. >=20 > I'm not objecting to breaking these improvements into two or three > pieces, if someone wants to do that. However, it shouldn't be > required to break them up; it's OK if someone wants to do it all at > once. (This stuff is not that hard, after all.) I was planning to > give it a shot at some point but obviously have not done so yet. Hi paul, P=C3=A1draig and All, Thanks for all your comments! For now, I am inclined to separate efficient read through fiemap and improve the write and allocation stuff via fallocate() or other ways = later. In this version, I created a new file src/sparse-core.c(not find out a be= tter name yet :( ) to place the original fiemap_copy(). Below is the revised fiemap_copy(), it honor make_holes to write all data= (a new function fill_with_holes_ok() implemented at sparse-core.c for this purpose) if --= sparse=3Dnever option is specified, it still does same thing as original for --sparse=3Dalways and= --sparse=3Dauto excecpt make use of external buf to avoid allocated another one while reading and writ= ing. >From 53e1975cb23a4d3c9d9f982d1e48437e95b071d8 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Fri, 16 Jul 2010 22:25:21 +0800 Subject: [PATCH 1/1] cp: Isolate fiemap_copy() to sparse-core.c * src/sparse-core.c: new file include fiemap_copy() * src/sparse-core.h: header file of sparse-core.c * src/copy.c (copy_reg): Now, `cp' attempt to get an fiemap of the source= file by default, fall back to a normal copy if the underlaying file system does not support it. We honor --sparse=3Dnever by writing all data but use fiemap if availab= le to efficiently read. * po/POTFILES.in: add sparse-core.c to it to pass po_check for translatab= le diagnostics. Signed-off-by: Jie Liu --- po/POTFILES.in | 1 + src/Makefile.am | 3 +- src/copy.c | 187 ++++------------------------------------- src/sparse-core.c | 240 +++++++++++++++++++++++++++++++++++++++++++++++= ++++++ src/sparse-core.h | 24 ++++++ 5 files changed, 284 insertions(+), 171 deletions(-) create mode 100644 src/sparse-core.c create mode 100644 src/sparse-core.h diff --git a/po/POTFILES.in b/po/POTFILES.in index c862877..025c9b0 100644 --- a/po/POTFILES.in +++ b/po/POTFILES.in @@ -110,6 +110,7 @@ src/shred.c src/shuf.c src/sleep.c src/sort.c +src/sparse-core.c src/split.c src/stat.c src/stdbuf.c diff --git a/src/Makefile.am b/src/Makefile.am index 7d56312..a838920 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -144,6 +144,7 @@ noinst_HEADERS =3D \ chown-core.h \ copy.h \ cp-hash.h \ + sparse-core.h \ dircolors.h \ fiemap.h \ fs.h \ @@ -459,7 +460,7 @@ uninstall-local: fi; \ fi -copy_sources =3D copy.c cp-hash.c +copy_sources =3D copy.c cp-hash.c sparse-core.c # Use `ginstall' in the definition of PROGRAMS and in dependencies to av= oid # confusion with the `install' target. The install rule transforms `gin= stall' diff --git a/src/copy.c b/src/copy.c index f48c74d..f6d9b5e 100644 --- a/src/copy.c +++ b/src/copy.c @@ -30,6 +30,7 @@ #endif #include "system.h" +#include "sparse-core.h" #include "acl.h" #include "backupfile.h" #include "buffer-lcm.h" @@ -63,10 +64,6 @@ #include -#ifndef HAVE_FIEMAP -# include "fiemap.h" -#endif - #ifndef HAVE_FCHOWN # define HAVE_FCHOWN false # define fchown(fd, uid, gid) (-1) @@ -153,153 +150,6 @@ clone_file (int dest_fd, int src_fd) #endif } -#ifdef __linux__ -# ifndef FS_IOC_FIEMAP -# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) -# endif -/* Perform a FIEMAP copy, if possible. - Call ioctl(2) with FS_IOC_FIEMAP (available in linux 2.6.27) to - obtain a map of file extents excluding holes. This avoids the - overhead of detecting holes in a hole-introducing/preserving copy, - and thus makes copying sparse files much more efficient. Upon a - successful copy, return true. If the initial ioctl fails, set - *NORMAL_COPY_REQUIRED to true and return false. Upon any other - failure, set *NORMAL_COPY_REQUIRED to false and return false. */ -static bool -fiemap_copy (int src_fd, int dest_fd, size_t buf_size, - off_t src_total_size, char const *src_name, - char const *dst_name, bool *normal_copy_required) -{ - bool last =3D false; - union { struct fiemap f; char c[4096]; } fiemap_buf; - struct fiemap *fiemap =3D &fiemap_buf.f; - struct fiemap_extent *fm_ext =3D &fiemap->fm_extents[0]; - enum { count =3D (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext= }; - verify (count !=3D 0); - - off_t last_ext_logical =3D 0; - uint64_t last_ext_len =3D 0; - uint64_t last_read_size =3D 0; - unsigned int i =3D 0; - *normal_copy_required =3D false; - - /* This is required at least to initialize fiemap->fm_start, - but also serves (in mid 2010) to appease valgrind, which - appears not to know the semantics of the FIEMAP ioctl. */ - memset (&fiemap_buf, 0, sizeof fiemap_buf); - - do - { - fiemap->fm_length =3D FIEMAP_MAX_OFFSET; - fiemap->fm_flags =3D FIEMAP_FLAG_SYNC; - fiemap->fm_extent_count =3D count; - - /* When ioctl(2) fails, fall back to the normal copy only if it - is the first time we met. */ - if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) - { - /* If the first ioctl fails, tell the caller that it is - ok to proceed with a normal copy. */ - if (i =3D=3D 0) - *normal_copy_required =3D true; - else - { - /* If the second or subsequent ioctl fails, diagnose it, - since it ends up causing the entire copy/cp to fail. *= / - error (0, errno, _("%s: FIEMAP ioctl failed"), quote (src_= name)); - } - return false; - } - - /* If 0 extents are returned, then more ioctls are not needed. */ - if (fiemap->fm_mapped_extents =3D=3D 0) - break; - - for (i =3D 0; i < fiemap->fm_mapped_extents; i++) - { - assert (fm_ext[i].fe_logical <=3D OFF_T_MAX); - - off_t ext_logical =3D fm_ext[i].fe_logical; - uint64_t ext_len =3D fm_ext[i].fe_length; - - if (lseek (src_fd, ext_logical, SEEK_SET) < 0) - { - error (0, errno, _("cannot lseek %s"), quote (src_name)); - return false; - } - - if (lseek (dest_fd, ext_logical, SEEK_SET) < 0) - { - error (0, errno, _("cannot lseek %s"), quote (dst_name)); - return false; - } - - if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST) - { - last_ext_logical =3D ext_logical; - last_ext_len =3D ext_len; - last =3D true; - } - - while (ext_len) - { - char buf[buf_size]; - - /* Avoid reading into the holes if the left extent - length is shorter than the buffer size. */ - if (ext_len < buf_size) - buf_size =3D ext_len; - - ssize_t n_read =3D read (src_fd, buf, buf_size); - if (n_read < 0) - { -#ifdef EINTR - if (errno =3D=3D EINTR) - continue; -#endif - error (0, errno, _("reading %s"), quote (src_name)); - return false; - } - - if (n_read =3D=3D 0) - { - /* Figure out how many bytes read from the last extent= . */ - last_read_size =3D last_ext_len - ext_len; - break; - } - - if (full_write (dest_fd, buf, n_read) !=3D n_read) - { - error (0, errno, _("writing %s"), quote (dst_name)); - return false; - } - - ext_len -=3D n_read; - } - } - - fiemap->fm_start =3D fm_ext[i - 1].fe_logical + fm_ext[i - 1].fe_l= ength; - - } while (! last); - - /* If a file ends up with holes, the sum of the last extent logical of= fset - and the read-returned size will be shorter than the actual size of = the - file. Use ftruncate to extend the length of the destination file. = */ - if (last_ext_logical + last_read_size < src_total_size) - { - if (ftruncate (dest_fd, src_total_size) < 0) - { - error (0, errno, _("failed to extend %s"), quote (dst_name)); - return false; - } - } - - return true; -} -#else -static bool fiemap_copy (ignored) { errno =3D=3D ENOTSUP; return false; = } -#endif - /* FIXME: describe */ /* FIXME: rewrite this to use a hash table so we avoid the quadratic performance hit that's probably noticeable only on trees deeper @@ -830,25 +680,6 @@ copy_reg (char const *src_name, char const *dst_name= , #endif } - if (make_holes) - { - bool require_normal_copy; - /* Perform efficient FIEMAP copy for sparse files, fall back t= o the - standard copy only if the ioctl(2) fails. */ - if (fiemap_copy (source_desc, dest_desc, buf_size, - src_open_sb.st_size, src_name, - dst_name, &require_normal_copy)) - goto preserve_metadata; - else - { - if (! require_normal_copy) - { - return_val =3D false; - goto close_src_and_dst_desc; - } - } - } - /* If not making a sparse file, try to use a more-efficient buffer size. */ if (! make_holes) @@ -877,6 +708,22 @@ copy_reg (char const *src_name, char const *dst_name= , buf_alloc =3D xmalloc (buf_size + buf_alignment_slop); buf =3D ptr_align (buf_alloc, buf_alignment); + bool require_normal_copy; + /* Perform efficient FIEMAP copy for sparse files, fall back to th= e + standard copy only if the ioctl(2) fails. */ + if (fiemap_copy (source_desc, dest_desc, buf, buf_size, + make_holes, src_name, dst_name, + src_open_sb.st_size, &require_normal_copy)) + goto preserve_metadata; + else + { + if (! require_normal_copy) + { + return_val =3D false; + goto close_src_and_dst_desc; + } + } + while (true) { word *wp =3D NULL; diff --git a/src/sparse-core.c b/src/sparse-core.c new file mode 100644 index 0000000..e122dc8 --- /dev/null +++ b/src/sparse-core.c @@ -0,0 +1,240 @@ +/* sparse-core.c -- core functions for efficient reading sparse files + Copyright (C) 2010 Free Software Foundation, Inc. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see .= */ + +#include +#include +#include +#include +#include +#include + +#include "system.h" +#include "sparse-core.h" +#include "error.h" +#include "quote.h" +#include "full-write.h" + +#ifndef HAVE_FIEMAP +# include "fiemap.h" +#endif + +static bool +fill_with_holes_ok(int dest_fd, const char *dst_name, + char *buf, size_t buf_size, + uint64_t holes_len) +{ + while (buf_size < holes_len) + { + if (full_write (dest_fd, buf, buf_size) !=3D buf_size) + { + error (0, errno, _("writing %s"), quote (dst_name)); + return false; + } + holes_len -=3D buf_size; + } + + if (0 < holes_len) + { + if (full_write (dest_fd, buf, holes_len) !=3D holes_len) + { + error (0, errno, _("writing %s"), quote (dst_name)); + return false; + } + } + + return true; +} + + +#ifdef __linux__ +# ifndef FS_IOC_FIEMAP +# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) +# endif + +/* Perform a FIEMAP copy, if possible. + Call ioctl(2) with FS_IOC_FIEMAP (available in linux 2.6.27) to + obtain a map of file extents excluding holes. This avoids the + overhead of detecting holes in a hole-introducing/preserving copy, + and thus makes copying sparse files much more efficient. Upon a + successful copy, return true. If the initial ioctl fails, set + *NORMAL_COPY_REQUIRED to true and return false. Upon any other + failure, set *NORMAL_COPY_REQUIRED to false and return false. */ + +extern bool +fiemap_copy (int src_fd, int dest_fd, char *buf, + size_t buf_size, bool make_holes, + char const *src_name, char const *dst_name, + off_t src_total_size, bool *normal_copy_required) +{ + bool last =3D false; + union { struct fiemap f; char c[4096]; } fiemap_buf; + struct fiemap *fiemap =3D &fiemap_buf.f; + struct fiemap_extent *fm_ext =3D &fiemap->fm_extents[0]; + enum { count =3D (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext= }; + verify (count !=3D 0); + + off_t prev_ext_logical =3D 0; + off_t last_ext_logical =3D 0; + uint64_t prev_ext_len =3D 0; + uint64_t last_ext_len =3D 0; + uint64_t last_read_size =3D 0; + unsigned int i =3D 0; + *normal_copy_required =3D false; + + if (! make_holes) + memset (buf, 0, buf_size); + + /* This is required at least to initialize fiemap->fm_start, + but also serves (in mid 2010) to appease valgrind, which + appears not to know the semantics of the FIEMAP ioctl. */ + memset (&fiemap_buf, 0, sizeof fiemap_buf); + + do + { + fiemap->fm_length =3D FIEMAP_MAX_OFFSET; + fiemap->fm_flags =3D FIEMAP_FLAG_SYNC; + fiemap->fm_extent_count =3D count; + + /* When ioctl(2) fails, fall back to the normal copy only if it + is the first time we met. */ + if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) + { + /* If the first ioctl fails, tell the caller that it is + ok to proceed with a normal copy. */ + if (i =3D=3D 0) + *normal_copy_required =3D true; + else + { + /* If the second or subsequent ioctl fails, diagnose it, + since it ends up causing the entire copy/cp to fail. *= / + error (0, errno, _("%s: FIEMAP ioctl failed"), quote (src_= name)); + } + return false; + } + + /* If 0 extents are returned, then more ioctls are not needed. */ + if (fiemap->fm_mapped_extents =3D=3D 0) + break; + + for (i =3D 0; i < fiemap->fm_mapped_extents; i++) + { + assert (fm_ext[i].fe_logical <=3D OFF_T_MAX); + + off_t ext_logical =3D fm_ext[i].fe_logical; + uint64_t ext_len =3D fm_ext[i].fe_length; + + if (lseek (src_fd, ext_logical, SEEK_SET) < 0) + { + error (0, errno, _("cannot lseek %s"), quote (src_name)); + return false; + } + + if (0 < i) + { + prev_ext_logical =3D fm_ext[i - 1].fe_logical, + prev_ext_len =3D fm_ext[i - 1].fe_length; + } + + if (! make_holes) + { + if (prev_ext_logical + prev_ext_len < ext_logical) + { + uint64_t holes_len =3D ext_logical - prev_ext_logical = - prev_ext_len; + if (! fill_with_holes_ok (dest_fd, dst_name, buf, buf_= size, holes_len)) + return false; + } + } + else + { + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0) + { + error (0, errno, _("cannot lseek %s"), quote (dst_name= )); + return false; + } + } + + if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST) + { + last_ext_logical =3D ext_logical; + last_ext_len =3D ext_len; + last =3D true; + } + + while (ext_len) + { + /* Avoid reading into the holes if the left extent + length is shorter than the buffer size. */ + if (ext_len < buf_size) + buf_size =3D ext_len; + + ssize_t n_read =3D read (src_fd, buf, buf_size); + if (n_read < 0) + { +#ifdef EINTR + if (errno =3D=3D EINTR) + continue; +#endif + error (0, errno, _("reading %s"), quote (src_name)); + return false; + } + + if (n_read =3D=3D 0) + { + /* Figure out how many bytes read from the last extent= . */ + last_read_size =3D last_ext_len - ext_len; + break; + } + + if (full_write (dest_fd, buf, n_read) !=3D n_read) + { + error (0, errno, _("writing %s"), quote (dst_name)); + return false; + } + + ext_len -=3D n_read; + } + } + + fiemap->fm_start =3D fm_ext[i - 1].fe_logical + fm_ext[i - 1].fe_l= ength; + + } while (! last); + + /* If a file ends up with holes, the sum of the last extent logical of= fset + and the read-returned size will be shorter than the actual size of = the + file. Use ftruncate to extend the length of the destination file. = */ + if (last_ext_logical + last_read_size < src_total_size) + { + if (! make_holes) + { + uint64_t holes_len =3D src_total_size - last_ext_logical - las= t_ext_len; + if (! fill_with_holes_ok (dest_fd, dst_name, buf, buf_size, ho= les_len)) + return false; + } + else + { + if (ftruncate (dest_fd, src_total_size) < 0) + { + error (0, errno, _("failed to extend %s"), quote (dst_name= )); + return false; + } + } + } + + return true; +} +#else +extern bool fiemap_copy (ignored) { errno =3D=3D ENOTSUP; return false; = } +#endif diff --git a/src/sparse-core.h b/src/sparse-core.h new file mode 100644 index 0000000..89c9ebf --- /dev/null +++ b/src/sparse-core.h @@ -0,0 +1,24 @@ +/* core functions for efficient reading sparse files + Copyright (C) 2010 Free Software Foundation, Inc. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see .= */ + +#ifndef SPARSE_CORE_H +# define SPARSE_CORE_H + +bool fiemap_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, + bool make_holes, char const *src_name, char const *dst= _name, + off_t src_total_size, bool *normal_copy_required); + +#endif /* SPARSE_CORE_H */ --=20 1.5.4.3 Thanks, -Jeff --=20 The knowledge you get, no matter how much it is, must be possessed yourse= lf and nourished with your own painstaking efforts and be your achievement through hard work. From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 16 11:53:29 2010 Received: (at submit) by debbugs.gnu.org; 16 Jul 2010 15:53:30 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZnE9-0006cn-5x for submit@debbugs.gnu.org; Fri, 16 Jul 2010 11:53:29 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZnE7-0006ce-7F for submit@debbugs.gnu.org; Fri, 16 Jul 2010 11:53:28 -0400 Received: from lists.gnu.org ([199.232.76.165]:51811) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OZnEK-0003Z8-RF for submit@debbugs.gnu.org; Fri, 16 Jul 2010 11:53:40 -0400 Received: from [140.186.70.92] (port=43291 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OZnEJ-0006go-EZ for bug-coreutils@gnu.org; Fri, 16 Jul 2010 11:53:40 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OZnEI-0008Ev-Ar for bug-coreutils@gnu.org; Fri, 16 Jul 2010 11:53:39 -0400 Received: from kiwi.cs.ucla.edu ([131.179.128.19]:56589) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZnEI-0008EC-2D for bug-coreutils@gnu.org; Fri, 16 Jul 2010 11:53:38 -0400 Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200]) by kiwi.cs.ucla.edu (8.13.8+Sun/8.13.8/UCLACS-6.0) with ESMTP id o6GFrRDO019987; Fri, 16 Jul 2010 08:53:28 -0700 (PDT) Message-ID: <4C408077.3020800@cs.ucla.edu> Date: Fri, 16 Jul 2010 08:53:27 -0700 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.10) Gecko/20100527 Thunderbird/3.0.5 MIME-Version: 1.0 To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4BE41FFF.4020809@oracle.com> <4BF683CC.3060407@oracle.com> <4BF6ABE7.3020600@oracle.com> <877hmpeivx.fsf@meyering.net> <4BFF763A.9010205@oracle.com> <8763286kni.fsf@meyering.net> <4C0275FA.1030901@oracle.com> <874ohpywri.fsf@meyering.net> <87y6exuc5i.fsf@meyering.net> <874ohdu5b8.fsf@meyering.net> <87eighs1t9.fsf@meyering.net> <4C0FA93C.6020205@oracle.com> <87iq5sp7eg.fsf@meyering.net> <4C108CB9.8010805@oracle.com> <4C11798B.6090500@cs.ucla.edu> <4C1184B4.2050309@oracle.com> <4C17F08B.4020203@cs.ucla.edu> <87sk4odlaa.fsf@meyering.net> <4C17FC61.2020401@cs.ucla.edu> <87zkyvcw7f.fsf@meyering.net> <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> In-Reply-To: <4C407197.4050908@oracle.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Solaris 10 (beta) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.1 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Joel Becker , Chris Mason , =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.1 (-----) On 07/16/10 07:49, jeff.liu wrote: > For now, I am inclined to separate efficient read through fiemap > and improve the write and allocation stuff via fallocate() or other ways later. I haven't had time to look at it carefully, but here's a very brief review. The code you sent, like what's in the fiemap branch, has a separate version of a chunk of copy.c that does both reading and writing and optimizes both reading and writing by invoking the fiemap ioctls at strategic locations. Instead, it would be better to have a module that separates out the efficient-read stuff by telling copy.c where the next significant input extent is, and then modify copy.c to use that module. On hosts that do not support fiemap, the module would simply report the entire input file as that file's only extent. Surely such an approach would be more modular, and would result in less duplication of code. I can write something along those lines if there's interest and if nobody else wants to take a crack at it. From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 16 17:34:19 2010 Received: (at submit) by debbugs.gnu.org; 16 Jul 2010 21:34:20 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZsXz-00020O-Dd for submit@debbugs.gnu.org; Fri, 16 Jul 2010 17:34:19 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZsXx-00020J-Ro for submit@debbugs.gnu.org; Fri, 16 Jul 2010 17:34:18 -0400 Received: from lists.gnu.org ([199.232.76.165]:35442) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OZsYC-0001iD-1P for submit@debbugs.gnu.org; Fri, 16 Jul 2010 17:34:32 -0400 Received: from [140.186.70.92] (port=50117 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OZsYA-0000Sw-H6 for bug-coreutils@gnu.org; Fri, 16 Jul 2010 17:34:31 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OZsY9-0005E3-CT for bug-coreutils@gnu.org; Fri, 16 Jul 2010 17:34:30 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:55096) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZsY9-0005Dy-3x for bug-coreutils@gnu.org; Fri, 16 Jul 2010 17:34:29 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o6GLYMv5019762 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 16 Jul 2010 21:34:24 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o6GL2gFW021614; Fri, 16 Jul 2010 21:34:20 GMT Received: from ca-server1.us.oracle.com by acsmt354.oracle.com with ESMTP id 412839821279315998; Fri, 16 Jul 2010 14:33:18 -0700 Received: from jlbec by ca-server1.us.oracle.com with local (Exim 4.69) (envelope-from ) id 1OZsWz-0002cE-Hw; Fri, 16 Jul 2010 14:33:17 -0700 Date: Fri, 16 Jul 2010 14:33:17 -0700 From: Joel Becker To: Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy Message-ID: <20100716213317.GC30859@mail.oracle.com> References: <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C408077.3020800@cs.ucla.edu> X-Burt-Line: Trees are cool. X-Red-Smith: Ninety feet between bases is perhaps as close as man has ever come to perfection. User-Agent: Mutt/1.5.20 (2009-06-14) X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A0B0204.4C40D05D.0120:SCFMA4539814,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , "jeff.liu" , Chris Mason , =?iso-8859-1?Q?P=E1draig?= Brady , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) On Fri, Jul 16, 2010 at 08:53:27AM -0700, Paul Eggert wrote: > I haven't had time to look at it carefully, but here's a very brief > review. The code you sent, like what's in the fiemap branch, has > a separate version of a chunk of copy.c that does both reading > and writing and optimizes both reading and writing by invoking the fiemap ioctls > at strategic locations. Instead, it would be better to have > a module that separates out the efficient-read stuff by telling > copy.c where the next significant input extent is, and then modify copy.c > to use that module. On hosts that do not support fiemap, the module > would simply report the entire input file as that file's only extent. Precisely. The sparse-core.c or whatever it is called shouldn't be doing the copy, it should just provide: handle = init_extent_scan(fd); while (get_next_extent(handle, &extent_start, &extent_len)) { ... } close_extent_scan(handle); Then copy.c just implements this loop and the '...' part. Joel -- "A narcissist is someone better looking than you are." - Gore Vidal Joel Becker Consulting Software Developer Oracle E-mail: joel.becker@oracle.com Phone: (650) 506-8127 From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 16 22:11:35 2010 Received: (at submit) by debbugs.gnu.org; 17 Jul 2010 02:11:36 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZwsI-0003it-Us for submit@debbugs.gnu.org; Fri, 16 Jul 2010 22:11:35 -0400 Received: from mx10.gnu.org ([199.232.76.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZwsH-0003io-Ey for submit@debbugs.gnu.org; Fri, 16 Jul 2010 22:11:34 -0400 Received: from lists.gnu.org ([199.232.76.165]:58511) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1OZwsV-0004nA-Tg for submit@debbugs.gnu.org; Fri, 16 Jul 2010 22:11:47 -0400 Received: from [140.186.70.92] (port=50579 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OZwsU-00035u-KX for bug-coreutils@gnu.org; Fri, 16 Jul 2010 22:11:47 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OZwsT-0006l0-9Z for bug-coreutils@gnu.org; Fri, 16 Jul 2010 22:11:46 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:29086) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OZwsT-0006kl-1f for bug-coreutils@gnu.org; Fri, 16 Jul 2010 22:11:45 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o6H2Bdj5026533 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sat, 17 Jul 2010 02:11:40 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o6H23QMw019287; Sat, 17 Jul 2010 02:11:38 GMT Received: from abhmt010.oracle.com by acsmt354.oracle.com with ESMTP id 413221521279332671; Fri, 16 Jul 2010 19:11:11 -0700 Received: from [221.223.114.1] (/221.223.114.1) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 16 Jul 2010 19:11:11 -0700 Message-ID: <4C411152.3070005@oracle.com> Date: Sat, 17 Jul 2010 10:11:30 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Joel Becker , Paul Eggert Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C188BE3.7080103@draigBrady.com> <87fx0ncqgr.fsf@meyering.net> <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> In-Reply-To: <20100716213317.GC30859@mail.oracle.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090201.4C41115B.003D:SCFMA4539814,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.8 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , bug-coreutils@gnu.org, Jim Meyering , Tao Ma , =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.8 (-----) Joel Becker wrote: > On Fri, Jul 16, 2010 at 08:53:27AM -0700, Paul Eggert wrote: >> I haven't had time to look at it carefully, but here's a very brief >> review. The code you sent, like what's in the fiemap branch, has >> a separate version of a chunk of copy.c that does both reading >> and writing and optimizes both reading and writing by invoking the fiemap ioctls >> at strategic locations. Instead, it would be better to have >> a module that separates out the efficient-read stuff by telling >> copy.c where the next significant input extent is, and then modify copy.c >> to use that module. On hosts that do not support fiemap, the module >> would simply report the entire input file as that file's only extent. > > Precisely. The sparse-core.c or whatever it is called shouldn't > be doing the copy, it should just provide: > > handle = init_extent_scan(fd); > while (get_next_extent(handle, &extent_start, &extent_len)) { > ... > } > close_extent_scan(handle); > > Then copy.c just implements this loop and the '...' part. > > Joel > yes, its better to separate copy and extent scan, and its not difficult to implement. But I was wondering to return an array of extents info or just return one extent info for each scan? I would like to work out an unique interface could work for both Linux and Solaris, for Solaris SEEK_DATA/HOLES stuff, looks its convinent to just return next extent info every time. But for fiemap, maybe its better to return an extents_info_array as currentt design to reduce the ioctl(2) calls. union { struct fiemap f; char c[4096]; } fiemap_buf; enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; Maybe we can encapsulae Solaris SEEK_DATA/SEEK_HOLE to return extents_info_array(if possible, not play with those APIs yet) if you guys aggree with above. then Joel's code example looks like: handle = init_extent_scan(fd); while (get_next_extents(handle, &extents_info)) { num = extents_info->extents_count; for (i = 0; i < num; i++) { extent_info = extents_info[i]; same_as_usual; } } close_extent_scan(handle); Thanks -Jeff -- The knowledge you get, no matter how much it is, must be possessed yourself and nourished with your own painstaking efforts and be your achievement through hard work. From debbugs-submit-bounces@debbugs.gnu.org Sat Jul 17 02:15:07 2010 Received: (at submit) by debbugs.gnu.org; 17 Jul 2010 06:15:07 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oa0fy-0005AD-NM for submit@debbugs.gnu.org; Sat, 17 Jul 2010 02:15:06 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oa0fx-00059d-2J for submit@debbugs.gnu.org; Sat, 17 Jul 2010 02:15:05 -0400 Received: from lists.gnu.org ([199.232.76.165]:43889) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1Oa0gC-0007MM-9i for submit@debbugs.gnu.org; Sat, 17 Jul 2010 02:15:20 -0400 Received: from [140.186.70.92] (port=54340 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Oa0gA-00025w-9l for bug-coreutils@gnu.org; Sat, 17 Jul 2010 02:15:19 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Oa0g9-0003nA-6q for bug-coreutils@gnu.org; Sat, 17 Jul 2010 02:15:18 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:47632) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oa0g9-0003n4-1D for bug-coreutils@gnu.org; Sat, 17 Jul 2010 02:15:17 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o6H6F7A2006712 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sat, 17 Jul 2010 06:15:09 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o6H4fiqw019670; Sat, 17 Jul 2010 06:15:06 GMT Received: from ca-server1.us.oracle.com by acsmt353.oracle.com with ESMTP id 413383961279347269; Fri, 16 Jul 2010 23:14:29 -0700 Received: from jlbec by ca-server1.us.oracle.com with local (Exim 4.69) (envelope-from ) id 1Oa0fM-0007HK-UR; Fri, 16 Jul 2010 23:14:29 -0700 Date: Fri, 16 Jul 2010 23:14:27 -0700 From: Joel Becker To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy Message-ID: <20100717061426.GA18756@mail.oracle.com> References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C411152.3070005@oracle.com> X-Burt-Line: Trees are cool. X-Red-Smith: Ninety feet between bases is perhaps as close as man has ever come to perfection. User-Agent: Mutt/1.5.20 (2009-06-14) X-Source-IP: acsmt355.oracle.com [141.146.40.155] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A0B0203.4C414A6B.010E:SCFMA4539814,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Jim Meyering , Chris Mason , =?iso-8859-1?Q?P=E1draig?= Brady , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) On Sat, Jul 17, 2010 at 10:11:30AM +0800, jeff.liu wrote: > Joel Becker wrote: > > On Fri, Jul 16, 2010 at 08:53:27AM -0700, Paul Eggert wrote: > >> I haven't had time to look at it carefully, but here's a very brief > >> review. The code you sent, like what's in the fiemap branch, has > >> a separate version of a chunk of copy.c that does both reading > >> and writing and optimizes both reading and writing by invoking the fiemap ioctls > >> at strategic locations. Instead, it would be better to have > >> a module that separates out the efficient-read stuff by telling > >> copy.c where the next significant input extent is, and then modify copy.c > >> to use that module. On hosts that do not support fiemap, the module > >> would simply report the entire input file as that file's only extent. > > > > Precisely. The sparse-core.c or whatever it is called shouldn't > > be doing the copy, it should just provide: > > > > handle = init_extent_scan(fd); > > while (get_next_extent(handle, &extent_start, &extent_len)) { > > ... > > } > > close_extent_scan(handle); > > > > Then copy.c just implements this loop and the '...' part. > > > > Joel > > > > yes, its better to separate copy and extent scan, and its not difficult to implement. But I was > wondering to return an array of extents info or just return one extent info for each scan? get_next_extent() just returns one extent, but the caller has no idea what is hanging off of handle. In fiemap, it could be a large array of extents you've cached during init_extent_scan(). For Solaris it might be some placeholder. > I would like to work out an unique interface could work for both Linux and Solaris, for Solaris > SEEK_DATA/HOLES stuff, looks its convinent to just return next extent info every time. > > But for fiemap, maybe its better to return an extents_info_array as currentt design to reduce the > ioctl(2) calls. You don't need multiple ioctl(2) calls. Here's a trivial example: void *init_extent_scan(int fd) { struct fiemap *handle; handle = malloc(sizeof(struct fiemap) + (EXTENTS_PER_IOCTL * sizeof(struct fiemap_extent))); handle->fm_extent_count = EXTENTS_PER_IOCTL; if (!ioctl(fd, FS_IOC_FIEMAP, handle)) return handle; if (lseek(fd, SEEK_HOLE) >= 0) return (void *)-1; return NULL; } loff_t get_next_extent(void *handle, loff_t *start, loff_t *len) { if (handle == (void *)-1) /* Do SEEK_HOLE thing */ else if (handle) return handle->fm_extents[next_one++]; } Joel -- Life's Little Instruction Book #497 "Go down swinging." Joel Becker Consulting Software Developer Oracle E-mail: joel.becker@oracle.com Phone: (650) 506-8127 From debbugs-submit-bounces@debbugs.gnu.org Sat Jul 17 02:23:07 2010 Received: (at submit) by debbugs.gnu.org; 17 Jul 2010 06:23:07 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oa0ni-0005DB-1L for submit@debbugs.gnu.org; Sat, 17 Jul 2010 02:23:06 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oa0ng-0005Cl-E6 for submit@debbugs.gnu.org; Sat, 17 Jul 2010 02:23:05 -0400 Received: from lists.gnu.org ([199.232.76.165]:34537) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1Oa0nv-0007V4-Bs for submit@debbugs.gnu.org; Sat, 17 Jul 2010 02:23:19 -0400 Received: from [140.186.70.92] (port=34727 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Oa0nt-0003Yl-OO for bug-coreutils@gnu.org; Sat, 17 Jul 2010 02:23:18 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Oa0ns-0004kK-LZ for bug-coreutils@gnu.org; Sat, 17 Jul 2010 02:23:17 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:34987) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oa0ns-0004kF-DA for bug-coreutils@gnu.org; Sat, 17 Jul 2010 02:23:16 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o6H6NBHp013349 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sat, 17 Jul 2010 06:23:13 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o6GL2g2D021617; Sat, 17 Jul 2010 06:23:10 GMT Received: from abhmt007.oracle.com by acsmt355.oracle.com with ESMTP id 413388821279347790; Fri, 16 Jul 2010 23:23:10 -0700 Received: from [221.223.104.244] (/221.223.104.244) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 16 Jul 2010 23:23:09 -0700 Message-ID: <4C414C5E.3060509@oracle.com> Date: Sat, 17 Jul 2010 14:23:26 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Joel Becker Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> In-Reply-To: <20100717061426.GA18756@mail.oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Source-IP: acsmt354.oracle.com [141.146.40.154] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090203.4C414C4F.0092:SCFMA4539814,ss=1,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.8 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Jim Meyering , Chris Mason , =?ISO-8859-1?Q?P=E1draig_Brady?= , Tao Ma X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.9 (-----) Joel Becker wrote: > On Sat, Jul 17, 2010 at 10:11:30AM +0800, jeff.liu wrote: >> Joel Becker wrote: >>> On Fri, Jul 16, 2010 at 08:53:27AM -0700, Paul Eggert wrote: >>>> I haven't had time to look at it carefully, but here's a very brief >>>> review. The code you sent, like what's in the fiemap branch, has >>>> a separate version of a chunk of copy.c that does both reading >>>> and writing and optimizes both reading and writing by invoking the fiemap ioctls >>>> at strategic locations. Instead, it would be better to have >>>> a module that separates out the efficient-read stuff by telling >>>> copy.c where the next significant input extent is, and then modify copy.c >>>> to use that module. On hosts that do not support fiemap, the module >>>> would simply report the entire input file as that file's only extent. >>> Precisely. The sparse-core.c or whatever it is called shouldn't >>> be doing the copy, it should just provide: >>> >>> handle = init_extent_scan(fd); >>> while (get_next_extent(handle, &extent_start, &extent_len)) { >>> ... >>> } >>> close_extent_scan(handle); >>> >>> Then copy.c just implements this loop and the '...' part. >>> >>> Joel >>> >> yes, its better to separate copy and extent scan, and its not difficult to implement. But I was >> wondering to return an array of extents info or just return one extent info for each scan? > > get_next_extent() just returns one extent, but the caller has no > idea what is hanging off of handle. In fiemap, it could be a large > array of extents you've cached during init_extent_scan(). For Solaris > it might be some placeholder. > >> I would like to work out an unique interface could work for both Linux and Solaris, for Solaris >> SEEK_DATA/HOLES stuff, looks its convinent to just return next extent info every time. >> >> But for fiemap, maybe its better to return an extents_info_array as currentt design to reduce the >> ioctl(2) calls. > > You don't need multiple ioctl(2) calls. Here's a trivial > example: > > void *init_extent_scan(int fd) > { > struct fiemap *handle; > > handle = malloc(sizeof(struct fiemap) + > (EXTENTS_PER_IOCTL * sizeof(struct fiemap_extent))); > handle->fm_extent_count = EXTENTS_PER_IOCTL; > if (!ioctl(fd, FS_IOC_FIEMAP, handle)) > return handle; > > if (lseek(fd, SEEK_HOLE) >= 0) > return (void *)-1; > > return NULL; > } > > loff_t get_next_extent(void *handle, loff_t *start, loff_t *len) > { > if (handle == (void *)-1) > /* Do SEEK_HOLE thing */ > else if (handle) > return handle->fm_extents[next_one++]; > } Thanks Joel, I understood your idea now! > > Joel > Regards, -Jeff -- The knowledge you get, no matter how much it is, must be possessed yourself and nourished with your own painstaking efforts and be your achievement through hard work. From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 20 11:24:05 2010 Received: (at submit) by debbugs.gnu.org; 20 Jul 2010 15:24:05 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ObEfs-0002yB-Q2 for submit@debbugs.gnu.org; Tue, 20 Jul 2010 11:24:05 -0400 Received: from mail.gnu.org ([199.232.76.166] helo=mx10.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ObEfp-0002xp-Fo for submit@debbugs.gnu.org; Tue, 20 Jul 2010 11:24:02 -0400 Received: from lists.gnu.org ([199.232.76.165]:46160) by monty-python.gnu.org with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1ObEgD-00032h-DW for submit@debbugs.gnu.org; Tue, 20 Jul 2010 11:24:25 -0400 Received: from [140.186.70.92] (port=48743 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1ObEgB-0003lO-0n for bug-coreutils@gnu.org; Tue, 20 Jul 2010 11:24:25 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1ObEg8-0006RZ-PQ for bug-coreutils@gnu.org; Tue, 20 Jul 2010 11:24:22 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:23734) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ObEg8-0006R8-DD for bug-coreutils@gnu.org; Tue, 20 Jul 2010 11:24:20 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o6KFO61M002887 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 20 Jul 2010 15:24:08 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o6JKXDqw013031; Tue, 20 Jul 2010 15:24:05 GMT Received: from abhmt020.oracle.com by acsmt354.oracle.com with ESMTP id 420385621279639441; Tue, 20 Jul 2010 08:24:01 -0700 Received: from [221.223.116.223] (/221.223.116.223) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 20 Jul 2010 08:24:00 -0700 Message-ID: <4C45BFA5.5000809@oracle.com> Date: Tue, 20 Jul 2010 23:24:21 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> In-Reply-To: <4C414C5E.3060509@oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Source-IP: acsmt353.oracle.com [141.146.40.153] X-Auth-Type: Internal IP X-CT-RefId: str=0001.0A090202.4C45BF96.00CD:SCFMA4539814,ss=1,pt=113351,fgs=0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by monty-python.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.6 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?ISO-8859-1?Q?P=E1draig_Brady?= , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.9 (-----) Hello All, Below is my patches to isolate the extents scan and fetch functions in a new module to improve its extendibility. It introduce a new file 'src/extent-scan.c' to place those functions, and then call those functions from extents_copy() at copy_reg() to process the regular file copy. In addition to this, another major change is to copy all data if '--sparse=never' option is specified. It write all data to destination file but using extents_copy() if available for efficient read source file, and it try to figure out the holes between the previous and current extents, and call fill_with_holes_ok() to write zeros as holes to the destination file if it is. Call file_with_holes_ok() to write zeros up to the source file size if hit the last extent of the source file and there is a hole behind it. I have not implement the solaris lseek(2) at the moment for lack of solaris environment, it need to delay a period of time. According to my tryout, it works for those 4 filesystems in common use, you all know. As usual, any comments are welcome! >From 70773fdf1d85ba070e054b0467a7a0e1e2b00ea8 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Tue, 20 Jul 2010 20:35:25 +0800 Subject: [PATCH 1/3] cp: delete fiemap_copy() related stuff from copy.c * delete fiemap_copy(), now it is implemented as a module. Signed-off-by: Jie Liu --- src/copy.c | 171 ------------------------------------------------------------ 1 files changed, 0 insertions(+), 171 deletions(-) diff --git a/src/copy.c b/src/copy.c index f48c74d..171499c 100644 --- a/src/copy.c +++ b/src/copy.c @@ -63,10 +63,6 @@ #include -#ifndef HAVE_FIEMAP -# include "fiemap.h" -#endif - #ifndef HAVE_FCHOWN # define HAVE_FCHOWN false # define fchown(fd, uid, gid) (-1) @@ -153,153 +149,6 @@ clone_file (int dest_fd, int src_fd) #endif } -#ifdef __linux__ -# ifndef FS_IOC_FIEMAP -# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) -# endif -/* Perform a FIEMAP copy, if possible. - Call ioctl(2) with FS_IOC_FIEMAP (available in linux 2.6.27) to - obtain a map of file extents excluding holes. This avoids the - overhead of detecting holes in a hole-introducing/preserving copy, - and thus makes copying sparse files much more efficient. Upon a - successful copy, return true. If the initial ioctl fails, set - *NORMAL_COPY_REQUIRED to true and return false. Upon any other - failure, set *NORMAL_COPY_REQUIRED to false and return false. */ -static bool -fiemap_copy (int src_fd, int dest_fd, size_t buf_size, - off_t src_total_size, char const *src_name, - char const *dst_name, bool *normal_copy_required) -{ - bool last = false; - union { struct fiemap f; char c[4096]; } fiemap_buf; - struct fiemap *fiemap = &fiemap_buf.f; - struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; - enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; - verify (count != 0); - - off_t last_ext_logical = 0; - uint64_t last_ext_len = 0; - uint64_t last_read_size = 0; - unsigned int i = 0; - *normal_copy_required = false; - - /* This is required at least to initialize fiemap->fm_start, - but also serves (in mid 2010) to appease valgrind, which - appears not to know the semantics of the FIEMAP ioctl. */ - memset (&fiemap_buf, 0, sizeof fiemap_buf); - - do - { - fiemap->fm_length = FIEMAP_MAX_OFFSET; - fiemap->fm_flags = FIEMAP_FLAG_SYNC; - fiemap->fm_extent_count = count; - - /* When ioctl(2) fails, fall back to the normal copy only if it - is the first time we met. */ - if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) - { - /* If the first ioctl fails, tell the caller that it is - ok to proceed with a normal copy. */ - if (i == 0) - *normal_copy_required = true; - else - { - /* If the second or subsequent ioctl fails, diagnose it, - since it ends up causing the entire copy/cp to fail. */ - error (0, errno, _("%s: FIEMAP ioctl failed"), quote (src_name)); - } - return false; - } - - /* If 0 extents are returned, then more ioctls are not needed. */ - if (fiemap->fm_mapped_extents == 0) - break; - - for (i = 0; i < fiemap->fm_mapped_extents; i++) - { - assert (fm_ext[i].fe_logical <= OFF_T_MAX); - - off_t ext_logical = fm_ext[i].fe_logical; - uint64_t ext_len = fm_ext[i].fe_length; - - if (lseek (src_fd, ext_logical, SEEK_SET) < 0) - { - error (0, errno, _("cannot lseek %s"), quote (src_name)); - return false; - } - - if (lseek (dest_fd, ext_logical, SEEK_SET) < 0) - { - error (0, errno, _("cannot lseek %s"), quote (dst_name)); - return false; - } - - if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST) - { - last_ext_logical = ext_logical; - last_ext_len = ext_len; - last = true; - } - - while (ext_len) - { - char buf[buf_size]; - - /* Avoid reading into the holes if the left extent - length is shorter than the buffer size. */ - if (ext_len < buf_size) - buf_size = ext_len; - - ssize_t n_read = read (src_fd, buf, buf_size); - if (n_read < 0) - { -#ifdef EINTR - if (errno == EINTR) - continue; -#endif - error (0, errno, _("reading %s"), quote (src_name)); - return false; - } - - if (n_read == 0) - { - /* Figure out how many bytes read from the last extent. */ - last_read_size = last_ext_len - ext_len; - break; - } - - if (full_write (dest_fd, buf, n_read) != n_read) - { - error (0, errno, _("writing %s"), quote (dst_name)); - return false; - } - - ext_len -= n_read; - } - } - - fiemap->fm_start = fm_ext[i - 1].fe_logical + fm_ext[i - 1].fe_length; - - } while (! last); - - /* If a file ends up with holes, the sum of the last extent logical offset - and the read-returned size will be shorter than the actual size of the - file. Use ftruncate to extend the length of the destination file. */ - if (last_ext_logical + last_read_size < src_total_size) - { - if (ftruncate (dest_fd, src_total_size) < 0) - { - error (0, errno, _("failed to extend %s"), quote (dst_name)); - return false; - } - } - - return true; -} -#else -static bool fiemap_copy (ignored) { errno == ENOTSUP; return false; } -#endif - /* FIXME: describe */ /* FIXME: rewrite this to use a hash table so we avoid the quadratic performance hit that's probably noticeable only on trees deeper @@ -830,25 +679,6 @@ copy_reg (char const *src_name, char const *dst_name, #endif } - if (make_holes) - { - bool require_normal_copy; - /* Perform efficient FIEMAP copy for sparse files, fall back to the - standard copy only if the ioctl(2) fails. */ - if (fiemap_copy (source_desc, dest_desc, buf_size, - src_open_sb.st_size, src_name, - dst_name, &require_normal_copy)) - goto preserve_metadata; - else - { - if (! require_normal_copy) - { - return_val = false; - goto close_src_and_dst_desc; - } - } - } - /* If not making a sparse file, try to use a more-efficient buffer size. */ if (! make_holes) @@ -977,7 +807,6 @@ copy_reg (char const *src_name, char const *dst_name, } } -preserve_metadata: if (x->preserve_timestamps) { struct timespec timespec[2]; -- 1.5.4.3 >From f083f1a52ec5baba90aa228c1053f4a32127b3b2 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Tue, 20 Jul 2010 23:10:29 +0800 Subject: [PATCH 2/3] cp: add a new module for scanning extents * src/extent-scan.c: Source code for scanning extents. Call init_extent_scan() to return an extents map. Call get_next_extent() to get the next extent for each iteration. Call close_extent_scan() to tear down the scan, it now do nothing. * src/extent-scan.h: Header file of extent-scan.c. * src/Makefile.am: Reference it. Signed-off-by: Jie Liu --- src/Makefile.am | 3 +- src/extent-scan.c | 137 +++++++++++++++++++++++++++++++++++++++++++++++++++++ src/extent-scan.h | 25 ++++++++++ 3 files changed, 164 insertions(+), 1 deletions(-) create mode 100644 src/extent-scan.c create mode 100644 src/extent-scan.h diff --git a/src/Makefile.am b/src/Makefile.am index 7d56312..fb8186c 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -145,6 +145,7 @@ noinst_HEADERS = \ copy.h \ cp-hash.h \ dircolors.h \ + extent-scan.h \ fiemap.h \ fs.h \ group-list.h \ @@ -459,7 +460,7 @@ uninstall-local: fi; \ fi -copy_sources = copy.c cp-hash.c +copy_sources = copy.c cp-hash.c extent-scan.c # Use `ginstall' in the definition of PROGRAMS and in dependencies to avoid # confusion with the `install' target. The install rule transforms `ginstall' diff --git a/src/extent-scan.c b/src/extent-scan.c new file mode 100644 index 0000000..c4085e0 --- /dev/null +++ b/src/extent-scan.c @@ -0,0 +1,137 @@ +/* extent-scan.c -- core functions for scanning extents + Copyright (C) 2010 Free Software Foundation, Inc. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + + Written by Jie Liu (jeff.liu@oracle.com). */ + +#include +#include +#include +#include + +#include "system.h" +#include "extent-scan.h" +#include "error.h" +#include "quote.h" + +#ifndef HAVE_FIEMAP +# include "fiemap.h" +#endif + +/* The number of extents currently scan returned. */ +static size_t current_scanned_extents_count = 0; + +#ifdef __linux__ +# ifndef FS_IOC_FIEMAP +# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) +# endif +/* Call ioctl(2) with FS_IOC_FIEMAP (available in linux 2.6.27) to + obtain a map of file extents excluding holes. */ +extern void * +init_extent_scan (int src_fd, const char *src_name, + bool *normal_copy_required, + bool *hit_last_extent) +{ + union { struct fiemap f; char c[4096]; } fiemap_buf; + struct fiemap *fiemap = &fiemap_buf.f; + struct fiemap_extent *fm_extents = &fiemap->fm_extents[0]; + enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_extents }; + verify (count != 0); + static uint64_t next_map_start = 0; + + /* This is required at least to initialize fiemap->fm_start, + * but also serves (in mid 2010) to appease valgrind, which + * appears not to know the semantics of the FIEMAP ioctl. */ + memset (&fiemap_buf, 0, sizeof fiemap_buf); + + fiemap->fm_start = next_map_start; + fiemap->fm_flags = FIEMAP_FLAG_SYNC; + fiemap->fm_extent_count = count; + fiemap->fm_length = FIEMAP_MAX_OFFSET - next_map_start; + + /* When ioctl(2) fails, fall back to the normal copy only if it + is the first time we met. */ + if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) + { + error (0, errno, _("%s: FIEMAP ioctl failed"), quote (src_name)); + + if (next_map_start == 0) + *normal_copy_required = true; + + return NULL; + } + + /* If 0 extents are returned, then more init_extent_scan() are not needed. */ + if (fiemap->fm_mapped_extents == 0) + { + *hit_last_extent = true; + return NULL; + } + + current_scanned_extents_count = fiemap->fm_mapped_extents; + unsigned int last_extent_index = current_scanned_extents_count - 1; + + if (fm_extents[last_extent_index].fe_flags & FIEMAP_EXTENT_LAST) + { + *hit_last_extent = true; + return fm_extents; + } + + next_map_start = fm_extents[last_extent_index].fe_logical + + fm_extents[last_extent_index].fe_length; + + return fm_extents; +} + +/* Return an extent's logical offset and length for each iteration. */ +extern bool +get_next_extent (void *scanned_extents, off_t *extent_logical, + uint64_t *extent_length) +{ + static size_t i = 0; + struct fiemap_extent *fm_extents = (struct fiemap_extent *) scanned_extents; + + if (i < current_scanned_extents_count) + { + *extent_logical = fm_extents[i].fe_logical; + *extent_length = fm_extents[i].fe_length; + i++; + return true; + } + + return false; +} + +extern void +close_extent_scan (void) +{ + return ; +} +#else +extent void * +init_extent_scan (int src_fd, const char *src_name, + bool *hit_last_extent, bool *normal_copy_required) +{ + *normal_copy_required = true; + errno = ENOTSUP; + (void) src_fd; + (void) src_name; + (void) hit_last_extent; + (void) normal_copy_required; + return false; +} +extern bool get_next_extent (ignored) { errno = ENOTSUP; return false; } +extern void close_extent_scan (ignored) { error = ENOTSUP; return ; } +#endif diff --git a/src/extent-scan.h b/src/extent-scan.h new file mode 100644 index 0000000..e7e373f --- /dev/null +++ b/src/extent-scan.h @@ -0,0 +1,25 @@ +/* core functions for efficient reading sparse files + Copyright (C) 2010 Free Software Foundation, Inc. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . */ + +#ifndef SPARSE_CORE_H +# define SPARSE_CORE_H + +void *init_extent_scan (int src_fd, const char *src_name, + bool *normal_copy_required, + bool *hit_last_extent); +bool get_next_extent (void *scanned_extents, off_t *extent_logical, uint64_t *extent_length); +void close_extent_scan (void); +#endif /* SPARSE_CORE_H */ -- 1.5.4.3 >From f23169c2c1721b8888dccb77000f79ddf9804df0 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Tue, 20 Jul 2010 23:11:08 +0800 Subject: [PATCH 3/3] cp: add extents_copy() for efficient sparse file copy * src/copy.c (copy_reg): Now, `cp' attempt to make use of the new extent scan module for efficient sparse file copy through extents_copy(), fall back to a normal copy if the underlaying file system does not support it. We honor --sparse=never by writing all data but use extents scan if available to read source file for the efficiently read. * src/copy.c: Add fill_with_holes_ok() for write zeros as holes to destination file. * po/POTFILES.in: add extent-scan.c to it. Signed-off-by: Jie Liu --- po/POTFILES.in | 1 + src/copy.c | 190 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 191 insertions(+), 0 deletions(-) diff --git a/po/POTFILES.in b/po/POTFILES.in index c862877..2ac1993 100644 --- a/po/POTFILES.in +++ b/po/POTFILES.in @@ -60,6 +60,7 @@ src/echo.c src/env.c src/expand.c src/expr.c +src/extent-scan.c src/factor.c src/false.c src/fmt.c diff --git a/src/copy.c b/src/copy.c index 171499c..6d89bbe 100644 --- a/src/copy.c +++ b/src/copy.c @@ -36,6 +36,7 @@ #include "copy.h" #include "cp-hash.h" #include "error.h" +#include "extent-scan.h" #include "fcntl--.h" #include "file-set.h" #include "filemode.h" @@ -149,6 +150,176 @@ clone_file (int dest_fd, int src_fd) #endif } +/* Write zeros as holes to the destination file. */ +static bool +fill_with_holes_ok (int dest_fd, const char *dst_name, + char *buf, size_t buf_size, + uint64_t holes_len) +{ + while (buf_size < holes_len) + { + if (full_write (dest_fd, buf, buf_size) != buf_size) + { + error (0, errno, _("writing %s"), quote (dst_name)); + return false; + } + holes_len -= buf_size; + } + + if (0 < holes_len) + { + if (full_write (dest_fd, buf, holes_len) != holes_len) + { + error (0, errno, _("writing %s"), quote (dst_name)); + return false; + } + } + + return true; +} + +/* Perform an efficient extents copy, if possible. This avoids + the overhead of detecting holes in hole-introducing/preserving + copy, and thus make copying sparse files much more efficient. + Upon a successful copy, return true. If the first initial extent + scan fails, set *NORMAL_COPY_REQUIRED to true and return false. + Upon any other failure, set *NORMAL_COPY_REQUIRED to false and + return false. */ +static bool +extents_copy (int source_desc, int dest_desc, + char *buf, size_t buf_size, + const char *src_name, const char *dst_name, + bool make_holes, size_t src_total_size, + bool *require_normal_copy) +{ + bool init_extent_scan_failed = false; + bool hit_last_extent = false; + void *scanned_extents; + off_t last_ext_logical = 0; + off_t ext_logical = 0; + uint64_t last_ext_len = 0; + uint64_t ext_len = 0; + uint64_t holes_len = 0; + uint64_t last_read_size = 0; + + if (! make_holes) + memset (buf, 0, buf_size); + + do + { + scanned_extents = init_extent_scan (source_desc, src_name, + &init_extent_scan_failed, + &hit_last_extent); + if (init_extent_scan_failed) + { + *require_normal_copy = true; + return false; + } + + while (get_next_extent (scanned_extents, &ext_logical, &ext_len)) + { + assert (ext_logical <= OFF_T_MAX); + + if (lseek (source_desc, ext_logical, SEEK_SET) < 0) + { + error (0, errno, _("cannot lseek %s"), quote (src_name)); + return false; + } + + if (make_holes) + { + if (lseek (dest_desc, ext_logical, SEEK_SET) < 0) + { + error (0, errno, _("cannot lseek %s"), quote (dst_name)); + return false; + } + } + else + { + /* If not making a sparse file, write zeros to the destination + file if there is a hole between the last and current extent. */ + if (last_ext_logical + last_ext_len < ext_logical) + { + holes_len = ext_logical - last_ext_logical - last_ext_len; + if (! fill_with_holes_ok (dest_desc, dst_name, buf, buf_size, holes_len)) + return false; + } + } + + last_ext_logical = ext_logical; + last_ext_len = ext_len; + + last_read_size = 0; + while (ext_len) + { + /* Avoid reading into the holes if the left extent + length is shorter than the buffer size. */ + if (ext_len < buf_size) + buf_size = ext_len; + + ssize_t n_read = read (source_desc, buf, buf_size); + if (n_read < 0) + { +#ifdef EINTR + if (errno == EINTR) + continue; +#endif + error (0, errno, _("reading %s"), quote (src_name)); + return false; + } + + if (n_read == 0) + { + last_read_size = last_ext_len - ext_len; + break; + } + + if (full_write (dest_desc, buf, n_read) != n_read) + { + error (0, errno, _("writing %s"), quote (dst_name)); + return false; + } + + ext_len -= n_read; + last_read_size += n_read; + } + } + } while (! hit_last_extent); + + close_extent_scan (); + + /* If a file ends up with holes, the sum of the last extent logical offset + and the read-returned size or the last extent length will be shorter than + the actual size of the file. Use ftruncate to extend the length of the + destination file if make_holes, or write zeros up to the actual size of the + file. */ + if (make_holes) + { + if (last_ext_logical + last_read_size < src_total_size) + { + if (ftruncate (dest_desc, src_total_size) < 0) + { + error (0, errno, _("failed to extend %s"), quote (dst_name)); + return false; + } + } + } + else + { + if (last_ext_logical + last_ext_len < src_total_size) + { + holes_len = src_total_size - last_ext_logical - last_ext_len; + if (0 < holes_len) + { + if (! fill_with_holes_ok (dest_desc, dst_name, buf, buf_size, holes_len)) + return false; + } + } + } + + return true; +} + /* FIXME: describe */ /* FIXME: rewrite this to use a hash table so we avoid the quadratic performance hit that's probably noticeable only on trees deeper @@ -707,6 +878,24 @@ copy_reg (char const *src_name, char const *dst_name, buf_alloc = xmalloc (buf_size + buf_alignment_slop); buf = ptr_align (buf_alloc, buf_alignment); + bool require_normal_copy; + /* Perform efficient extents copy for sparse files, fall back to the + standard copy only if the first initial extent scan fails. If the + `--sparse=never' option was specified, we writing all data but use + extents copy if available to efficiently read. */ + if (extents_copy (source_desc, dest_desc, buf, buf_size, + src_name, dst_name, make_holes, + src_open_sb.st_size, &require_normal_copy)) + goto preserve_metadata; + else + { + if (! require_normal_copy) + { + return_val = false; + goto close_src_and_dst_desc; + } + } + while (true) { word *wp = NULL; @@ -807,6 +996,7 @@ copy_reg (char const *src_name, char const *dst_name, } } +preserve_metadata: if (x->preserve_timestamps) { struct timespec timespec[2]; -- 1.5.4.3 Thanks, -Jeff -- The knowledge you get, no matter how much it is, must be possessed yourself and nourished with your own painstaking efforts and be your achievement through hard work. From debbugs-submit-bounces@debbugs.gnu.org Mon Sep 20 03:16:32 2010 Received: (at submit) by debbugs.gnu.org; 20 Sep 2010 07:16:33 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oxac4-0001TN-3r for submit@debbugs.gnu.org; Mon, 20 Sep 2010 03:16:32 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oxac1-0001TI-Hr for submit@debbugs.gnu.org; Mon, 20 Sep 2010 03:16:30 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OxaeP-0007Cy-Hx for submit@debbugs.gnu.org; Mon, 20 Sep 2010 03:18:58 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:42511) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OxaeN-0007Ca-1s for submit@debbugs.gnu.org; Mon, 20 Sep 2010 03:18:57 -0400 Received: from [140.186.70.92] (port=41485 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Oxae8-00065D-C8 for bug-coreutils@gnu.org; Mon, 20 Sep 2010 03:18:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Oxade-00078v-Pr for bug-coreutils@gnu.org; Mon, 20 Sep 2010 03:18:12 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:64026) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oxade-00078S-FC for bug-coreutils@gnu.org; Mon, 20 Sep 2010 03:18:10 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o8K7I2Hc022624 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Mon, 20 Sep 2010 07:18:03 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o8K19S2K020327; Mon, 20 Sep 2010 07:18:00 GMT Received: from abhmt005.oracle.com by acsmt354.oracle.com with ESMTP id 618345331284967079; Mon, 20 Sep 2010 00:17:59 -0700 Received: from [10.182.120.185] (/10.182.120.185) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 20 Sep 2010 00:17:58 -0700 Message-ID: <4C970AA9.5070304@oracle.com> Date: Mon, 20 Sep 2010 15:18:01 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> In-Reply-To: <4C45BFA5.5000809@oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?ISO-8859-1?Q?P=E1draig_Brady?= , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.2 (------) Hi Jim and All, Do you have any comments for the current implementation? Sorry for my so delayed asking. Thanks, -Jeff jeff.liu wrote: > Hello All, > > Below is my patches to isolate the extents scan and fetch functions in a new module to improve its > extendibility. > > It introduce a new file 'src/extent-scan.c' to place those functions, and then call those functions > from extents_copy() at copy_reg() to process the regular file copy. > > In addition to this, another major change is to copy all data if '--sparse=never' option is > specified. It write all data to destination file but using extents_copy() if available for > efficient read source file, and it try to figure out the holes between the previous and current > extents, and call fill_with_holes_ok() to write zeros as holes to the destination file if it is. > Call file_with_holes_ok() to write zeros up to the source file size if hit the last extent of the > source file and there is a hole behind it. > > I have not implement the solaris lseek(2) at the moment for lack of solaris environment, it need to > delay a period of time. > > According to my tryout, it works for those 4 filesystems in common use, you all know. > As usual, any comments are welcome! > > > From 70773fdf1d85ba070e054b0467a7a0e1e2b00ea8 Mon Sep 17 00:00:00 2001 > From: Jie Liu > Date: Tue, 20 Jul 2010 20:35:25 +0800 > Subject: [PATCH 1/3] cp: delete fiemap_copy() related stuff from copy.c > > * delete fiemap_copy(), now it is implemented as a module. > > Signed-off-by: Jie Liu > --- > src/copy.c | 171 ------------------------------------------------------------ > 1 files changed, 0 insertions(+), 171 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index f48c74d..171499c 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -63,10 +63,6 @@ > > #include > > -#ifndef HAVE_FIEMAP > -# include "fiemap.h" > -#endif > - > #ifndef HAVE_FCHOWN > # define HAVE_FCHOWN false > # define fchown(fd, uid, gid) (-1) > @@ -153,153 +149,6 @@ clone_file (int dest_fd, int src_fd) > #endif > } > > -#ifdef __linux__ > -# ifndef FS_IOC_FIEMAP > -# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) > -# endif > -/* Perform a FIEMAP copy, if possible. > - Call ioctl(2) with FS_IOC_FIEMAP (available in linux 2.6.27) to > - obtain a map of file extents excluding holes. This avoids the > - overhead of detecting holes in a hole-introducing/preserving copy, > - and thus makes copying sparse files much more efficient. Upon a > - successful copy, return true. If the initial ioctl fails, set > - *NORMAL_COPY_REQUIRED to true and return false. Upon any other > - failure, set *NORMAL_COPY_REQUIRED to false and return false. */ > -static bool > -fiemap_copy (int src_fd, int dest_fd, size_t buf_size, > - off_t src_total_size, char const *src_name, > - char const *dst_name, bool *normal_copy_required) > -{ > - bool last = false; > - union { struct fiemap f; char c[4096]; } fiemap_buf; > - struct fiemap *fiemap = &fiemap_buf.f; > - struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; > - enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; > - verify (count != 0); > - > - off_t last_ext_logical = 0; > - uint64_t last_ext_len = 0; > - uint64_t last_read_size = 0; > - unsigned int i = 0; > - *normal_copy_required = false; > - > - /* This is required at least to initialize fiemap->fm_start, > - but also serves (in mid 2010) to appease valgrind, which > - appears not to know the semantics of the FIEMAP ioctl. */ > - memset (&fiemap_buf, 0, sizeof fiemap_buf); > - > - do > - { > - fiemap->fm_length = FIEMAP_MAX_OFFSET; > - fiemap->fm_flags = FIEMAP_FLAG_SYNC; > - fiemap->fm_extent_count = count; > - > - /* When ioctl(2) fails, fall back to the normal copy only if it > - is the first time we met. */ > - if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) > - { > - /* If the first ioctl fails, tell the caller that it is > - ok to proceed with a normal copy. */ > - if (i == 0) > - *normal_copy_required = true; > - else > - { > - /* If the second or subsequent ioctl fails, diagnose it, > - since it ends up causing the entire copy/cp to fail. */ > - error (0, errno, _("%s: FIEMAP ioctl failed"), quote (src_name)); > - } > - return false; > - } > - > - /* If 0 extents are returned, then more ioctls are not needed. */ > - if (fiemap->fm_mapped_extents == 0) > - break; > - > - for (i = 0; i < fiemap->fm_mapped_extents; i++) > - { > - assert (fm_ext[i].fe_logical <= OFF_T_MAX); > - > - off_t ext_logical = fm_ext[i].fe_logical; > - uint64_t ext_len = fm_ext[i].fe_length; > - > - if (lseek (src_fd, ext_logical, SEEK_SET) < 0) > - { > - error (0, errno, _("cannot lseek %s"), quote (src_name)); > - return false; > - } > - > - if (lseek (dest_fd, ext_logical, SEEK_SET) < 0) > - { > - error (0, errno, _("cannot lseek %s"), quote (dst_name)); > - return false; > - } > - > - if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST) > - { > - last_ext_logical = ext_logical; > - last_ext_len = ext_len; > - last = true; > - } > - > - while (ext_len) > - { > - char buf[buf_size]; > - > - /* Avoid reading into the holes if the left extent > - length is shorter than the buffer size. */ > - if (ext_len < buf_size) > - buf_size = ext_len; > - > - ssize_t n_read = read (src_fd, buf, buf_size); > - if (n_read < 0) > - { > -#ifdef EINTR > - if (errno == EINTR) > - continue; > -#endif > - error (0, errno, _("reading %s"), quote (src_name)); > - return false; > - } > - > - if (n_read == 0) > - { > - /* Figure out how many bytes read from the last extent. */ > - last_read_size = last_ext_len - ext_len; > - break; > - } > - > - if (full_write (dest_fd, buf, n_read) != n_read) > - { > - error (0, errno, _("writing %s"), quote (dst_name)); > - return false; > - } > - > - ext_len -= n_read; > - } > - } > - > - fiemap->fm_start = fm_ext[i - 1].fe_logical + fm_ext[i - 1].fe_length; > - > - } while (! last); > - > - /* If a file ends up with holes, the sum of the last extent logical offset > - and the read-returned size will be shorter than the actual size of the > - file. Use ftruncate to extend the length of the destination file. */ > - if (last_ext_logical + last_read_size < src_total_size) > - { > - if (ftruncate (dest_fd, src_total_size) < 0) > - { > - error (0, errno, _("failed to extend %s"), quote (dst_name)); > - return false; > - } > - } > - > - return true; > -} > -#else > -static bool fiemap_copy (ignored) { errno == ENOTSUP; return false; } > -#endif > - > /* FIXME: describe */ > /* FIXME: rewrite this to use a hash table so we avoid the quadratic > performance hit that's probably noticeable only on trees deeper > @@ -830,25 +679,6 @@ copy_reg (char const *src_name, char const *dst_name, > #endif > } > > - if (make_holes) > - { > - bool require_normal_copy; > - /* Perform efficient FIEMAP copy for sparse files, fall back to the > - standard copy only if the ioctl(2) fails. */ > - if (fiemap_copy (source_desc, dest_desc, buf_size, > - src_open_sb.st_size, src_name, > - dst_name, &require_normal_copy)) > - goto preserve_metadata; > - else > - { > - if (! require_normal_copy) > - { > - return_val = false; > - goto close_src_and_dst_desc; > - } > - } > - } > - > /* If not making a sparse file, try to use a more-efficient > buffer size. */ > if (! make_holes) > @@ -977,7 +807,6 @@ copy_reg (char const *src_name, char const *dst_name, > } > } > > -preserve_metadata: > if (x->preserve_timestamps) > { > struct timespec timespec[2]; From debbugs-submit-bounces@debbugs.gnu.org Mon Sep 20 07:32:45 2010 Received: (at submit) by debbugs.gnu.org; 20 Sep 2010 11:32:45 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oxec0-0003ld-TH for submit@debbugs.gnu.org; Mon, 20 Sep 2010 07:32:45 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oxebx-0003lY-Au for submit@debbugs.gnu.org; Mon, 20 Sep 2010 07:32:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OxeeL-0001FZ-Sb for submit@debbugs.gnu.org; Mon, 20 Sep 2010 07:35:11 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:40930) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OxeeL-0001FT-OF for submit@debbugs.gnu.org; Mon, 20 Sep 2010 07:35:09 -0400 Received: from [140.186.70.92] (port=52210 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OxeeK-0005bn-C1 for bug-coreutils@gnu.org; Mon, 20 Sep 2010 07:35:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OxeeI-0001Ek-UB for bug-coreutils@gnu.org; Mon, 20 Sep 2010 07:35:08 -0400 Received: from mx.meyering.net ([82.230.74.64]:53248) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OxeeI-0001EA-C3 for bug-coreutils@gnu.org; Mon, 20 Sep 2010 07:35:06 -0400 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 1CFABD85C; Mon, 20 Sep 2010 13:35:05 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4C970AA9.5070304@oracle.com> (jeff liu's message of "Mon, 20 Sep 2010 15:18:01 +0800") References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> Date: Mon, 20 Sep 2010 13:35:05 +0200 Message-ID: <87iq20ejh2.fsf@meyering.net> Lines: 93 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.2 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?utf-8?Q?P=C3=A1draig?= Brady , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.2 (-----) jeff.liu wrote: > Hi Jim and All, > > Do you have any comments for the current implementation? Hi Jeff, Thanks for the reminder. I've just gone back and looked at those patches: http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/20534/focus=21008 There are some superficial problems. First, whenever you move code from one place to another, the commit that performs the move should be careful to induce no other change. In this case, the change should remove code from copy.c and create the new .c file with code that is essentially identical. You'll have to remove a "static" attribute on the primary function(s), and add #include directives in the new file, but those are inevitable. Also, in copy.c, you will remove the function body and associated #include directives, adding an #include of the new .h file. Of course, this change must also update src/Makefile.am, and the result should pass "make distcheck". But perhaps you require new functions like init_extent_scan in order to use the abstracted function properly. In that case, your first commit would add the new functions in copy.c and make use of them there. Then you would move all of the functions to their new file in the next commit, making no semantic change. Note however, that this copying code is intended to be usable in a multi-threaded application, and thus must avoid using internal static state. Your patch adds a few new static variables, each of which would cause trouble in such an application: +static size_t current_scanned_extents_count = 0; + static uint64_t next_map_start = 0; + static size_t i = 0; While you're rearranging your patch along the lines above, please eliminate those static variables, too. Also, this new function should be adjusted: +/* Write zeros as holes to the destination file. */ +static bool +fill_with_holes_ok (int dest_fd, const char *dst_name, + char *buf, size_t buf_size, + uint64_t holes_len) Its signature is unnecessarily complicated for a function that does nothing more than write N zero bytes to a file descriptor. Also, the function name is misleading (as is its holes_len parameter), since it certainly does not create holes. Hmm... now I'm suspicious: could the second use of your fill_with_holes_ok write from an uninitialized "buf"? It appears that is possible, but I confess not to have applied the patch. What do you think of this signature, static bool write_zeros (int fd, uint64_t n_bytes) That would require a buffer full of zeros, preferably of optimal size. It could use a body like this, { static char zero[32 * 1024]; while (n_bytes) { uint64_t n = MIN (sizeof zero, n_bytes); if (full_write (fd, zero, n)) != n) return false; n_bytes -= n; } return true; } or even calloc an IO_BUFSIZE-byte buffer on the first call and use that. Yes, using calloc appears better, since this code will end up being used relatively infrequently. Or perhaps better still, do use calloc, but if the allocation fails, fall back on using a smaller static buffer, of size say 1024 bytes. Of course, simplifying the function means each caller will have to diagnose failure, but imho, that's preferable in this case. Jim From debbugs-submit-bounces@debbugs.gnu.org Tue Sep 21 02:53:36 2010 Received: (at submit) by debbugs.gnu.org; 21 Sep 2010 06:53:36 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OxwjP-00044p-FY for submit@debbugs.gnu.org; Tue, 21 Sep 2010 02:53:35 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OxwjN-00044k-0M for submit@debbugs.gnu.org; Tue, 21 Sep 2010 02:53:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Oxwln-0005Sv-Fn for submit@debbugs.gnu.org; Tue, 21 Sep 2010 02:56:04 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:36595) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oxwln-0005Sn-BL for submit@debbugs.gnu.org; Tue, 21 Sep 2010 02:56:03 -0400 Received: from [140.186.70.92] (port=33458 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Oxwll-0005V8-Bw for bug-coreutils@gnu.org; Tue, 21 Sep 2010 02:56:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Oxweh-0004Hs-Bj for bug-coreutils@gnu.org; Tue, 21 Sep 2010 02:48:44 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:41223) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Oxweh-0004Hj-4S for bug-coreutils@gnu.org; Tue, 21 Sep 2010 02:48:43 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o8L6mXfv013607 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 21 Sep 2010 06:48:35 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o8L6mV5i009345; Tue, 21 Sep 2010 06:48:31 GMT Received: from abhmt017.oracle.com by acsmt353.oracle.com with ESMTP id 622042621285051628; Mon, 20 Sep 2010 23:47:08 -0700 Received: from [10.182.120.185] (/10.182.120.185) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 20 Sep 2010 23:47:06 -0700 Message-ID: <4C9854EE.2020405@oracle.com> Date: Tue, 21 Sep 2010 14:47:10 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87iq20ejh2.fsf@meyering.net> In-Reply-To: <87iq20ejh2.fsf@meyering.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.2 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) Hi Jim, Thanks for your prompt response and kindly suggestion! I totally agree with your review comments, I will post next round patches according to that soon. Regards, -Jeff Jim Meyering wrote: > jeff.liu wrote: >> Hi Jim and All, >> >> Do you have any comments for the current implementation? > > Hi Jeff, > > Thanks for the reminder. > I've just gone back and looked at those patches: > > http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/20534/focus=21008 > > There are some superficial problems. > > First, whenever you move code from one place to another, > the commit that performs the move should be careful to > induce no other change. In this case, the change should > remove code from copy.c and create the new .c file with > code that is essentially identical. You'll have to remove > a "static" attribute on the primary function(s), and add > #include directives in the new file, but those are inevitable. > Also, in copy.c, you will remove the function body > and associated #include directives, adding an #include > of the new .h file. Of course, this change must also update > src/Makefile.am, and the result should pass "make distcheck". > > But perhaps you require new functions like init_extent_scan > in order to use the abstracted function properly. > In that case, your first commit would add the new functions > in copy.c and make use of them there. Then you would move > all of the functions to their new file in the next commit, > making no semantic change. > > Note however, that this copying code is intended to be usable in a > multi-threaded application, and thus must avoid using internal static > state. Your patch adds a few new static variables, each of which > would cause trouble in such an application: > > +static size_t current_scanned_extents_count = 0; > + static uint64_t next_map_start = 0; > + static size_t i = 0; > > While you're rearranging your patch along the lines above, > please eliminate those static variables, too. > > Also, this new function should be adjusted: > > +/* Write zeros as holes to the destination file. */ > +static bool > +fill_with_holes_ok (int dest_fd, const char *dst_name, > + char *buf, size_t buf_size, > + uint64_t holes_len) > > Its signature is unnecessarily complicated for a function > that does nothing more than write N zero bytes to a file descriptor. > Also, the function name is misleading (as is its holes_len parameter), > since it certainly does not create holes. > > Hmm... now I'm suspicious: could the second use of your fill_with_holes_ok > write from an uninitialized "buf"? It appears that is possible, > but I confess not to have applied the patch. > > What do you think of this signature, > > static bool > write_zeros (int fd, uint64_t n_bytes) > > That would require a buffer full of zeros, preferably of optimal size. > It could use a body like this, > > { > static char zero[32 * 1024]; > while (n_bytes) > { > uint64_t n = MIN (sizeof zero, n_bytes); > if (full_write (fd, zero, n)) != n) > return false; > n_bytes -= n; > } > return true; > } > > or even calloc an IO_BUFSIZE-byte buffer on the first call > and use that. Yes, using calloc appears better, since this code > will end up being used relatively infrequently. Or perhaps better > still, do use calloc, but if the allocation fails, fall back on > using a smaller static buffer, of size say 1024 bytes. > > Of course, simplifying the function means each caller > will have to diagnose failure, but imho, that's preferable > in this case. > > Jim > > > From debbugs-submit-bounces@debbugs.gnu.org Sun Sep 26 04:20:11 2010 Received: (at submit) by debbugs.gnu.org; 26 Sep 2010 08:20:11 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OzmSw-0000BH-By for submit@debbugs.gnu.org; Sun, 26 Sep 2010 04:20:11 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OzmSt-0000B2-GZ for submit@debbugs.gnu.org; Sun, 26 Sep 2010 04:20:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OzmVW-0006Nh-4s for submit@debbugs.gnu.org; Sun, 26 Sep 2010 04:22:52 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:60348) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OzmVV-0006Na-QN for submit@debbugs.gnu.org; Sun, 26 Sep 2010 04:22:50 -0400 Received: from [140.186.70.92] (port=48729 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OzmVT-0002j1-4A for bug-coreutils@gnu.org; Sun, 26 Sep 2010 04:22:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OzmVQ-0006Mn-8j for bug-coreutils@gnu.org; Sun, 26 Sep 2010 04:22:47 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:51370) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OzmVP-0006Mf-Vx for bug-coreutils@gnu.org; Sun, 26 Sep 2010 04:22:44 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o8Q8MWCl005714 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sun, 26 Sep 2010 08:22:34 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o8Q8MVTs023262; Sun, 26 Sep 2010 08:22:31 GMT Received: from abhmt013.oracle.com by acsmt353.oracle.com with ESMTP id 637295791285489248; Sun, 26 Sep 2010 01:20:48 -0700 Received: from [10.182.120.185] (/10.182.120.185) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sun, 26 Sep 2010 01:20:47 -0700 Message-ID: <4C9F025A.9090300@oracle.com> Date: Sun, 26 Sep 2010 16:20:42 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87iq20ejh2.fsf@meyering.net> In-Reply-To: <87iq20ejh2.fsf@meyering.net> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.2 (------) Hi Jim, Sorry for the delay. This is the new patch to isolate the stuff regarding to extents reading to a new module. and teach cp(1) to make use of it. Changes: ======== extent-scan.h/extent-scan.c: 1. Removed all the *static* variables from this module. I'd like to introduce two new data structures "struct extent_scan" and "struct extent_info" which are used to reserve the information goes through a file reading and the info(logical offset/length/flags) regarding to each extent scan-returned separately, it is intended to work as a wrap for either fiemap or SEEK_DATA/SEEK_HOLE info. /* Structure used to reserve extent information. */ struct extent_info { /* Logical offset of an extent. */ off_t ext_logical; /* Extent length. */ uint64_t ext_length; /* Extent flags, use it for FIEMAP only. */ uint32_t ext_flags; }; /* Structure used to reserve extent scan information. */ struct extent_scan { /* File descriptor of extent scan run against. */ int fd; /* File name for debugging purpose. */ char *fname; /* Next extent scan start offset. */ off_t scan_start; /* How many extents info returned for a scan. */ uint32_t ei_count; /* If true, fall back to a normal copy, either set by the failure of ioctl(2) for FIEMAP or lseek(2) with SEEK_DATA. */ bool initial_scan_failed; /* If ture, the total extent scan per file has been finished. */ bool hit_last_extent; /* Extent information. */ struct extent_info *ext_info; }; 2. Functions in the new module: void open_extent_scan(int fd, char const *fname, struct extent_scan **scan); Call this function first, it do allocate the spaces for "extent_scan" and initialize some of entries in this structure if necessary. The returned variable will be used as the input argument of get_extents_info() which shown as below. bool get_extents_info(struct extent_scan *scan); maybe need to figure out a better name? This function accept an initialized extent_scan and allocated the spaces for extent_info based on the number of extents through either call fiemap ioctl(2) or lseek(2) SEEK_DATA/SEEK_HOLE stuff. Fill the extent_info with the logical offset and length per extent, and set the extent_info->ext_flags for fiemap, or just set it to ZERO for lseek. void close_extent_scan(struct extent_scan *scan); Call this function after every get_extents_info() done to release the memory of extent_info, but keep the extent_scan space. After a file copy done successfully, call it again to release the extent_scan. copy.c: ====== 1. Replace fill_with_holes_ok() with write_zeros (int fd, uint64_t len), but is it better to add an char const *src_name to this function for debugging purpose? 2. Replace fiemap_copy() with extent_copy() According to my tryout, it works fine, both make syntax-check and make distcheck also run passed. I hope you guys would like this implementation. :) Any comments are appreciated! >From 8ca00082e68b3bef4ee0ed042cb7a70dcaf154e8 Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Sun, 26 Sep 2010 16:03:48 +0800 Subject: [PATCH 1/1] cp: add a new module for scanning extents * src/extent-scan.c: Source code for scanning extents. Call open_extent_scan() to allocate and return an initialized extent scan. Call get_extents_info() to get a number of extents for each iteration. Call close_extent_scan() to release the space allocated extent_info per extent scan, and extent_scan per file scan. * src/extent-scan.h: Header file of extent-scan.c. * src/Makefile.am: Reference it and link it to copy_source. * src/copy.c: Make use of the new module, replace fiemap_copy() with extent_copy(). Signed-off-by: Jie Liu --- src/Makefile.am | 2 +- src/copy.c | 186 +++++++++++++++++++++++++++++------------------------ src/extent-scan.c | 137 +++++++++++++++++++++++++++++++++++++++ src/extent-scan.h | 71 ++++++++++++++++++++ 4 files changed, 312 insertions(+), 84 deletions(-) create mode 100644 src/extent-scan.c create mode 100644 src/extent-scan.h diff --git a/src/Makefile.am b/src/Makefile.am index 7d56312..2412b16 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -459,7 +459,7 @@ uninstall-local: fi; \ fi -copy_sources = copy.c cp-hash.c +copy_sources = copy.c cp-hash.c extent-scan.c # Use `ginstall' in the definition of PROGRAMS and in dependencies to avoid # confusion with the `install' target. The install rule transforms `ginstall' diff --git a/src/copy.c b/src/copy.c index f48c74d..0c54365 100644 --- a/src/copy.c +++ b/src/copy.c @@ -35,6 +35,7 @@ #include "buffer-lcm.h" #include "copy.h" #include "cp-hash.h" +#include "extent-scan.h" #include "error.h" #include "fcntl--.h" #include "file-set.h" @@ -63,10 +64,6 @@ #include -#ifndef HAVE_FIEMAP -# include "fiemap.h" -#endif - #ifndef HAVE_FCHOWN # define HAVE_FCHOWN false # define fchown(fd, uid, gid) (-1) @@ -153,74 +150,67 @@ clone_file (int dest_fd, int src_fd) #endif } -#ifdef __linux__ -# ifndef FS_IOC_FIEMAP -# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) -# endif -/* Perform a FIEMAP copy, if possible. - Call ioctl(2) with FS_IOC_FIEMAP (available in linux 2.6.27) to - obtain a map of file extents excluding holes. This avoids the - overhead of detecting holes in a hole-introducing/preserving copy, - and thus makes copying sparse files much more efficient. Upon a - successful copy, return true. If the initial ioctl fails, set - *NORMAL_COPY_REQUIRED to true and return false. Upon any other - failure, set *NORMAL_COPY_REQUIRED to false and return false. */ static bool -fiemap_copy (int src_fd, int dest_fd, size_t buf_size, - off_t src_total_size, char const *src_name, - char const *dst_name, bool *normal_copy_required) +write_zeros (int fd, uint64_t n_bytes) { - bool last = false; - union { struct fiemap f; char c[4096]; } fiemap_buf; - struct fiemap *fiemap = &fiemap_buf.f; - struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; - enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; - verify (count != 0); + char *zeros = calloc (IO_BUFSIZE, sizeof (char)); + if (! zeros) + { + /* Try a small buffer. */ + static char zeros[1024]; + } + while (n_bytes) + { + uint64_t n = MIN (sizeof zeros, n_bytes); + if ((full_write (fd, zeros, n)) != n) + return false; + n_bytes -= n; + } + + return true; +} + +/* Perform an efficient extent copy, if possible. This avoids + the overhead of detecting holes in hole-introducing/preserving + copy, and thus makes copying sparse files much more efficient. + Upon a successful copy, return true. If the initial extent scan + fails, set *NORMAL_COPY_REQUIRED to true and return false. + Upon any other failure, set *NORMAL_COPY_REQUIRED to false and + return false. */ +static bool +extent_copy (int src_fd, int dest_fd, size_t buf_size, + off_t src_total_size, bool make_holes, + char const *src_name, char const *dst_name, + bool *require_normal_copy) +{ + struct extent_scan *scan; off_t last_ext_logical = 0; uint64_t last_ext_len = 0; uint64_t last_read_size = 0; - unsigned int i = 0; - *normal_copy_required = false; + unsigned int i; + bool ok = true; - /* This is required at least to initialize fiemap->fm_start, - but also serves (in mid 2010) to appease valgrind, which - appears not to know the semantics of the FIEMAP ioctl. */ - memset (&fiemap_buf, 0, sizeof fiemap_buf); + open_extent_scan (src_fd, src_name, &scan); do { - fiemap->fm_length = FIEMAP_MAX_OFFSET; - fiemap->fm_flags = FIEMAP_FLAG_SYNC; - fiemap->fm_extent_count = count; - - /* When ioctl(2) fails, fall back to the normal copy only if it - is the first time we met. */ - if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) + ok = get_extents_info (scan); + if (! ok) { - /* If the first ioctl fails, tell the caller that it is - ok to proceed with a normal copy. */ - if (i == 0) - *normal_copy_required = true; - else - { - /* If the second or subsequent ioctl fails, diagnose it, - since it ends up causing the entire copy/cp to fail. */ - error (0, errno, _("%s: FIEMAP ioctl failed"), quote (src_name)); - } + if (scan->hit_last_extent) + break; + + if (scan->initial_scan_failed) + *require_normal_copy = true; + return false; } - /* If 0 extents are returned, then more ioctls are not needed. */ - if (fiemap->fm_mapped_extents == 0) - break; - - for (i = 0; i < fiemap->fm_mapped_extents; i++) + for (i = 0; i < scan->ei_count; i++) { - assert (fm_ext[i].fe_logical <= OFF_T_MAX); - - off_t ext_logical = fm_ext[i].fe_logical; - uint64_t ext_len = fm_ext[i].fe_length; + off_t ext_logical = scan->ext_info[i].ext_logical; + uint64_t ext_len = scan->ext_info[i].ext_length; if (lseek (src_fd, ext_logical, SEEK_SET) < 0) { @@ -228,19 +218,30 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, return false; } - if (lseek (dest_fd, ext_logical, SEEK_SET) < 0) + if (make_holes) { - error (0, errno, _("cannot lseek %s"), quote (dst_name)); - return false; + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0) + { + error (0, errno, _("cannot lseek %s"), quote (dst_name)); + return false; + } } - - if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST) + else { - last_ext_logical = ext_logical; - last_ext_len = ext_len; - last = true; + /* If not making a sparse file, write zeros to the destination + file if there is a hole between the last and current extent. */ + if (last_ext_logical + last_ext_len < ext_logical) + { + uint64_t holes_len = ext_logical - last_ext_logical - last_ext_len; + if (! write_zeros (dest_fd, holes_len)) + return false; + } } + last_ext_logical = ext_logical; + last_ext_len = ext_len; + last_read_size = 0; + while (ext_len) { char buf[buf_size]; @@ -258,12 +259,12 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, continue; #endif error (0, errno, _("reading %s"), quote (src_name)); - return false; + return false; } if (n_read == 0) { - /* Figure out how many bytes read from the last extent. */ + /* Figure out how many bytes read from the previous extent. */ last_read_size = last_ext_len - ext_len; break; } @@ -278,27 +279,44 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, } } - fiemap->fm_start = fm_ext[i - 1].fe_logical + fm_ext[i - 1].fe_length; + /* Release the space allocated to scan->ext_info. */ + close_extent_scan (scan); + } while (! scan->hit_last_extent); - } while (! last); + /* Release the space allocated to scan->fname and scan. */ + close_extent_scan (scan); /* If a file ends up with holes, the sum of the last extent logical offset - and the read-returned size will be shorter than the actual size of the - file. Use ftruncate to extend the length of the destination file. */ - if (last_ext_logical + last_read_size < src_total_size) + and the read-returned size or the last extent length will be shorter than + the actual size of the file. Use ftruncate to extend the length of the + destination file if make_holes, or write zeros up to the actual size of the + file. */ + if (make_holes) { - if (ftruncate (dest_fd, src_total_size) < 0) + if (last_ext_logical + last_read_size < src_total_size) { - error (0, errno, _("failed to extend %s"), quote (dst_name)); - return false; + if (ftruncate (dest_fd, src_total_size) < 0) + { + error (0, errno, _("failed to extend %s"), quote (dst_name)); + return false; + } + } + } + else + { + if (last_ext_logical + last_ext_len < src_total_size) + { + uint64_t holes_len = src_total_size - last_ext_logical - last_ext_len; + if (0 < holes_len) + { + if (! write_zeros (dest_fd, holes_len)) + return false; + } } } return true; } -#else -static bool fiemap_copy (ignored) { errno == ENOTSUP; return false; } -#endif /* FIXME: describe */ /* FIXME: rewrite this to use a hash table so we avoid the quadratic @@ -833,11 +851,13 @@ copy_reg (char const *src_name, char const *dst_name, if (make_holes) { bool require_normal_copy; - /* Perform efficient FIEMAP copy for sparse files, fall back to the - standard copy only if the ioctl(2) fails. */ - if (fiemap_copy (source_desc, dest_desc, buf_size, - src_open_sb.st_size, src_name, - dst_name, &require_normal_copy)) + /* Perform efficient extent copy for sparse file, fall back to the + standard copy only if the initial extent scan fails. If the + '--sparse=never' option was specified, we writing all data but + use extent copy if available to efficiently read. */ + if (extent_copy (source_desc, dest_desc, buf_size, + src_open_sb.st_size, make_holes, + src_name, dst_name, &require_normal_copy)) goto preserve_metadata; else { diff --git a/src/extent-scan.c b/src/extent-scan.c new file mode 100644 index 0000000..ff7d622 --- /dev/null +++ b/src/extent-scan.c @@ -0,0 +1,137 @@ +/* extent-scan.c -- core functions for scanning extents + Copyright (C) 2010 Free Software Foundation, Inc. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + + Written by Jie Liu (jeff.liu@oracle.com). */ + +#include +#include +#include +#include +#include + +#include "system.h" +#include "extent-scan.h" +#include "error.h" +#include "quote.h" + +#ifndef HAVE_FIEMAP +# include "fiemap.h" +#endif + +/* Allocate space for struct extent_scan, initialize the entries if + necessary and return it as the input argument of get_extents_info(). */ +extern void +open_extent_scan (int src_fd, char const *src_name, + struct extent_scan **scan) +{ + struct extent_scan *ext_scan = xmalloc (sizeof *ext_scan); + + ext_scan->fd = src_fd; + ext_scan->fname = xstrdup (src_name); + ext_scan->ei_count = 0; + ext_scan->scan_start = 0; + ext_scan->initial_scan_failed = false; + ext_scan->hit_last_extent = false; + + *scan = ext_scan; +} + +#ifdef __linux__ +# ifndef FS_IOC_FIEMAP +# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) +# endif +/* Call ioctl(2) with FS_IOC_FIEMAP (available in linux 2.6.27) to + obtain a map of file extents excluding holes. */ +extern bool +get_extents_info (struct extent_scan *scan) +{ + union { struct fiemap f; char c[4096]; } fiemap_buf; + struct fiemap *fiemap = &fiemap_buf.f; + struct fiemap_extent *fm_extents = &fiemap->fm_extents[0]; + enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_extents }; + verify (count != 0); + unsigned int i; + + /* This is required at least to initialize fiemap->fm_start, + but also serves (in mid 2010) to appease valgrind, which + appears not to know the semantics of the FIEMAP ioctl. */ + memset (&fiemap_buf, 0, sizeof fiemap_buf); + + fiemap->fm_start = scan->scan_start; + fiemap->fm_flags = FIEMAP_FLAG_SYNC; + fiemap->fm_extent_count = count; + fiemap->fm_length = FIEMAP_MAX_OFFSET - scan->scan_start; + + /* Fall back to the standard copy if call ioctl(2) failed for the + the first time. */ + if (ioctl (scan->fd, FS_IOC_FIEMAP, fiemap) < 0) + { + error (0, errno, _("%s: FIEMAP ioctl failed"), quote (scan->fname)); + + if (scan->scan_start == 0) + scan->initial_scan_failed = true; + return false; + } + + /* If 0 extents are returned, then more get_extent_table() are not needed. */ + if (fiemap->fm_mapped_extents == 0) + { + scan->hit_last_extent = true; + return false; + } + + scan->ei_count = fiemap->fm_mapped_extents; + scan->ext_info = xcalloc (scan->ei_count, sizeof scan->ext_info[0]); + + for (i = 0; i < scan->ei_count; i++) + { + assert (fm_extents[i].fe_logical <= OFF_T_MAX); + + scan->ext_info[i].ext_logical = fm_extents[i].fe_logical; + scan->ext_info[i].ext_length = fm_extents[i].fe_length; + scan->ext_info[i].ext_flags = fm_extents[i].fe_flags; + } + + i--; + if (scan->ext_info[i].ext_flags & FIEMAP_EXTENT_LAST) + { + scan->hit_last_extent = true; + return true; + } + + scan->scan_start = fm_extents[i].fe_logical + fm_extents[i].fe_length; + + return true; +} +#else +extern bool get_extents_info (ignored) { errno = ENOTSUP; return false; } +#endif + +extern void +close_extent_scan (struct extent_scan *scan) +{ + if (scan) + { + /* If not hit the last extent, only release extent_info. */ + if (scan->ext_info && ! scan->hit_last_extent) + { + free (scan->ext_info); + return; + } + free (scan->fname); + free (scan); + } +} diff --git a/src/extent-scan.h b/src/extent-scan.h new file mode 100644 index 0000000..3caae88 --- /dev/null +++ b/src/extent-scan.h @@ -0,0 +1,71 @@ +/* core functions for efficient reading sparse files + Copyright (C) 2010 Free Software Foundation, Inc. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + + Written by Jie Liu (jeff.liu@oracle.com). */ + +#ifndef EXTENT_SCAN_H +# define EXTENT_SCAN_H + +/* Structure used to reserve information of each extent. */ +struct extent_info +{ + /* Logical offset of an extent. */ + off_t ext_logical; + + /* Extent length. */ + uint64_t ext_length; + + /* Extent flags, use it for FIEMAP only, or set it to zero. */ + uint32_t ext_flags; +}; + +/* Structure used to reserve extent scan information per file. */ +struct extent_scan +{ + /* File descriptor of extent scan run against. */ + int fd; + + /* File name for debugging purpose. */ + char *fname; + + /* Next scan start offset. */ + off_t scan_start; + + /* How many extent info returned for a scan. */ + uint32_t ei_count; + + /* If true, fall back to a normal copy, either + set by the failure of ioctl(2) for FIEMAP or + lseek(2) with SEEK_DATA. */ + bool initial_scan_failed; + + /* If ture, the total extent scan per file has been finished. */ + bool hit_last_extent; + + /* Extent information. */ + struct extent_info *ext_info; +}; + +void +open_extent_scan (int src_fd, char const *src_name, + struct extent_scan **scan); + +bool +get_extents_info (struct extent_scan *scan); + +void +close_extent_scan (struct extent_scan *scan); +#endif /* EXTENT_SCAN_H */ -- 1.5.4.3 Regards, -Jeff Jim Meyering wrote: > jeff.liu wrote: >> Hi Jim and All, >> >> Do you have any comments for the current implementation? > > Hi Jeff, > > Thanks for the reminder. > I've just gone back and looked at those patches: > > http://thread.gmane.org/gmane.comp.gnu.coreutils.bugs/20534/focus=21008 > > There are some superficial problems. > > First, whenever you move code from one place to another, > the commit that performs the move should be careful to > induce no other change. In this case, the change should > remove code from copy.c and create the new .c file with > code that is essentially identical. You'll have to remove > a "static" attribute on the primary function(s), and add > #include directives in the new file, but those are inevitable. > Also, in copy.c, you will remove the function body > and associated #include directives, adding an #include > of the new .h file. Of course, this change must also update > src/Makefile.am, and the result should pass "make distcheck". > > But perhaps you require new functions like init_extent_scan > in order to use the abstracted function properly. > In that case, your first commit would add the new functions > in copy.c and make use of them there. Then you would move > all of the functions to their new file in the next commit, > making no semantic change. > > Note however, that this copying code is intended to be usable in a > multi-threaded application, and thus must avoid using internal static > state. Your patch adds a few new static variables, each of which > would cause trouble in such an application: > > +static size_t current_scanned_extents_count = 0; > + static uint64_t next_map_start = 0; > + static size_t i = 0; > > While you're rearranging your patch along the lines above, > please eliminate those static variables, too. > > Also, this new function should be adjusted: > > +/* Write zeros as holes to the destination file. */ > +static bool > +fill_with_holes_ok (int dest_fd, const char *dst_name, > + char *buf, size_t buf_size, > + uint64_t holes_len) > > Its signature is unnecessarily complicated for a function > that does nothing more than write N zero bytes to a file descriptor. > Also, the function name is misleading (as is its holes_len parameter), > since it certainly does not create holes. > > Hmm... now I'm suspicious: could the second use of your fill_with_holes_ok > write from an uninitialized "buf"? It appears that is possible, > but I confess not to have applied the patch. > > What do you think of this signature, > > static bool > write_zeros (int fd, uint64_t n_bytes) > > That would require a buffer full of zeros, preferably of optimal size. > It could use a body like this, > > { > static char zero[32 * 1024]; > while (n_bytes) > { > uint64_t n = MIN (sizeof zero, n_bytes); > if (full_write (fd, zero, n)) != n) > return false; > n_bytes -= n; > } > return true; > } > > or even calloc an IO_BUFSIZE-byte buffer on the first call > and use that. Yes, using calloc appears better, since this code > will end up being used relatively infrequently. Or perhaps better > still, do use calloc, but if the allocation fails, fall back on > using a smaller static buffer, of size say 1024 bytes. > > Of course, simplifying the function means each caller > will have to diagnose failure, but imho, that's preferable > in this case. > > Jim > > > From debbugs-submit-bounces@debbugs.gnu.org Sun Sep 26 04:44:22 2010 Received: (at submit) by debbugs.gnu.org; 26 Sep 2010 08:44:23 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OzmqM-0000Ma-Bv for submit@debbugs.gnu.org; Sun, 26 Sep 2010 04:44:22 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OzmqJ-0000MT-T8 for submit@debbugs.gnu.org; Sun, 26 Sep 2010 04:44:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Ozmsy-000125-5j for submit@debbugs.gnu.org; Sun, 26 Sep 2010 04:47:05 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:56408) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Ozmsx-00011s-Sw for submit@debbugs.gnu.org; Sun, 26 Sep 2010 04:47:04 -0400 Received: from [140.186.70.92] (port=60561 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Ozmsv-0007DE-SS for bug-coreutils@gnu.org; Sun, 26 Sep 2010 04:47:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Ozmsu-00011N-4f for bug-coreutils@gnu.org; Sun, 26 Sep 2010 04:47:01 -0400 Received: from mx.meyering.net ([82.230.74.64]:33340) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Ozmst-00010n-U7 for bug-coreutils@gnu.org; Sun, 26 Sep 2010 04:47:00 -0400 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 685F6E412; Sun, 26 Sep 2010 10:46:56 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4C9F025A.9090300@oracle.com> (jeff liu's message of "Sun, 26 Sep 2010 16:20:42 +0800") References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87iq20ejh2.fsf@meyering.net> <4C9F025A.9090300@oracle.com> Date: Sun, 26 Sep 2010 10:46:56 +0200 Message-ID: <87sk0whoxr.fsf@meyering.net> Lines: 65 MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?utf-8?Q?P=C3=A1draig?= Brady , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) Hi Jeff, This function has problems: - the inner "zeros" declaration shadows the outer one and ends up being useless. - the "sizeof zeros" resolves to 4 or 8. obviously not what you intended. ... > static bool > +write_zeros (int fd, uint64_t n_bytes) > { > - bool last = false; > - union { struct fiemap f; char c[4096]; } fiemap_buf; > - struct fiemap *fiemap = &fiemap_buf.f; > - struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; > - enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; > - verify (count != 0); > + char *zeros = calloc (IO_BUFSIZE, sizeof (char)); > + if (! zeros) > + { > + /* Try a small buffer. */ > + static char zeros[1024]; > + } > > + while (n_bytes) > + { > + uint64_t n = MIN (sizeof zeros, n_bytes); > + if ((full_write (fd, zeros, n)) != n) > + return false; > + n_bytes -= n; > + } > + > + return true; > +} Please use the following instead. I'll review the rest tomorrow or Tuesday. static bool write_zeros (int fd, uint64_t n_bytes) { static char *zeros; static size_t nz = IO_BUFSIZE; if (zeros == NULL) { static char fallback[1024]; zeros = calloc (nz, 1); if (zeros == NULL) { zeros = fallback; nz = sizeof fallback; } } while (n_bytes) { uint64_t n = MIN (nz, n_bytes); if ((full_write (fd, zeros, n)) != n) return false; n_bytes -= n; } return true; } From debbugs-submit-bounces@debbugs.gnu.org Sun Sep 26 04:51:48 2010 Received: (at submit) by debbugs.gnu.org; 26 Sep 2010 08:51:48 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OzmxX-0000Pg-PG for submit@debbugs.gnu.org; Sun, 26 Sep 2010 04:51:48 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OzmxV-0000Pb-7H for submit@debbugs.gnu.org; Sun, 26 Sep 2010 04:51:45 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Ozn09-0001y9-Dm for submit@debbugs.gnu.org; Sun, 26 Sep 2010 04:54:30 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:40860) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Ozn09-0001y5-Bc for submit@debbugs.gnu.org; Sun, 26 Sep 2010 04:54:29 -0400 Received: from [140.186.70.92] (port=34085 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Ozn08-0000PO-4c for bug-coreutils@gnu.org; Sun, 26 Sep 2010 04:54:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1Ozn06-0001xL-T2 for bug-coreutils@gnu.org; Sun, 26 Sep 2010 04:54:27 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:61094) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Ozn06-0001x4-KK for bug-coreutils@gnu.org; Sun, 26 Sep 2010 04:54:26 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o8Q8sK5N015824 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sun, 26 Sep 2010 08:54:22 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o8Q8sIkb031956; Sun, 26 Sep 2010 08:54:19 GMT Received: from abhmt005.oracle.com by acsmt354.oracle.com with ESMTP id 630617611285491184; Sun, 26 Sep 2010 01:53:04 -0700 Received: from [10.182.120.185] (/10.182.120.185) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Sun, 26 Sep 2010 01:53:04 -0700 Message-ID: <4C9F09EE.8080801@oracle.com> Date: Sun, 26 Sep 2010 16:53:02 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87iq20ejh2.fsf@meyering.net> <4C9F025A.9090300@oracle.com> <87sk0whoxr.fsf@meyering.net> In-Reply-To: <87sk0whoxr.fsf@meyering.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?ISO-8859-1?Q?P=E1draig_Brady?= , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) Hi Jim, Thanks for your prompt response, I will fix this issue when all review done. Regards, -Jeff Jim Meyering wrote: > Hi Jeff, > > This function has problems: > - the inner "zeros" declaration shadows the outer one > and ends up being useless. > - the "sizeof zeros" resolves to 4 or 8. obviously not what you intended. > > ... >> static bool >> +write_zeros (int fd, uint64_t n_bytes) >> { >> - bool last = false; >> - union { struct fiemap f; char c[4096]; } fiemap_buf; >> - struct fiemap *fiemap = &fiemap_buf.f; >> - struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; >> - enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; >> - verify (count != 0); >> + char *zeros = calloc (IO_BUFSIZE, sizeof (char)); >> + if (! zeros) >> + { >> + /* Try a small buffer. */ >> + static char zeros[1024]; >> + } >> >> + while (n_bytes) >> + { >> + uint64_t n = MIN (sizeof zeros, n_bytes); >> + if ((full_write (fd, zeros, n)) != n) >> + return false; >> + n_bytes -= n; >> + } >> + >> + return true; >> +} > > Please use the following instead. > I'll review the rest tomorrow or Tuesday. > > static bool > write_zeros (int fd, uint64_t n_bytes) > { > static char *zeros; > static size_t nz = IO_BUFSIZE; > > if (zeros == NULL) > { > static char fallback[1024]; > zeros = calloc (nz, 1); > if (zeros == NULL) > { > zeros = fallback; > nz = sizeof fallback; > } > } > > while (n_bytes) > { > uint64_t n = MIN (nz, n_bytes); > if ((full_write (fd, zeros, n)) != n) > return false; > n_bytes -= n; > } > > return true; > } From debbugs-submit-bounces@debbugs.gnu.org Mon Sep 27 23:06:44 2010 Received: (at submit) by debbugs.gnu.org; 28 Sep 2010 03:06:44 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0QWh-0003v9-Qm for submit@debbugs.gnu.org; Mon, 27 Sep 2010 23:06:44 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0QWf-0003v4-FD for submit@debbugs.gnu.org; Mon, 27 Sep 2010 23:06:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1P0QZN-0004eZ-QQ for submit@debbugs.gnu.org; Mon, 27 Sep 2010 23:09:30 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:40354) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0QZN-0004eV-O0 for submit@debbugs.gnu.org; Mon, 27 Sep 2010 23:09:29 -0400 Received: from [140.186.70.92] (port=35860 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P0QZM-0002Q4-M9 for bug-coreutils@gnu.org; Mon, 27 Sep 2010 23:09:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1P0QZL-0004eG-3Y for bug-coreutils@gnu.org; Mon, 27 Sep 2010 23:09:28 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:64598) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0QZK-0004e5-UB for bug-coreutils@gnu.org; Mon, 27 Sep 2010 23:09:27 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o8S39Jhq022609 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 28 Sep 2010 03:09:20 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o8S0va9W028903; Tue, 28 Sep 2010 03:09:18 GMT Received: from abhmt007.oracle.com by acsmt354.oracle.com with ESMTP id 641757361285643304; Mon, 27 Sep 2010 20:08:24 -0700 Received: from [10.182.120.185] (/10.182.120.185) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 27 Sep 2010 20:08:23 -0700 Message-ID: <4CA15C28.4040504@oracle.com> Date: Tue, 28 Sep 2010 11:08:24 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87iq20ejh2.fsf@meyering.net> <4C9F025A.9090300@oracle.com> <87sk0whoxr.fsf@meyering.net> <4C9F09EE.8080801@oracle.com> In-Reply-To: <4C9F09EE.8080801@oracle.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?ISO-8859-1?Q?P=E1draig_Brady?= , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) jeff.liu wrote: > Hi Jim, > > Thanks for your prompt response, I will fix this issue when all review done. Hi Jim, For my current implementation, I just have another thought to remove the "char *fname" from struct extent_scan, and add a new item "int errno" to save the errno set by ioctl(2) or lseek(2) if either call failed. at first, I am intended to use "char *fname" for the debugging purpose inside, however, maybe its better to do such things outside of the module according to the return value and errno. this change can not only reduce the memory allocation for 'fname' but also make a neatly open_extent_scan() interface. /* Structure used to reserve extent scan information per file. */ struct extent_scan { .... int errno; .... }; void open_extent_scan (int src_fd, struct extent_scan **scan); Is it better? Thanks, -Jeff > > > Regards, > -Jeff > > Jim Meyering wrote: >> Hi Jeff, >> >> This function has problems: >> - the inner "zeros" declaration shadows the outer one >> and ends up being useless. >> - the "sizeof zeros" resolves to 4 or 8. obviously not what you intended. >> >> ... >>> static bool >>> +write_zeros (int fd, uint64_t n_bytes) >>> { >>> - bool last = false; >>> - union { struct fiemap f; char c[4096]; } fiemap_buf; >>> - struct fiemap *fiemap = &fiemap_buf.f; >>> - struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; >>> - enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; >>> - verify (count != 0); >>> + char *zeros = calloc (IO_BUFSIZE, sizeof (char)); >>> + if (! zeros) >>> + { >>> + /* Try a small buffer. */ >>> + static char zeros[1024]; >>> + } >>> >>> + while (n_bytes) >>> + { >>> + uint64_t n = MIN (sizeof zeros, n_bytes); >>> + if ((full_write (fd, zeros, n)) != n) >>> + return false; >>> + n_bytes -= n; >>> + } >>> + >>> + return true; >>> +} >> Please use the following instead. >> I'll review the rest tomorrow or Tuesday. >> >> static bool >> write_zeros (int fd, uint64_t n_bytes) >> { >> static char *zeros; >> static size_t nz = IO_BUFSIZE; >> >> if (zeros == NULL) >> { >> static char fallback[1024]; >> zeros = calloc (nz, 1); >> if (zeros == NULL) >> { >> zeros = fallback; >> nz = sizeof fallback; >> } >> } >> >> while (n_bytes) >> { >> uint64_t n = MIN (nz, n_bytes); >> if ((full_write (fd, zeros, n)) != n) >> return false; >> n_bytes -= n; >> } >> >> return true; >> } > > > > From debbugs-submit-bounces@debbugs.gnu.org Tue Sep 28 04:20:39 2010 Received: (at submit) by debbugs.gnu.org; 28 Sep 2010 08:20:40 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0VQU-0005xp-TR for submit@debbugs.gnu.org; Tue, 28 Sep 2010 04:20:39 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0VQS-0005xk-3K for submit@debbugs.gnu.org; Tue, 28 Sep 2010 04:20:37 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1P0VT9-0004Fm-Fl for submit@debbugs.gnu.org; Tue, 28 Sep 2010 04:23:26 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:52916) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0VT9-0004Fh-8W for submit@debbugs.gnu.org; Tue, 28 Sep 2010 04:23:23 -0400 Received: from [140.186.70.92] (port=59108 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P0VT7-0006NZ-De for bug-coreutils@gnu.org; Tue, 28 Sep 2010 04:23:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1P0VT5-0004Es-0X for bug-coreutils@gnu.org; Tue, 28 Sep 2010 04:23:21 -0400 Received: from mx.meyering.net ([82.230.74.64]:34882) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0VT4-0004Eh-Bj for bug-coreutils@gnu.org; Tue, 28 Sep 2010 04:23:18 -0400 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 8EB29E93C; Tue, 28 Sep 2010 10:23:16 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4C9F025A.9090300@oracle.com> (jeff liu's message of "Sun, 26 Sep 2010 16:20:42 +0800") References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87iq20ejh2.fsf@meyering.net> <4C9F025A.9090300@oracle.com> Date: Tue, 28 Sep 2010 10:23:16 +0200 Message-ID: <87k4m6s2dn.fsf@meyering.net> Lines: 147 MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?utf-8?Q?P=C3=A1draig?= Brady , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) jeff.liu wrote: > Sorry for the delay. > > This is the new patch to isolate the stuff regarding to extents reading to a new module. and teach > cp(1) to make use of it. Jeff, I applied your patch to my rebased fiemap-copy branch. My first step was to run the usual ./bootstrap && ./configure && make && make check "make check" failed on due to a double free in your new code: (x86_64, Fedora 13, ext4 working directory) To get details, I made this temporary modification: diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap index b6b1103..a8643bc 100755 --- a/tests/cp/sparse-fiemap +++ b/tests/cp/sparse-fiemap @@ -82,7 +82,7 @@ for i in $(seq 1 2 21); do -e 'for (1..'$j') { sysseek (*F, $n, 1)' \ -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' > j1 || fail=1 # sync - cp --sparse=always j1 j2 || fail=1 + valgrind cp --sparse=always j1 j2 || fail=1 # sync # Technically we may need the 'sync' uses above, but # uncommenting them makes this test take much longer. Then I reran make check. That showed that 4 invocations of cp failed, each with errors like the ones below. I'll look at the actual code once you've fixed these bugs: ==13203== Command: cp --sparse=always j1 j2 ==13203== ==13203== Invalid read of size 1 ==13203== at 0x404960: extent_copy (copy.c:284) ==13203== by 0x405A92: copy_reg (copy.c:848) ==13203== by 0x4084F2: copy_internal (copy.c:2189) ==13203== by 0x40901E: copy (copy.c:2475) ==13203== by 0x403AC9: do_copy (cp.c:757) ==13203== by 0x4041F4: main (cp.c:1162) ==13203== Address 0x4c35b8d is 29 bytes inside a block of size 40 free'd ==13203== at 0x4A04D72: free (vg_replace_malloc.c:325) ==13203== by 0x4096A9: close_extent_scan (extent-scan.c:135) ==13203== by 0x40495B: extent_copy (copy.c:283) ==13203== by 0x405A92: copy_reg (copy.c:848) ==13203== by 0x4084F2: copy_internal (copy.c:2189) ==13203== by 0x40901E: copy (copy.c:2475) ==13203== by 0x403AC9: do_copy (cp.c:757) ==13203== by 0x4041F4: main (cp.c:1162) ==13203== ==13203== Invalid read of size 8 ==13203== at 0x409664: close_extent_scan (extent-scan.c:129) ==13203== by 0x40497D: extent_copy (copy.c:287) ==13203== by 0x405A92: copy_reg (copy.c:848) ==13203== by 0x4084F2: copy_internal (copy.c:2189) ==13203== by 0x40901E: copy (copy.c:2475) ==13203== by 0x403AC9: do_copy (cp.c:757) ==13203== by 0x4041F4: main (cp.c:1162) ==13203== Address 0x4c35b90 is 32 bytes inside a block of size 40 free'd ==13203== at 0x4A04D72: free (vg_replace_malloc.c:325) ==13203== by 0x4096A9: close_extent_scan (extent-scan.c:135) ==13203== by 0x40495B: extent_copy (copy.c:283) ==13203== by 0x405A92: copy_reg (copy.c:848) ==13203== by 0x4084F2: copy_internal (copy.c:2189) ==13203== by 0x40901E: copy (copy.c:2475) ==13203== by 0x403AC9: do_copy (cp.c:757) ==13203== by 0x4041F4: main (cp.c:1162) ==13203== ==13203== Invalid read of size 1 ==13203== at 0x409671: close_extent_scan (extent-scan.c:129) ==13203== by 0x40497D: extent_copy (copy.c:287) ==13203== by 0x405A92: copy_reg (copy.c:848) ==13203== by 0x4084F2: copy_internal (copy.c:2189) ==13203== by 0x40901E: copy (copy.c:2475) ==13203== by 0x403AC9: do_copy (cp.c:757) ==13203== by 0x4041F4: main (cp.c:1162) ==13203== Address 0x4c35b8d is 29 bytes inside a block of size 40 free'd ==13203== at 0x4A04D72: free (vg_replace_malloc.c:325) ==13203== by 0x4096A9: close_extent_scan (extent-scan.c:135) ==13203== by 0x40495B: extent_copy (copy.c:283) ==13203== by 0x405A92: copy_reg (copy.c:848) ==13203== by 0x4084F2: copy_internal (copy.c:2189) ==13203== by 0x40901E: copy (copy.c:2475) ==13203== by 0x403AC9: do_copy (cp.c:757) ==13203== by 0x4041F4: main (cp.c:1162) ==13203== ==13203== Invalid read of size 8 ==13203== at 0x409692: close_extent_scan (extent-scan.c:134) ==13203== by 0x40497D: extent_copy (copy.c:287) ==13203== by 0x405A92: copy_reg (copy.c:848) ==13203== by 0x4084F2: copy_internal (copy.c:2189) ==13203== by 0x40901E: copy (copy.c:2475) ==13203== by 0x403AC9: do_copy (cp.c:757) ==13203== by 0x4041F4: main (cp.c:1162) ==13203== Address 0x4c35b78 is 8 bytes inside a block of size 40 free'd ==13203== at 0x4A04D72: free (vg_replace_malloc.c:325) ==13203== by 0x4096A9: close_extent_scan (extent-scan.c:135) ==13203== by 0x40495B: extent_copy (copy.c:283) ==13203== by 0x405A92: copy_reg (copy.c:848) ==13203== by 0x4084F2: copy_internal (copy.c:2189) ==13203== by 0x40901E: copy (copy.c:2475) ==13203== by 0x403AC9: do_copy (cp.c:757) ==13203== by 0x4041F4: main (cp.c:1162) ==13203== ==13203== Invalid free() / delete / delete[] ==13203== at 0x4A04D72: free (vg_replace_malloc.c:325) ==13203== by 0x40969D: close_extent_scan (extent-scan.c:134) ==13203== by 0x40497D: extent_copy (copy.c:287) ==13203== by 0x405A92: copy_reg (copy.c:848) ==13203== by 0x4084F2: copy_internal (copy.c:2189) ==13203== by 0x40901E: copy (copy.c:2475) ==13203== by 0x403AC9: do_copy (cp.c:757) ==13203== by 0x4041F4: main (cp.c:1162) ==13203== Address 0x4c35be0 is 0 bytes inside a block of size 3 free'd ==13203== at 0x4A04D72: free (vg_replace_malloc.c:325) ==13203== by 0x40969D: close_extent_scan (extent-scan.c:134) ==13203== by 0x40495B: extent_copy (copy.c:283) ==13203== by 0x405A92: copy_reg (copy.c:848) ==13203== by 0x4084F2: copy_internal (copy.c:2189) ==13203== by 0x40901E: copy (copy.c:2475) ==13203== by 0x403AC9: do_copy (cp.c:757) ==13203== by 0x4041F4: main (cp.c:1162) ==13203== ==13203== Invalid free() / delete / delete[] ==13203== at 0x4A04D72: free (vg_replace_malloc.c:325) ==13203== by 0x4096A9: close_extent_scan (extent-scan.c:135) ==13203== by 0x40497D: extent_copy (copy.c:287) ==13203== by 0x405A92: copy_reg (copy.c:848) ==13203== by 0x4084F2: copy_internal (copy.c:2189) ==13203== by 0x40901E: copy (copy.c:2475) ==13203== by 0x403AC9: do_copy (cp.c:757) ==13203== by 0x4041F4: main (cp.c:1162) ==13203== Address 0x4c35b70 is 0 bytes inside a block of size 40 free'd ==13203== at 0x4A04D72: free (vg_replace_malloc.c:325) ==13203== by 0x4096A9: close_extent_scan (extent-scan.c:135) ==13203== by 0x40495B: extent_copy (copy.c:283) ==13203== by 0x405A92: copy_reg (copy.c:848) ==13203== by 0x4084F2: copy_internal (copy.c:2189) ==13203== by 0x40901E: copy (copy.c:2475) ==13203== by 0x403AC9: do_copy (cp.c:757) ==13203== by 0x4041F4: main (cp.c:1162) From debbugs-submit-bounces@debbugs.gnu.org Tue Sep 28 04:24:35 2010 Received: (at submit) by debbugs.gnu.org; 28 Sep 2010 08:24:36 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0VUJ-0005zV-AC for submit@debbugs.gnu.org; Tue, 28 Sep 2010 04:24:35 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0VUH-0005zQ-Bk for submit@debbugs.gnu.org; Tue, 28 Sep 2010 04:24:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1P0VX0-00057c-IM for submit@debbugs.gnu.org; Tue, 28 Sep 2010 04:27:23 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:47150) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0VX0-00057T-FN for submit@debbugs.gnu.org; Tue, 28 Sep 2010 04:27:22 -0400 Received: from [140.186.70.92] (port=59950 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P0VWy-00079i-4Z for bug-coreutils@gnu.org; Tue, 28 Sep 2010 04:27:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1P0VWv-00056K-2t for bug-coreutils@gnu.org; Tue, 28 Sep 2010 04:27:20 -0400 Received: from mx.meyering.net ([82.230.74.64]:34903) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0VWu-000567-T2 for bug-coreutils@gnu.org; Tue, 28 Sep 2010 04:27:17 -0400 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 41A0FE93C; Tue, 28 Sep 2010 10:27:16 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4CA15C28.4040504@oracle.com> (jeff liu's message of "Tue, 28 Sep 2010 11:08:24 +0800") References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87iq20ejh2.fsf@meyering.net> <4C9F025A.9090300@oracle.com> <87sk0whoxr.fsf@meyering.net> <4C9F09EE.8080801@oracle.com> <4CA15C28.4040504@oracle.com> Date: Tue, 28 Sep 2010 10:27:16 +0200 Message-ID: <87eices26z.fsf@meyering.net> Lines: 35 MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -5.3 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?utf-8?Q?P=C3=A1draig?= Brady , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) jeff.liu wrote: > jeff.liu wrote: >> Hi Jim, >> >> Thanks for your prompt response, I will fix this issue when all review done. > Hi Jim, > > For my current implementation, I just have another thought to remove the "char *fname" from struct > extent_scan, and add a new item "int errno" to save the errno set by ioctl(2) or lseek(2) if either > call failed. > at first, I am intended to use "char *fname" for the debugging purpose inside, however, maybe its > better to do such things outside of the module according to the return value and errno. this change > can not only reduce the memory allocation for 'fname' but also make a neatly open_extent_scan() > interface. > > /* Structure used to reserve extent scan information per file. */ > struct extent_scan > { > .... > int errno; > .... > }; > > void > open_extent_scan (int src_fd, struct extent_scan **scan); > > > Is it better? That would be better. Since there is only one diagnostic using that file name you can do the same from the caller. In fact, why store errno in that struct at all? Can't you just ensure that errno is set to the proper value when it fails? From debbugs-submit-bounces@debbugs.gnu.org Wed Sep 29 03:33:29 2010 Received: (at submit) by debbugs.gnu.org; 29 Sep 2010 07:33:30 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0rAP-0001ZZ-AR for submit@debbugs.gnu.org; Wed, 29 Sep 2010 03:33:29 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0rAN-0001ZU-Ev for submit@debbugs.gnu.org; Wed, 29 Sep 2010 03:33:27 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1P0rD9-0000uP-CV for submit@debbugs.gnu.org; Wed, 29 Sep 2010 03:36:20 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:34456) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0rD9-0000uL-AN for submit@debbugs.gnu.org; Wed, 29 Sep 2010 03:36:19 -0400 Received: from [140.186.70.92] (port=57100 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P0rD3-0003Ky-EK for bug-coreutils@gnu.org; Wed, 29 Sep 2010 03:36:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1P0rD2-0000sr-1K for bug-coreutils@gnu.org; Wed, 29 Sep 2010 03:36:13 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:54092) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0rD1-0000rz-SY for bug-coreutils@gnu.org; Wed, 29 Sep 2010 03:36:12 -0400 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o8T7a3HS005301 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 29 Sep 2010 07:36:04 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o8T1cTmJ026401; Wed, 29 Sep 2010 07:36:02 GMT Received: from abhmt016.oracle.com by acsmt353.oracle.com with ESMTP id 639645281285745716; Wed, 29 Sep 2010 00:35:16 -0700 Received: from [10.182.120.185] (/10.182.120.185) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 29 Sep 2010 00:35:16 -0700 Message-ID: <4CA2EC35.6000600@oracle.com> Date: Wed, 29 Sep 2010 15:35:17 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87iq20ejh2.fsf@meyering.net> <4C9F025A.9090300@oracle.com> <87sk0whoxr.fsf@meyering.net> <4C9F09EE.8080801@oracle.com> <4CA15C28.4040504@oracle.com> <87eices26z.fsf@meyering.net> In-Reply-To: <87eices26z.fsf@meyering.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?ISO-8859-1?Q?P=E1draig_Brady?= , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) Jim Meyering wrote: > jeff.liu wrote: >> jeff.liu wrote: >>> Hi Jim, >>> >>> Thanks for your prompt response, I will fix this issue when all review done. >> Hi Jim, >> >> For my current implementation, I just have another thought to remove the "char *fname" from struct >> extent_scan, and add a new item "int errno" to save the errno set by ioctl(2) or lseek(2) if either >> call failed. >> at first, I am intended to use "char *fname" for the debugging purpose inside, however, maybe its >> better to do such things outside of the module according to the return value and errno. this change >> can not only reduce the memory allocation for 'fname' but also make a neatly open_extent_scan() >> interface. >> >> /* Structure used to reserve extent scan information per file. */ >> struct extent_scan >> { >> .... >> int errno; >> .... >> }; >> >> void >> open_extent_scan (int src_fd, struct extent_scan **scan); >> >> >> Is it better? > > That would be better. > Since there is only one diagnostic using that file name > you can do the same from the caller. > > In fact, why store errno in that struct at all? > Can't you just ensure that errno is set to the proper value when it fails? Indeed, there is no need to store 'errno'. Thanks, -Jeff > > > From debbugs-submit-bounces@debbugs.gnu.org Wed Sep 29 04:11:05 2010 Received: (at submit) by debbugs.gnu.org; 29 Sep 2010 08:11:05 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0rkl-0001oR-Q0 for submit@debbugs.gnu.org; Wed, 29 Sep 2010 04:11:04 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0rkj-0001o3-22 for submit@debbugs.gnu.org; Wed, 29 Sep 2010 04:11:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1P0rnP-0006oj-2e for submit@debbugs.gnu.org; Wed, 29 Sep 2010 04:13:53 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:45036) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0rnO-0006od-TO for submit@debbugs.gnu.org; Wed, 29 Sep 2010 04:13:47 -0400 Received: from [140.186.70.92] (port=38349 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P0rnI-0003yQ-IW for bug-coreutils@gnu.org; Wed, 29 Sep 2010 04:13:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1P0rnA-0006mT-Sy for bug-coreutils@gnu.org; Wed, 29 Sep 2010 04:13:40 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:58337) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P0rnA-0006mB-Hs for bug-coreutils@gnu.org; Wed, 29 Sep 2010 04:13:32 -0400 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o8T8DO3m030067 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 29 Sep 2010 08:13:26 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o8T2lppn002827; Wed, 29 Sep 2010 08:13:23 GMT Received: from abhmt015.oracle.com by acsmt355.oracle.com with ESMTP id 639744831285747904; Wed, 29 Sep 2010 01:11:44 -0700 Received: from [10.182.120.185] (/10.182.120.185) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 29 Sep 2010 01:11:41 -0700 Message-ID: <4CA2F4BD.20907@oracle.com> Date: Wed, 29 Sep 2010 16:11:41 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87iq20ejh2.fsf@meyering.net> <4C9F025A.9090300@oracle.com> <87k4m6s2dn.fsf@meyering.net> In-Reply-To: <87k4m6s2dn.fsf@meyering.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?ISO-8859-1?Q?P=E1draig_Brady?= , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) Jim Meyering wrote: > jeff.liu wrote: >> Sorry for the delay. >> >> This is the new patch to isolate the stuff regarding to extents reading to a new module. and teach >> cp(1) to make use of it. > > Jeff, > > I applied your patch to my rebased fiemap-copy branch. > My first step was to run the usual > > ./bootstrap && ./configure && make && make check > > "make check" failed on due to a double free in your new code: > (x86_64, Fedora 13, ext4 working directory) > > To get details, I made this temporary modification: Hi Jim, I am sorry for the fault, it fixed at the patch below. Would you please revie at your convenience? Changes: ======== 1. fix write_zeros() as Jim's comments, thanks for pointing this out. 2. remove char const *fname from struct extent_scan. 3. change the signature of open_extent_scan() from "void open_extent_scan(struct extent_scan **scan)" to "void open_extent_scan(struct extent_scan *scan)"; the reason is I'd like to reduce once memory allocation for the extent_scan variable, instead, using stack to save it. 4. remove close_extent_scan() from a function defined at extent-scan.c to extent-scan.h as a Macro definination, but it does nothing for now, since initial extent scan defined at stack. 5. add a macro "free_extents_info()" defined at extent-scan.h to release the memory allocated to extent info which should be called combine with get_extents_info(), it just one line, so IMHO, define it as macro should be ok. I have done the memory check via `valgrind`, no issue found. make test against cp/sparse-fiemap failed at the extent compare stage, but the file content is identical to each other by comparing those two files "j1/j2" manually. Is it make sense if we verify them through diff(1) since the testing file is in small size? or we have to merge the contig extents from the output of `filefrag', I admit I have not dig into the filefrag-extent-compare at the moment, I need to recall the perl language syntax. :-P. >From 50a3338db06442fa2d789fd65175172d140cc96e Mon Sep 17 00:00:00 2001 From: Jie Liu Date: Wed, 29 Sep 2010 15:35:43 +0800 Subject: [PATCH 1/1] cp: add a new module for scanning extents * src/extent-scan.c: Source code for scanning extents. Call open_extent_scan() to initialize extent scan. Call get_extents_info() to get a number of extents for each iteration. * src/extent-scan.h: Header file of extent-scan.c. Wrap free_extent_info() as macro define to release the space allocated extent_info per extent scan. Wrap close_extent_scan() as macro define but do nothing at the moment. * src/Makefile.am: Reference it and link it to copy_source. * src/copy.c: Make use of the new module, replace fiemap_copy() with extent_copy(). Signed-off-by: Jie Liu --- src/Makefile.am | 2 +- src/copy.c | 197 ++++++++++++++++++++++++++++++---------------------- src/extent-scan.c | 113 ++++++++++++++++++++++++++++++ src/extent-scan.h | 68 ++++++++++++++++++ 4 files changed, 296 insertions(+), 84 deletions(-) create mode 100644 src/extent-scan.c create mode 100644 src/extent-scan.h diff --git a/src/Makefile.am b/src/Makefile.am index 7d56312..2412b16 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -459,7 +459,7 @@ uninstall-local: fi; \ fi -copy_sources = copy.c cp-hash.c +copy_sources = copy.c cp-hash.c extent-scan.c # Use `ginstall' in the definition of PROGRAMS and in dependencies to avoid # confusion with the `install' target. The install rule transforms `ginstall' diff --git a/src/copy.c b/src/copy.c index f48c74d..42ec300 100644 --- a/src/copy.c +++ b/src/copy.c @@ -35,6 +35,7 @@ #include "buffer-lcm.h" #include "copy.h" #include "cp-hash.h" +#include "extent-scan.h" #include "error.h" #include "fcntl--.h" #include "file-set.h" @@ -63,10 +64,6 @@ #include -#ifndef HAVE_FIEMAP -# include "fiemap.h" -#endif - #ifndef HAVE_FCHOWN # define HAVE_FCHOWN false # define fchown(fd, uid, gid) (-1) @@ -153,74 +150,79 @@ clone_file (int dest_fd, int src_fd) #endif } -#ifdef __linux__ -# ifndef FS_IOC_FIEMAP -# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) -# endif -/* Perform a FIEMAP copy, if possible. - Call ioctl(2) with FS_IOC_FIEMAP (available in linux 2.6.27) to - obtain a map of file extents excluding holes. This avoids the - overhead of detecting holes in a hole-introducing/preserving copy, - and thus makes copying sparse files much more efficient. Upon a - successful copy, return true. If the initial ioctl fails, set - *NORMAL_COPY_REQUIRED to true and return false. Upon any other - failure, set *NORMAL_COPY_REQUIRED to false and return false. */ static bool -fiemap_copy (int src_fd, int dest_fd, size_t buf_size, - off_t src_total_size, char const *src_name, - char const *dst_name, bool *normal_copy_required) +write_zeros (int fd, uint64_t n_bytes) { - bool last = false; - union { struct fiemap f; char c[4096]; } fiemap_buf; - struct fiemap *fiemap = &fiemap_buf.f; - struct fiemap_extent *fm_ext = &fiemap->fm_extents[0]; - enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_ext }; - verify (count != 0); + static char *zeros; + static size_t nz = IO_BUFSIZE; + + if (zeros == NULL) + { + static char fallback[1024]; + zeros = calloc (nz, 1); + if (zeros == NULL) + { + zeros = fallback; + nz = sizeof fallback; + } + } + + while (n_bytes) + { + uint64_t n = MIN (sizeof nz, n_bytes); + if ((full_write (fd, zeros, n)) != n) + return false; + n_bytes -= n; + } + + return true; +} +/* Perform an efficient extent copy, if possible. This avoids + the overhead of detecting holes in hole-introducing/preserving + copy, and thus makes copying sparse files much more efficient. + Upon a successful copy, return true. If the initial extent scan + fails, set *NORMAL_COPY_REQUIRED to true and return false. + Upon any other failure, set *NORMAL_COPY_REQUIRED to false and + return false. */ +static bool +extent_copy (int src_fd, int dest_fd, size_t buf_size, + off_t src_total_size, bool make_holes, + char const *src_name, char const *dst_name, + bool *require_normal_copy) +{ + struct extent_scan scan; off_t last_ext_logical = 0; uint64_t last_ext_len = 0; uint64_t last_read_size = 0; - unsigned int i = 0; - *normal_copy_required = false; + unsigned int i; + bool ok = true; - /* This is required at least to initialize fiemap->fm_start, - but also serves (in mid 2010) to appease valgrind, which - appears not to know the semantics of the FIEMAP ioctl. */ - memset (&fiemap_buf, 0, sizeof fiemap_buf); + open_extent_scan (src_fd, &scan); do { - fiemap->fm_length = FIEMAP_MAX_OFFSET; - fiemap->fm_flags = FIEMAP_FLAG_SYNC; - fiemap->fm_extent_count = count; - - /* When ioctl(2) fails, fall back to the normal copy only if it - is the first time we met. */ - if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0) + ok = get_extents_info (&scan); + if (! ok) { - /* If the first ioctl fails, tell the caller that it is - ok to proceed with a normal copy. */ - if (i == 0) - *normal_copy_required = true; - else + if (scan.hit_last_extent) + break; + + if (scan.initial_scan_failed) { - /* If the second or subsequent ioctl fails, diagnose it, - since it ends up causing the entire copy/cp to fail. */ - error (0, errno, _("%s: FIEMAP ioctl failed"), quote (src_name)); + close_extent_scan (&scan); + *require_normal_copy = true; + return false; } + + error (0, errno, _("failed to get extents info %s"), quote (src_name)); return false; } - /* If 0 extents are returned, then more ioctls are not needed. */ - if (fiemap->fm_mapped_extents == 0) - break; - - for (i = 0; i < fiemap->fm_mapped_extents; i++) + for (i = 0; i < scan.ei_count; i++) { - assert (fm_ext[i].fe_logical <= OFF_T_MAX); - - off_t ext_logical = fm_ext[i].fe_logical; - uint64_t ext_len = fm_ext[i].fe_length; + off_t ext_logical = scan.ext_info[i].ext_logical; + uint64_t ext_len = scan.ext_info[i].ext_length; if (lseek (src_fd, ext_logical, SEEK_SET) < 0) { @@ -228,27 +230,37 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, return false; } - if (lseek (dest_fd, ext_logical, SEEK_SET) < 0) + if (make_holes) { - error (0, errno, _("cannot lseek %s"), quote (dst_name)); - return false; + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0) + { + error (0, errno, _("cannot lseek %s"), quote (dst_name)); + return false; + } } - - if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST) + else { - last_ext_logical = ext_logical; - last_ext_len = ext_len; - last = true; + /* If not making a sparse file, write zeros to the destination + file if there is a hole between the last and current extent. */ + if (last_ext_logical + last_ext_len < ext_logical) + { + uint64_t holes_len = ext_logical - last_ext_logical - last_ext_len; + if (! write_zeros (dest_fd, holes_len)) + return false; + } } + last_ext_logical = ext_logical; + last_ext_len = ext_len; + last_read_size = 0; + while (ext_len) { char buf[buf_size]; /* Avoid reading into the holes if the left extent length is shorter than the buffer size. */ - if (ext_len < buf_size) - buf_size = ext_len; + buf_size = MIN (ext_len, buf_size); ssize_t n_read = read (src_fd, buf, buf_size); if (n_read < 0) @@ -258,12 +270,12 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, continue; #endif error (0, errno, _("reading %s"), quote (src_name)); - return false; + return false; } if (n_read == 0) { - /* Figure out how many bytes read from the last extent. */ + /* Figure out how many bytes read from the previous extent. */ last_read_size = last_ext_len - ext_len; break; } @@ -278,27 +290,44 @@ fiemap_copy (int src_fd, int dest_fd, size_t buf_size, } } - fiemap->fm_start = fm_ext[i - 1].fe_logical + fm_ext[i - 1].fe_length; + /* Release the space allocated to scan->ext_info. */ + free_extents_info (&scan); + } while (! scan.hit_last_extent); - } while (! last); + /* Do nothing now. */ + close_extent_scan (&scan); /* If a file ends up with holes, the sum of the last extent logical offset - and the read-returned size will be shorter than the actual size of the - file. Use ftruncate to extend the length of the destination file. */ - if (last_ext_logical + last_read_size < src_total_size) + and the read-returned size or the last extent length will be shorter than + the actual size of the file. Use ftruncate to extend the length of the + destination file if make_holes, or write zeros up to the actual size of the + file. */ + if (make_holes) { - if (ftruncate (dest_fd, src_total_size) < 0) + if (last_ext_logical + last_read_size < src_total_size) { - error (0, errno, _("failed to extend %s"), quote (dst_name)); - return false; + if (ftruncate (dest_fd, src_total_size) < 0) + { + error (0, errno, _("failed to extend %s"), quote (dst_name)); + return false; + } + } + } + else + { + if (last_ext_logical + last_ext_len < src_total_size) + { + uint64_t holes_len = src_total_size - last_ext_logical - last_ext_len; + if (0 < holes_len) + { + if (! write_zeros (dest_fd, holes_len)) + return false; + } } } return true; } -#else -static bool fiemap_copy (ignored) { errno == ENOTSUP; return false; } -#endif /* FIXME: describe */ /* FIXME: rewrite this to use a hash table so we avoid the quadratic @@ -833,11 +862,13 @@ copy_reg (char const *src_name, char const *dst_name, if (make_holes) { bool require_normal_copy; - /* Perform efficient FIEMAP copy for sparse files, fall back to the - standard copy only if the ioctl(2) fails. */ - if (fiemap_copy (source_desc, dest_desc, buf_size, - src_open_sb.st_size, src_name, - dst_name, &require_normal_copy)) + /* Perform efficient extent copy for sparse file, fall back to the + standard copy only if the initial extent scan fails. If the + '--sparse=never' option was specified, we writing all data but + use extent copy if available to efficiently read. */ + if (extent_copy (source_desc, dest_desc, buf_size, + src_open_sb.st_size, make_holes, + src_name, dst_name, &require_normal_copy)) goto preserve_metadata; else { diff --git a/src/extent-scan.c b/src/extent-scan.c new file mode 100644 index 0000000..f371b87 --- /dev/null +++ b/src/extent-scan.c @@ -0,0 +1,113 @@ +/* extent-scan.c -- core functions for scanning extents + Copyright (C) 2010 Free Software Foundation, Inc. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + + Written by Jie Liu (jeff.liu@oracle.com). */ + +#include +#include +#include +#include +#include + +#include "system.h" +#include "extent-scan.h" +#include "error.h" +#include "quote.h" + +#ifndef HAVE_FIEMAP +# include "fiemap.h" +#endif + +/* Allocate space for struct extent_scan, initialize the entries if + necessary and return it as the input argument of get_extents_info(). */ +extern void +open_extent_scan (int src_fd, struct extent_scan *scan) +{ + scan->fd = src_fd; + scan->ei_count = 0; + scan->scan_start = 0; + scan->initial_scan_failed = false; + scan->hit_last_extent = false; +} + +#ifdef __linux__ +# ifndef FS_IOC_FIEMAP +# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap) +# endif +/* Call ioctl(2) with FS_IOC_FIEMAP (available in linux 2.6.27) to + obtain a map of file extents excluding holes. */ +extern bool +get_extents_info (struct extent_scan *scan) +{ + union { struct fiemap f; char c[4096]; } fiemap_buf; + struct fiemap *fiemap = &fiemap_buf.f; + struct fiemap_extent *fm_extents = &fiemap->fm_extents[0]; + enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_extents }; + verify (count != 0); + unsigned int i; + + /* This is required at least to initialize fiemap->fm_start, + but also serves (in mid 2010) to appease valgrind, which + appears not to know the semantics of the FIEMAP ioctl. */ + memset (&fiemap_buf, 0, sizeof fiemap_buf); + + fiemap->fm_start = scan->scan_start; + fiemap->fm_flags = FIEMAP_FLAG_SYNC; + fiemap->fm_extent_count = count; + fiemap->fm_length = FIEMAP_MAX_OFFSET - scan->scan_start; + + /* Fall back to the standard copy if call ioctl(2) failed for the + the first time. */ + if (ioctl (scan->fd, FS_IOC_FIEMAP, fiemap) < 0) + { + if (scan->scan_start == 0) + scan->initial_scan_failed = true; + return false; + } + + /* If 0 extents are returned, then more get_extent_table() are not needed. */ + if (fiemap->fm_mapped_extents == 0) + { + scan->hit_last_extent = true; + return false; + } + + scan->ei_count = fiemap->fm_mapped_extents; + scan->ext_info = xnmalloc (scan->ei_count, sizeof (struct extent_info)); + + for (i = 0; i < scan->ei_count; i++) + { + assert (fm_extents[i].fe_logical <= OFF_T_MAX); + + scan->ext_info[i].ext_logical = fm_extents[i].fe_logical; + scan->ext_info[i].ext_length = fm_extents[i].fe_length; + scan->ext_info[i].ext_flags = fm_extents[i].fe_flags; + } + + i--; + if (scan->ext_info[i].ext_flags & FIEMAP_EXTENT_LAST) + { + scan->hit_last_extent = true; + return true; + } + + scan->scan_start = fm_extents[i].fe_logical + fm_extents[i].fe_length; + + return true; +} +#else +extern bool get_extents_info (ignored) { errno = ENOTSUP; return false; } +#endif diff --git a/src/extent-scan.h b/src/extent-scan.h new file mode 100644 index 0000000..07c2e5b --- /dev/null +++ b/src/extent-scan.h @@ -0,0 +1,68 @@ +/* core functions for efficient reading sparse files + Copyright (C) 2010 Free Software Foundation, Inc. + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + + Written by Jie Liu (jeff.liu@oracle.com). */ + +#ifndef EXTENT_SCAN_H +# define EXTENT_SCAN_H + +/* Structure used to reserve information of each extent. */ +struct extent_info +{ + /* Logical offset of an extent. */ + off_t ext_logical; + + /* Extent length. */ + uint64_t ext_length; + + /* Extent flags, use it for FIEMAP only, or set it to zero. */ + uint32_t ext_flags; +}; + +/* Structure used to reserve extent scan information per file. */ +struct extent_scan +{ + /* File descriptor of extent scan run against. */ + int fd; + + /* Next scan start offset. */ + off_t scan_start; + + /* How many extent info returned for a scan. */ + uint32_t ei_count; + + /* If true, fall back to a normal copy, either + set by the failure of ioctl(2) for FIEMAP or + lseek(2) with SEEK_DATA. */ + bool initial_scan_failed; + + /* If ture, the total extent scan per file has been finished. */ + bool hit_last_extent; + + /* Extent information. */ + struct extent_info *ext_info; +}; + +void +open_extent_scan (int src_fd, struct extent_scan *scan); + +bool +get_extents_info (struct extent_scan *scan); + +#define free_extents_info(ext_scan) free ((ext_scan)->ext_info) +#define close_extent_scan(ext_scan) /* empty */ + +#endif /* EXTENT_SCAN_H */ -- 1.5.4.3 Regards, -Jeff From debbugs-submit-bounces@debbugs.gnu.org Mon Oct 11 16:55:58 2010 Received: (at submit) by debbugs.gnu.org; 11 Oct 2010 20:55:58 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P5PPa-00042h-02 for submit@debbugs.gnu.org; Mon, 11 Oct 2010 16:55:58 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P5PPX-00042c-FG for submit@debbugs.gnu.org; Mon, 11 Oct 2010 16:55:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1P5PSp-0004Gd-GU for submit@debbugs.gnu.org; Mon, 11 Oct 2010 16:59:20 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:47617) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1P5PSp-0004GZ-8d for submit@debbugs.gnu.org; Mon, 11 Oct 2010 16:59:19 -0400 Received: from [140.186.70.92] (port=34674 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P5PSn-0008T5-LO for bug-coreutils@gnu.org; Mon, 11 Oct 2010 16:59:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1P5PSl-0004GB-Pp for bug-coreutils@gnu.org; Mon, 11 Oct 2010 16:59:17 -0400 Received: from mx.meyering.net ([82.230.74.64]:53731) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1P5PSl-0004G0-90 for bug-coreutils@gnu.org; Mon, 11 Oct 2010 16:59:15 -0400 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 91244DA77; Mon, 11 Oct 2010 22:59:14 +0200 (CEST) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4CA2F4BD.20907@oracle.com> (jeff liu's message of "Wed, 29 Sep 2010 16:11:41 +0800") References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87iq20ejh2.fsf@meyering.net> <4C9F025A.9090300@oracle.com> <87k4m6s2dn.fsf@meyering.net> <4CA2F4BD.20907@oracle.com> Date: Mon, 11 Oct 2010 22:59:14 +0200 Message-ID: <87iq18bg4d.fsf@meyering.net> Lines: 377 MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?utf-8?Q?P=C3=A1draig?= Brady , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.3 (-----) jeff.liu wrote: > Jim Meyering wrote: >> jeff.liu wrote: >>> Sorry for the delay. >>> >>> This is the new patch to isolate the stuff regarding to extents reading to a new module. and teach >>> cp(1) to make use of it. >> >> Jeff, >> >> I applied your patch to my rebased fiemap-copy branch. >> My first step was to run the usual >> >> ./bootstrap && ./configure && make && make check >> >> "make check" failed on due to a double free in your new code: >> (x86_64, Fedora 13, ext4 working directory) >> >> To get details, I made this temporary modification: > Hi Jim, > > I am sorry for the fault, it fixed at the patch below. > Would you please revie at your convenience? Thanks, Here are 5 changes on top of yours. I'll definitely adjust logs and maybe merge one or two before pushing anything. Just to be sure people understand, this series will not be in the upcoming release. Quick summary: - don't let write failure go unreported - make "distcheck" pass once again - rename functions to start with "extent_scan_" - remove unnecessary #include directives (part of "make syntax-check") None of this is pushed yet, but at least now it passes "make distcheck". >From ef3c2fe3760b11bc143b36246ee458ec86c484c9 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Mon, 11 Oct 2010 11:19:02 +0200 Subject: [PATCH 1/5] rename extent_scan member * extent-scan.h [struct extent_scan]: Rename member: s/hit_last_extent/hit_final_extent/. "final" is clearer, since "last" can be interpreted as "preceding". --- src/copy.c | 4 ++-- src/extent-scan.c | 6 +++--- src/extent-scan.h | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/copy.c b/src/copy.c index 43eeb74..1e1360e 100644 --- a/src/copy.c +++ b/src/copy.c @@ -208,7 +208,7 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, bool ok = get_extents_info (&scan); if (! ok) { - if (scan.hit_last_extent) + if (scan.hit_final_extent) break; if (scan.initial_scan_failed) @@ -302,7 +302,7 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, /* Release the space allocated to scan->ext_info. */ free_extents_info (&scan); - } while (! scan.hit_last_extent); + } while (! scan.hit_final_extent); /* Do nothing now. */ close_extent_scan (&scan); diff --git a/src/extent-scan.c b/src/extent-scan.c index f371b87..b0345f5 100644 --- a/src/extent-scan.c +++ b/src/extent-scan.c @@ -40,7 +40,7 @@ open_extent_scan (int src_fd, struct extent_scan *scan) scan->ei_count = 0; scan->scan_start = 0; scan->initial_scan_failed = false; - scan->hit_last_extent = false; + scan->hit_final_extent = false; } #ifdef __linux__ @@ -81,7 +81,7 @@ get_extents_info (struct extent_scan *scan) /* If 0 extents are returned, then more get_extent_table() are not needed. */ if (fiemap->fm_mapped_extents == 0) { - scan->hit_last_extent = true; + scan->hit_final_extent = true; return false; } @@ -100,7 +100,7 @@ get_extents_info (struct extent_scan *scan) i--; if (scan->ext_info[i].ext_flags & FIEMAP_EXTENT_LAST) { - scan->hit_last_extent = true; + scan->hit_final_extent = true; return true; } diff --git a/src/extent-scan.h b/src/extent-scan.h index 07c2e5b..0c9c199 100644 --- a/src/extent-scan.h +++ b/src/extent-scan.h @@ -50,7 +50,7 @@ struct extent_scan bool initial_scan_failed; /* If ture, the total extent scan per file has been finished. */ - bool hit_last_extent; + bool hit_final_extent; /* Extent information. */ struct extent_info *ext_info; -- 1.7.3.1.104.gc752e >From 7f38fe3ab8d1ee08f8ca4a96457df39da5bd1f70 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Mon, 11 Oct 2010 11:44:12 +0200 Subject: [PATCH 2/5] rename extent-scan functions to start with extent_scan_ --- src/copy.c | 12 +++++------- src/extent-scan.c | 10 +++++----- src/extent-scan.h | 22 ++++++++++++---------- 3 files changed, 22 insertions(+), 22 deletions(-) diff --git a/src/copy.c b/src/copy.c index 1e1360e..a7d10b8 100644 --- a/src/copy.c +++ b/src/copy.c @@ -201,11 +201,11 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, uint64_t last_ext_len = 0; uint64_t last_read_size = 0; - open_extent_scan (src_fd, &scan); + extent_scan_init (src_fd, &scan); do { - bool ok = get_extents_info (&scan); + bool ok = extent_scan_read (&scan); if (! ok) { if (scan.hit_final_extent) @@ -213,7 +213,6 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, if (scan.initial_scan_failed) { - close_extent_scan (&scan); *require_normal_copy = true; return false; } @@ -301,11 +300,10 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, } /* Release the space allocated to scan->ext_info. */ - free_extents_info (&scan); - } while (! scan.hit_final_extent); + extent_scan_free (&scan); - /* Do nothing now. */ - close_extent_scan (&scan); + } + while (! scan.hit_final_extent); /* If a file ends up with holes, the sum of the last extent logical offset and the read-returned size or the last extent length will be shorter than diff --git a/src/extent-scan.c b/src/extent-scan.c index b0345f5..97bb792 100644 --- a/src/extent-scan.c +++ b/src/extent-scan.c @@ -32,9 +32,9 @@ #endif /* Allocate space for struct extent_scan, initialize the entries if - necessary and return it as the input argument of get_extents_info(). */ + necessary and return it as the input argument of extent_scan_read(). */ extern void -open_extent_scan (int src_fd, struct extent_scan *scan) +extent_scan_init (int src_fd, struct extent_scan *scan) { scan->fd = src_fd; scan->ei_count = 0; @@ -50,14 +50,13 @@ open_extent_scan (int src_fd, struct extent_scan *scan) /* Call ioctl(2) with FS_IOC_FIEMAP (available in linux 2.6.27) to obtain a map of file extents excluding holes. */ extern bool -get_extents_info (struct extent_scan *scan) +extent_scan_read (struct extent_scan *scan) { union { struct fiemap f; char c[4096]; } fiemap_buf; struct fiemap *fiemap = &fiemap_buf.f; struct fiemap_extent *fm_extents = &fiemap->fm_extents[0]; enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_extents }; verify (count != 0); - unsigned int i; /* This is required at least to initialize fiemap->fm_start, but also serves (in mid 2010) to appease valgrind, which @@ -88,6 +87,7 @@ get_extents_info (struct extent_scan *scan) scan->ei_count = fiemap->fm_mapped_extents; scan->ext_info = xnmalloc (scan->ei_count, sizeof (struct extent_info)); + unsigned int i; for (i = 0; i < scan->ei_count; i++) { assert (fm_extents[i].fe_logical <= OFF_T_MAX); @@ -109,5 +109,5 @@ get_extents_info (struct extent_scan *scan) return true; } #else -extern bool get_extents_info (ignored) { errno = ENOTSUP; return false; } +extern bool extent_scan_read (ignored) { errno = ENOTSUP; return false; } #endif diff --git a/src/extent-scan.h b/src/extent-scan.h index 0c9c199..3119c8d 100644 --- a/src/extent-scan.h +++ b/src/extent-scan.h @@ -19,7 +19,7 @@ #ifndef EXTENT_SCAN_H # define EXTENT_SCAN_H -/* Structure used to reserve information of each extent. */ +/* Structure used to store information of each extent. */ struct extent_info { /* Logical offset of an extent. */ @@ -44,25 +44,27 @@ struct extent_scan /* How many extent info returned for a scan. */ uint32_t ei_count; - /* If true, fall back to a normal copy, either - set by the failure of ioctl(2) for FIEMAP or - lseek(2) with SEEK_DATA. */ + /* If true, fall back to a normal copy, either set by the + failure of ioctl(2) for FIEMAP or lseek(2) with SEEK_DATA. */ bool initial_scan_failed; - /* If ture, the total extent scan per file has been finished. */ + /* If true, the total extent scan per file has been finished. */ bool hit_final_extent; - /* Extent information. */ + /* Extent information: a malloc'd array of ei_count structs. */ struct extent_info *ext_info; }; void -open_extent_scan (int src_fd, struct extent_scan *scan); +extent_scan_init (int src_fd, struct extent_scan *scan); bool -get_extents_info (struct extent_scan *scan); +extent_scan_read (struct extent_scan *scan); -#define free_extents_info(ext_scan) free ((ext_scan)->ext_info) -#define close_extent_scan(ext_scan) /* empty */ +static inline void +extent_scan_free (struct extent_scan *scan) +{ + free (scan->ext_info); +} #endif /* EXTENT_SCAN_H */ -- 1.7.3.1.104.gc752e >From e33ec433eb36b1a777f9591a63bcaee1b9e6c1bf Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Mon, 11 Oct 2010 11:55:46 +0200 Subject: [PATCH 3/5] distribute extent-scan.h, too * src/Makefile.am (copy_sources): Also distribute extent-scan.h. --- src/Makefile.am | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/src/Makefile.am b/src/Makefile.am index 7187596..de4c828 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -450,7 +450,7 @@ uninstall-local: fi; \ fi -copy_sources = copy.c cp-hash.c extent-scan.c +copy_sources = copy.c cp-hash.c extent-scan.c extent-scan.h # Use `ginstall' in the definition of PROGRAMS and in dependencies to avoid # confusion with the `install' target. The install rule transforms `ginstall' -- 1.7.3.1.104.gc752e >From b0a1374189800a6e8edc2cfb5154199fe970ccd7 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Mon, 11 Oct 2010 11:55:58 +0200 Subject: [PATCH 4/5] formatting --- src/extent-scan.c | 7 ++++++- src/extent-scan.h | 6 ++---- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/src/extent-scan.c b/src/extent-scan.c index 97bb792..5160975 100644 --- a/src/extent-scan.c +++ b/src/extent-scan.c @@ -109,5 +109,10 @@ extent_scan_read (struct extent_scan *scan) return true; } #else -extern bool extent_scan_read (ignored) { errno = ENOTSUP; return false; } +extern bool +extent_scan_read (struct extent_scan *scan ATTRIBUTE_UNUSED) +{ + errno = ENOTSUP; + return false; +} #endif diff --git a/src/extent-scan.h b/src/extent-scan.h index 3119c8d..ac9e500 100644 --- a/src/extent-scan.h +++ b/src/extent-scan.h @@ -55,11 +55,9 @@ struct extent_scan struct extent_info *ext_info; }; -void -extent_scan_init (int src_fd, struct extent_scan *scan); +void extent_scan_init (int src_fd, struct extent_scan *scan); -bool -extent_scan_read (struct extent_scan *scan); +bool extent_scan_read (struct extent_scan *scan); static inline void extent_scan_free (struct extent_scan *scan) -- 1.7.3.1.104.gc752e >From f4513c41e656af44859587060ce9658241988cb1 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Mon, 11 Oct 2010 12:00:07 +0200 Subject: [PATCH 5/5] extent-scan.c: don't include error.h or quote.h * src/extent-scan.c: Don't include error.h or quote.h. Neither is used. --- src/extent-scan.c | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/src/extent-scan.c b/src/extent-scan.c index 5160975..3bb0d53 100644 --- a/src/extent-scan.c +++ b/src/extent-scan.c @@ -24,8 +24,6 @@ #include "system.h" #include "extent-scan.h" -#include "error.h" -#include "quote.h" #ifndef HAVE_FIEMAP # include "fiemap.h" -- 1.7.3.1.104.gc752e From debbugs-submit-bounces@debbugs.gnu.org Tue Oct 12 01:24:22 2010 Received: (at submit) by debbugs.gnu.org; 12 Oct 2010 05:24:22 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P5XLZ-0000Qy-G5 for submit@debbugs.gnu.org; Tue, 12 Oct 2010 01:24:22 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1P5XLW-0000Qt-FG for submit@debbugs.gnu.org; Tue, 12 Oct 2010 01:24:19 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1P5XOo-0003Hj-4c for submit@debbugs.gnu.org; Tue, 12 Oct 2010 01:27:44 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:40843) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1P5XOm-0003Gz-VM for submit@debbugs.gnu.org; Tue, 12 Oct 2010 01:27:42 -0400 Received: from [140.186.70.92] (port=38686 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1P5XOi-0000Vq-RX for bug-coreutils@gnu.org; Tue, 12 Oct 2010 01:27:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1P5XDr-0001Qz-5X for bug-coreutils@gnu.org; Tue, 12 Oct 2010 01:16:24 -0400 Received: from rcsinet10.oracle.com ([148.87.113.121]:52020) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1P5XDq-0001Gu-Ty for bug-coreutils@gnu.org; Tue, 12 Oct 2010 01:16:23 -0400 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id o9C5GDig025165 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 12 Oct 2010 05:16:14 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id o9BKTOB0021852; Tue, 12 Oct 2010 05:16:12 GMT Received: from abhmt021.oracle.com by acsmt353.oracle.com with ESMTP id 682368651286860511; Mon, 11 Oct 2010 22:15:11 -0700 Received: from [10.182.120.185] (/10.182.120.185) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 11 Oct 2010 22:15:10 -0700 Message-ID: <4CB3EEDF.2090206@oracle.com> Date: Tue, 12 Oct 2010 13:15:11 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87iq20ejh2.fsf@meyering.net> <4C9F025A.9090300@oracle.com> <87k4m6s2dn.fsf@meyering.net> <4CA2F4BD.20907@oracle.com> <87iq18bg4d.fsf@meyering.net> In-Reply-To: <87iq18bg4d.fsf@meyering.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?ISO-8859-1?Q?P=E1draig_Brady?= , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) Jim Meyering wrote: > jeff.liu wrote: > >> Jim Meyering wrote: >>> jeff.liu wrote: >>>> Sorry for the delay. >>>> >>>> This is the new patch to isolate the stuff regarding to extents reading to a new module. and teach >>>> cp(1) to make use of it. >>> Jeff, >>> >>> I applied your patch to my rebased fiemap-copy branch. >>> My first step was to run the usual >>> >>> ./bootstrap && ./configure && make && make check >>> >>> "make check" failed on due to a double free in your new code: >>> (x86_64, Fedora 13, ext4 working directory) >>> >>> To get details, I made this temporary modification: >> Hi Jim, >> >> I am sorry for the fault, it fixed at the patch below. >> Would you please revie at your convenience? > > Thanks, > > Here are 5 changes on top of yours. > I'll definitely adjust logs and maybe merge one or two before > pushing anything. Just to be sure people understand, this series > will not be in the upcoming release. > > Quick summary: > - don't let write failure go unreported > - make "distcheck" pass once again > - rename functions to start with "extent_scan_" > - remove unnecessary #include directives (part of "make syntax-check") > > None of this is pushed yet, but at least now it passes "make distcheck". Thanks for the update, the new functions name looks better than me. Regards, -Jeff > > From ef3c2fe3760b11bc143b36246ee458ec86c484c9 Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Mon, 11 Oct 2010 11:19:02 +0200 > Subject: [PATCH 1/5] rename extent_scan member > > * extent-scan.h [struct extent_scan]: Rename member: > s/hit_last_extent/hit_final_extent/. "final" is clearer, > since "last" can be interpreted as "preceding". > --- > src/copy.c | 4 ++-- > src/extent-scan.c | 6 +++--- > src/extent-scan.h | 2 +- > 3 files changed, 6 insertions(+), 6 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index 43eeb74..1e1360e 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -208,7 +208,7 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, > bool ok = get_extents_info (&scan); > if (! ok) > { > - if (scan.hit_last_extent) > + if (scan.hit_final_extent) > break; > > if (scan.initial_scan_failed) > @@ -302,7 +302,7 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, > > /* Release the space allocated to scan->ext_info. */ > free_extents_info (&scan); > - } while (! scan.hit_last_extent); > + } while (! scan.hit_final_extent); > > /* Do nothing now. */ > close_extent_scan (&scan); > diff --git a/src/extent-scan.c b/src/extent-scan.c > index f371b87..b0345f5 100644 > --- a/src/extent-scan.c > +++ b/src/extent-scan.c > @@ -40,7 +40,7 @@ open_extent_scan (int src_fd, struct extent_scan *scan) > scan->ei_count = 0; > scan->scan_start = 0; > scan->initial_scan_failed = false; > - scan->hit_last_extent = false; > + scan->hit_final_extent = false; > } > > #ifdef __linux__ > @@ -81,7 +81,7 @@ get_extents_info (struct extent_scan *scan) > /* If 0 extents are returned, then more get_extent_table() are not needed. */ > if (fiemap->fm_mapped_extents == 0) > { > - scan->hit_last_extent = true; > + scan->hit_final_extent = true; > return false; > } > > @@ -100,7 +100,7 @@ get_extents_info (struct extent_scan *scan) > i--; > if (scan->ext_info[i].ext_flags & FIEMAP_EXTENT_LAST) > { > - scan->hit_last_extent = true; > + scan->hit_final_extent = true; > return true; > } > > diff --git a/src/extent-scan.h b/src/extent-scan.h > index 07c2e5b..0c9c199 100644 > --- a/src/extent-scan.h > +++ b/src/extent-scan.h > @@ -50,7 +50,7 @@ struct extent_scan > bool initial_scan_failed; > > /* If ture, the total extent scan per file has been finished. */ > - bool hit_last_extent; > + bool hit_final_extent; > > /* Extent information. */ > struct extent_info *ext_info; > -- > 1.7.3.1.104.gc752e > > > From 7f38fe3ab8d1ee08f8ca4a96457df39da5bd1f70 Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Mon, 11 Oct 2010 11:44:12 +0200 > Subject: [PATCH 2/5] rename extent-scan functions to start with extent_scan_ > > --- > src/copy.c | 12 +++++------- > src/extent-scan.c | 10 +++++----- > src/extent-scan.h | 22 ++++++++++++---------- > 3 files changed, 22 insertions(+), 22 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index 1e1360e..a7d10b8 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -201,11 +201,11 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, > uint64_t last_ext_len = 0; > uint64_t last_read_size = 0; > > - open_extent_scan (src_fd, &scan); > + extent_scan_init (src_fd, &scan); > > do > { > - bool ok = get_extents_info (&scan); > + bool ok = extent_scan_read (&scan); > if (! ok) > { > if (scan.hit_final_extent) > @@ -213,7 +213,6 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, > > if (scan.initial_scan_failed) > { > - close_extent_scan (&scan); > *require_normal_copy = true; > return false; > } > @@ -301,11 +300,10 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, > } > > /* Release the space allocated to scan->ext_info. */ > - free_extents_info (&scan); > - } while (! scan.hit_final_extent); > + extent_scan_free (&scan); > > - /* Do nothing now. */ > - close_extent_scan (&scan); > + } > + while (! scan.hit_final_extent); > > /* If a file ends up with holes, the sum of the last extent logical offset > and the read-returned size or the last extent length will be shorter than > diff --git a/src/extent-scan.c b/src/extent-scan.c > index b0345f5..97bb792 100644 > --- a/src/extent-scan.c > +++ b/src/extent-scan.c > @@ -32,9 +32,9 @@ > #endif > > /* Allocate space for struct extent_scan, initialize the entries if > - necessary and return it as the input argument of get_extents_info(). */ > + necessary and return it as the input argument of extent_scan_read(). */ > extern void > -open_extent_scan (int src_fd, struct extent_scan *scan) > +extent_scan_init (int src_fd, struct extent_scan *scan) > { > scan->fd = src_fd; > scan->ei_count = 0; > @@ -50,14 +50,13 @@ open_extent_scan (int src_fd, struct extent_scan *scan) > /* Call ioctl(2) with FS_IOC_FIEMAP (available in linux 2.6.27) to > obtain a map of file extents excluding holes. */ > extern bool > -get_extents_info (struct extent_scan *scan) > +extent_scan_read (struct extent_scan *scan) > { > union { struct fiemap f; char c[4096]; } fiemap_buf; > struct fiemap *fiemap = &fiemap_buf.f; > struct fiemap_extent *fm_extents = &fiemap->fm_extents[0]; > enum { count = (sizeof fiemap_buf - sizeof *fiemap) / sizeof *fm_extents }; > verify (count != 0); > - unsigned int i; > > /* This is required at least to initialize fiemap->fm_start, > but also serves (in mid 2010) to appease valgrind, which > @@ -88,6 +87,7 @@ get_extents_info (struct extent_scan *scan) > scan->ei_count = fiemap->fm_mapped_extents; > scan->ext_info = xnmalloc (scan->ei_count, sizeof (struct extent_info)); > > + unsigned int i; > for (i = 0; i < scan->ei_count; i++) > { > assert (fm_extents[i].fe_logical <= OFF_T_MAX); > @@ -109,5 +109,5 @@ get_extents_info (struct extent_scan *scan) > return true; > } > #else > -extern bool get_extents_info (ignored) { errno = ENOTSUP; return false; } > +extern bool extent_scan_read (ignored) { errno = ENOTSUP; return false; } > #endif > diff --git a/src/extent-scan.h b/src/extent-scan.h > index 0c9c199..3119c8d 100644 > --- a/src/extent-scan.h > +++ b/src/extent-scan.h > @@ -19,7 +19,7 @@ > #ifndef EXTENT_SCAN_H > # define EXTENT_SCAN_H > > -/* Structure used to reserve information of each extent. */ > +/* Structure used to store information of each extent. */ > struct extent_info > { > /* Logical offset of an extent. */ > @@ -44,25 +44,27 @@ struct extent_scan > /* How many extent info returned for a scan. */ > uint32_t ei_count; > > - /* If true, fall back to a normal copy, either > - set by the failure of ioctl(2) for FIEMAP or > - lseek(2) with SEEK_DATA. */ > + /* If true, fall back to a normal copy, either set by the > + failure of ioctl(2) for FIEMAP or lseek(2) with SEEK_DATA. */ > bool initial_scan_failed; > > - /* If ture, the total extent scan per file has been finished. */ > + /* If true, the total extent scan per file has been finished. */ > bool hit_final_extent; > > - /* Extent information. */ > + /* Extent information: a malloc'd array of ei_count structs. */ > struct extent_info *ext_info; > }; > > void > -open_extent_scan (int src_fd, struct extent_scan *scan); > +extent_scan_init (int src_fd, struct extent_scan *scan); > > bool > -get_extents_info (struct extent_scan *scan); > +extent_scan_read (struct extent_scan *scan); > > -#define free_extents_info(ext_scan) free ((ext_scan)->ext_info) > -#define close_extent_scan(ext_scan) /* empty */ > +static inline void > +extent_scan_free (struct extent_scan *scan) > +{ > + free (scan->ext_info); > +} > > #endif /* EXTENT_SCAN_H */ > -- > 1.7.3.1.104.gc752e > > > From e33ec433eb36b1a777f9591a63bcaee1b9e6c1bf Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Mon, 11 Oct 2010 11:55:46 +0200 > Subject: [PATCH 3/5] distribute extent-scan.h, too > > * src/Makefile.am (copy_sources): Also distribute extent-scan.h. > --- > src/Makefile.am | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/src/Makefile.am b/src/Makefile.am > index 7187596..de4c828 100644 > --- a/src/Makefile.am > +++ b/src/Makefile.am > @@ -450,7 +450,7 @@ uninstall-local: > fi; \ > fi > > -copy_sources = copy.c cp-hash.c extent-scan.c > +copy_sources = copy.c cp-hash.c extent-scan.c extent-scan.h > > # Use `ginstall' in the definition of PROGRAMS and in dependencies to avoid > # confusion with the `install' target. The install rule transforms `ginstall' > -- > 1.7.3.1.104.gc752e > > > From b0a1374189800a6e8edc2cfb5154199fe970ccd7 Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Mon, 11 Oct 2010 11:55:58 +0200 > Subject: [PATCH 4/5] formatting > > --- > src/extent-scan.c | 7 ++++++- > src/extent-scan.h | 6 ++---- > 2 files changed, 8 insertions(+), 5 deletions(-) > > diff --git a/src/extent-scan.c b/src/extent-scan.c > index 97bb792..5160975 100644 > --- a/src/extent-scan.c > +++ b/src/extent-scan.c > @@ -109,5 +109,10 @@ extent_scan_read (struct extent_scan *scan) > return true; > } > #else > -extern bool extent_scan_read (ignored) { errno = ENOTSUP; return false; } > +extern bool > +extent_scan_read (struct extent_scan *scan ATTRIBUTE_UNUSED) > +{ > + errno = ENOTSUP; > + return false; > +} > #endif > diff --git a/src/extent-scan.h b/src/extent-scan.h > index 3119c8d..ac9e500 100644 > --- a/src/extent-scan.h > +++ b/src/extent-scan.h > @@ -55,11 +55,9 @@ struct extent_scan > struct extent_info *ext_info; > }; > > -void > -extent_scan_init (int src_fd, struct extent_scan *scan); > +void extent_scan_init (int src_fd, struct extent_scan *scan); > > -bool > -extent_scan_read (struct extent_scan *scan); > +bool extent_scan_read (struct extent_scan *scan); > > static inline void > extent_scan_free (struct extent_scan *scan) > -- > 1.7.3.1.104.gc752e > > > From f4513c41e656af44859587060ce9658241988cb1 Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Mon, 11 Oct 2010 12:00:07 +0200 > Subject: [PATCH 5/5] extent-scan.c: don't include error.h or quote.h > > * src/extent-scan.c: Don't include error.h or quote.h. Neither is used. > --- > src/extent-scan.c | 2 -- > 1 files changed, 0 insertions(+), 2 deletions(-) > > diff --git a/src/extent-scan.c b/src/extent-scan.c > index 5160975..3bb0d53 100644 > --- a/src/extent-scan.c > +++ b/src/extent-scan.c > @@ -24,8 +24,6 @@ > > #include "system.h" > #include "extent-scan.h" > -#include "error.h" > -#include "quote.h" > > #ifndef HAVE_FIEMAP > # include "fiemap.h" > -- > 1.7.3.1.104.gc752e > > > From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 22 07:59:31 2011 Received: (at submit) by debbugs.gnu.org; 22 Jan 2011 12:59:31 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Pgd3y-0007Sg-Bz for submit@debbugs.gnu.org; Sat, 22 Jan 2011 07:59:31 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Pgd3w-0007SV-6A for submit@debbugs.gnu.org; Sat, 22 Jan 2011 07:59:29 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PgdBb-00049f-DR for submit@debbugs.gnu.org; Sat, 22 Jan 2011 08:07:24 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:34605) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PgdBb-00049Y-9l for submit@debbugs.gnu.org; Sat, 22 Jan 2011 08:07:23 -0500 Received: from [140.186.70.92] (port=59060 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PgdBZ-0008WU-Ld for bug-coreutils@gnu.org; Sat, 22 Jan 2011 08:07:23 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PgdBY-00048z-1X for bug-coreutils@gnu.org; Sat, 22 Jan 2011 08:07:21 -0500 Received: from mx.meyering.net ([82.230.74.64]:36455) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PgdBX-00048X-IZ for bug-coreutils@gnu.org; Sat, 22 Jan 2011 08:07:19 -0500 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id A9C0F604AB; Sat, 22 Jan 2011 14:07:16 +0100 (CET) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4C970AA9.5070304@oracle.com> (jeff liu's message of "Mon, 20 Sep 2010 15:18:01 +0800") References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> Date: Sat, 22 Jan 2011 14:07:16 +0100 Message-ID: <87ipxhdr3v.fsf@meyering.net> Lines: 280 MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Spam-Score: -5.7 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , bug-coreutils@gnu.org, Joel Becker , Tao Ma , =?iso-8859-1?Q?P=E1draig?= Brady , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.7 (-----) jeff.liu wrote: > Hi Jim and All, > > Do you have any comments for the current implementation? There have been several releases since we last talked about this, but now is a good time to revive it. I've rebased the fiemap-copy branch and made a few changes: (somewhat sloppy 2nd log entry with the "*") copy.c: shorten a comment to fit in 80 columns * src/copy.c (copy_reg): Remove useless else-after-goto. copy: call extent_copy also when make_holes is false, ... copy: tweak variable name; improve a comment copy: don't allocate a separate buffer just for extent-based copy I pushed the result as the new fiemap-copy-2 branch: http://git.savannah.gnu.org/cgit/coreutils.git/log/?h=fiemap-copy-2 Here are the five most recent commits. The last one is the most interesting. Here's its full log entry: copy: don't allocate a separate buffer just for extent-based copy * src/copy.c (copy_reg): Move use of extent_scan to just *after* we allocate the main copying buffer, so we can... (extent_scan): Take a new parameter, BUF, and use that rather than allocating a private buffer. Update caller. >From ffd02ad91ac22b18c0a07c433e7e9983aed81542 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Tue, 11 Jan 2011 22:49:34 +0100 Subject: [PATCH 1/5] copy.c: shorten a comment to fit in 80 columns --- src/copy.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/src/copy.c b/src/copy.c index 30c1b56..270009b 100644 --- a/src/copy.c +++ b/src/copy.c @@ -287,7 +287,7 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, if (n_read == 0) { - /* Figure out how many bytes read from the previous extent. */ + /* Record number of bytes read from the previous extent. */ last_read_size = last_ext_len - ext_len; break; } -- 1.7.3.5.38.gb312b >From f880d4e43c47fa0b08757d911e00c69de07296ab Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sat, 22 Jan 2011 12:30:21 +0100 Subject: [PATCH 2/5] * src/copy.c (copy_reg): Remove useless else-after-goto. --- src/copy.c | 10 ++++------ 1 files changed, 4 insertions(+), 6 deletions(-) diff --git a/src/copy.c b/src/copy.c index 270009b..71da00d 100644 --- a/src/copy.c +++ b/src/copy.c @@ -879,13 +879,11 @@ copy_reg (char const *src_name, char const *dst_name, src_open_sb.st_size, make_holes, src_name, dst_name, &require_normal_copy)) goto preserve_metadata; - else + + if (! require_normal_copy) { - if (! require_normal_copy) - { - return_val = false; - goto close_src_and_dst_desc; - } + return_val = false; + goto close_src_and_dst_desc; } } -- 1.7.3.5.38.gb312b >From 237c2325b3d11e1b1a576978b884df3423a075b1 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sat, 22 Jan 2011 12:36:03 +0100 Subject: [PATCH 3/5] copy: call extent_copy also when make_holes is false, ... so that we benefit from using extents also when reading a sparse input file with --sparse=never. * src/copy.c (copy_reg): Remove erroneous test of "make_holes" so that we call extent_copy also when make_holes is false. Otherwise, what's the point of that parameter? --- src/copy.c | 29 +++++++++++++---------------- 1 files changed, 13 insertions(+), 16 deletions(-) diff --git a/src/copy.c b/src/copy.c index 71da00d..be7fdba 100644 --- a/src/copy.c +++ b/src/copy.c @@ -868,23 +868,20 @@ copy_reg (char const *src_name, char const *dst_name, #endif } - if (make_holes) + bool require_normal_copy; + /* Perform efficient extent copy for sparse file, fall back to the + standard copy only if the initial extent scan fails. If the + '--sparse=never' option was specified, we writing all data but + use extent copy if available to efficiently read. */ + if (extent_copy (source_desc, dest_desc, buf_size, + src_open_sb.st_size, make_holes, + src_name, dst_name, &require_normal_copy)) + goto preserve_metadata; + + if (! require_normal_copy) { - bool require_normal_copy; - /* Perform efficient extent copy for sparse file, fall back to the - standard copy only if the initial extent scan fails. If the - '--sparse=never' option was specified, we writing all data but - use extent copy if available to efficiently read. */ - if (extent_copy (source_desc, dest_desc, buf_size, - src_open_sb.st_size, make_holes, - src_name, dst_name, &require_normal_copy)) - goto preserve_metadata; - - if (! require_normal_copy) - { - return_val = false; - goto close_src_and_dst_desc; - } + return_val = false; + goto close_src_and_dst_desc; } /* If not making a sparse file, try to use a more-efficient -- 1.7.3.5.38.gb312b >From b3dfab326ad8d917ac1eaba10e0852bf695f93ae Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sat, 22 Jan 2011 12:55:58 +0100 Subject: [PATCH 4/5] copy: tweak variable name; improve a comment * src/copy.c (copy_reg): Rename a variable to make more sense from caller's perspective: s/require_normal_copy/normal_copy_required/. This is an output-only variable, and the original name could make it look like an input (or i&o) variable. --- src/copy.c | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/copy.c b/src/copy.c index be7fdba..fae8dbe 100644 --- a/src/copy.c +++ b/src/copy.c @@ -868,17 +868,17 @@ copy_reg (char const *src_name, char const *dst_name, #endif } - bool require_normal_copy; - /* Perform efficient extent copy for sparse file, fall back to the + bool normal_copy_required; + /* Perform an efficient extent-based copy, falling back to the standard copy only if the initial extent scan fails. If the - '--sparse=never' option was specified, we writing all data but - use extent copy if available to efficiently read. */ + '--sparse=never' option is specified, write all data but use + any extents to read more efficiently. */ if (extent_copy (source_desc, dest_desc, buf_size, src_open_sb.st_size, make_holes, - src_name, dst_name, &require_normal_copy)) + src_name, dst_name, &normal_copy_required)) goto preserve_metadata; - if (! require_normal_copy) + if (! normal_copy_required) { return_val = false; goto close_src_and_dst_desc; -- 1.7.3.5.38.gb312b >From bdf7c351a37ed6eeaa6bce98cb82902073bcc6c3 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sat, 22 Jan 2011 13:09:08 +0100 Subject: [PATCH 5/5] copy: don't allocate a separate buffer just for extent-based copy * src/copy.c (copy_reg): Move use of extent_scan to just *after* we allocate the main copying buffer, so we can... (extent_scan): Take a new parameter, BUF, and use that rather than allocating a private buffer. Update caller. --- src/copy.c | 36 +++++++++++++++++------------------- 1 files changed, 17 insertions(+), 19 deletions(-) diff --git a/src/copy.c b/src/copy.c index fae8dbe..c9cc2f7 100644 --- a/src/copy.c +++ b/src/copy.c @@ -194,7 +194,7 @@ write_zeros (int fd, uint64_t n_bytes) Upon any other failure, set *NORMAL_COPY_REQUIRED to false and return false. */ static bool -extent_copy (int src_fd, int dest_fd, size_t buf_size, +extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, off_t src_total_size, bool make_holes, char const *src_name, char const *dst_name, bool *require_normal_copy) @@ -268,8 +268,6 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, while (ext_len) { - char buf[buf_size]; - /* Avoid reading into the holes if the left extent length is shorter than the buffer size. */ buf_size = MIN (ext_len, buf_size); @@ -868,22 +866,6 @@ copy_reg (char const *src_name, char const *dst_name, #endif } - bool normal_copy_required; - /* Perform an efficient extent-based copy, falling back to the - standard copy only if the initial extent scan fails. If the - '--sparse=never' option is specified, write all data but use - any extents to read more efficiently. */ - if (extent_copy (source_desc, dest_desc, buf_size, - src_open_sb.st_size, make_holes, - src_name, dst_name, &normal_copy_required)) - goto preserve_metadata; - - if (! normal_copy_required) - { - return_val = false; - goto close_src_and_dst_desc; - } - /* If not making a sparse file, try to use a more-efficient buffer size. */ if (! make_holes) @@ -912,6 +894,22 @@ copy_reg (char const *src_name, char const *dst_name, buf_alloc = xmalloc (buf_size + buf_alignment_slop); buf = ptr_align (buf_alloc, buf_alignment); + bool normal_copy_required; + /* Perform an efficient extent-based copy, falling back to the + standard copy only if the initial extent scan fails. If the + '--sparse=never' option is specified, write all data but use + any extents to read more efficiently. */ + if (extent_copy (source_desc, dest_desc, buf, buf_size, + src_open_sb.st_size, make_holes, + src_name, dst_name, &normal_copy_required)) + goto preserve_metadata; + + if (! normal_copy_required) + { + return_val = false; + goto close_src_and_dst_desc; + } + while (true) { word *wp = NULL; -- 1.7.3.5.38.gb312b From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 25 02:24:32 2011 Received: (at submit) by debbugs.gnu.org; 25 Jan 2011 07:24:32 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PhdGR-0000xI-PX for submit@debbugs.gnu.org; Tue, 25 Jan 2011 02:24:32 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PhdGN-0000x5-Rq for submit@debbugs.gnu.org; Tue, 25 Jan 2011 02:24:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PhdOA-0004Di-GE for submit@debbugs.gnu.org; Tue, 25 Jan 2011 02:32:31 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:49229) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PhdOA-0004DX-Ay for submit@debbugs.gnu.org; Tue, 25 Jan 2011 02:32:30 -0500 Received: from [140.186.70.92] (port=59345 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PhdO8-000257-77 for bug-coreutils@gnu.org; Tue, 25 Jan 2011 02:32:29 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PhdO6-0004D1-9o for bug-coreutils@gnu.org; Tue, 25 Jan 2011 02:32:28 -0500 Received: from rcsinet10.oracle.com ([148.87.113.121]:44500) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PhdO6-0004Ci-1u for bug-coreutils@gnu.org; Tue, 25 Jan 2011 02:32:26 -0500 Received: from acsinet15.oracle.com (acsinet15.oracle.com [141.146.126.227]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id p0P7WHCp008796 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 25 Jan 2011 07:32:19 GMT Received: from acsmt354.oracle.com (acsmt354.oracle.com [141.146.40.154]) by acsinet15.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id p0P5OdLr017084; Tue, 25 Jan 2011 07:32:17 GMT Received: from abhmt009.oracle.com by acsmt354.oracle.com with ESMTP id 950938891295940722; Mon, 24 Jan 2011 23:32:02 -0800 Received: from [10.159.221.15] (/10.159.221.15) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 24 Jan 2011 23:32:00 -0800 Message-ID: <4D3E7C6A.6030708@oracle.com> Date: Tue, 25 Jan 2011 15:31:54 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87ipxhdr3v.fsf@meyering.net> In-Reply-To: <87ipxhdr3v.fsf@meyering.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , =?ISO-8859-1?Q?P=E1draig_Brady?= , Paul Eggert , bug-coreutils@gnu.org, Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) Hi Jim, Thanks for your time to help consolidating the code! Is this patchset acceptable to merge into the next official release? AFAICS, the tests passed on all filesystems except ext4, but the result is ok by comparing the file contents, can we take this risk? Another thing is to add solaris SEEK_DATA support to extent_scan.c as we discussed before, not sure if anyone working on this now. If not, I will take some time to follow up but have to delay about 2 weeks since I will on vacation for the chinese new year start from next week. Btw, do you have plan to post extent_scan module to gnulib upstream? so that other file archive projects(like tar(1)) can benefit from it. Any thing I can do for this patchset please just let me know. :) Regards, -Jeff Jim Meyering wrote: > jeff.liu wrote: >> Hi Jim and All, >> >> Do you have any comments for the current implementation? > > There have been several releases since we last talked about this, > but now is a good time to revive it. > > I've rebased the fiemap-copy branch and made a few changes: > (somewhat sloppy 2nd log entry with the "*") > > copy.c: shorten a comment to fit in 80 columns > * src/copy.c (copy_reg): Remove useless else-after-goto. > copy: call extent_copy also when make_holes is false, ... > copy: tweak variable name; improve a comment > copy: don't allocate a separate buffer just for extent-based copy > > I pushed the result as the new fiemap-copy-2 branch: > > http://git.savannah.gnu.org/cgit/coreutils.git/log/?h=fiemap-copy-2 > > Here are the five most recent commits. > The last one is the most interesting. > Here's its full log entry: > > copy: don't allocate a separate buffer just for extent-based copy > * src/copy.c (copy_reg): Move use of extent_scan to just *after* > we allocate the main copying buffer, so we can... > (extent_scan): Take a new parameter, BUF, and use that rather > than allocating a private buffer. Update caller. > > > From ffd02ad91ac22b18c0a07c433e7e9983aed81542 Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Tue, 11 Jan 2011 22:49:34 +0100 > Subject: [PATCH 1/5] copy.c: shorten a comment to fit in 80 columns > > --- > src/copy.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index 30c1b56..270009b 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -287,7 +287,7 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, > > if (n_read == 0) > { > - /* Figure out how many bytes read from the previous extent. */ > + /* Record number of bytes read from the previous extent. */ > last_read_size = last_ext_len - ext_len; > break; > } > -- > 1.7.3.5.38.gb312b > > > From f880d4e43c47fa0b08757d911e00c69de07296ab Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Sat, 22 Jan 2011 12:30:21 +0100 > Subject: [PATCH 2/5] * src/copy.c (copy_reg): Remove useless else-after-goto. > > --- > src/copy.c | 10 ++++------ > 1 files changed, 4 insertions(+), 6 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index 270009b..71da00d 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -879,13 +879,11 @@ copy_reg (char const *src_name, char const *dst_name, > src_open_sb.st_size, make_holes, > src_name, dst_name, &require_normal_copy)) > goto preserve_metadata; > - else > + > + if (! require_normal_copy) > { > - if (! require_normal_copy) > - { > - return_val = false; > - goto close_src_and_dst_desc; > - } > + return_val = false; > + goto close_src_and_dst_desc; > } > } > > -- > 1.7.3.5.38.gb312b > > > From 237c2325b3d11e1b1a576978b884df3423a075b1 Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Sat, 22 Jan 2011 12:36:03 +0100 > Subject: [PATCH 3/5] copy: call extent_copy also when make_holes is false, ... > > so that we benefit from using extents also when reading a sparse > input file with --sparse=never. > * src/copy.c (copy_reg): Remove erroneous test of "make_holes" > so that we call extent_copy also when make_holes is false. > Otherwise, what's the point of that parameter? > --- > src/copy.c | 29 +++++++++++++---------------- > 1 files changed, 13 insertions(+), 16 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index 71da00d..be7fdba 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -868,23 +868,20 @@ copy_reg (char const *src_name, char const *dst_name, > #endif > } > > - if (make_holes) > + bool require_normal_copy; > + /* Perform efficient extent copy for sparse file, fall back to the > + standard copy only if the initial extent scan fails. If the > + '--sparse=never' option was specified, we writing all data but > + use extent copy if available to efficiently read. */ > + if (extent_copy (source_desc, dest_desc, buf_size, > + src_open_sb.st_size, make_holes, > + src_name, dst_name, &require_normal_copy)) > + goto preserve_metadata; > + > + if (! require_normal_copy) > { > - bool require_normal_copy; > - /* Perform efficient extent copy for sparse file, fall back to the > - standard copy only if the initial extent scan fails. If the > - '--sparse=never' option was specified, we writing all data but > - use extent copy if available to efficiently read. */ > - if (extent_copy (source_desc, dest_desc, buf_size, > - src_open_sb.st_size, make_holes, > - src_name, dst_name, &require_normal_copy)) > - goto preserve_metadata; > - > - if (! require_normal_copy) > - { > - return_val = false; > - goto close_src_and_dst_desc; > - } > + return_val = false; > + goto close_src_and_dst_desc; > } > > /* If not making a sparse file, try to use a more-efficient > -- > 1.7.3.5.38.gb312b > > > From b3dfab326ad8d917ac1eaba10e0852bf695f93ae Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Sat, 22 Jan 2011 12:55:58 +0100 > Subject: [PATCH 4/5] copy: tweak variable name; improve a comment > > * src/copy.c (copy_reg): Rename a variable to make more sense from > caller's perspective: s/require_normal_copy/normal_copy_required/. > This is an output-only variable, and the original name could make > it look like an input (or i&o) variable. > --- > src/copy.c | 12 ++++++------ > 1 files changed, 6 insertions(+), 6 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index be7fdba..fae8dbe 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -868,17 +868,17 @@ copy_reg (char const *src_name, char const *dst_name, > #endif > } > > - bool require_normal_copy; > - /* Perform efficient extent copy for sparse file, fall back to the > + bool normal_copy_required; > + /* Perform an efficient extent-based copy, falling back to the > standard copy only if the initial extent scan fails. If the > - '--sparse=never' option was specified, we writing all data but > - use extent copy if available to efficiently read. */ > + '--sparse=never' option is specified, write all data but use > + any extents to read more efficiently. */ > if (extent_copy (source_desc, dest_desc, buf_size, > src_open_sb.st_size, make_holes, > - src_name, dst_name, &require_normal_copy)) > + src_name, dst_name, &normal_copy_required)) > goto preserve_metadata; > > - if (! require_normal_copy) > + if (! normal_copy_required) > { > return_val = false; > goto close_src_and_dst_desc; > -- > 1.7.3.5.38.gb312b > > > From bdf7c351a37ed6eeaa6bce98cb82902073bcc6c3 Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Sat, 22 Jan 2011 13:09:08 +0100 > Subject: [PATCH 5/5] copy: don't allocate a separate buffer just for extent-based copy > > * src/copy.c (copy_reg): Move use of extent_scan to just *after* > we allocate the main copying buffer, so we can... > (extent_scan): Take a new parameter, BUF, and use that rather > than allocating a private buffer. Update caller. > --- > src/copy.c | 36 +++++++++++++++++------------------- > 1 files changed, 17 insertions(+), 19 deletions(-) > > diff --git a/src/copy.c b/src/copy.c > index fae8dbe..c9cc2f7 100644 > --- a/src/copy.c > +++ b/src/copy.c > @@ -194,7 +194,7 @@ write_zeros (int fd, uint64_t n_bytes) > Upon any other failure, set *NORMAL_COPY_REQUIRED to false and > return false. */ > static bool > -extent_copy (int src_fd, int dest_fd, size_t buf_size, > +extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, > off_t src_total_size, bool make_holes, > char const *src_name, char const *dst_name, > bool *require_normal_copy) > @@ -268,8 +268,6 @@ extent_copy (int src_fd, int dest_fd, size_t buf_size, > > while (ext_len) > { > - char buf[buf_size]; > - > /* Avoid reading into the holes if the left extent > length is shorter than the buffer size. */ > buf_size = MIN (ext_len, buf_size); > @@ -868,22 +866,6 @@ copy_reg (char const *src_name, char const *dst_name, > #endif > } > > - bool normal_copy_required; > - /* Perform an efficient extent-based copy, falling back to the > - standard copy only if the initial extent scan fails. If the > - '--sparse=never' option is specified, write all data but use > - any extents to read more efficiently. */ > - if (extent_copy (source_desc, dest_desc, buf_size, > - src_open_sb.st_size, make_holes, > - src_name, dst_name, &normal_copy_required)) > - goto preserve_metadata; > - > - if (! normal_copy_required) > - { > - return_val = false; > - goto close_src_and_dst_desc; > - } > - > /* If not making a sparse file, try to use a more-efficient > buffer size. */ > if (! make_holes) > @@ -912,6 +894,22 @@ copy_reg (char const *src_name, char const *dst_name, > buf_alloc = xmalloc (buf_size + buf_alignment_slop); > buf = ptr_align (buf_alloc, buf_alignment); > > + bool normal_copy_required; > + /* Perform an efficient extent-based copy, falling back to the > + standard copy only if the initial extent scan fails. If the > + '--sparse=never' option is specified, write all data but use > + any extents to read more efficiently. */ > + if (extent_copy (source_desc, dest_desc, buf, buf_size, > + src_open_sb.st_size, make_holes, > + src_name, dst_name, &normal_copy_required)) > + goto preserve_metadata; > + > + if (! normal_copy_required) > + { > + return_val = false; > + goto close_src_and_dst_desc; > + } > + > while (true) > { > word *wp = NULL; > -- > 1.7.3.5.38.gb312b > > > From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 25 02:51:48 2011 Received: (at submit) by debbugs.gnu.org; 25 Jan 2011 07:51:48 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Phdgq-0001ZX-Hn for submit@debbugs.gnu.org; Tue, 25 Jan 2011 02:51:48 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Phdgo-0001ZL-8B for submit@debbugs.gnu.org; Tue, 25 Jan 2011 02:51:47 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Phdob-000132-Oe for submit@debbugs.gnu.org; Tue, 25 Jan 2011 02:59:50 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:46431) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Phdob-00012x-KX for submit@debbugs.gnu.org; Tue, 25 Jan 2011 02:59:49 -0500 Received: from [140.186.70.92] (port=51024 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Phdoa-0004kz-Js for bug-coreutils@gnu.org; Tue, 25 Jan 2011 02:59:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PhdoZ-00011r-3U for bug-coreutils@gnu.org; Tue, 25 Jan 2011 02:59:48 -0500 Received: from mx.meyering.net ([82.230.74.64]:43868) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PhdoY-00011I-Qy for bug-coreutils@gnu.org; Tue, 25 Jan 2011 02:59:46 -0500 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id D4BF1600E1; Tue, 25 Jan 2011 08:59:44 +0100 (CET) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4D3E7C6A.6030708@oracle.com> (jeff liu's message of "Tue, 25 Jan 2011 15:31:54 +0800") References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87ipxhdr3v.fsf@meyering.net> <4D3E7C6A.6030708@oracle.com> Date: Tue, 25 Jan 2011 08:59:44 +0100 Message-ID: <87sjwhv2fj.fsf@meyering.net> Lines: 33 MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Spam-Score: -5.7 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , =?iso-8859-1?Q?P=E1draig?= Brady , Paul Eggert , bug-coreutils@gnu.org, Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.7 (-----) jeff.liu wrote: > AFAICS, the tests passed on all filesystems except ext4, Really? The vast majority of my testing is with ext4 on Fedora 14, and I have seen no failure -- otherwise I would have mentioned that as a known problem. What type of system/kernel are you using? Was your ext4 partition created long ago? With what options? Did "make check" fail? If so, please provide details. If something else failed, please give me enough information to reproduce it. > but the result is ok by comparing the file > contents, can we take this risk? > Is this patchset acceptable to merge into the next official release? An ext4 failure sounds ominous. > Another thing is to add solaris SEEK_DATA support to extent_scan.c as > we discussed before, not sure > if anyone working on this now. If not, I will take some time to follow > up but have to delay about 2 > weeks since I will on vacation for the chinese new year start from next week. > > Btw, do you have plan to post extent_scan module to gnulib upstream? > so that other file archive > projects(like tar(1)) can benefit from it. I do not plan to do that right away. > Any thing I can do for this patchset please just let me know. :) From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 25 06:40:56 2011 Received: (at submit) by debbugs.gnu.org; 25 Jan 2011 11:40:56 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PhhGZ-0006fH-5T for submit@debbugs.gnu.org; Tue, 25 Jan 2011 06:40:55 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PhhGW-0006f2-6e for submit@debbugs.gnu.org; Tue, 25 Jan 2011 06:40:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PhhOJ-0000b0-H1 for submit@debbugs.gnu.org; Tue, 25 Jan 2011 06:48:56 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:47408) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PhhOJ-0000at-DY for submit@debbugs.gnu.org; Tue, 25 Jan 2011 06:48:55 -0500 Received: from [140.186.70.92] (port=51529 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PhhOI-0003PW-4u for bug-coreutils@gnu.org; Tue, 25 Jan 2011 06:48:55 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PhhOG-0000aD-HK for bug-coreutils@gnu.org; Tue, 25 Jan 2011 06:48:53 -0500 Received: from rcsinet10.oracle.com ([148.87.113.121]:29612) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PhhOG-0000a2-98 for bug-coreutils@gnu.org; Tue, 25 Jan 2011 06:48:52 -0500 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id p0PBmhiK024179 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 25 Jan 2011 11:48:45 GMT Received: from acsmt353.oracle.com (acsmt353.oracle.com [141.146.40.153]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id p0P5I8RV028326; Tue, 25 Jan 2011 11:48:40 GMT Received: from abhmt015.oracle.com by acsmt354.oracle.com with ESMTP id 989886591295956013; Tue, 25 Jan 2011 03:46:53 -0800 Received: from [221.223.118.51] (/221.223.118.51) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 25 Jan 2011 03:46:52 -0800 Message-ID: <4D3EB82A.60203@oracle.com> Date: Tue, 25 Jan 2011 19:46:50 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87ipxhdr3v.fsf@meyering.net> <4D3E7C6A.6030708@oracle.com> <87sjwhv2fj.fsf@meyering.net> In-Reply-To: <87sjwhv2fj.fsf@meyering.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Spam-Score: -5.9 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , =?ISO-8859-1?Q?P=E1draig_Brady?= , Paul Eggert , bug-coreutils@gnu.org, Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.9 (-----) Jim Meyering wrote: > jeff.liu wrote: >> AFAICS, the tests passed on all filesystems except ext4, > > Really? > The vast majority of my testing is with ext4 on Fedora 14, and I have seen > no failure -- otherwise I would have mentioned that as a known problem. I have mentioned this issue at: http://osdir.com/ml/bug-coreutils-gnu/2010-09/msg00092.html "make test against cp/sparse-fiemap failed at the extent compare stage, but the file content is identical to each other by comparing those two files "j1/j2" manually. Is it make sense if we verify them through diff(1) since the testing file is in small size?" > > What type of system/kernel are you using? 2.6.33-RC3 && 2.6.36 > Was your ext4 partition created long ago? With what options? fiemap copy works well if run `cp' against physical ext4 partition. > Did "make check" fail? If so, please provide details. Yeah, I will show the detail of 'make check' at below. btw, I just checked out the new branch and tried to compile it but ran into an error: date.c:30:28: error: parse-datetime.h: No such file or directory date.c: In function 'batch_convert': date.c:284: warning: implicit declaration of function 'parse_datetime' I guess 'parse-datetime.h' is shipped with gnulib? For now, I can not pull the latest gnulib code since the remote host does not response. For a quick reply, I ran 'make check' against the previous code base before your latest commit. sudo make check TESTS=cp/sparse-fiemap VERBOSE=yes tests/cp/sparse-fiemap.log: =========================== .... ... + mount -oloop blob mnt + cd mnt + echo test + test -s f + test 0 = 1 + dd if=/dev/zero of=sparse bs=1k count=1 seek=1G 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 4.6234e-05 s, 22.1 MB/s + timeout 10 cp --sparse=always sparse fiemap ++ stat --printf %s sparse ++ stat --printf %s fiemap + test 1099511628800 = 1099511628800 + perl -e 1 ++ seq 1 2 21 + for i in '$(seq 1 2 21)' + for j in 1 2 31 100 + perl -e 'BEGIN { $n = 1 * 1024; *F = *STDOUT }' -e 'for (1..1) { sysseek (*F, $n, 1)' -e '&& syswrite (*F, chr($_)x$n) or die "$!"}' + cp --sparse=always j1 j2 + cmp j1 j2 + filefrag -v j1 + grep extent j1: 2 extents found + filefrag -v j1 + filefrag -v j2 + f ff1 + perl /home/jeff/opensource_dev/fiemap_copy/tests/filefrag-extent-compare + awk '/^ *[0-9]/ {printf "%d %d ", $2 ,NF < 5 ? $NF : $5 } END {print ""}' + sed 's/ [a-z,][a-z,]*$//' ff1 + f ff2 + sed 's/ [a-z,][a-z,]*$//' ff2 + awk '/^ *[0-9]/ {printf "%d %d ", $2 ,NF < 5 ? $NF : $5 } END {print ""}' @a and @b have different lengths, even after adjustment + fail=1 + break + test 1 = 1 + break + Exit 1 + set +e + exit 1 + exit 1 + remove_tmp_ + __st=1 + cleanup_ + cd / + umount /home/jeff/opensource_dev/fiemap_copy/tests/gt-sparse-fiemap.cXgC/mnt + cd /home/jeff/opensource_dev/fiemap_copy/tests + chmod -R u+rwx /home/jeff/opensource_dev/fiemap_copy/tests/gt-sparse-fiemap.cXgC + rm -rf /home/jeff/opensource_dev/fiemap_copy/tests/gt-sparse-fiemap.cXgC + exit 1 Thanks, -Jeff > If something else failed, please give me enough information > to reproduce it. > >> but the result is ok by comparing the file >> contents, can we take this risk? > >> Is this patchset acceptable to merge into the next official release? > > An ext4 failure sounds ominous. > >> Another thing is to add solaris SEEK_DATA support to extent_scan.c as >> we discussed before, not sure >> if anyone working on this now. If not, I will take some time to follow >> up but have to delay about 2 >> weeks since I will on vacation for the chinese new year start from next week. >> >> Btw, do you have plan to post extent_scan module to gnulib upstream? >> so that other file archive >> projects(like tar(1)) can benefit from it. > > I do not plan to do that right away. > >> Any thing I can do for this patchset please just let me know. :) > > > From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 25 12:10:00 2011 Received: (at submit) by debbugs.gnu.org; 25 Jan 2011 17:10:00 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PhmP1-0006xd-11 for submit@debbugs.gnu.org; Tue, 25 Jan 2011 12:09:59 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PhmOy-0006xC-Ie for submit@debbugs.gnu.org; Tue, 25 Jan 2011 12:09:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PhmWh-0004xm-IY for submit@debbugs.gnu.org; Tue, 25 Jan 2011 12:17:56 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:38817) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PhmWh-0004xh-FL for submit@debbugs.gnu.org; Tue, 25 Jan 2011 12:17:55 -0500 Received: from [140.186.70.92] (port=41842 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PhmWf-0002Og-Pf for bug-coreutils@gnu.org; Tue, 25 Jan 2011 12:17:55 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PhmWd-0004vr-Tg for bug-coreutils@gnu.org; Tue, 25 Jan 2011 12:17:53 -0500 Received: from mx.meyering.net ([82.230.74.64]:54937) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PhmWd-0004vR-NO for bug-coreutils@gnu.org; Tue, 25 Jan 2011 12:17:51 -0500 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 854C060472; Tue, 25 Jan 2011 18:17:50 +0100 (CET) From: Jim Meyering To: "jeff.liu" Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4D3EB82A.60203@oracle.com> (jeff liu's message of "Tue, 25 Jan 2011 19:46:50 +0800") References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87ipxhdr3v.fsf@meyering.net> <4D3E7C6A.6030708@oracle.com> <87sjwhv2fj.fsf@meyering.net> <4D3EB82A.60203@oracle.com> Date: Tue, 25 Jan 2011 18:17:50 +0100 Message-ID: <87pqrksy0x.fsf@meyering.net> Lines: 55 MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Spam-Score: -5.8 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Eggert , bug-coreutils@gnu.org, Paul, =?iso-8859-1?Q?P=E1draig?= Brady , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.8 (-----) jeff.liu wrote: > Jim Meyering wrote: >> jeff.liu wrote: >>> AFAICS, the tests passed on all filesystems except ext4, >> >> Really? >> The vast majority of my testing is with ext4 on Fedora 14, and I have seen >> no failure -- otherwise I would have mentioned that as a known problem. > > I have mentioned this issue at: > http://osdir.com/ml/bug-coreutils-gnu/2010-09/msg00092.html > > "make test against cp/sparse-fiemap failed at the extent compare > stage, but the file content is > identical to each other by comparing those two files "j1/j2" manually. > Is it make sense if we verify them through diff(1) since the testing > file is in small size?" No. The whole point of the test is to verify that the extents have been preserved in the copy. Diff doesn't know about extents. >> What type of system/kernel are you using? > 2.6.33-RC3 && 2.6.36 >> Was your ext4 partition created long ago? With what options? > fiemap copy works well if run `cp' against physical ext4 partition. >> Did "make check" fail? If so, please provide details. > Yeah, I will show the detail of 'make check' at below. What version of filefrag are you using? Mine comes from e2fsprogs-1.41.12-6.fc14.x86_64 > btw, I just checked out the new branch and tried to compile it but ran > into an error: > date.c:30:28: error: parse-datetime.h: No such file or directory > date.c: In function 'batch_convert': > date.c:284: warning: implicit declaration of function 'parse_datetime' > > I guess 'parse-datetime.h' is shipped with gnulib? For now, I can not > pull the latest gnulib code > since the remote host does not response. Did you run ./bootstrap ? That is a requirement whenever the coreutils pulls in a change to the gnulib submodule. > For a quick reply, I ran 'make check' against the previous code base > before your latest commit. > > sudo make check TESTS=cp/sparse-fiemap VERBOSE=yes ... > + filefrag -v j2 ... > + awk '/^ *[0-9]/ {printf "%d %d ", $2 ,NF < 5 ? $NF : $5 } END {print ""}' > @a and @b have different lengths, even after adjustment > + fail=1 From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 25 22:52:54 2011 Received: (at submit) by debbugs.gnu.org; 26 Jan 2011 03:52:55 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PhwRC-0007fX-JN for submit@debbugs.gnu.org; Tue, 25 Jan 2011 22:52:54 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PhwRA-0007fJ-2M for submit@debbugs.gnu.org; Tue, 25 Jan 2011 22:52:52 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PhwYz-0005iP-IS for submit@debbugs.gnu.org; Tue, 25 Jan 2011 23:00:58 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:40637) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PhwYz-0005iL-G0 for submit@debbugs.gnu.org; Tue, 25 Jan 2011 23:00:57 -0500 Received: from [140.186.70.92] (port=51258 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PhwYy-0001GA-8w for bug-coreutils@gnu.org; Tue, 25 Jan 2011 23:00:57 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PhwYw-0005i4-W5 for bug-coreutils@gnu.org; Tue, 25 Jan 2011 23:00:56 -0500 Received: from rcsinet10.oracle.com ([148.87.113.121]:27430) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PhwYw-0005hs-Ps for bug-coreutils@gnu.org; Tue, 25 Jan 2011 23:00:54 -0500 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id p0Q40i3I004884 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 26 Jan 2011 04:00:45 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id p0PLZ55o002618; Wed, 26 Jan 2011 04:00:41 GMT Received: from abhmt014.oracle.com by acsmt354.oracle.com with ESMTP id 953696311296014331; Tue, 25 Jan 2011 19:58:51 -0800 Received: from [10.191.6.208] (/10.191.6.208) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Tue, 25 Jan 2011 19:58:50 -0800 Message-ID: <4D3F9BEE.6060701@oracle.com> Date: Wed, 26 Jan 2011 11:58:38 +0800 From: "jeff.liu" User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87ipxhdr3v.fsf@meyering.net> <4D3E7C6A.6030708@oracle.com> <87sjwhv2fj.fsf@meyering.net> <4D3EB82A.60203@oracle.com> <87pqrksy0x.fsf@meyering.net> In-Reply-To: <87pqrksy0x.fsf@meyering.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Spam-Score: -6.3 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul@debbugs.gnu.org, Eggert , bug-coreutils@gnu.org, =?ISO-8859-1?Q?P=E1draig_Brady?= , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.3 (------) Jim Meyering wrote: > jeff.liu wrote: >> Jim Meyering wrote: >>> jeff.liu wrote: >>>> AFAICS, the tests passed on all filesystems except ext4, >>> Really? >>> The vast majority of my testing is with ext4 on Fedora 14, and I have seen >>> no failure -- otherwise I would have mentioned that as a known problem. >> I have mentioned this issue at: >> http://osdir.com/ml/bug-coreutils-gnu/2010-09/msg00092.html >> >> "make test against cp/sparse-fiemap failed at the extent compare >> stage, but the file content is >> identical to each other by comparing those two files "j1/j2" manually. >> Is it make sense if we verify them through diff(1) since the testing >> file is in small size?" > > No. The whole point of the test is to verify that the extents have > been preserved in the copy. Diff doesn't know about extents. > >>> What type of system/kernel are you using? >> 2.6.33-RC3 && 2.6.36 >>> Was your ext4 partition created long ago? With what options? >> fiemap copy works well if run `cp' against physical ext4 partition. >>> Did "make check" fail? If so, please provide details. >> Yeah, I will show the detail of 'make check' at below. > > What version of filefrag are you using? Mine comes from E2fsprogs version 1.41.12 shipped with ubuntu8.0.4. I updated the filefrag(8) to the upstream one but still no luck. :( the kernel I have tried are 2.6.28/2.6.33-RC3/2.6.36. > Mine comes from e2fsprogs-1.41.12-6.fc14.x86_64 > >> btw, I just checked out the new branch and tried to compile it but ran >> into an error: >> date.c:30:28: error: parse-datetime.h: No such file or directory >> date.c: In function 'batch_convert': >> date.c:284: warning: implicit declaration of function 'parse_datetime' >> >> I guess 'parse-datetime.h' is shipped with gnulib? For now, I can not >> pull the latest gnulib code >> since the remote host does not response. > > Did you run ./bootstrap ? sure, yesterday it failed due to my proxy issue. Thanks, -Jeff > That is a requirement whenever the coreutils > pulls in a change to the gnulib submodule. > >> For a quick reply, I ran 'make check' against the previous code base >> before your latest commit. >> >> sudo make check TESTS=cp/sparse-fiemap VERBOSE=yes > ... >> + filefrag -v j2 > ... >> + awk '/^ *[0-9]/ {printf "%d %d ", $2 ,NF < 5 ? $NF : $5 } END {print ""}' >> @a and @b have different lengths, even after adjustment >> + fail=1 > > > From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 26 09:10:43 2011 Received: (at submit) by debbugs.gnu.org; 26 Jan 2011 14:10:43 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Pi654-0006QW-Sm for submit@debbugs.gnu.org; Wed, 26 Jan 2011 09:10:43 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Pi653-0006QL-Pa for submit@debbugs.gnu.org; Wed, 26 Jan 2011 09:10:42 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Pi6Ct-0002wM-An for submit@debbugs.gnu.org; Wed, 26 Jan 2011 09:18:49 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,HTML_MESSAGE, RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:43399) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Pi6Ct-0002wF-0v for submit@debbugs.gnu.org; Wed, 26 Jan 2011 09:18:47 -0500 Received: from [140.186.70.92] (port=60765 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Pi6Cr-0007vr-5e for bug-coreutils@gnu.org; Wed, 26 Jan 2011 09:18:46 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Pi6Co-0002uc-B9 for bug-coreutils@gnu.org; Wed, 26 Jan 2011 09:18:44 -0500 Received: from rcsinet10.oracle.com ([148.87.113.121]:40653) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Pi6Cn-0002tV-T3 for bug-coreutils@gnu.org; Wed, 26 Jan 2011 09:18:42 -0500 Received: from rcsinet13.oracle.com (rcsinet13.oracle.com [148.87.113.125]) by rcsinet10.oracle.com (Switch-3.4.2/Switch-3.4.2) with ESMTP id p0QEINw2021441 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Wed, 26 Jan 2011 14:18:26 GMT Received: from acsmt355.oracle.com (acsmt355.oracle.com [141.146.40.155]) by rcsinet13.oracle.com (Switch-3.4.2/Switch-3.4.1) with ESMTP id p0QEII1k028595; Wed, 26 Jan 2011 14:18:20 GMT Received: from abhmt016.oracle.com by acsmt355.oracle.com with ESMTP id 994094891296051411; Wed, 26 Jan 2011 06:16:51 -0800 Received: from [192.168.1.101] (/221.223.119.120) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 26 Jan 2011 06:16:50 -0800 Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy Mime-Version: 1.0 (Apple Message framework v1081) Content-Type: multipart/alternative; boundary=Apple-Mail-1--83383736 From: Jeff liu In-Reply-To: <4D3F9BEE.6060701@oracle.com> Date: Wed, 26 Jan 2011 22:16:36 +0800 Message-Id: References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87ipxhdr3v.fsf@meyering.net> <4D3E7C6A.6030708@oracle.com> <87sjwhv2fj.fsf@meyering.net> <4D3EB82A.60203@oracle.com> <87pqrksy0x.fsf@meyering.net> <4D3F9BEE.6060701@oracle.com> To: "jeff.liu" X-Mailer: Apple Mail (2.1081) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Spam-Score: -6.4 (------) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul@debbugs.gnu.org, Eggert , bug-coreutils@gnu.org, Jim Meyering , =?GB2312?Q?P=A8=A2draig_Brady?= , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.4 (------) --Apple-Mail-1--83383736 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=GB2312 =D4=DA 2011-1-26=A3=AC=C9=CF=CE=E711:58=A3=AC jeff.liu =D0=B4=B5=C0=A3=BA > Jim Meyering wrote: >> jeff.liu wrote: >>> Jim Meyering wrote: >>>> jeff.liu wrote: >>>>> AFAICS, the tests passed on all filesystems except ext4, >>>> Really? >>>> The vast majority of my testing is with ext4 on Fedora 14, and I = have seen >>>> no failure -- otherwise I would have mentioned that as a known = problem. >>> I have mentioned this issue at: >>> http://osdir.com/ml/bug-coreutils-gnu/2010-09/msg00092.html >>>=20 >>> "make test against cp/sparse-fiemap failed at the extent compare >>> stage, but the file content is >>> identical to each other by comparing those two files "j1/j2" = manually. >>> Is it make sense if we verify them through diff(1) since the testing >>> file is in small size?" >>=20 >> No. The whole point of the test is to verify that the extents have >> been preserved in the copy. Diff doesn't know about extents. >>=20 >>>> What type of system/kernel are you using? >>> 2.6.33-RC3 && 2.6.36 >>>> Was your ext4 partition created long ago? With what options? >>> fiemap copy works well if run `cp' against physical ext4 partition. >>>> Did "make check" fail? If so, please provide details. >>> Yeah, I will show the detail of 'make check' at below. >>=20 >> What version of filefrag are you using? > Mine comes from E2fsprogs version 1.41.12 shipped with ubuntu8.0.4. > I updated the filefrag(8) to the upstream one but still no luck. :( > the kernel I have tried are 2.6.28/2.6.33-RC3/2.6.36. >=20 Hi Jim, The issue I observed before is weird. Now make check passed against the following combination: 1. Refresh installed host in Ubuntu10.0.4,=20 filefrag comes from E2fsprogs 1.41.11 && Kernel: 2.6.32-16 2. filefrag in e2fsprogs-1.4.12 && kernel-2.6.36. $ sudo make check TESTS=3Dcp/sparse-fiemap VERBOSE=3Dyes [sudo] password for jeff:=20 make check-TESTS make[1]: Entering directory `/home/jeff/opensoure_dev/coreutils/tests' make[2]: Entering directory `/home/jeff/opensoure_dev/coreutils/tests' PASS: cp/sparse-fiemap =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 1 test passed =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D make[2]: Leaving directory `/home/jeff/opensoure_dev/coreutils/tests' make[1]: Leaving directory `/home/jeff/opensoure_dev/coreutils/tests' Thanks, -Jeff >> Mine comes from e2fsprogs-1.41.12-6.fc14.x86_64 >>=20 >>> btw, I just checked out the new branch and tried to compile it but = ran >>> into an error: >>> date.c:30:28: error: parse-datetime.h: No such file or directory >>> date.c: In function 'batch_convert': >>> date.c:284: warning: implicit declaration of function = 'parse_datetime' >>>=20 >>> I guess 'parse-datetime.h' is shipped with gnulib? For now, I can = not >>> pull the latest gnulib code >>> since the remote host does not response. >>=20 >> Did you run ./bootstrap ? > sure, yesterday it failed due to my proxy issue. >=20 >=20 > Thanks, > -Jeff >> That is a requirement whenever the coreutils >> pulls in a change to the gnulib submodule. >>=20 >>> For a quick reply, I ran 'make check' against the previous code base >>> before your latest commit. >>>=20 >>> sudo make check TESTS=3Dcp/sparse-fiemap VERBOSE=3Dyes >> ... >>> + filefrag -v j2 >> ... >>> + awk '/^ *[0-9]/ {printf "%d %d ", $2 ,NF < 5 ? $NF : $5 } END = {print ""}' >>> @a and @b have different lengths, even after adjustment >>> + fail=3D1 >>=20 >>=20 >>=20 >=20 >=20 >=20 >=20 --Apple-Mail-1--83383736 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=GB2312
Jim = Meyering wrote:
jeff.liu = wrote:
Jim Meyering = wrote:
jeff.liu = wrote:
AFAICS, the tests passed on all = filesystems except = ext4,
Really?
The = vast majority of my testing is with ext4 on Fedora 14, and I have = seen
no = failure -- otherwise I would have mentioned that as a known = problem.
I have mentioned this issue = at:
http:= //osdir.com/ml/bug-coreutils-gnu/2010-09/msg00092.html

"make test against = cp/sparse-fiemap failed at the extent = compare
stage, but the file content = is
identical to each other by comparing those two files = "j1/j2" manually.
Is it make sense if we verify = them through diff(1) since the = testing
file is in small = size?"

No.  The = whole point of the test is to verify that the extents = have
been preserved in the = copy.  Diff doesn't know about extents.

What type of system/kernel are = you using?
2.6.33-RC3 && = 2.6.36
Was your ext4 partition created = long ago?  With what = options?
fiemap copy works well if run = `cp' against physical ext4 = partition.
Did = "make check" fail?  If so, please provide = details.
Yeah, I will show the detail of = 'make check' at below.

What version of = filefrag are you using?
Mine comes from E2fsprogs = version 1.41.12 shipped with ubuntu8.0.4.
I updated the filefrag(8) = to the upstream one but still no luck. :(
the kernel I have tried are = 2.6.28/2.6.33-RC3/2.6.36.

Hi = Jim,

The issue I observed before is = weird.

Now  make check passed against the = following combination:
1. Refresh installed host in = Ubuntu10.0.4, 
filefrag comes from E2fsprogs 1.41.11 = && Kernel: 2.6.32-16
2. filefrag in = e2fsprogs-1.4.12 && kernel-2.6.36.

$ sudo make check = TESTS=3Dcp/sparse-fiemap VERBOSE=3Dyes
[sudo] password for = jeff: 
make  check-TESTS
make[1]: Entering directory = `/home/jeff/opensoure_dev/coreutils/tests'
make[2]: Entering = directory `/home/jeff/opensoure_dev/coreutils/tests'
PASS: = cp/sparse-fiemap
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
1 test = passed
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
make[2]: Leaving = directory `/home/jeff/opensoure_dev/coreutils/tests'
make[1]: = Leaving directory = `/home/jeff/opensoure_dev/coreutils/tests'

<= div>
Thanks,
-Jeff


Mine comes from = e2fsprogs-1.41.12-6.fc14.x86_64

btw, I just checked out the new branch and tried to = compile it but ran
into an = error:
date.c:30:28: error: parse-datetime.h: No such file or = directory
date.c: In function = 'batch_convert':
date.c:284: warning: implicit = declaration of function = 'parse_datetime'

I guess 'parse-datetime.h' is = shipped with gnulib? For now, I can = not
pull the latest gnulib = code
since the remote host does not = response.

Did you run = ./bootstrap ?
sure, yesterday it failed due to my proxy = issue.


Thanks,
-Jeff
That is = a requirement whenever the coreutils
pulls in a change to the gnulib = submodule.

For a quick reply, I ran 'make check' against the previous = code base
before your latest = commit.

sudo make check = TESTS=3Dcp/sparse-fiemap = VERBOSE=3Dyes
...
+ filefrag -v j2
...
+ awk '/^ *[0-9]/ {printf "%d %d ", $2 ,NF < 5 ? $NF : = $5 } END {print ""}'
@a and @b have different = lengths, even after adjustment
+ = fail=3D1








= --Apple-Mail-1--83383736-- From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 28 13:09:50 2011 Received: (at submit) by debbugs.gnu.org; 28 Jan 2011 18:09:51 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Pisla-0004OJ-K4 for submit@debbugs.gnu.org; Fri, 28 Jan 2011 13:09:50 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PislY-0004O6-3T for submit@debbugs.gnu.org; Fri, 28 Jan 2011 13:09:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PistL-0005p4-V5 for submit@debbugs.gnu.org; Fri, 28 Jan 2011 13:18:01 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:50021) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PistL-0005oi-PX for submit@debbugs.gnu.org; Fri, 28 Jan 2011 13:17:51 -0500 Received: from [140.186.70.92] (port=47752 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PistB-0007Km-OL for bug-coreutils@gnu.org; Fri, 28 Jan 2011 13:17:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Pishn-0003Em-Rj for bug-coreutils@gnu.org; Fri, 28 Jan 2011 13:05:56 -0500 Received: from mx.meyering.net ([82.230.74.64]:35210) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Pishn-0003EP-Lq for bug-coreutils@gnu.org; Fri, 28 Jan 2011 13:05:55 -0500 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 86FB76008B; Fri, 28 Jan 2011 19:05:52 +0100 (CET) From: Jim Meyering To: Jeff liu Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: (Jeff liu's message of "Wed, 26 Jan 2011 22:16:36 +0800") References: <4C1898D0.3000204@draigBrady.com> <4C3D6008.90302@oracle.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87ipxhdr3v.fsf@meyering.net> <4D3E7C6A.6030708@oracle.com> <87sjwhv2fj.fsf@meyering.net> <4D3EB82A.60203@oracle.com> <87pqrksy0x.fsf@meyering.net> <4D3F9BEE.6060701@oracle.com> Date: Fri, 28 Jan 2011 19:05:52 +0100 Message-ID: <87vd18c39b.fsf@meyering.net> Lines: 15 MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 82.230.74.64 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Received-From: 199.232.76.165 X-Spam-Score: -5.8 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul@debbugs.gnu.org, Eggert , bug-coreutils@gnu.org, =?iso-8859-1?Q?P=E1draig?=, Brady , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.8 (-----) Jeff liu wrote: > Now make check passed against the following combination: > 1. Refresh installed host in Ubuntu10.0.4, > filefrag comes from E2fsprogs 1.41.11 && Kernel: 2.6.32-16 > 2. filefrag in e2fsprogs-1.4.12 && kernel-2.6.36. [passes] Glad to here it passes for you, now. FYI, I have spent pretty much time on cp over the last couple days, factoring out the hole-inducing code and making extent_copy use it. Part of the motivation was to fix cp --sparse=always, which was broken on the branch. It would not induce holes when going through extent_copy. I've added a couple more tests and will post the series as soon I've cleaned things up a little more. From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 29 04:39:57 2011 Received: (at submit) by debbugs.gnu.org; 29 Jan 2011 09:39:57 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Pj7Hg-0005nV-3z for submit@debbugs.gnu.org; Sat, 29 Jan 2011 04:39:57 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Pj7Hc-0005nJ-8U for submit@debbugs.gnu.org; Sat, 29 Jan 2011 04:39:54 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Pj7PY-0005zD-Cr for submit@debbugs.gnu.org; Sat, 29 Jan 2011 04:48:07 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:56143) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Pj7PW-0005yx-Ul for submit@debbugs.gnu.org; Sat, 29 Jan 2011 04:48:04 -0500 Received: from [140.186.70.92] (port=46263 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Pj7PT-0006lA-OU for bug-coreutils@gnu.org; Sat, 29 Jan 2011 04:48:02 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Pj7PQ-0005xx-17 for bug-coreutils@gnu.org; Sat, 29 Jan 2011 04:47:59 -0500 Received: from mx.meyering.net ([82.230.74.64]:58290) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Pj7PP-0005xW-AT for bug-coreutils@gnu.org; Sat, 29 Jan 2011 04:47:55 -0500 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id A4C7060156; Sat, 29 Jan 2011 10:47:53 +0100 (CET) From: Jim Meyering To: Jeff liu Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <87vd18c39b.fsf@meyering.net> (Jim Meyering's message of "Fri, 28 Jan 2011 19:05:52 +0100") References: <4C1898D0.3000204@draigBrady.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87ipxhdr3v.fsf@meyering.net> <4D3E7C6A.6030708@oracle.com> <87sjwhv2fj.fsf@meyering.net> <4D3EB82A.60203@oracle.com> <87pqrksy0x.fsf@meyering.net> <4D3F9BEE.6060701@oracle.com> <87vd18c39b.fsf@meyering.net> Date: Sat, 29 Jan 2011 10:47:53 +0100 Message-ID: <87ipx89h2u.fsf@meyering.net> Lines: 1028 MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 82.230.74.64 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Received-From: 199.232.76.165 X-Spam-Score: -5.8 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul@debbugs.gnu.org, Eggert , bug-coreutils@gnu.org, =?us-ascii?Q?=3D=3FUTF-8=3FQ=3FP=3DC3=3DA1draig?=@debbugs.gnu.org, ?= Brady , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.8 (-----) Jim Meyering wrote: > Jeff liu wrote: >> Now make check passed against the following combination: >> 1. Refresh installed host in Ubuntu10.0.4, >> filefrag comes from E2fsprogs 1.41.11 && Kernel: 2.6.32-16 >> 2. filefrag in e2fsprogs-1.4.12 && kernel-2.6.36. > [passes] > > Glad to here it passes for you, now. > FYI, I have spent pretty much time on cp over the last > couple days, factoring out the hole-inducing code and > making extent_copy use it. Part of the motivation was > to fix cp --sparse=always, which was broken on the branch. > It would not induce holes when going through extent_copy. > I've added a couple more tests and will post the series as > soon I've cleaned things up a little more. Here are 9 more patches, just pushed to the fiemap-copy-2 branch: http://git.savannah.gnu.org/cgit/coreutils.git/log/?h=fiemap-copy-2 The first and last add tests, and the others consolidate, clean up, and fix a few bugs. 1/9 tests: ensure that FIEMAP-enabled cp copies a sparse file efficiently Ensure that copying a sparse 1TiB file completes in less than 3 seconds That can only succeed with FIEMAP (or --reflink=, which is off by default) 2/9 fiemap copy: rename some locals The _logical suffix was not useful. Change it to _start 3/9 fiemap copy: simplify post-loop logic; improve comments 4/9 fiemap copy: avoid a performance hit due to very small buffer I didn't measure this, but once you see it, it's an obvious bug. Using an arbitrarily small buffer size is bound to cause trouble. 5/9 fiemap copy: avoid leak-on-error Failing from within the loop, we have to free the extent buffer. 6/9 copy: factor sparse-copying code into its own function, because we're going to have to use it from within extent_copy, too. I realized that cp --sparse=always could no longer create holes in the destination. Factoring this out is the first step. 7/9 copy: remove obsolete comment unrelated to the rest, but hard to pull out since it's in moved code 8/9 copy: make extent_copy use sparse_copy, rather than its own code Now that sparse_copy is separate, and used by copy_reg, adapt it so that it can also be used by extent_copy. 9/9 tests: cp/fiemap: exercise previously-failing parts This is a hole-inducing test that would have failed with previous fiemap-based copying code. I may change or remove the sparse_copy_finalize function, which just calls ftruncate, especially now that it's used from only one place (initially I was using it from each sparse_copy caller, but that didn't work out), and don't particularly like the added lseek call that is performed for each file copied, but keeping track of total written/offset byte counts and inflicting the need to do that on both callers seems like too much added code/complexity to justify avoiding that single lseek call. >From 8e4f0efd3ad17f1dd7a561369da22dfaf43ab3e8 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Fri, 28 Jan 2011 22:31:23 +0100 Subject: [PATCH 1/9] tests: ensure that FIEMAP-enabled cp copies a sparse file efficiently * tests/cp/fiemap-perf: New file. * tests/Makefile.am (TESTS): Add it. --- tests/Makefile.am | 1 + tests/cp/fiemap-perf | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 0 deletions(-) create mode 100755 tests/cp/fiemap-perf diff --git a/tests/Makefile.am b/tests/Makefile.am index 847f181..7855ac5 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -320,6 +320,7 @@ TESTS = \ cp/dir-vs-file \ cp/existing-perm-race \ cp/fail-perm \ + cp/fiemap-perf \ cp/file-perm-race \ cp/into-self \ cp/link \ diff --git a/tests/cp/fiemap-perf b/tests/cp/fiemap-perf new file mode 100755 index 0000000..429e59b --- /dev/null +++ b/tests/cp/fiemap-perf @@ -0,0 +1,32 @@ +#!/bin/sh +# ensure that a sparse file is copied efficiently, by default + +# Copyright (C) 2011 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +. "${srcdir=.}/init.sh"; path_prepend_ ../src +print_ver_ cp + +# Require a fiemap-enabled FS. +df -T -t btrfs -t xfs -t ext4 -t ocfs2 . \ + || skip_ "this file system lacks FIEMAP support" + +# Create a large-but-sparse file. +timeout 1 dd bs=1 seek=1T of=f < /dev/null || framework_failure_ + +# Nothing can read (much less write) that many bytes in so little time. +timeout 3 cp f f2 || framework_failure_ + +Exit $fail -- 1.7.3.5.44.g960a >From dd380c3d672f78adb4cb907e8658db6b3962a281 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Thu, 27 Jan 2011 18:28:25 +0100 Subject: [PATCH 2/9] fiemap copy: rename some locals (extent_copy): Rename locals: s/*ext_logical/*ext_start/ --- src/copy.c | 22 +++++++++++----------- 1 files changed, 11 insertions(+), 11 deletions(-) diff --git a/src/copy.c b/src/copy.c index c9cc2f7..e164ab7 100644 --- a/src/copy.c +++ b/src/copy.c @@ -200,7 +200,7 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, bool *require_normal_copy) { struct extent_scan scan; - off_t last_ext_logical = 0; + off_t last_ext_start = 0; uint64_t last_ext_len = 0; uint64_t last_read_size = 0; @@ -228,10 +228,10 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, unsigned int i; for (i = 0; i < scan.ei_count; i++) { - off_t ext_logical = scan.ext_info[i].ext_logical; + off_t ext_start = scan.ext_info[i].ext_logical; uint64_t ext_len = scan.ext_info[i].ext_length; - if (lseek (src_fd, ext_logical, SEEK_SET) < 0) + if (lseek (src_fd, ext_start, SEEK_SET) < 0) { error (0, errno, _("cannot lseek %s"), quote (src_name)); return false; @@ -239,7 +239,7 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, if (make_holes) { - if (lseek (dest_fd, ext_logical, SEEK_SET) < 0) + if (lseek (dest_fd, ext_start, SEEK_SET) < 0) { error (0, errno, _("cannot lseek %s"), quote (dst_name)); return false; @@ -249,10 +249,10 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, { /* We're not inducing holes; write zeros to the destination file if there is a hole between the last and current extent. */ - if (last_ext_logical + last_ext_len < ext_logical) + if (last_ext_start + last_ext_len < ext_start) { - uint64_t hole_size = (ext_logical - - last_ext_logical + uint64_t hole_size = (ext_start + - last_ext_start - last_ext_len); if (! write_zeros (dest_fd, hole_size)) { @@ -262,7 +262,7 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, } } - last_ext_logical = ext_logical; + last_ext_start = ext_start; last_ext_len = ext_len; last_read_size = 0; @@ -313,7 +313,7 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, file. */ if (make_holes) { - if (last_ext_logical + last_read_size < src_total_size) + if (last_ext_start + last_read_size < src_total_size) { if (ftruncate (dest_fd, src_total_size) < 0) { @@ -324,9 +324,9 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, } else { - if (last_ext_logical + last_ext_len < src_total_size) + if (last_ext_start + last_ext_len < src_total_size) { - uint64_t holes_len = src_total_size - last_ext_logical - last_ext_len; + uint64_t holes_len = src_total_size - last_ext_start - last_ext_len; if (0 < holes_len) { if (! write_zeros (dest_fd, holes_len)) -- 1.7.3.5.44.g960a >From d1067e37b0e4b945ab901e98d6eedb249fa2a42c Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Thu, 27 Jan 2011 19:00:48 +0100 Subject: [PATCH 3/9] fiemap copy: simplify post-loop logic; improve comments * src/copy.c (extent_copy): Avoid duplication in post-loop extend-to-desired-length code. --- src/copy.c | 44 +++++++++++++++----------------------------- 1 files changed, 15 insertions(+), 29 deletions(-) diff --git a/src/copy.c b/src/copy.c index e164ab7..ab18a76 100644 --- a/src/copy.c +++ b/src/copy.c @@ -268,8 +268,8 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, while (ext_len) { - /* Avoid reading into the holes if the left extent - length is shorter than the buffer size. */ + /* Don't read from a following hole if EXT_LEN + is smaller than the buffer size. */ buf_size = MIN (ext_len, buf_size); ssize_t n_read = read (src_fd, buf, buf_size); @@ -285,7 +285,7 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, if (n_read == 0) { - /* Record number of bytes read from the previous extent. */ + /* Record number of bytes read from this extent-at-EOF. */ last_read_size = last_ext_len - ext_len; break; } @@ -306,33 +306,19 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, } while (! scan.hit_final_extent); - /* If a file ends up with holes, the sum of the last extent logical offset - and the read-returned size or the last extent length will be shorter than - the actual size of the file. Use ftruncate to extend the length of the - destination file if make_holes, or write zeros up to the actual size of the - file. */ - if (make_holes) - { - if (last_ext_start + last_read_size < src_total_size) - { - if (ftruncate (dest_fd, src_total_size) < 0) - { - error (0, errno, _("failed to extend %s"), quote (dst_name)); - return false; - } - } - } - else - { - if (last_ext_start + last_ext_len < src_total_size) - { - uint64_t holes_len = src_total_size - last_ext_start - last_ext_len; - if (0 < holes_len) - { - if (! write_zeros (dest_fd, holes_len)) - return false; - } - } + /* When the source file ends with a hole, the sum of the last extent start + offset and (the read-returned size or the last extent length) is smaller + than the actual size of the file. In that case, extend the destination + file to the required length. When MAKE_HOLES is set, use ftruncate; + otherwise, use write_zeros. */ + uint64_t eof_hole_len = (src_total_size - last_ext_start + - (last_read_size ? last_read_size : last_ext_len)); + if (eof_hole_len && (make_holes + ? ftruncate (dest_fd, src_total_size) + : ! write_zeros (dest_fd, eof_hole_len))) + { + error (0, errno, _("failed to extend %s"), quote (dst_name)); + return false; } return true; -- 1.7.3.5.44.g960a >From 33f4a4a549afb3de94e546091c91586a1ece67ba Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Thu, 27 Jan 2011 17:30:08 +0100 Subject: [PATCH 4/9] fiemap copy: avoid a performance hit due to very small buffer * src/copy.c (extent_copy): Don't let what should have been a temporary reduction of buf_size (to handle a short ext_len) become permanent and thus impact the performance of all further iterations. --- src/copy.c | 5 ++--- 1 files changed, 2 insertions(+), 3 deletions(-) diff --git a/src/copy.c b/src/copy.c index ab18a76..9a3a8f7 100644 --- a/src/copy.c +++ b/src/copy.c @@ -270,9 +270,8 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, { /* Don't read from a following hole if EXT_LEN is smaller than the buffer size. */ - buf_size = MIN (ext_len, buf_size); - - ssize_t n_read = read (src_fd, buf, buf_size); + size_t b_size = MIN (ext_len, buf_size); + ssize_t n_read = read (src_fd, buf, b_size); if (n_read < 0) { #ifdef EINTR -- 1.7.3.5.44.g960a >From 47c8476ec9629239c82caf50b1c68b7bc58ba2d6 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Thu, 27 Jan 2011 17:49:04 +0100 Subject: [PATCH 5/9] fiemap copy: avoid leak-on-error * src/copy.c (extent_copy): Don't leak an extent_scan buffer on failed lseek, read, or write. --- src/copy.c | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/src/copy.c b/src/copy.c index 9a3a8f7..208e463 100644 --- a/src/copy.c +++ b/src/copy.c @@ -234,6 +234,8 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, if (lseek (src_fd, ext_start, SEEK_SET) < 0) { error (0, errno, _("cannot lseek %s"), quote (src_name)); + fail: + extent_scan_free (&scan); return false; } @@ -242,7 +244,7 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, if (lseek (dest_fd, ext_start, SEEK_SET) < 0) { error (0, errno, _("cannot lseek %s"), quote (dst_name)); - return false; + goto fail; } } else @@ -257,7 +259,7 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, if (! write_zeros (dest_fd, hole_size)) { error (0, errno, _("%s: write failed"), quote (dst_name)); - return false; + goto fail; } } } @@ -279,7 +281,7 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, continue; #endif error (0, errno, _("reading %s"), quote (src_name)); - return false; + goto fail; } if (n_read == 0) @@ -292,7 +294,7 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, if (full_write (dest_fd, buf, n_read) != n_read) { error (0, errno, _("writing %s"), quote (dst_name)); - return false; + goto fail; } ext_len -= n_read; -- 1.7.3.5.44.g960a >From c0b7bc3864c06ea12c2740056e28623449fb63a7 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Thu, 27 Jan 2011 20:57:17 +0100 Subject: [PATCH 6/9] copy: factor sparse-copying code into its own function, because we're going to have to use it from within extent_copy, too. * src/copy.c (sparse_copy): New function, factored out of... (copy_reg): ...here. Remove now-unused locals. --- src/copy.c | 212 ++++++++++++++++++++++++++++++++---------------------------- 1 files changed, 114 insertions(+), 98 deletions(-) diff --git a/src/copy.c b/src/copy.c index 208e463..cc8f68f 100644 --- a/src/copy.c +++ b/src/copy.c @@ -134,6 +134,116 @@ utimens_symlink (char const *file, struct timespec const *timespec) return err; } +/* Copy the regular file open on SRC_FD/SRC_NAME to DST_FD/DST_NAME, + honoring the MAKE_HOLES setting and using the BUF_SIZE-byte buffer + BUF for temporary storage. Return true upon successful completion; + print a diagnostic and return false upon error. */ +static bool +sparse_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, + bool make_holes, + char const *src_name, char const *dst_name) +{ + typedef uintptr_t word; + off_t n_read_total = 0; + bool last_write_made_hole = false; + + while (true) + { + word *wp = NULL; + + ssize_t n_read = read (src_fd, buf, buf_size); + if (n_read < 0) + { +#ifdef EINTR + if (errno == EINTR) + continue; +#endif + error (0, errno, _("reading %s"), quote (src_name)); + return false; + } + if (n_read == 0) + break; + + n_read_total += n_read; + + if (make_holes) + { + char *cp; + + /* Sentinel to stop loop. */ + buf[n_read] = '\1'; +#ifdef lint + /* Usually, buf[n_read] is not the byte just before a "word" + (aka uintptr_t) boundary. In that case, the word-oriented + test below (*wp++ == 0) would read some uninitialized bytes + after the sentinel. To avoid false-positive reports about + this condition (e.g., from a tool like valgrind), set the + remaining bytes -- to any value. */ + memset (buf + n_read + 1, 0, sizeof (word) - 1); +#endif + + /* Find first nonzero *word*, or the word with the sentinel. */ + + wp = (word *) buf; + while (*wp++ == 0) + continue; + + /* Find the first nonzero *byte*, or the sentinel. */ + + cp = (char *) (wp - 1); + while (*cp++ == 0) + continue; + + if (cp <= buf + n_read) + /* Clear to indicate that a normal write is needed. */ + wp = NULL; + else + { + /* We found the sentinel, so the whole input block was zero. + Make a hole. */ + if (lseek (dest_fd, n_read, SEEK_CUR) < 0) + { + error (0, errno, _("cannot lseek %s"), quote (dst_name)); + return false; + } + last_write_made_hole = true; + } + } + + if (!wp) + { + size_t n = n_read; + if (full_write (dest_fd, buf, n) != n) + { + error (0, errno, _("writing %s"), quote (dst_name)); + return false; + } + last_write_made_hole = false; + + /* It is tempting to return early here upon a short read from a + regular file. That would save the final read syscall for each + file. Unfortunately that doesn't work for certain files in + /proc with linux kernels from at least 2.6.9 .. 2.6.29. */ + } + } + + /* If the file ends with a `hole', we need to do something to record + the length of the file. On modern systems, calling ftruncate does + the job. On systems without native ftruncate support, we have to + write a byte at the ending position. Otherwise the kernel would + truncate the file at the end of the last write operation. */ + if (last_write_made_hole) + { + if (ftruncate (dest_fd, n_read_total) < 0) + { + error (0, errno, _("truncating %s"), quote (dst_name)); + return false; + } + } + + return true; +} + /* Perform the O(1) btrfs clone operation, if possible. Upon success, return 0. Otherwise, return -1 and set errno. */ static inline int @@ -824,7 +934,6 @@ copy_reg (char const *src_name, char const *dst_name, if (data_copy_required) { typedef uintptr_t word; - off_t n_read_total = 0; /* Choose a suitable buffer size; it may be adjusted later. */ size_t buf_alignment = lcm (getpagesize (), sizeof (word)); @@ -832,7 +941,6 @@ copy_reg (char const *src_name, char const *dst_name, size_t buf_size = io_blksize (sb); /* Deal with sparse files. */ - bool last_write_made_hole = false; bool make_holes = false; if (S_ISREG (sb.st_mode)) @@ -897,103 +1005,11 @@ copy_reg (char const *src_name, char const *dst_name, goto close_src_and_dst_desc; } - while (true) + if ( ! sparse_copy (source_desc, dest_desc, buf, buf_size, + make_holes, src_name, dst_name)) { - word *wp = NULL; - - ssize_t n_read = read (source_desc, buf, buf_size); - if (n_read < 0) - { -#ifdef EINTR - if (errno == EINTR) - continue; -#endif - error (0, errno, _("reading %s"), quote (src_name)); - return_val = false; - goto close_src_and_dst_desc; - } - if (n_read == 0) - break; - - n_read_total += n_read; - - if (make_holes) - { - char *cp; - - /* Sentinel to stop loop. */ - buf[n_read] = '\1'; -#ifdef lint - /* Usually, buf[n_read] is not the byte just before a "word" - (aka uintptr_t) boundary. In that case, the word-oriented - test below (*wp++ == 0) would read some uninitialized bytes - after the sentinel. To avoid false-positive reports about - this condition (e.g., from a tool like valgrind), set the - remaining bytes -- to any value. */ - memset (buf + n_read + 1, 0, sizeof (word) - 1); -#endif - - /* Find first nonzero *word*, or the word with the sentinel. */ - - wp = (word *) buf; - while (*wp++ == 0) - continue; - - /* Find the first nonzero *byte*, or the sentinel. */ - - cp = (char *) (wp - 1); - while (*cp++ == 0) - continue; - - if (cp <= buf + n_read) - /* Clear to indicate that a normal write is needed. */ - wp = NULL; - else - { - /* We found the sentinel, so the whole input block was zero. - Make a hole. */ - if (lseek (dest_desc, n_read, SEEK_CUR) < 0) - { - error (0, errno, _("cannot lseek %s"), quote (dst_name)); - return_val = false; - goto close_src_and_dst_desc; - } - last_write_made_hole = true; - } - } - - if (!wp) - { - size_t n = n_read; - if (full_write (dest_desc, buf, n) != n) - { - error (0, errno, _("writing %s"), quote (dst_name)); - return_val = false; - goto close_src_and_dst_desc; - } - last_write_made_hole = false; - - /* It is tempting to return early here upon a short read from a - regular file. That would save the final read syscall for each - file. Unfortunately that doesn't work for certain files in - /proc with linux kernels from at least 2.6.9 .. 2.6.29. */ - } - } - - /* If the file ends with a `hole', we need to do something to record - the length of the file. On modern systems, calling ftruncate does - the job. On systems without native ftruncate support, we have to - write a byte at the ending position. Otherwise the kernel would - truncate the file at the end of the last write operation. */ - - if (last_write_made_hole) - { - if (ftruncate (dest_desc, n_read_total) < 0) - { - error (0, errno, _("truncating %s"), quote (dst_name)); - return_val = false; - goto close_src_and_dst_desc; - } + return_val = false; + goto close_src_and_dst_desc; } } -- 1.7.3.5.44.g960a >From 80038c3cba2dee9c6c41ab6a28a1233a538ee2ee Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Thu, 27 Jan 2011 21:01:07 +0100 Subject: [PATCH 7/9] copy: remove obsolete comment * src/copy.c (sparse_copy): Remove now-obsolete comment about how we used to work around lack of ftruncate. Combine nested if conditions into one. --- src/copy.c | 21 +++++++++------------ 1 files changed, 9 insertions(+), 12 deletions(-) diff --git a/src/copy.c b/src/copy.c index cc8f68f..4bfdce6 100644 --- a/src/copy.c +++ b/src/copy.c @@ -137,7 +137,10 @@ utimens_symlink (char const *file, struct timespec const *timespec) /* Copy the regular file open on SRC_FD/SRC_NAME to DST_FD/DST_NAME, honoring the MAKE_HOLES setting and using the BUF_SIZE-byte buffer BUF for temporary storage. Return true upon successful completion; - print a diagnostic and return false upon error. */ + print a diagnostic and return false upon error. + Note that for best results, BUF should be "well"-aligned. + BUF must have sizeof(uintptr_t)-1 bytes of additional space + beyond BUF[BUF_SIZE-1]. */ static bool sparse_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, bool make_holes, @@ -227,18 +230,12 @@ sparse_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, } } - /* If the file ends with a `hole', we need to do something to record - the length of the file. On modern systems, calling ftruncate does - the job. On systems without native ftruncate support, we have to - write a byte at the ending position. Otherwise the kernel would - truncate the file at the end of the last write operation. */ - if (last_write_made_hole) + /* If the file ends with a `hole', we need to do something to record the + length of the file. On modern systems, calling ftruncate does the job. */ + if (last_write_made_hole && ftruncate (dest_fd, n_read_total) < 0) { - if (ftruncate (dest_fd, n_read_total) < 0) - { - error (0, errno, _("truncating %s"), quote (dst_name)); - return false; - } + error (0, errno, _("truncating %s"), quote (dst_name)); + return false; } return true; -- 1.7.3.5.44.g960a >From f161ba3fcd5832d1344224ec41627cace5d73544 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Fri, 28 Jan 2011 21:19:50 +0100 Subject: [PATCH 8/9] copy: make extent_copy use sparse_copy, rather than its own code * src/copy.c (extent_copy): Before this change, extent_copy would fail to create holes, thus breaking --sparse=auto and --sparse=always. I.e., copying a large enough file of all zeros, cp --sparse=always should introduce a hole, but with extent_copy, it would not. --- src/copy.c | 109 +++++++++++++++++++++++++++--------------------------------- 1 files changed, 49 insertions(+), 60 deletions(-) diff --git a/src/copy.c b/src/copy.c index 4bfdce6..96bb35b 100644 --- a/src/copy.c +++ b/src/copy.c @@ -136,25 +136,28 @@ utimens_symlink (char const *file, struct timespec const *timespec) /* Copy the regular file open on SRC_FD/SRC_NAME to DST_FD/DST_NAME, honoring the MAKE_HOLES setting and using the BUF_SIZE-byte buffer - BUF for temporary storage. Return true upon successful completion; + BUF for temporary storage. Copy no more than MAX_N_READ bytes. + Return true upon successful completion; print a diagnostic and return false upon error. Note that for best results, BUF should be "well"-aligned. BUF must have sizeof(uintptr_t)-1 bytes of additional space - beyond BUF[BUF_SIZE-1]. */ + beyond BUF[BUF_SIZE-1]. + Set *LAST_WRITE_MADE_HOLE to true if the final operation on + DEST_FD introduced a hole. */ static bool sparse_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, bool make_holes, - char const *src_name, char const *dst_name) + char const *src_name, char const *dst_name, + uintmax_t max_n_read, bool *last_write_made_hole) { typedef uintptr_t word; - off_t n_read_total = 0; - bool last_write_made_hole = false; + *last_write_made_hole = false; - while (true) + while (max_n_read) { word *wp = NULL; - ssize_t n_read = read (src_fd, buf, buf_size); + ssize_t n_read = read (src_fd, buf, MIN (max_n_read, buf_size)); if (n_read < 0) { #ifdef EINTR @@ -166,8 +169,7 @@ sparse_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, } if (n_read == 0) break; - - n_read_total += n_read; + max_n_read -= n_read; if (make_holes) { @@ -209,7 +211,7 @@ sparse_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, error (0, errno, _("cannot lseek %s"), quote (dst_name)); return false; } - last_write_made_hole = true; + *last_write_made_hole = true; } } @@ -221,7 +223,7 @@ sparse_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, error (0, errno, _("writing %s"), quote (dst_name)); return false; } - last_write_made_hole = false; + *last_write_made_hole = false; /* It is tempting to return early here upon a short read from a regular file. That would save the final read syscall for each @@ -230,9 +232,16 @@ sparse_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, } } - /* If the file ends with a `hole', we need to do something to record the - length of the file. On modern systems, calling ftruncate does the job. */ - if (last_write_made_hole && ftruncate (dest_fd, n_read_total) < 0) + return true; +} + +/* If the file ends with a `hole' (i.e., if sparse_copy set wrote_hole_at_eof), + call this function to record the length of the output file. */ +static bool +sparse_copy_finalize (int dest_fd, char const *dst_name) +{ + off_t len = lseek (dest_fd, 0, SEEK_CUR); + if (0 <= len && ftruncate (dest_fd, len) < 0) { error (0, errno, _("truncating %s"), quote (dst_name)); return false; @@ -309,10 +318,10 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, struct extent_scan scan; off_t last_ext_start = 0; uint64_t last_ext_len = 0; - uint64_t last_read_size = 0; extent_scan_init (src_fd, &scan); + bool wrote_hole_at_eof = true; do { bool ok = extent_scan_read (&scan); @@ -356,8 +365,9 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, } else { - /* We're not inducing holes; write zeros to the destination file - if there is a hole between the last and current extent. */ + /* When not inducing holes and when there is a hole between + the end of the previous extent and the beginning of the + current one, write zeros to the destination file. */ if (last_ext_start + last_ext_len < ext_start) { uint64_t hole_size = (ext_start @@ -373,39 +383,11 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, last_ext_start = ext_start; last_ext_len = ext_len; - last_read_size = 0; - - while (ext_len) - { - /* Don't read from a following hole if EXT_LEN - is smaller than the buffer size. */ - size_t b_size = MIN (ext_len, buf_size); - ssize_t n_read = read (src_fd, buf, b_size); - if (n_read < 0) - { -#ifdef EINTR - if (errno == EINTR) - continue; -#endif - error (0, errno, _("reading %s"), quote (src_name)); - goto fail; - } - - if (n_read == 0) - { - /* Record number of bytes read from this extent-at-EOF. */ - last_read_size = last_ext_len - ext_len; - break; - } - - if (full_write (dest_fd, buf, n_read) != n_read) - { - error (0, errno, _("writing %s"), quote (dst_name)); - goto fail; - } - ext_len -= n_read; - } + if ( ! sparse_copy (src_fd, dest_fd, buf, buf_size, + make_holes, src_name, dst_name, ext_len, + &wrote_hole_at_eof)) + return false; } /* Release the space allocated to scan->ext_info. */ @@ -414,16 +396,19 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, } while (! scan.hit_final_extent); - /* When the source file ends with a hole, the sum of the last extent start - offset and (the read-returned size or the last extent length) is smaller - than the actual size of the file. In that case, extend the destination - file to the required length. When MAKE_HOLES is set, use ftruncate; - otherwise, use write_zeros. */ - uint64_t eof_hole_len = (src_total_size - last_ext_start - - (last_read_size ? last_read_size : last_ext_len)); - if (eof_hole_len && (make_holes - ? ftruncate (dest_fd, src_total_size) - : ! write_zeros (dest_fd, eof_hole_len))) + /* When the source file ends with a hole, we have to do a little more work, + since the above copied only up to and including the final extent. + In order to complete the copy, we may have to insert a hole or write + zeros in the destination corresponding to the source file's hole-at-EOF. + + In addition, if the final extent was a block of zeros at EOF and we've + just converted them to a hole in the destination, we must call ftruncate + here in order to record the proper length in the destination. */ + off_t dest_len = lseek (dest_fd, 0, SEEK_CUR); + if ((dest_len < src_total_size || wrote_hole_at_eof) + && (make_holes + ? ftruncate (dest_fd, src_total_size) + : ! write_zeros (dest_fd, src_total_size - dest_len))) { error (0, errno, _("failed to extend %s"), quote (dst_name)); return false; @@ -1002,8 +987,12 @@ copy_reg (char const *src_name, char const *dst_name, goto close_src_and_dst_desc; } + bool wrote_hole_at_eof; if ( ! sparse_copy (source_desc, dest_desc, buf, buf_size, - make_holes, src_name, dst_name)) + make_holes, src_name, dst_name, UINTMAX_MAX, + &wrote_hole_at_eof) + || (wrote_hole_at_eof && + ! sparse_copy_finalize (dest_desc, dst_name))) { return_val = false; goto close_src_and_dst_desc; -- 1.7.3.5.44.g960a >From 7f154dcfc5641c9616921d4c5ac5005bcb2507eb Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Thu, 27 Jan 2011 15:17:42 +0100 Subject: [PATCH 9/9] tests: cp/fiemap: exercise previously-failing parts * tests/cp/fiemap-2: New test. * tests/Makefile.am (TESTS): Add it. --- tests/Makefile.am | 1 + tests/cp/fiemap-2 | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+), 0 deletions(-) create mode 100755 tests/cp/fiemap-2 diff --git a/tests/Makefile.am b/tests/Makefile.am index 7855ac5..40d35ac 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -321,6 +321,7 @@ TESTS = \ cp/existing-perm-race \ cp/fail-perm \ cp/fiemap-perf \ + cp/fiemap-2 \ cp/file-perm-race \ cp/into-self \ cp/link \ diff --git a/tests/cp/fiemap-2 b/tests/cp/fiemap-2 new file mode 100755 index 0000000..d40505b --- /dev/null +++ b/tests/cp/fiemap-2 @@ -0,0 +1,54 @@ +#!/bin/sh +# Exercise a few more corners of the fiemap-copying code. + +# Copyright (C) 2011 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +. "${srcdir=.}/init.sh"; path_prepend_ ../src +print_ver_ cp + +# Require a fiemap-enabled FS. +df -T -t btrfs -t xfs -t ext4 -t ocfs2 . \ + || skip_ "this file system lacks FIEMAP support" + +# Exercise the code that handles a file ending in a hole. +printf x > k || framework_failure_ +dd bs=1k seek=128 of=k < /dev/null || framework_failure_ + +# The first time through the outer loop, the input file, K, ends with a hole. +# The second time through, we append a byte so that it does not. +for append in no yes; do + test $append = yes && printf y >> k + for i in always never; do + cp --sparse=$i k k2 || fail=1 + cmp k k2 || fail=1 + done +done + +# Ensure that --sparse=always can restore holes. +rm -f k +# Create a file starting with an "x", followed by 257K-1 0 bytes. +printf x > k || framework_failure_ +dd bs=1k seek=1 of=k count=255 < /dev/zero || framework_failure_ + +# cp should detect the all-zero blocks and convert some of them to holes. +# How many it detects/converts currently depends on io_blksize. +# Currently, on my F14/ext4 desktop, this K starts off with size 256KiB, +# (note that the K in the preceding test starts off with size 4KiB). +# cp from coreutils-8.9 with --sparse=always reduces the size to 32KiB. +cp --sparse=always k k2 || fail=1 +test $(stat -c %b k2) -lt $(stat -c %b k) || fail=1 + +Exit $fail -- 1.7.3.5.44.g960a From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 29 07:01:32 2011 Received: (at submit) by debbugs.gnu.org; 29 Jan 2011 12:01:32 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Pj9Uh-0000T6-N1 for submit@debbugs.gnu.org; Sat, 29 Jan 2011 07:01:32 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Pj9Ue-0000Sr-7u for submit@debbugs.gnu.org; Sat, 29 Jan 2011 07:01:29 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Pj9cb-0005sj-Ho for submit@debbugs.gnu.org; Sat, 29 Jan 2011 07:09:43 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:50340) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Pj9cb-0005rr-Ex for submit@debbugs.gnu.org; Sat, 29 Jan 2011 07:09:41 -0500 Received: from [140.186.70.92] (port=55109 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1Pj9SM-0006WS-UZ for bug-coreutils@gnu.org; Sat, 29 Jan 2011 06:59:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Pj9SL-00048R-1X for bug-coreutils@gnu.org; Sat, 29 Jan 2011 06:59:06 -0500 Received: from mail1.slb.deg.dub.stisp.net ([84.203.253.98]:11068) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1Pj9SK-000483-LV for bug-coreutils@gnu.org; Sat, 29 Jan 2011 06:59:05 -0500 Received: (qmail 66989 invoked from network); 29 Jan 2011 11:59:02 -0000 Received: from unknown (HELO ?192.168.2.25?) (84.203.137.218) by mail1.slb.deg.dub.stisp.net with SMTP; 29 Jan 2011 11:59:02 -0000 Message-ID: <4D440097.50204@draigBrady.com> Date: Sat, 29 Jan 2011 11:57:11 +0000 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.8) Gecko/20100227 Thunderbird/3.0.3 MIME-Version: 1.0 To: Jim Meyering Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy References: <4C1898D0.3000204@draigBrady.com> <4C3DF7B3.4050104@cs.ucla.edu> <4C3E4D88.3040305@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87ipxhdr3v.fsf@meyering.net> <4D3E7C6A.6030708@oracle.com> <87sjwhv2fj.fsf@meyering.net> <4D3EB82A.60203@oracle.com> <87pqrksy0x.fsf@meyering.net> <4D3F9BEE.6060701@oracle.com> <87vd18c39b.fsf@meyering.net> <87ipx89h2u.fsf@meyering.net> In-Reply-To: <87ipx89h2u.fsf@meyering.net> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: FreeBSD 4.6-4.9 X-Received-From: 84.203.253.98 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Received-From: 199.232.76.165 X-Spam-Score: -4.7 (----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul@debbugs.gnu.org, Eggert , bug-coreutils@gnu.org, Jeff liu , =?us-ascii?Q?=3D=3FUTF-8=3FQ=3FP=3DC3=3DA1draig?=@debbugs.gnu.org, Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.7 (----) On 29/01/11 09:47, Jim Meyering wrote: > Jim Meyering wrote: >> Jeff liu wrote: >>> Now make check passed against the following combination: >>> 1. Refresh installed host in Ubuntu10.0.4, >>> filefrag comes from E2fsprogs 1.41.11 && Kernel: 2.6.32-16 >>> 2. filefrag in e2fsprogs-1.4.12 && kernel-2.6.36. >> [passes] >> >> Glad to here it passes for you, now. >> FYI, I have spent pretty much time on cp over the last >> couple days, factoring out the hole-inducing code and >> making extent_copy use it. Part of the motivation was >> to fix cp --sparse=always, which was broken on the branch. >> It would not induce holes when going through extent_copy. >> I've added a couple more tests and will post the series as >> soon I've cleaned things up a little more. > > Here are 9 more patches, just pushed to the fiemap-copy-2 branch: > > http://git.savannah.gnu.org/cgit/coreutils.git/log/?h=fiemap-copy-2 > > The first and last add tests, and the others consolidate, > clean up, and fix a few bugs. > > 1/9 tests: ensure that FIEMAP-enabled cp copies a sparse file efficiently > Ensure that copying a sparse 1TiB file completes in less than 3 seconds > That can only succeed with FIEMAP (or --reflink=, which is off by default) > > 2/9 fiemap copy: rename some locals > The _logical suffix was not useful. Change it to _start > > 3/9 fiemap copy: simplify post-loop logic; improve comments > > 4/9 fiemap copy: avoid a performance hit due to very small buffer > I didn't measure this, but once you see it, it's an obvious bug. > Using an arbitrarily small buffer size is bound to cause trouble. > > 5/9 fiemap copy: avoid leak-on-error > Failing from within the loop, we have to free the extent buffer. > > 6/9 copy: factor sparse-copying code into its own function, because > we're going to have to use it from within extent_copy, too. > I realized that cp --sparse=always could no longer create holes > in the destination. Factoring this out is the first step. > > 7/9 copy: remove obsolete comment > unrelated to the rest, but hard to pull out since it's in moved code > > 8/9 copy: make extent_copy use sparse_copy, rather than its own code > Now that sparse_copy is separate, and used by copy_reg, adapt it > so that it can also be used by extent_copy. > > 9/9 tests: cp/fiemap: exercise previously-failing parts > This is a hole-inducing test that would have failed with previous > fiemap-based copying code. > > I may change or remove the sparse_copy_finalize function, which just calls > ftruncate, especially now that it's used from only one place (initially > I was using it from each sparse_copy caller, but that didn't work out), > and don't particularly like the added lseek call that is performed for > each file copied, but keeping track of total written/offset byte counts > and inflicting the need to do that on both callers seems like too much > added code/complexity to justify avoiding that single lseek call. > >>>From 8e4f0efd3ad17f1dd7a561369da22dfaf43ab3e8 Mon Sep 17 00:00:00 2001 > From: Jim Meyering > Date: Fri, 28 Jan 2011 22:31:23 +0100 > Subject: [PATCH 1/9] tests: ensure that FIEMAP-enabled cp copies a sparse file efficiently > > * tests/cp/fiemap-perf: New file. > * tests/Makefile.am (TESTS): Add it. > --- > tests/Makefile.am | 1 + > tests/cp/fiemap-perf | 32 ++++++++++++++++++++++++++++++++ > 2 files changed, 33 insertions(+), 0 deletions(-) > create mode 100755 tests/cp/fiemap-perf > > diff --git a/tests/Makefile.am b/tests/Makefile.am > index 847f181..7855ac5 100644 > --- a/tests/Makefile.am > +++ b/tests/Makefile.am > @@ -320,6 +320,7 @@ TESTS = \ > cp/dir-vs-file \ > cp/existing-perm-race \ > cp/fail-perm \ > + cp/fiemap-perf \ > cp/file-perm-race \ > cp/into-self \ > cp/link \ > diff --git a/tests/cp/fiemap-perf b/tests/cp/fiemap-perf > new file mode 100755 > index 0000000..429e59b > --- /dev/null > +++ b/tests/cp/fiemap-perf > @@ -0,0 +1,32 @@ > +#!/bin/sh > +# ensure that a sparse file is copied efficiently, by default > + > +# Copyright (C) 2011 Free Software Foundation, Inc. > + > +# This program is free software: you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation, either version 3 of the License, or > +# (at your option) any later version. > + > +# This program is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > + > +# You should have received a copy of the GNU General Public License > +# along with this program. If not, see . > + > +. "${srcdir=.}/init.sh"; path_prepend_ ../src > +print_ver_ cp > + > +# Require a fiemap-enabled FS. > +df -T -t btrfs -t xfs -t ext4 -t ocfs2 . \ > + || skip_ "this file system lacks FIEMAP support" > + > +# Create a large-but-sparse file. > +timeout 1 dd bs=1 seek=1T of=f < /dev/null || framework_failure_ > + > +# Nothing can read (much less write) that many bytes in so little time. > +timeout 3 cp f f2 || framework_failure_ I'm a bit worried with a 1s timeout. The following will only give false negatives over 100GB/s timeout 10 truncate -s1T f || framework_failure_ timeout 10 cp f f2 || framework_failure_ I wouldn't worry about filling file systems either, as we're already limiting to ext4 etc. >>>From 7f154dcfc5641c9616921d4c5ac5005bcb2507eb Mon Sep 17 00:00:00 20: Jim Meyering > Date: Thu, 27 Jan 2011 15:17:42 +0100 > Subject: [PATCH 9/9] tests: cp/fiemap: exercise previously-failing parts > +# Ensure that --sparse=always can restore holes. > +rm -f k > +# Create a file starting with an "x", followed by 257K-1 0 bytes. > +printf x > k || framework_failure_ > +dd bs=1k seek=1 of=k count=255 < /dev/zero || framework_failure_ S/257/256/ ? From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 29 11:01:59 2011 Received: (at submit) by debbugs.gnu.org; 29 Jan 2011 16:01:59 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PjDFP-0007Uf-Af for submit@debbugs.gnu.org; Sat, 29 Jan 2011 11:01:59 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PjDFN-0007UT-Ec for submit@debbugs.gnu.org; Sat, 29 Jan 2011 11:01:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PjDNL-0003ei-Ic for submit@debbugs.gnu.org; Sat, 29 Jan 2011 11:10:13 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:50141) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PjDNL-0003ee-GO for submit@debbugs.gnu.org; Sat, 29 Jan 2011 11:10:11 -0500 Received: from [140.186.70.92] (port=47513 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PjDNK-0004QR-4F for bug-coreutils@gnu.org; Sat, 29 Jan 2011 11:10:11 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PjDNJ-0003eK-62 for bug-coreutils@gnu.org; Sat, 29 Jan 2011 11:10:10 -0500 Received: from mx.meyering.net ([82.230.74.64]:51319) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PjDNI-0003eC-UY for bug-coreutils@gnu.org; Sat, 29 Jan 2011 11:10:09 -0500 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 83C336005A; Sat, 29 Jan 2011 17:10:07 +0100 (CET) From: Jim Meyering To: =?iso-8859-1?Q?P=E1draig?= Brady Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <4D440097.50204@draigBrady.com> (=?iso-8859-1?Q?=22P=E1draig?= Brady"'s message of "Sat, 29 Jan 2011 11:57:11 +0000") References: <4C1898D0.3000204@draigBrady.com> <4C3E68EC.4000408@oracle.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87ipxhdr3v.fsf@meyering.net> <4D3E7C6A.6030708@oracle.com> <87sjwhv2fj.fsf@meyering.net> <4D3EB82A.60203@oracle.com> <87pqrksy0x.fsf@meyering.net> <4D3F9BEE.6060701@oracle.com> <87vd18c39b.fsf@meyering.net> <87ipx89h2u.fsf@meyering.net> <4D440097.50204@draigBrady.com> Date: Sat, 29 Jan 2011 17:10:07 +0100 Message-ID: <878vy3ady8.fsf@meyering.net> Lines: 62 MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 82.230.74.64 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Received-From: 199.232.76.165 X-Spam-Score: -5.8 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Jeff liu , Paul Eggert , bug-coreutils@gnu.org, Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.8 (-----) P=E1draig Brady wrote: ... >> +# Require a fiemap-enabled FS. >> +df -T -t btrfs -t xfs -t ext4 -t ocfs2 . \ >> + || skip_ "this file system lacks FIEMAP support" >> + >> +# Create a large-but-sparse file. >> +timeout 1 dd bs=3D1 seek=3D1T of=3Df < /dev/null || framework_failure_ >> + >> +# Nothing can read (much less write) that many bytes in so little time. >> +timeout 3 cp f f2 || framework_failure_ > > I'm a bit worried with a 1s timeout. > The following will only give false negatives over 100GB/s > > timeout 10 truncate -s1T f || framework_failure_ > timeout 10 cp f f2 || framework_failure_ Thanks. Using truncate there is better, and 10 seconds is more consistent with many other timeout-using tests. While fixing that I noticed another problem: the latter command should be setting fail=3D1. diff --git a/tests/cp/fiemap-perf b/tests/cp/fiemap-perf index 429e59b..6c588cb 100755 --- a/tests/cp/fiemap-perf +++ b/tests/cp/fiemap-perf @@ -24,9 +24,9 @@ df -T -t btrfs -t xfs -t ext4 -t ocfs2 . \ || skip_ "this file system lacks FIEMAP support" # Create a large-but-sparse file. -timeout 1 dd bs=3D1 seek=3D1T of=3Df < /dev/null || framework_failure_ +timeout 10 truncate -s1T f || framework_failure_ # Nothing can read (much less write) that many bytes in so little time. -timeout 3 cp f f2 || framework_failure_ +timeout 10 cp f f2 || fail=3D1 Exit $fail > I wouldn't worry about filling file systems either, > as we're already limiting to ext4 etc. Nor would I. >> Subject: [PATCH 9/9] tests: cp/fiemap: exercise previously-failing parts >> +# Ensure that --sparse=3Dalways can restore holes. >> +rm -f k >> +# Create a file starting with an "x", followed by 257K-1 0 bytes. >> +printf x > k || framework_failure_ >> +dd bs=3D1k seek=3D1 of=3Dk count=3D255 < /dev/zero || framework_failure_ > > S/257/256/ ? Good catch. Initially I had count=3D256 and the comment was correct, but I changed to 255 and didn't adjust. I've corrected the comment as you suggest. I've amended those two commits, rebased, and pushed to a new branch: fiemap-copy-3. From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 30 15:24:05 2011 Received: (at submit) by debbugs.gnu.org; 30 Jan 2011 20:24:06 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Pjdob-0004v5-7A for submit@debbugs.gnu.org; Sun, 30 Jan 2011 15:24:05 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1PjdoZ-0004uT-7E for submit@debbugs.gnu.org; Sun, 30 Jan 2011 15:24:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Pjdwa-0002uN-Le for submit@debbugs.gnu.org; Sun, 30 Jan 2011 15:32:21 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([199.232.76.165]:38289) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Pjdwa-0002uI-JO for submit@debbugs.gnu.org; Sun, 30 Jan 2011 15:32:20 -0500 Received: from [140.186.70.92] (port=38973 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PjdwZ-0003xE-Je for bug-coreutils@gnu.org; Sun, 30 Jan 2011 15:32:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PjdwY-0002u0-Et for bug-coreutils@gnu.org; Sun, 30 Jan 2011 15:32:19 -0500 Received: from mx.meyering.net ([82.230.74.64]:48038) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PjdwY-0002tl-5F for bug-coreutils@gnu.org; Sun, 30 Jan 2011 15:32:18 -0500 Received: by rho.meyering.net (Acme Bit-Twister, from userid 1000) id 773E86017B; Sun, 30 Jan 2011 21:32:16 +0100 (CET) From: Jim Meyering To: =?iso-8859-1?Q?P=E1draig?= Brady Subject: Re: bug#6131: [PATCH]: fiemap support for efficient sparse file copy In-Reply-To: <878vy3ady8.fsf@meyering.net> (Jim Meyering's message of "Sat, 29 Jan 2011 17:10:07 +0100") References: <4C1898D0.3000204@draigBrady.com> <4C3F90FC.5060907@cs.ucla.edu> <4C407197.4050908@oracle.com> <4C408077.3020800@cs.ucla.edu> <20100716213317.GC30859@mail.oracle.com> <4C411152.3070005@oracle.com> <20100717061426.GA18756@mail.oracle.com> <4C414C5E.3060509@oracle.com> <4C45BFA5.5000809@oracle.com> <4C970AA9.5070304@oracle.com> <87ipxhdr3v.fsf@meyering.net> <4D3E7C6A.6030708@oracle.com> <87sjwhv2fj.fsf@meyering.net> <4D3EB82A.60203@oracle.com> <87pqrksy0x.fsf@meyering.net> <4D3F9BEE.6060701@oracle.com> <87vd18c39b.fsf@meyering.net> <87ipx89h2u.fsf@meyering.net> <4D440097.50204@draigBrady.com> <878vy3ady8.fsf@meyering.net> Date: Sun, 30 Jan 2011 21:32:16 +0100 Message-ID: <87sjwa2kvj.fsf@meyering.net> Lines: 42 MIME-Version: 1.0 Content-Type: text/plain X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 82.230.74.64 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 2) X-Received-From: 199.232.76.165 X-Spam-Score: -5.8 (-----) X-Debbugs-Envelope-To: submit Cc: Sunil Mushran , Paul Eggert , 6131-done@debbugs.gnu.org, bug-coreutils@gnu.org, Jeff liu , Chris Mason X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.8 (-----) Jim Meyering wrote: ... > I've amended those two commits, rebased, and pushed to > a new branch: fiemap-copy-3. As I mentioned, everything is now on "master". Odd how that works. Minutes after pushing everything, I wondered if there was a NEWS entry for this feature... No. So I wrote this: >From 5b11cd01790473b1964a0f91eca5205dfdcae773 Mon Sep 17 00:00:00 2001 From: Jim Meyering Date: Sun, 30 Jan 2011 21:27:12 +0100 Subject: [PATCH] doc: NEWS: mention cp's improvement * NEWS (New Features): cp now copies sparse files efficiently. --- NEWS | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-) diff --git a/NEWS b/NEWS index 6e7efe1..042bfed 100644 --- a/NEWS +++ b/NEWS @@ -21,6 +21,15 @@ GNU coreutils NEWS -*- outline -*- ** New features + cp now copies sparse files efficiently on file systems with FIEMAP + support (ext4, btrfs, xfs, ocfs2). Before, it had to read 2^20 bytes + when copying a 1MiB sparse file. Now, it copies bytes only for the + non-sparse sections of a file. Similarly, to induce a hole in the + output file, it had to detect a long sequence of zero bytes. Now, + it knows precisely where each hole in an input file is, and can + reproduce them efficiently in the output file. mv also benefits + when it resorts to copying, e.g., between file systems. + join now supports -o 'auto' which will automatically infer the output format from the first line in each file, to ensure the same number of fields are output for each line. -- 1.7.3.5.44.g960a From unknown Mon Jun 23 13:11:06 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Mon, 28 Feb 2011 12:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator