Package: coreutils;
Reported by: "jeff.liu" <jeff.liu <at> oracle.com>
Date: Fri, 7 May 2010 14:16:02 UTC
Severity: normal
Tags: patch
Done: Jim Meyering <jim <at> meyering.net>
Bug is archived. No further changes may be made.
View this message in rfc822 format
From: "jeff.liu" <jeff.liu <at> oracle.com> To: Jim Meyering <jim <at> meyering.net> Cc: Sunil Mushran <sunil.mushran <at> oracle.com>, Tao Ma <tao.ma <at> oracle.com>, bug-coreutils <at> gnu.org, Joel Becker <Joel.Becker <at> oracle.com>, Chris Mason <chris.mason <at> oracle.com> Subject: bug#6131: [PATCH]: fiemap support for efficient sparse file copy Date: Thu, 27 May 2010 22:08:43 +0800
Jim Meyering wrote:
> jeff.liu wrote:
>> This is the revised version, it fixed the fiemap-start offset calculation
>> approach to remove it out
>> of the 'for (i = 0; i < fiemap->fm_mapped_extents; i++)' loop.
>
> Hi Jeff,
>
> I've included below the state of my local changes.
> Unfortunately, with that 5-patch series, there is always a test failure
> on F13/ext4. Maybe someone who knows more about extents can provide an
> explanation?
>
> Here's a small example to demonstrate:
>
> Create a file with many extents:
>
> perl -e 'BEGIN { $n = 19 * 1024; *F = *STDOUT }' \
> -e 'for (1..100) { sysseek (*F, $n, 1)' \
> -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1
>
> Using the patched "cp", repeat the following 10 or 20 times:
>
> ./cp --sparse=always j1 j2; sync
> filefrag -v j1 | awk '/^ / {print $1,$2}' > ff1 || fail=1
> filefrag -v j2 | awk '/^ / {print $1,$2}' > ff2 || fail=1
> diff -u ff1 ff2 || fail=1
>
> Usually there is no diff output, but occasionally it'll print this:
> [hmm... today it consistently prints these differences every other time.]
Woo!!!
I just run this test on btrfs/ext4/ocfs2 against mainline kernel(Linux jeff-laptop
2.6.33-rc5-00238-gb04da8b-dirty) on my laptop.
Only btrfs always works well for me, Ext4 has the same issue like yours.
jeff <at> jeff-laptop:/ext4/test$ for ((i=0; i < 100; i++)); do ./1.sh; done
--- ff1 2010-05-27 22:03:25.263480260 +0800
+++ ff2 2010-05-27 22:03:25.315476210 +0800
@@ -26,77 +26,78 @@
24 232
25 242
26 251
-27 261
-28 270
-29 280
-30 289
-31 299
-32 308
-33 318
-34 327
-35 337
-36 346
-37 356
-38 365
-39 375
-40 384
-41 394
-42 403
-43 413
-44 422
-45 432
-46 441
-47 451
-48 460
-49 470
-50 479
-51 489
-52 498
-53 508
-54 512
-55 517
-56 527
-57 536
-58 546
-59 555
-60 565
-61 574
-62 584
-63 593
-64 603
-65 612
-66 622
-67 631
-68 641
-69 650
-70 660
-71 669
-72 679
-73 688
-74 698
-75 707
-76 717
-77 726
-78 736
-79 745
-80 755
-81 764
-82 774
-83 783
-84 793
-85 802
-86 812
-87 821
-88 831
-89 840
-90 850
-91 859
-92 869
-93 878
-94 888
-95 897
-96 907
-97 916
-98 926
-99 935
-100 945
+27 256
+28 261
+29 270
+30 280
+31 289
+32 299
+33 308
+34 318
+35 327
+36 337
+37 346
+38 356
+39 365
+40 375
+41 384
+42 394
+43 403
+44 413
+45 422
+46 432
+47 441
+48 451
+49 460
+50 470
+51 479
+52 489
+53 498
+54 508
+55 512
+56 517
+57 527
+58 536
+59 546
+60 555
+61 565
+62 574
+63 584
+64 593
+65 603
+66 612
+67 622
+68 631
+69 641
+70 650
+71 660
+72 669
+73 679
+74 688
+75 698
+76 707
+77 717
+78 726
+79 736
+80 745
+81 755
+82 764
+83 774
+84 783
+85 793
+86 802
+87 812
+88 821
+89 831
+90 840
+91 850
+92 859
+93 869
+94 878
+95 888
+96 897
+97 907
+98 916
+99 926
+100 935
+101 945
OCFS2's show different for many times(repeat 100 times).
But I cannot figure out this issue from the patch at the moment. :(
HEROS, who can give some hints from the kernel's point of view?
jeff <at> jeff-laptop:/ocfs2/test$ for ((i=0; i < 100; i++)); do ./1.sh; done
--- ff1 2010-05-27 21:49:58.288483759 +0800
+++ ff2 2010-05-27 21:49:58.304545778 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 519
--- ff1 2010-05-27 21:50:05.260477055 +0800
+++ ff2 2010-05-27 21:50:05.276532299 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 815
--- ff1 2010-05-27 21:50:13.048484948 +0800
+++ ff2 2010-05-27 21:50:13.076486205 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 161
--- ff1 2010-05-27 21:50:19.748471818 +0800
+++ ff2 2010-05-27 21:50:19.764488230 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 457
--- ff1 2010-05-27 21:50:25.384540820 +0800
+++ ff2 2010-05-27 21:50:25.396508694 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 753
--- ff1 2010-05-27 21:50:33.504500173 +0800
+++ ff2 2010-05-27 21:50:33.536480827 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 99
--- ff1 2010-05-27 21:50:40.012512743 +0800
+++ ff2 2010-05-27 21:50:40.028511067 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 395
--- ff1 2010-05-27 21:50:46.680505132 +0800
+++ ff2 2010-05-27 21:50:46.708501290 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 691
--- ff1 2010-05-27 21:50:55.576480477 +0800
+++ ff2 2010-05-27 21:50:55.596475937 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 37
--- ff1 2010-05-27 21:51:03.491482451 +0800
+++ ff2 2010-05-27 21:51:03.515503404 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 333
--- ff1 2010-05-27 21:51:11.567497188 +0800
+++ ff2 2010-05-27 21:51:11.603479937 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 629
--- ff1 2010-05-27 21:51:20.983471486 +0800
+++ ff2 2010-05-27 21:51:21.011472813 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 925
--- ff1 2010-05-27 21:51:29.548508415 +0800
+++ ff2 2010-05-27 21:51:29.588486206 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 271
--- ff1 2010-05-27 21:51:35.927492089 +0800
+++ ff2 2010-05-27 21:51:35.959488178 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 567
--- ff1 2010-05-27 21:51:40.200480478 +0800
+++ ff2 2010-05-27 21:51:40.240471818 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 863
--- ff1 2010-05-27 21:51:45.772499475 +0800
+++ ff2 2010-05-27 21:51:45.820498986 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 209
--- ff1 2010-05-27 21:51:49.919519607 +0800
+++ ff2 2010-05-27 21:51:49.947475956 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 505
--- ff1 2010-05-27 21:51:55.195482312 +0800
+++ ff2 2010-05-27 21:51:55.231483778 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 801
--- ff1 2010-05-27 21:52:00.911479588 +0800
+++ ff2 2010-05-27 21:52:00.939473232 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 147
--- ff1 2010-05-27 21:52:06.208488091 +0800
+++ ff2 2010-05-27 21:52:06.252482433 +0800
@@ -1,2 +1,3 @@
ext logical
0 0
+1 443
....
...
>
> $ diff -u ff1 ff2 || fail=1
> --- ff1 2010-05-22 18:42:26.943501382 +0200
> +++ ff2 2010-05-22 18:42:27.020876155 +0200
> @@ -53,49 +53,50 @@ ext logical
> 51 489
> 52 498
> 53 508
> -54 517
> -55 527
> -56 536
> -57 546
> -58 555
> -59 565
> -60 574
> -61 584
> -62 593
> -63 603
> -64 612
> -65 622
> -66 631
> -67 641
> -68 650
> -69 660
> -70 669
> -71 679
> -72 688
> -73 698
> -74 707
> -75 717
> -76 726
> -77 736
> -78 745
> -79 755
> -80 764
> -81 774
> -82 783
> -83 793
> -84 802
> -85 812
> -86 821
> -87 831
> -88 840
> -89 850
> -90 859
> -91 869
> -92 878
> -93 888
> -94 897
> -95 907
> -96 916
> -97 926
> -98 935
> -99 945
> +54 512
> +55 517
> +56 527
> +57 536
> +58 546
> +59 555
> +60 565
> +61 574
> +62 584
> +63 593
> +64 603
> +65 612
> +66 622
> +67 631
> +68 641
> +69 650
> +70 660
> +71 669
> +72 679
> +73 688
> +74 698
> +75 707
> +76 717
> +77 726
> +78 736
> +79 745
> +80 755
> +81 764
> +82 774
> +83 783
> +84 793
> +85 802
> +86 812
> +87 821
> +88 831
> +89 840
> +90 850
> +91 859
> +92 869
> +93 878
> +94 888
> +95 897
> +96 907
> +97 916
> +98 926
> +99 935
> +100 945
>
>
> From 2ef44bcb0bbb38cca738aa90e067caee312da939 Mon Sep 17 00:00:00 2001
> From: Jie Liu <jeff.liu <at> oracle.com>
> Date: Thu, 13 May 2010 22:09:30 +0800
> Subject: [PATCH 1/5] cp: Add FIEMAP support for efficient sparse file copy
>
> * src/fiemap.h: Add fiemap.h for fiemap ioctl(2) support.
> Copied from linux's include/linux/fiemap.h, with minor formatting changes.
> * src/copy.c (copy_reg): Now, when `cp' invoked with --sparse=[WHEN] option, we
> will try to do FIEMAP-copy if the underlaying file system support it, fall back
> to a normal copy if it fails.
> ---
> src/copy.c | 159 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> src/fiemap.h | 102 +++++++++++++++++++++++++++++++++++++
> 2 files changed, 261 insertions(+), 0 deletions(-)
> create mode 100644 src/fiemap.h
>
> diff --git a/src/copy.c b/src/copy.c
> index c16cef6..0e54729 100644
> --- a/src/copy.c
> +++ b/src/copy.c
> @@ -63,6 +63,10 @@
>
> #include <sys/ioctl.h>
>
> +#ifndef HAVE_FIEMAP
> +# include "fiemap.h"
> +#endif
> +
> #ifndef HAVE_FCHOWN
> # define HAVE_FCHOWN false
> # define fchown(fd, uid, gid) (-1)
> @@ -149,6 +153,141 @@ clone_file (int dest_fd, int src_fd)
> #endif
> }
>
> +#ifdef __linux__
> +# ifndef FS_IOC_FIEMAP
> +# define FS_IOC_FIEMAP _IOWR ('f', 11, struct fiemap)
> +# endif
> +/* Perform FIEMAP(available in mainline 2.6.27) copy if possible.
> + Call ioctl(2) with FS_IOC_FIEMAP to efficiently map file allocation
> + excepts holes. So the overhead to deal with holes with lseek(2) in
> + normal copy could be saved. This would result in much faster backups
> + for any kind of sparse file. */
> +static bool
> +fiemap_copy_ok (int src_fd, int dest_fd, size_t buf_size,
> + off_t src_total_size, char const *src_name,
> + char const *dst_name, bool *normal_copy_required)
> +{
> + bool fail = false;
> + bool last = false;
> + char fiemap_buf[4096];
> + struct fiemap *fiemap = (struct fiemap *)fiemap_buf;
> + struct fiemap_extent *fm_ext = &fiemap->fm_extents[0];
> + uint32_t count = (sizeof (fiemap_buf) - sizeof (*fiemap)) /
> + sizeof (struct fiemap_extent);
> + off_t last_ext_logical = 0;
> + uint64_t last_ext_len = 0;
> + uint64_t last_read_size = 0;
> + unsigned int i = 0;
> +
> + /* This is required at least to initialize fiemap->fm_start,
> + but also serves (in May 2010) to appease valgrind, which
> + appears not to know the semantics of the FIEMAP ioctl. */
> + memset (fiemap_buf, 0, sizeof fiemap_buf);
> +
> + do
> + {
> + fiemap->fm_length = FIEMAP_MAX_OFFSET;
> + fiemap->fm_extent_count = count;
> +
> + /* When ioctl(2) fails, fall back to the normal copy only if it
> + is the first time we met. */
> + if (ioctl (src_fd, FS_IOC_FIEMAP, fiemap) < 0)
> + {
> + /* If `i > 0', then at least one ioctl(2) has been performed before. */
> + if (i == 0)
> + *normal_copy_required = true;
> + return false;
> + }
> +
> + /* If 0 extents are returned, then more ioctls are not needed. */
> + if (fiemap->fm_mapped_extents == 0)
> + break;
> +
> + for (i = 0; i < fiemap->fm_mapped_extents; i++)
> + {
> + assert (fm_ext[i].fe_logical <= OFF_T_MAX);
> +
> + off_t ext_logical = fm_ext[i].fe_logical;
> + uint64_t ext_len = fm_ext[i].fe_length;
> +
> + if (lseek (src_fd, ext_logical, SEEK_SET) < 0LL)
> + {
> + error (0, errno, _("cannot lseek %s"), quote (src_name));
> + return fail;
> + }
> +
> + if (lseek (dest_fd, ext_logical, SEEK_SET) < 0LL)
> + {
> + error (0, errno, _("cannot lseek %s"), quote (dst_name));
> + return fail;
> + }
> +
> + if (fm_ext[i].fe_flags & FIEMAP_EXTENT_LAST)
> + {
> + last_ext_logical = ext_logical;
> + last_ext_len = ext_len;
> + last = true;
> + }
> +
> + while (0 < ext_len)
> + {
> + char buf[buf_size];
> +
> + /* Avoid reading into the holes if the left extent
> + length is shorter than the buffer size. */
> + if (ext_len < buf_size)
> + buf_size = ext_len;
> +
> + ssize_t n_read = read (src_fd, buf, buf_size);
> + if (n_read < 0)
> + {
> +#ifdef EINTR
> + if (errno == EINTR)
> + continue;
> +#endif
> + error (0, errno, _("reading %s"), quote (src_name));
> + return fail;
> + }
> +
> + if (n_read == 0)
> + {
> + /* Figure out how many bytes read from the last extent. */
> + last_read_size = last_ext_len - ext_len;
> + break;
> + }
> +
> + if (full_write (dest_fd, buf, n_read) != n_read)
> + {
> + error (0, errno, _("writing %s"), quote (dst_name));
> + return fail;
> + }
> +
> + ext_len -= n_read;
> + }
> + }
> +
> + fiemap->fm_start = fm_ext[i - 1].fe_logical + fm_ext[i - 1].fe_length;
> +
> + } while (! last);
> +
> + /* If a file ends up with holes, the sum of the last extent logical offset
> + and the read-returned size will be shorter than the actual size of the
> + file. Use ftruncate to extend the length of the destination file. */
> + if (last_ext_logical + last_read_size < src_total_size)
> + {
> + if (ftruncate (dest_fd, src_total_size) < 0)
> + {
> + error (0, errno, _("extending %s"), quote (dst_name));
> + return fail;
> + }
> + }
> +
> + return ! fail;
> +}
> +#else
> +static bool fiemap_copy_ok (ignored) { errno == ENOTSUP; return false; }
> +#endif
> +
> /* FIXME: describe */
> /* FIXME: rewrite this to use a hash table so we avoid the quadratic
> performance hit that's probably noticeable only on trees deeper
> @@ -679,6 +818,25 @@ copy_reg (char const *src_name, char const *dst_name,
> #endif
> }
>
> + if (make_holes)
> + {
> + bool require_normal_copy = false;
> + /* Perform efficient FIEMAP copy for sparse files, fall back to the
> + standard copy only if the ioctl(2) fails. */
> + if (fiemap_copy_ok (source_desc, dest_desc, buf_size,
> + src_open_sb.st_size, src_name,
> + dst_name, &require_normal_copy))
> + goto preserve_metadata;
> + else
> + {
> + if (! require_normal_copy)
> + {
> + return_val = false;
> + goto close_src_and_dst_desc;
> + }
> + }
> + }
> +
> /* If not making a sparse file, try to use a more-efficient
> buffer size. */
> if (! make_holes)
> @@ -807,6 +965,7 @@ copy_reg (char const *src_name, char const *dst_name,
> }
> }
>
> +preserve_metadata:
> if (x->preserve_timestamps)
> {
> struct timespec timespec[2];
> diff --git a/src/fiemap.h b/src/fiemap.h
> new file mode 100644
> index 0000000..d33293b
> --- /dev/null
> +++ b/src/fiemap.h
> @@ -0,0 +1,102 @@
> +/* FS_IOC_FIEMAP ioctl infrastructure.
> + Some portions copyright (C) 2007 Cluster File Systems, Inc
> + Authors: Mark Fasheh <mfasheh <at> suse.com>
> + Kalpak Shah <kalpak.shah <at> sun.com>
> + Andreas Dilger <adilger <at> sun.com>. */
> +
> +/* Copy from kernel, modified to respect GNU code style by Jie Liu. */
> +
> +#ifndef _LINUX_FIEMAP_H
> +# define _LINUX_FIEMAP_H
> +
> +# include <linux/types.h>
> +
> +struct fiemap_extent
> +{
> + /* Logical offset in bytes for the start of the extent
> + from the beginning of the file. */
> + uint64_t fe_logical;
> +
> + /* Physical offset in bytes for the start of the extent
> + from the beginning of the disk. */
> + uint64_t fe_physical;
> +
> + /* Length in bytes for this extent. */
> + uint64_t fe_length;
> +
> + uint64_t fe_reserved64[2];
> +
> + /* FIEMAP_EXTENT_* flags for this extent. */
> + uint32_t fe_flags;
> +
> + uint32_t fe_reserved[3];
> +};
> +
> +struct fiemap
> +{
> + /* Logical offset(inclusive) at which to start mapping(in). */
> + uint64_t fm_start;
> +
> + /* Logical length of mapping which userspace wants(in). */
> + uint64_t fm_length;
> +
> + /* FIEMAP_FLAG_* flags for request(in/out). */
> + uint32_t fm_flags;
> +
> + /* Number of extents that were mapped(out). */
> + uint32_t fm_mapped_extents;
> +
> + /* Size of fm_extents array(in). */
> + uint32_t fm_extent_count;
> +
> + uint32_t fm_reserved;
> +
> + /* Array of mapped extents(out). */
> + struct fiemap_extent fm_extents[0];
> +};
> +
> +/* The maximum offset can be mapped for a file. */
> +# define FIEMAP_MAX_OFFSET (~0ULL)
> +
> +/* Sync file data before map. */
> +# define FIEMAP_FLAG_SYNC 0x00000001
> +
> +/* Map extented attribute tree. */
> +# define FIEMAP_FLAG_XATTR 0x00000002
> +
> +# define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR)
> +
> +/* Last extent in file. */
> +# define FIEMAP_EXTENT_LAST 0x00000001
> +
> +/* Data location unknown. */
> +# define FIEMAP_EXTENT_UNKNOWN 0x00000002
> +
> +/* Location still pending, Sets EXTENT_UNKNOWN. */
> +# define FIEMAP_EXTENT_DELALLOC 0x00000004
> +
> +/* Data can not be read while fs is unmounted. */
> +# define FIEMAP_EXTENT_ENCODED 0x00000008
> +
> +/* Data is encrypted by fs. Sets EXTENT_NO_BYPASS. */
> +# define FIEMAP_EXTENT_DATA_ENCRYPTED 0x00000080
> +
> +/* Extent offsets may not be block aligned. */
> +# define FIEMAP_EXTENT_NOT_ALIGNED 0x00000100
> +
> +/* Data mixed with metadata. Sets EXTENT_NOT_ALIGNED. */
> +# define FIEMAP_EXTENT_DATA_INLINE 0x00000200
> +
> +/* Multiple files in block. Set EXTENT_NOT_ALIGNED. */
> +# define FIEMAP_EXTENT_DATA_TAIL 0x00000400
> +
> +/* Space allocated, but not data (i.e. zero). */
> +# define FIEMAP_EXTENT_UNWRITTEN 0x00000800
> +
> +/* File does not natively support extents. Result merged for efficiency. */
> +# define FIEMAP_EXTENT_MERGED 0x00001000
> +
> +/* Space shared with other files. */
> +# define FIEMAP_EXTENT_SHARED 0x00002000
> +
> +#endif
> --
> 1.7.1.348.gb26ba
>
>
> From 260b5b89e33da2b9a5ea5bcd9dba874f503d2937 Mon Sep 17 00:00:00 2001
> From: Jie Liu <jeff.liu <at> oracle.com>
> Date: Thu, 13 May 2010 22:17:53 +0800
> Subject: [PATCH 2/5] tests: add a new test for FIEMAP-copy
>
> * tests/cp/sparse-fiemap: Add a new test for FIEMAP-copy against a
> loopbacked ext4 partition.
> * tests/Makefile.am (sparse-fiemap): Reference the new test.
> ---
> tests/Makefile.am | 1 +
> tests/cp/sparse-fiemap | 56 ++++++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 57 insertions(+), 0 deletions(-)
> create mode 100755 tests/cp/sparse-fiemap
>
> diff --git a/tests/Makefile.am b/tests/Makefile.am
> index c458574..f7840c8 100644
> --- a/tests/Makefile.am
> +++ b/tests/Makefile.am
> @@ -25,6 +25,7 @@ root_tests = \
> cp/special-bits \
> cp/cp-mv-enotsup-xattr \
> cp/capability \
> + cp/sparse-fiemap \
> dd/skip-seek-past-dev \
> install/install-C-root \
> ls/capability \
> diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap
> new file mode 100755
> index 0000000..945c94b
> --- /dev/null
> +++ b/tests/cp/sparse-fiemap
> @@ -0,0 +1,56 @@
> +#!/bin/sh
> +# Test cp --sparse=always through fiemap copy
> +
> +# Copyright (C) 2006-2010 Free Software Foundation, Inc.
> +
> +# This program is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation, either version 3 of the License, or
> +# (at your option) any later version.
> +
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +
> +if test "$VERBOSE" = yes; then
> + set -x
> + cp --version
> +fi
> +
> +. $srcdir/test-lib.sh
> +require_root_
> +
> +cwd=`pwd`
> +cleanup_() { cd /; umount "$cwd/mnt"; }
> +
> +skip=0
> +# Create an ext4 loopback file system
> +dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1
> +mkdir mnt
> +mkfs -t ext4 -F blob ||
> + skip_test_ "failed to create ext4 file system"
> +mount -oloop blob mnt || skip=1
> +echo test > mnt/f || skip=1
> +test -s mnt/f || skip=1
> +
> +test $skip = 1 &&
> + skip_test_ "insufficient mount/ext4 support"
> +
> +# Create a 1TiB sparse file
> +dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure
> +
> +cd mnt || fail=1
> +
> +# It takes many minutes to copy this sparse file using the old method.
> +# By contrast, it takes far less than 1 second using FIEMAP-copy.
> +timeout 10 cp --sparse=always sparse fiemap || fail=1
> +
> +# Ensure that the sparse file copied through fiemap has the same size
> +# in bytes as the original.
> +test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1
> +
> +Exit $fail
> --
> 1.7.1.348.gb26ba
>
>
> From 738031aed78b6323969f799d3d5fbf19e0cfc91a Mon Sep 17 00:00:00 2001
> From: Jim Meyering <meyering <at> redhat.com>
> Date: Fri, 21 May 2010 18:28:42 +0200
> Subject: [PATCH 3/5] tests: exercise more of the new FIEMAP copying code
>
> * tests/cp/sparse-fiemap: Ensure that a file with many extents (more
> than fit in copy.c's internal 4KiB buffer) is copied properly.
> ---
> tests/cp/sparse-fiemap | 38 ++++++++++++++++++++++++++++++++++++++
> 1 files changed, 38 insertions(+), 0 deletions(-)
>
> diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap
> index 945c94b..b1643be 100755
> --- a/tests/cp/sparse-fiemap
> +++ b/tests/cp/sparse-fiemap
> @@ -53,4 +53,42 @@ timeout 10 cp --sparse=always sparse fiemap || fail=1
> # in bytes as the original.
> test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1
>
> +# =================================================
> +# Ensure that we exercise the FIEMAP-copying code enough
> +# to provoke at least two iterations of the do...while loop
> +# in which it calls ioctl (fd, FS_IOC_FIEMAP,...
> +# This also verifies that non-trivial extents are preserved.
> +
> +$PERL -e 1 || skip_test_ 'skipping part of this test; you lack perl'
> +
> +$PERL -e 'BEGIN { $n = 16 * 1024; *F = *STDOUT }' \
> + -e 'for (1..100) { sysseek (*F, $n, 1)' \
> + -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1
> +
> +cp --sparse=always j1 j2 || fail=1
> +cmp j1 j2 || fail=1
> +
> +filefrag j1 | grep extent \
> + || skip_test_ 'skipping part of this test; you lack filefrag'
> +
> +# Here is sample filefrag output:
> +# $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \
> +# -e 'for (1..5) { sysseek(*F,$n,1)' \
> +# -e '&& syswrite *F,"."x$n or die "$!"}' > j
> +# $ filefrag -v j
> +# Filesystem type is: ef53
> +# File size of j is 163840 (40 blocks, blocksize 4096)
> +# ext logical physical expected length flags
> +# 0 4 6258884 4
> +# 1 12 6258892 6258887 4
> +# 2 20 6258900 6258895 4
> +# 3 28 6258908 6258903 4
> +# 4 36 6258916 6258911 4 eof
> +# j: 6 extents found
> +
> +# exclude the physical block numbers; they always differ
> +filefrag -v j1 | awk '/^ / {print $1,$2,$NF}' > ff1 || fail=1
> +filefrag -v j2 | awk '/^ / {print $1,$2,$NF}' > ff2 || fail=1
> +compare ff1 ff2 || fail=1
> +
> Exit $fail
> --
> 1.7.1.348.gb26ba
>
>
> From e84f881cb5c2eb92f3b6d5bddaef50c3e811bc30 Mon Sep 17 00:00:00 2001
> From: Jim Meyering <meyering <at> redhat.com>
> Date: Sat, 22 May 2010 10:22:58 +0200
> Subject: [PATCH 4/5] tests: require root access only if current partition is not ext4
>
> * tests/cp/sparse-fiemap: Don't require root access if current
> partition is ext4.
> ---
> tests/cp/sparse-fiemap | 44 +++++++++++++++++++++++++-------------------
> 1 files changed, 25 insertions(+), 19 deletions(-)
>
> diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap
> index b1643be..371bced 100755
> --- a/tests/cp/sparse-fiemap
> +++ b/tests/cp/sparse-fiemap
> @@ -22,28 +22,34 @@ if test "$VERBOSE" = yes; then
> fi
>
> . $srcdir/test-lib.sh
> -require_root_
>
> -cwd=`pwd`
> -cleanup_() { cd /; umount "$cwd/mnt"; }
> -
> -skip=0
> -# Create an ext4 loopback file system
> -dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1
> -mkdir mnt
> -mkfs -t ext4 -F blob ||
> - skip_test_ "failed to create ext4 file system"
> -mount -oloop blob mnt || skip=1
> -echo test > mnt/f || skip=1
> -test -s mnt/f || skip=1
> -
> -test $skip = 1 &&
> - skip_test_ "insufficient mount/ext4 support"
> +if df -T -t ext4 . ; then
> + : # Current dir is on an ext4 partition. Good!
> +else
> + # It's not; we need to create one, hence we need root access.
> + require_root_
> +
> + cwd=$PWD
> + cleanup_() { cd /; umount "$cwd/mnt"; }
> +
> + skip=0
> + # Create an ext4 loopback file system
> + dd if=/dev/zero of=blob bs=8192 count=1000 || skip=1
> + mkdir mnt
> + mkfs -t ext4 -F blob ||
> + skip_test_ "failed to create ext4 file system"
> + mount -oloop blob mnt || skip=1
> + echo test > mnt/f || skip=1
> + test -s mnt/f || skip=1
> +
> + test $skip = 1 &&
> + skip_test_ "insufficient mount/ext4 support"
> +
> + cd mnt || fail=1
> +fi
>
> # Create a 1TiB sparse file
> -dd if=/dev/zero of=mnt/sparse bs=1k count=1 seek=1G || framework_failure
> -
> -cd mnt || fail=1
> +dd if=/dev/zero of=sparse bs=1k count=1 seek=1G || framework_failure
>
> # It takes many minutes to copy this sparse file using the old method.
> # By contrast, it takes far less than 1 second using FIEMAP-copy.
> --
> 1.7.1.348.gb26ba
>
>
> From 213c9247a406b267bc5819ccbe3b729e9557c675 Mon Sep 17 00:00:00 2001
> From: Jim Meyering <meyering <at> redhat.com>
> Date: Sat, 22 May 2010 10:21:46 +0200
> Subject: [PATCH 5/5] tests: fiemap test improvement
>
> * tests/cp/sparse-fiemap: More tests.
> ---
> tests/cp/sparse-fiemap | 69 +++++++++++++++++++++++++-----------------------
> 1 files changed, 36 insertions(+), 33 deletions(-)
>
> diff --git a/tests/cp/sparse-fiemap b/tests/cp/sparse-fiemap
> index 371bced..ef3742e 100755
> --- a/tests/cp/sparse-fiemap
> +++ b/tests/cp/sparse-fiemap
> @@ -38,14 +38,14 @@ else
> mkdir mnt
> mkfs -t ext4 -F blob ||
> skip_test_ "failed to create ext4 file system"
> - mount -oloop blob mnt || skip=1
> - echo test > mnt/f || skip=1
> - test -s mnt/f || skip=1
> + mount -oloop blob mnt || skip=1
> + cd mnt || skip=1
> + echo test > f || skip=1
> + test -s f || skip=1
>
> test $skip = 1 &&
> skip_test_ "insufficient mount/ext4 support"
>
> - cd mnt || fail=1
> fi
>
> # Create a 1TiB sparse file
> @@ -67,34 +67,37 @@ test $(stat --printf %s sparse) = $(stat --printf %s fiemap) || fail=1
>
> $PERL -e 1 || skip_test_ 'skipping part of this test; you lack perl'
>
> -$PERL -e 'BEGIN { $n = 16 * 1024; *F = *STDOUT }' \
> - -e 'for (1..100) { sysseek (*F, $n, 1)' \
> - -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1
> -
> -cp --sparse=always j1 j2 || fail=1
> -cmp j1 j2 || fail=1
> -
> -filefrag j1 | grep extent \
> - || skip_test_ 'skipping part of this test; you lack filefrag'
> -
> -# Here is sample filefrag output:
> -# $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \
> -# -e 'for (1..5) { sysseek(*F,$n,1)' \
> -# -e '&& syswrite *F,"."x$n or die "$!"}' > j
> -# $ filefrag -v j
> -# Filesystem type is: ef53
> -# File size of j is 163840 (40 blocks, blocksize 4096)
> -# ext logical physical expected length flags
> -# 0 4 6258884 4
> -# 1 12 6258892 6258887 4
> -# 2 20 6258900 6258895 4
> -# 3 28 6258908 6258903 4
> -# 4 36 6258916 6258911 4 eof
> -# j: 6 extents found
> -
> -# exclude the physical block numbers; they always differ
> -filefrag -v j1 | awk '/^ / {print $1,$2,$NF}' > ff1 || fail=1
> -filefrag -v j2 | awk '/^ / {print $1,$2,$NF}' > ff2 || fail=1
> -compare ff1 ff2 || fail=1
> +for i in $(seq 20); do
> + for j in 1 2 31 100; do
> + $PERL -e 'BEGIN { $n = '$i' * 1024; *F = *STDOUT }' \
> + -e 'for (1..'$j') { sysseek (*F, $n, 1)' \
> + -e '&& syswrite (*F, "."x$n) or die "$!"}' > j1 || fail=1
> +
> + cp --sparse=always j1 j2 || fail=1
> + cmp j1 j2 || fail=1
> + filefrag -v j1 | grep extent \
> + || skip_test_ 'skipping part of this test; you lack filefrag'
> +
> + # Here is sample filefrag output:
> + # $ perl -e 'BEGIN{$n=16*1024; *F=*STDOUT}' \
> + # -e 'for (1..5) { sysseek(*F,$n,1)' \
> + # -e '&& syswrite *F,"."x$n or die "$!"}' > j
> + # $ filefrag -v j
> + # Filesystem type is: ef53
> + # File size of j is 163840 (40 blocks, blocksize 4096)
> + # ext logical physical expected length flags
> + # 0 4 6258884 4
> + # 1 12 6258892 6258887 4
> + # 2 20 6258900 6258895 4
> + # 3 28 6258908 6258903 4
> + # 4 36 6258916 6258911 4 eof
> + # j: 6 extents found
> +
> + # exclude the physical block numbers; they always differ
> + filefrag -v j1 | awk '/^ / {print $1,$2}' > ff1 || fail=1
> + filefrag -v j2 | awk '/^ / {print $1,$2}' > ff2 || fail=1
> + compare ff1 ff2 || fail=1
> + done
> +done
>
> Exit $fail
> --
> 1.7.1.348.gb26ba
>
>
>
--
With Windows 7, Microsoft is asserting legal control over your computer and is using this power to
abuse computer users.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.