From unknown Fri Jun 20 07:13:22 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#21790 <21790@debbugs.gnu.org> To: bug#21790 <21790@debbugs.gnu.org> Subject: Status: [PATCH] coreutils/cp: handle EOF extents correctly Reply-To: bug#21790 <21790@debbugs.gnu.org> Date: Fri, 20 Jun 2025 14:13:22 +0000 retitle 21790 [PATCH] coreutils/cp: handle EOF extents correctly reassign 21790 coreutils submitter 21790 Dmitry Monakhov severity 21790 normal tag 21790 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 30 12:01:50 2015 Received: (at submit) by debbugs.gnu.org; 30 Oct 2015 16:01:50 +0000 Received: from localhost ([127.0.0.1]:46368 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1ZsC7f-0000wL-R2 for submit@debbugs.gnu.org; Fri, 30 Oct 2015 12:01:49 -0400 Received: from eggs.gnu.org ([208.118.235.92]:53966) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Zs5uS-0006cU-H7 for submit@debbugs.gnu.org; Fri, 30 Oct 2015 05:24:04 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Zs5uR-0006B2-6S for submit@debbugs.gnu.org; Fri, 30 Oct 2015 05:23:44 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:38746) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zs5uR-0006Ay-38 for submit@debbugs.gnu.org; Fri, 30 Oct 2015 05:23:43 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:35736) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zs5uP-0004Hd-NB for bug-coreutils@gnu.org; Fri, 30 Oct 2015 05:23:42 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Zs5uL-0006AN-QT for bug-coreutils@gnu.org; Fri, 30 Oct 2015 05:23:41 -0400 Received: from mailhub.sw.ru ([195.214.232.25]:13129 helo=relay.sw.ru) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zs5uL-0006A9-DD for bug-coreutils@gnu.org; Fri, 30 Oct 2015 05:23:37 -0400 Received: from mct2.qa.sw.ru ([10.29.1.63]) by relay.sw.ru (8.13.4/8.13.4) with ESMTP id t9U92htJ024090; Fri, 30 Oct 2015 12:02:44 +0300 (MSK) From: Dmitry Monakhov To: bug-coreutils@gnu.org Subject: [PATCH] coreutils/cp: handle EOF extents correctly Date: Fri, 30 Oct 2015 13:02:39 +0400 Message-Id: <1446195759-32225-1-git-send-email-dmonakhov@openvz.org> X-Mailer: git-send-email 1.9.3 X-detected-operating-system: by eggs.gnu.org: OpenBSD 3.x X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Fri, 30 Oct 2015 12:01:46 -0400 Cc: Dmitry Monakhov , vvs@parallels.com, devel@openvz.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.0 (----) fallocate can allocate extens beyond EOF via FALLOC_FL_KEEP_SIZE. Currenly sparse engine tries to copy such extents which is wrong and result in silent data corruption (leave file with incorrect size). ##TESTCASE echo blabla > sparse_falloc.in truncate -s 2M sparse_falloc.in fallocate -n -o 4M -l 1M sparse_falloc.in cp sparse_falloc.in sparse_falloc.out cmp sparse_falloc.in sparse_falloc.out Signed-off-by: Dmitry Monakhov --- src/copy.c | 7 ++++++- tests/cp/sparse-unwritten.sh | 29 +++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 1 deletion(-) create mode 100755 tests/cp/sparse-unwritten.sh diff --git a/src/copy.c b/src/copy.c index edf022e..de197a4 100644 --- a/src/copy.c +++ b/src/copy.c @@ -430,7 +430,12 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, ext_start = last_ext_start + scan.ext_info[i].ext_length; ext_len = 0; } - + /* Is this extent beyond EOF? */ + if (ext_start + ext_len >= src_total_size) + { + wrote_hole_at_eof = true; + break; + } ext_hole_size = ext_start - last_ext_start - last_ext_len; wrote_hole_at_eof = false; diff --git a/tests/cp/sparse-unwritten.sh b/tests/cp/sparse-unwritten.sh new file mode 100755 index 0000000..4666535 --- /dev/null +++ b/tests/cp/sparse-unwritten.sh @@ -0,0 +1,29 @@ +#!/bin/sh +# Test cp for a file which has extents beyond EOF + +# Copyright (C) 2006-2015 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +. "${srcdir=.}/tests/init.sh"; path_prepend_ ./src +print_ver_ cp +require_sparse_support_ + +echo blabla > sparse_falloc.in || framework_failure_ +truncate -s 2M sparse_falloc.in || framework_failure_ +fallocate -n -o 4M -l 1M sparse_falloc.in || framework_failure_ +cp sparse_falloc.in sparse_falloc.out || fail=1 +cmp sparse_falloc.in sparse_falloc.out || fail=1 + +Exit $fail -- 2.1.4 From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 30 12:57:10 2015 Received: (at 21790) by debbugs.gnu.org; 30 Oct 2015 16:57:10 +0000 Received: from localhost ([127.0.0.1]:46428 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1ZsCzF-0003s0-IU for submit@debbugs.gnu.org; Fri, 30 Oct 2015 12:57:09 -0400 Received: from mail1.vodafone.ie ([213.233.128.43]:64798) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1ZsCzD-0003rs-MW for 21790@debbugs.gnu.org; Fri, 30 Oct 2015 12:57:08 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: As8dAIKgM1ZtTElv/2dsb2JhbABeKAECgj5SHzRvgUm5SIYJHYV8AoE1TAEBAQEBAYELhDYBAQQyAUYQCw0BCgkWDwkDAgECAUUGAQwIAQGIMAG4fIs1AQEBAQYCASCFfIV5iUABBJZDhR2ROJMKY4QEPjSFfgEBAQ Received: from unknown (HELO localhost.localdomain) ([109.76.73.111]) by mail1.vodafone.ie with ESMTP; 30 Oct 2015 16:57:05 +0000 Subject: Re: bug#21790: [PATCH] coreutils/cp: handle EOF extents correctly To: Dmitry Monakhov , 21790@debbugs.gnu.org References: <1446195759-32225-1-git-send-email-dmonakhov@openvz.org> From: =?UTF-8?Q?P=c3=a1draig_Brady?= Message-ID: <5633A160.1030405@draigBrady.com> Date: Fri, 30 Oct 2015 16:57:04 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <1446195759-32225-1-git-send-email-dmonakhov@openvz.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 21790 Cc: vvs@parallels.com, devel@openvz.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) On 30/10/15 09:02, Dmitry Monakhov wrote: > fallocate can allocate extens beyond EOF via FALLOC_FL_KEEP_SIZE. > Currenly sparse engine tries to copy such extents which is wrong and > result in silent data corruption (leave file with incorrect size). > > ##TESTCASE > echo blabla > sparse_falloc.in > truncate -s 2M sparse_falloc.in > fallocate -n -o 4M -l 1M sparse_falloc.in > cp sparse_falloc.in sparse_falloc.out > cmp sparse_falloc.in sparse_falloc.out Ouch. Thanks for the analysis and patch. It looks correct. I'll analyze further before applying. thanks! Pádraig. From debbugs-submit-bounces@debbugs.gnu.org Fri Oct 30 14:55:17 2015 Received: (at 21790) by debbugs.gnu.org; 30 Oct 2015 18:55:17 +0000 Received: from localhost ([127.0.0.1]:46502 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1ZsEpY-0008Rp-NO for submit@debbugs.gnu.org; Fri, 30 Oct 2015 14:55:17 -0400 Received: from mail1.vodafone.ie ([213.233.128.43]:46255) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1ZsEpE-0008Qq-52 for 21790@debbugs.gnu.org; Fri, 30 Oct 2015 14:55:14 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AtQaALC7M1ZtTElv/2dsb2JhbABeKAECgj5SHzRvgl24NIYJHYV8AoE1TAEBAQEBAYELhDYBAQR5EAsNAQoJFg8JAwIBAgFFBgEMCAEBiDABuGWLKgEBAQEBAQEDAQEBAQEVCYV8hXmJQAWWQ4JSgWFqiWGHVwySfmOEBD40hX4BAQE Received: from unknown (HELO localhost.localdomain) ([109.76.73.111]) by mail1.vodafone.ie with ESMTP; 30 Oct 2015 18:54:54 +0000 Subject: Re: bug#21790: [PATCH] coreutils/cp: handle EOF extents correctly To: Dmitry Monakhov , 21790@debbugs.gnu.org References: <1446195759-32225-1-git-send-email-dmonakhov@openvz.org> <5633A160.1030405@draigBrady.com> From: =?UTF-8?Q?P=c3=a1draig_Brady?= X-Enigmail-Draft-Status: N1110 Message-ID: <5633BCFD.3070302@draigBrady.com> Date: Fri, 30 Oct 2015 18:54:53 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <5633A160.1030405@draigBrady.com> Content-Type: multipart/mixed; boundary="------------030501090103080105090206" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 21790 Cc: vvs@parallels.com, devel@openvz.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) This is a multi-part message in MIME format. --------------030501090103080105090206 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit On 30/10/15 16:57, Pádraig Brady wrote: > On 30/10/15 09:02, Dmitry Monakhov wrote: >> fallocate can allocate extens beyond EOF via FALLOC_FL_KEEP_SIZE. >> Currenly sparse engine tries to copy such extents which is wrong and >> result in silent data corruption (leave file with incorrect size). >> >> ##TESTCASE >> echo blabla > sparse_falloc.in >> truncate -s 2M sparse_falloc.in >> fallocate -n -o 4M -l 1M sparse_falloc.in >> cp sparse_falloc.in sparse_falloc.out >> cmp sparse_falloc.in sparse_falloc.out > > Ouch. Thanks for the analysis and patch. > It looks correct. I'll analyze further before applying. This doesn't handle the --sparse==never case (which is broken with and without the patch). Also one might have an extent spanning the file size boundary, in which this patch could miss the remaining data? Also currently if the source file is being extended while being copied, we continue to read, whereas we now wont. Theoretically this is an issue when st_size doesn't match what's available to be read, though maybe not a practical issue since we don't use this path for a zero st_size, nor sources that don't support fiemap anyway. This might be an opportune time to rip out the fiemap stuff in favor of SEEK{DATA,HOLE} anyway, which I intended to do during this cycle. For fix backporting sake though, we should apply the minimal fix to the fiemap code first. Attached is the current minimally tested patch. thanks, Pádraig. --------------030501090103080105090206 Content-Type: text/x-patch; name="cp-trailing-extents.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="cp-trailing-extents.patch" diff --git a/src/copy.c b/src/copy.c index edf022e..f251f32 100644 --- a/src/copy.c +++ b/src/copy.c @@ -431,6 +431,14 @@ extent_copy (int src_fd, int dest_fd, char *buf, siz= e_t buf_size, ext_len =3D 0; } =20 + /* Truncate extent to EOF. */ + if (ext_start + ext_len >=3D src_total_size) + { + if (src_total_size < ext_start) + ext_start =3D src_total_size; + ext_len =3D src_total_size - ext_start; + } + ext_hole_size =3D ext_start - last_ext_start - last_ext_len; =20 wrote_hole_at_eof =3D false; @@ -493,14 +501,17 @@ extent_copy (int src_fd, int dest_fd, char *buf, si= ze_t buf_size, off_t n_read; empty_extent =3D false; last_ext_len =3D ext_len; + bool read_hole; =20 if ( ! sparse_copy (src_fd, dest_fd, buf, buf_size, sparse_mode =3D=3D SPARSE_ALWAYS ? hol= e_size: 0, true, src_name, dst_name, ext_len, &n_= read, - &wrote_hole_at_eof)) + &read_hole)) goto fail; =20 dest_pos =3D ext_start + n_read; + if (n_read) + wrote_hole_at_eof =3D read_hole; } =20 /* If the file ends with unwritten extents not accounted for i= n the --------------030501090103080105090206-- From debbugs-submit-bounces@debbugs.gnu.org Fri Nov 20 13:27:52 2015 Received: (at 21790-done) by debbugs.gnu.org; 20 Nov 2015 18:27:52 +0000 Received: from localhost ([127.0.0.1]:46391 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1ZzqPW-0000Gl-Q0 for submit@debbugs.gnu.org; Fri, 20 Nov 2015 13:27:51 -0500 Received: from mail2.vodafone.ie ([213.233.128.44]:1576) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1ZzqPA-0000G9-IT for 21790-done@debbugs.gnu.org; Fri, 20 Nov 2015 13:27:48 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AoIFAJplT1ZtTdHX/2dsb2JhbABegmlSU2+CX7ULhQyEDR2CQoMtAQICgUhMAQEBAQEBgQuENAEBAQQSZxALDQEDAwECAQkWDwkDAgECAT0IBgEMBgIBAQURCIgQAaYekBiKPwEBAQEBAQEBAQEBAQEBAQEBARIJhVmFeYR7hD4Fh0aFV4hvQIJYgWFqiWlJg3eDGQyEEIc0g1SDcmOEBD40g2EkgSYBAQE Received: from unknown (HELO localhost.localdomain) ([109.77.209.215]) by mail2.vodafone.ie with ESMTP; 20 Nov 2015 18:26:57 +0000 Subject: Re: bug#21790: [PATCH] coreutils/cp: handle EOF extents correctly To: Dmitry Monakhov , 21790-done@debbugs.gnu.org References: <1446195759-32225-1-git-send-email-dmonakhov@openvz.org> <5633A160.1030405@draigBrady.com> <5633BCFD.3070302@draigBrady.com> From: =?UTF-8?Q?P=c3=a1draig_Brady?= Message-ID: <564F65F1.2080504@draigBrady.com> Date: Fri, 20 Nov 2015 18:26:57 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <5633BCFD.3070302@draigBrady.com> Content-Type: multipart/mixed; boundary="------------020304060909060601010805" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 21790-done Cc: vvs@parallels.com, devel@openvz.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) This is a multi-part message in MIME format. --------------020304060909060601010805 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit On 30/10/15 18:54, Pádraig Brady wrote: > On 30/10/15 16:57, Pádraig Brady wrote: >> On 30/10/15 09:02, Dmitry Monakhov wrote: >>> fallocate can allocate extens beyond EOF via FALLOC_FL_KEEP_SIZE. >>> Currenly sparse engine tries to copy such extents which is wrong and >>> result in silent data corruption (leave file with incorrect size). >>> >>> ##TESTCASE >>> echo blabla > sparse_falloc.in >>> truncate -s 2M sparse_falloc.in >>> fallocate -n -o 4M -l 1M sparse_falloc.in >>> cp sparse_falloc.in sparse_falloc.out >>> cmp sparse_falloc.in sparse_falloc.out >> >> Ouch. Thanks for the analysis and patch. >> It looks correct. I'll analyze further before applying. > > This doesn't handle the --sparse==never case > (which is broken with and without the patch). > > Also one might have an extent spanning the file size boundary, > in which this patch could miss the remaining data? > > Also currently if the source file is being extended > while being copied, we continue to read, whereas we now wont. > Theoretically this is an issue when st_size doesn't match > what's available to be read, though maybe not a practical issue > since we don't use this path for a zero st_size, nor > sources that don't support fiemap anyway. > > This might be an opportune time to rip out the fiemap stuff > in favor of SEEK{DATA,HOLE} anyway, which I intended to do > during this cycle. For fix backporting sake though, > we should apply the minimal fix to the fiemap code first. > > Attached is the current minimally tested patch. I'll apply the attached in your name soon. Please review. thanks, Pádraig. --------------020304060909060601010805 Content-Type: text/x-patch; name="cp-sparse-extents.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="cp-sparse-extents.patch" >From f79f3a83e4e43b70d2a209e6d1ec8185dac16625 Mon Sep 17 00:00:00 2001 From: Dmitry Monakhov Date: Fri, 30 Oct 2015 22:04:46 +0000 Subject: [PATCH] copy: fix copying of extents beyond the apparent file size fallocate can allocate extents beyond EOF via FALLOC_FL_KEEP_SIZE. Where there is a gap (hole) between the extents, and EOF is within that gap, the final hole wasn't reproduced, resulting in silent data corruption in the copied file (size too small). * src/copy.c (extent_copy): Ensure we don't process extents beyond the apparent file size, since processing and allocating those is not currently supported. * tests/cp/fiemap-extents.sh: Rename from tests/cp/fiemap-empty.sh and renable parts checking the extents at and beyond EOF. * tests/local.mk: Reference the renamed test. * NEWS: Mention the bug fix. --- NEWS | 5 +++ src/copy.c | 18 +++++++- tests/cp/fiemap-empty.sh | 102 --------------------------------------------- tests/cp/fiemap-extents.sh | 81 +++++++++++++++++++++++++++++++++++ tests/local.mk | 2 +- 5 files changed, 104 insertions(+), 104 deletions(-) delete mode 100755 tests/cp/fiemap-empty.sh create mode 100755 tests/cp/fiemap-extents.sh diff --git a/NEWS b/NEWS index fc5e927..814e29d 100644 --- a/NEWS +++ b/NEWS @@ -4,6 +4,11 @@ GNU coreutils NEWS -*- outline -*- ** Bug fixes + cp now correctly copies files with a hole at the end of the file, + and extents allocated beyond the apparent size of the file. + That combination resulted in the trailing hole not being reproduced. + [bug introduced in coreutils-8.10] + ls no longer prematurely wraps lines when printing short file names. [bug introduced in 5.1.0] diff --git a/src/copy.c b/src/copy.c index dc1cd29..4fb3fb2 100644 --- a/src/copy.c +++ b/src/copy.c @@ -432,6 +432,19 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, ext_len = 0; } + /* Truncate extent to EOF. + Generally this will trigger with an extent starting after + src_total_size, and result in creating a hole or zeros until EOF. + Though in a file with changing extents since src_total_size + was determined, we might have an extent spanning that size, + in which case we'll only copy data up to that size. */ + if (src_total_size < ext_start + ext_len) + { + if (src_total_size < ext_start) + ext_start = src_total_size; + ext_len = src_total_size - ext_start; + } + ext_hole_size = ext_start - last_ext_start - last_ext_len; wrote_hole_at_eof = false; @@ -495,14 +508,17 @@ extent_copy (int src_fd, int dest_fd, char *buf, size_t buf_size, off_t n_read; empty_extent = false; last_ext_len = ext_len; + bool read_hole; if ( ! sparse_copy (src_fd, dest_fd, buf, buf_size, sparse_mode == SPARSE_ALWAYS ? hole_size: 0, true, src_name, dst_name, ext_len, &n_read, - &wrote_hole_at_eof)) + &read_hole)) goto fail; dest_pos = ext_start + n_read; + if (n_read) + wrote_hole_at_eof = read_hole; } /* If the file ends with unwritten extents not accounted for in the diff --git a/tests/cp/fiemap-empty.sh b/tests/cp/fiemap-empty.sh deleted file mode 100755 index b3b2cd7..0000000 --- a/tests/cp/fiemap-empty.sh +++ /dev/null @@ -1,102 +0,0 @@ -#!/bin/sh -# Test cp reads unwritten extents efficiently - -# Copyright (C) 2011-2015 Free Software Foundation, Inc. - -# This program is free software: you can redistribute it and/or modify -# it under the terms of the GNU General Public License as published by -# the Free Software Foundation, either version 3 of the License, or -# (at your option) any later version. - -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# GNU General Public License for more details. - -# You should have received a copy of the GNU General Public License -# along with this program. If not, see . - -. "${srcdir=.}/tests/init.sh"; path_prepend_ ./src -print_ver_ cp - -# FIXME: enable any part of this test that is still relevant, -# or, if none are relevant (now that cp does not handle unwritten -# extents), just remove the test altogether. -# Note also if checking allocations may need to sync first on BTRFS at least -skip_ 'disabled for now' - -touch fiemap_chk -fiemap_capable_ fiemap_chk || - skip_ 'this file system lacks FIEMAP support' -rm fiemap_chk - -# TODO: rather than requiring $(fallocate), possible add -# this functionality to truncate --alloc -fallocate --help >/dev/null || skip_ 'The fallocate utility is required' -fallocate -l 1 -n falloc.test || - skip_ 'this file system lacks FALLOCATE support' -rm falloc.test - -# Require more space than we'll actually use, so that -# tests run in parallel do not run out of space. -# Otherwise, with inadequate space, simply running the following -# fallocate command would induce a temporary disk-full condition, -# which would cause failure of unrelated tests run in parallel. -require_file_system_bytes_free_ 800000000 - -fallocate -l 600MiB space.test || - skip_ 'this test needs at least 600MiB free space' - -# Disable this test on old BTRFS (e.g. Fedora 14) -# which reports ordinary extents for unwritten ones. -filefrag space.test || skip_ 'the 'filefrag' utility is missing' -filefrag -v space.test | grep -F 'unwritten' > /dev/null || - skip_ 'this file system does not report empty extents as "unwritten"' - -rm space.test - -# Ensure we read a large empty file quickly -fallocate -l 300MiB empty.big || framework_failure_ -timeout 3 cp --sparse=always empty.big cp.test || fail=1 -test $(stat -c %s empty.big) = $(stat -c %s cp.test) || fail=1 -rm empty.big cp.test - -# Ensure we handle extents beyond file size correctly. -# Note until we support fallocate, we will not maintain -# the file allocation. FIXME: amend this test when fallocate is supported. -fallocate -l 10MiB -n unwritten.withdata || framework_failure_ -dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=unwritten.withdata -cp unwritten.withdata cp.test || fail=1 -test $(stat -c %s unwritten.withdata) = $(stat -c %s cp.test) || fail=1 -cmp unwritten.withdata cp.test || fail=1 -rm unwritten.withdata cp.test - -# The following to generate unaccounted extents followed by a hole, is not -# supported by ext4 at least. The ftruncate discards all extents not -# accounted for in the size. -# fallocate -l 10MiB -n unacc.withholes -# dd count=10 if=/dev/urandom conv=notrunc iflag=fullblock of=unacc.withholes -# truncate -s20M unacc.withholes - -# Ensure we handle a hole after empty extents correctly. -# Since all extents are accounted for in the size, -# we can maintain the allocation independently from -# fallocate() support. -fallocate -l 10MiB empty.withholes -truncate -s 20M empty.withholes -sectors_per_block=$(expr $(stat -c %o .) / 512) -cp empty.withholes cp.test || fail=1 -test $(stat -c %s empty.withholes) = $(stat -c %s cp.test) || fail=1 -# These are usually equal but can vary by an IO block due to alignment -alloc_diff=$(expr $(stat -c %b empty.withholes) - $(stat -c %b cp.test)) -alloc_diff=$(echo $alloc_diff | tr -d -- -) # abs() -test $alloc_diff -le $sectors_per_block || fail=1 -# Again with SPARSE_ALWAYS -cp --sparse=always empty.withholes cp.test || fail=1 -test $(stat -c %s empty.withholes) = $(stat -c %s cp.test) || fail=1 -# cp.test should take 0 space, but allowing for some systems -# that store default extended attributes in data blocks -test $(stat -c %b cp.test) -le $sectors_per_block || fail=1 -rm empty.withholes cp.test - -Exit $fail diff --git a/tests/cp/fiemap-extents.sh b/tests/cp/fiemap-extents.sh new file mode 100755 index 0000000..55ec5df --- /dev/null +++ b/tests/cp/fiemap-extents.sh @@ -0,0 +1,81 @@ +#!/bin/sh +# Test cp handles extents correctly + +# Copyright (C) 2011-2015 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + +. "${srcdir=.}/tests/init.sh"; path_prepend_ ./src +print_ver_ cp + +require_sparse_support_ + +touch fiemap_chk +fiemap_capable_ fiemap_chk || + skip_ 'this file system lacks FIEMAP support' +rm fiemap_chk + +fallocate --help >/dev/null || skip_ 'The fallocate utility is required' +touch falloc.test || framework_failure_ +fallocate -l 1 -o 0 -n falloc.test || + skip_ 'this file system lacks FALLOCATE support' +rm falloc.test + +# We don't currently handle unwritten extents specially +if false; then +# Require more space than we'll actually use, so that +# tests run in parallel do not run out of space. +# Otherwise, with inadequate space, simply running the following +# fallocate command would induce a temporary disk-full condition, +# which would cause failure of unrelated tests run in parallel. +require_file_system_bytes_free_ 800000000 + +fallocate -l 600MiB space.test || + skip_ 'this test needs at least 600MiB free space' + +# Disable this test on old BTRFS (e.g. Fedora 14) +# which reports ordinary extents for unwritten ones. +filefrag space.test || skip_ 'the 'filefrag' utility is missing' +filefrag -v space.test | grep -F 'unwritten' > /dev/null || + skip_ 'this file system does not report empty extents as "unwritten"' + +rm space.test + +# Ensure we read a large empty file quickly +fallocate -l 300MiB empty.big || framework_failure_ +timeout 3 cp --sparse=always empty.big cp.test || fail=1 +test $(stat -c %s empty.big) = $(stat -c %s cp.test) || fail=1 +rm empty.big cp.test +fi + +# Ensure we handle extents beyond file size correctly. +# Note until we support fallocate, we will not maintain +# the file allocation. FIXME: amend this test if fallocate is supported. +# Note currently this only uses fiemap logic when the allocation (-l) +# is smaller than the size, thus identifying the file as sparse. +# Note the '-l 1' case is an effective noop, and just checks +# a file with a trailing hole is copied correctly. +for sparse_mode in always auto never; do + for alloc in '-l 4MiB ' '-l 1MiB -o 4MiB' '-l 1'; do + dd count=10 if=/dev/urandom iflag=fullblock of=unwritten.withdata + truncate -s 2MiB unwritten.withdata || framework_failure_ + fallocate $alloc -n unwritten.withdata || framework_failure_ + cp --sparse=$sparse_mode unwritten.withdata cp.test || fail=1 + test $(stat -c %s unwritten.withdata) = $(stat -c %s cp.test) || fail=1 + cmp unwritten.withdata cp.test || fail=1 + rm unwritten.withdata cp.test + done +done + +Exit $fail diff --git a/tests/local.mk b/tests/local.mk index adf96f0..89fdbb0 100644 --- a/tests/local.mk +++ b/tests/local.mk @@ -446,7 +446,7 @@ all_tests = \ tests/cp/existing-perm-dir.sh \ tests/cp/existing-perm-race.sh \ tests/cp/fail-perm.sh \ - tests/cp/fiemap-empty.sh \ + tests/cp/fiemap-extents.sh \ tests/cp/fiemap-FMR.sh \ tests/cp/fiemap-perf.sh \ tests/cp/fiemap-2.sh \ -- 2.5.0 --------------020304060909060601010805-- From unknown Fri Jun 20 07:13:22 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sat, 19 Dec 2015 12:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator