From debbugs-submit-bounces@debbugs.gnu.org Thu May 14 08:20:28 2015 Received: (at submit) by debbugs.gnu.org; 14 May 2015 12:20:28 +0000 Received: from localhost ([127.0.0.1]:44570 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Yss7n-0000vX-Eu for submit@debbugs.gnu.org; Thu, 14 May 2015 08:20:27 -0400 Received: from eggs.gnu.org ([208.118.235.92]:49321) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Yss7l-0000vJ-8N for submit@debbugs.gnu.org; Thu, 14 May 2015 08:20:25 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Yss7a-0008BN-Vz for submit@debbugs.gnu.org; Thu, 14 May 2015 08:20:19 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05 autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:41317) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Yss7a-0008BH-TZ for submit@debbugs.gnu.org; Thu, 14 May 2015 08:20:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34552) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Yss7V-0007MR-3K for bug-coreutils@gnu.org; Thu, 14 May 2015 08:20:14 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Yss7P-0007wg-84 for bug-coreutils@gnu.org; Thu, 14 May 2015 08:20:09 -0400 Received: from achille.seblu.net ([62.210.204.83]:42377) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Yss7P-0007vP-0j for bug-coreutils@gnu.org; Thu, 14 May 2015 08:20:03 -0400 Received: from murphy (slice.seblu.net [81.57.151.16]) by achille.seblu.net (Postfix) with ESMTPSA id 53BCA50C0772 for ; Thu, 14 May 2015 14:20:00 +0200 (CEST) Message-ID: <1431605994.5455.81.camel@seblu.net> Subject: Test du/2g.sh fails with btrfs From: =?ISO-8859-1?Q?S=E9bastien?= Luttringer To: bug-coreutils@gnu.org Date: Thu, 14 May 2015 14:19:54 +0200 Content-Type: multipart/signed; micalg="pgp-sha384"; protocol="application/pgp-signature"; boundary="=-z3+c0JrlDqVX80Zj5LZ8" X-Mailer: Evolution 3.16.1 Mime-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.1 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.1 (----) --=-z3+c0JrlDqVX80Zj5LZ8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, In the same spirit than my yesterday report[1], the following test fail on btrfs when it works on tmpfs. $ RUN_VERY_EXPENSIVE_TESTS=3Dyes ./tests/du/2g.sh creating a 2GB file: 100% complete--- exp 2015-05-14 14:03:36.124772165 +02= 00+++ out 2015-05-14 14:03:36.123772158 +0200@@ -1 +1 @@-~2M+1572864=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0big You can find the full output of the test suite here [2]. Regards, [1] http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D20570[2] http://pastebin= .com/akytxtts --=C2=A0S=C3=A9bastien "Seblu" Luttringer https://seblu.net | Twitter: @seblu42 GPG: 0x2072D77A --=-z3+c0JrlDqVX80Zj5LZ8 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCQAGBQJVVJLqAAoJEF9wJCj3DgkDMo4P/3a0S8gnXCJ4toHD/i/Px7Jn ToXy/lVL/AkJ3aLkDhVCndM9PuGDcAjPMtUjm4bxvYHi59kR1x24Tk+bduZldFz/ /Wabd0GkjbG5g7kqfFSUFYFbpQiq7Lum6jFYrK9pHRag7JnHakRku8sNsJ2KUMRT JHm5W/YPo0CHap2o6l3ykrEnY2JeXiTpi8RPyqmCl9H+8XAuxjRRi+uEsec7rovt uxgeGhPFCyKLxCG1p9qpX59fOMT+R6bOLJ2rEYGU5m2qNhK3Cqyxa5WFcOZ/vFOA npfqBq8xNxKFscAkLJiC9SFqnqoWT/9hZe5TuSx6A8820iiW3y5T+VIRrlbt6T7e YAYD/Bu26WzFkcp98Zo9Y15Z0W8BLu/F8GYKiMTdUiCgzTRyNb628ZKoANX6gS4l HUWA10jTokCovqKR6hbE9RXhktyBmzryQvZWQHdxnU+uLdWDvLd5ZPzjccJDY7yK jnaXWQ/pm/5MiExdXeXnSsbutv7h6qBQH+IZ+FvjuScISa51diUz0FbOgXdHNuVQ GQs10yNvm/RGvhbPVEpwsj2DY1kivhoDjUhVUEkN02JhusczC5iCtbFzaTYVYNMc 7+DYWVDyunaIN4ekRSTIrDvGwsoqZq28YY8F4ddlc/iWbVtetScuMKMQ9RlKRkjO 8N9WHDB8GwzCTh6VgH+u =TLoh -----END PGP SIGNATURE----- --=-z3+c0JrlDqVX80Zj5LZ8-- From debbugs-submit-bounces@debbugs.gnu.org Thu May 14 09:15:11 2015 Received: (at control) by debbugs.gnu.org; 14 May 2015 13:15:12 +0000 Received: from localhost ([127.0.0.1]:44593 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Yssyk-0002Hy-5P for submit@debbugs.gnu.org; Thu, 14 May 2015 09:15:11 -0400 Received: from mail1.vodafone.ie ([213.233.128.43]:58118) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1Yssyg-0002H5-EA; Thu, 14 May 2015 09:15:08 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ap0KALaeVFVtTLyB/2dsb2JhbABcgw9UWQWDHrkyiX2FdgKBPEwBAQEBAQGBC0EFg1wBAQEEI2YLDQQDAQIBCSECAg8CPggGAQwGAgEBiCwBAwWwHYVqnkwBAQEBAQEBAwEBAQEBAQEBARmLOoR0GIJogUUFhmOOB4FAXogvhheHfwkHhm4jggYfgVM9MQGCRQEBAQ Received: from unknown (HELO localhost.localdomain) ([109.76.188.129]) by mail1.vodafone.ie with ESMTP; 14 May 2015 14:14:59 +0100 Message-ID: <55549FD2.60704@draigBrady.com> Date: Thu, 14 May 2015 14:14:58 +0100 From: =?UTF-8?B?UMOhZHJhaWcgQnJhZHk=?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: =?UTF-8?B?U8OpYmFzdGllbiBMdXR0cmluZ2Vy?= , 20570@debbugs.gnu.org Subject: Re: bug#20574: Test du/2g.sh fails with btrfs References: <1431605994.5455.81.camel@seblu.net> In-Reply-To: <1431605994.5455.81.camel@seblu.net> Content-Type: multipart/mixed; boundary="------------050504060408070507010805" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) This is a multi-part message in MIME format. --------------050504060408070507010805 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit forcemerge 20570 20574 stop On 14/05/15 13:19, Sébastien Luttringer wrote: > Hello, > > In the same spirit than my yesterday report[1], the following test > fail on btrfs when it works on tmpfs. > > $ RUN_VERY_EXPENSIVE_TESTS=yes ./tests/du/2g.sh > creating a 2GB file: 100% complete--- exp 2015-05-14 14:03:36.124772165 +0200+++ out 2015-05-14 14:03:36.123772158 +0200@@ -1 +1 @@-~2M+1572864 big > You can find the full output of the test suite here [2]. > Regards, > [1] http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20570 > [2] http://pastebin.com/akytxtts Drats I only tested on a 1G BTRFS loopback partition and so missed that. Updated patch is attached. cheers, Pádraig. --------------050504060408070507010805 Content-Type: text/x-patch; name="btrfs-alloc.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="btrfs-alloc.patch" >From 7f0f58fdd68e7ca86dd5db7effb2c7f94adc4a08 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?P=C3=A1draig=20Brady?= Date: Thu, 14 May 2015 11:43:54 +0100 Subject: [PATCH] tests: fix async allocation race on BTRFS * tests/dd/sparse.sh: Sync files before checking allocations, which may be done asynchronously on NFS and BTRFS at least. Also mark this test as very expensive on remote file systems. * tests/du/2g.sh: Likewise, also use fallocate if available to efficiently allocate the large file, otherwise skip on remote file systems. * tests/tail-2/inotify-rotate-resources.sh: Use the more standard is_local_dir_() to check remoteness. * tests/cp/fiemap-empty.sh: Comment on the sync issue for this currerntly unused test. Fixes http://bugs.gnu.org/20570 --- tests/cp/fiemap-empty.sh | 1 + tests/dd/sparse.sh | 9 ++++++++- tests/du/2g.sh | 26 ++++++++++++++++---------- tests/tail-2/inotify-rotate-resources.sh | 2 +- 4 files changed, 26 insertions(+), 12 deletions(-) diff --git a/tests/cp/fiemap-empty.sh b/tests/cp/fiemap-empty.sh index a5eaac5..b3b2cd7 100755 --- a/tests/cp/fiemap-empty.sh +++ b/tests/cp/fiemap-empty.sh @@ -22,6 +22,7 @@ print_ver_ cp # FIXME: enable any part of this test that is still relevant, # or, if none are relevant (now that cp does not handle unwritten # extents), just remove the test altogether. +# Note also if checking allocations may need to sync first on BTRFS at least skip_ 'disabled for now' touch fiemap_chk diff --git a/tests/dd/sparse.sh b/tests/dd/sparse.sh index 4fdabad..626b405 100755 --- a/tests/dd/sparse.sh +++ b/tests/dd/sparse.sh @@ -17,6 +17,7 @@ . "${srcdir=.}/tests/init.sh"; path_prepend_ ./src print_ver_ dd +is_local_dir_ . || very_expensive_ require_sparse_support_ # Ensure basic sparse generation works @@ -50,6 +51,9 @@ dd if=/dev/zero of=file.in bs=1M count=1 seek=1 conv=notrunc || fail=1 kb_alloc() { du -k "$1"|cut -f1; } +# sync out data for async allocators like NFS/BTRFS +# sync file.in || fail=1 + # If our just-created input file appears to be too small, # skip the remaining tests. On at least Solaris 10 with NFS, # file.in is reported to occupy <= 1KiB for about 50 seconds @@ -58,7 +62,10 @@ if test $(kb_alloc file.in) -gt 3000; then # Ensure NUL blocks smaller than the block size are not made sparse. # Here, with a 2MiB block size, dd's conv=sparse must *not* introduce a hole. - dd if=file.in of=file.out bs=2M conv=sparse + dd if=file.in of=file.out bs=2M conv=sparse || fail=1 + + # Intermittently BTRFS returns 0 allocation for file.out unless synced + sync file.out || framework_failure_ test 2500 -lt $(kb_alloc file.out) || fail=1 # Note we recreate a sparse file first to avoid diff --git a/tests/du/2g.sh b/tests/du/2g.sh index 5f04488..12c2eed 100755 --- a/tests/du/2g.sh +++ b/tests/du/2g.sh @@ -24,7 +24,6 @@ print_ver_ du # Creating a 2GB file counts as 'very expensive'. very_expensive_ - # Get number of free kilobytes on current partition, so we can # skip this test if there is insufficient free space. free_kb=$(df -k --output=avail . | tail -n1) @@ -42,15 +41,22 @@ test $min_kb -lt $free_kb || } big=big -rm -f $big -{ - for i in $(seq 100); do - # Note: 2147483648 == 2^31. Print floor(2^31/100) per iteration. - printf %21474836s x || fail=1 - done - # After the final iteration, append the remaining 48 bytes. - printf %48s x || fail=1 -} > $big || fail=1 + +if ! fallocate -l2G $big; then + rm -f $big + { + is_local_dir_ . || skip 'Not writing 2GB data to remote' + for i in $(seq 100); do + # Note: 2147483648 == 2^31. Print floor(2^31/100) per iteration. + printf %21474836s x || fail=1 + done + # After the final iteration, append the remaining 48 bytes. + printf %48s x || fail=1 + } > $big || fail=1 +fi + +# The allocation may be done asynchronously (BTRFS for example) +sync $big || framework_failure_ du -k $big > out1 || fail=1 rm -f $big diff --git a/tests/tail-2/inotify-rotate-resources.sh b/tests/tail-2/inotify-rotate-resources.sh index c13959e..d6629a8 100755 --- a/tests/tail-2/inotify-rotate-resources.sh +++ b/tests/tail-2/inotify-rotate-resources.sh @@ -25,7 +25,7 @@ grep '^#define HAVE_INOTIFY 1' "$CONFIG_HEADER" >/dev/null \ require_strace_ 'inotify_add_watch,inotify_rm_watch' # Quickly skip on remote file systems -df --local . >/dev/null 2>&1 || +is_local_dir . >/dev/null 2>&1 || skip_ 'inotify not used on remote file system' check_tail_output() -- 2.3.4 --------------050504060408070507010805-- From unknown Tue Sep 09 13:17:18 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 12 Jun 2015 11:24:06 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator