GNU bug report logs - #75590
ports.test: "SEEK_DATA while in hole" fails on some hosts

Previous Next

To reply to this bug, email your comments to 75590 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Rob Browning <rlb <at> defaultvalue.org>
To: bug-guile <at> gnu.org
Subject: ports.test: "SEEK_DATA while in hole" fails on some hosts
Date: Wed, 15 Jan 2025 11:30:49 -0600

This test failed on one of the debian buildds[1] like this:

  Running popen.test
  Backtrace:
             4 (primitive-load "/build/reproducible-path/guile-3.0-3.0.10+really3.0.10/test-suite/tests/ports.test")
  In ice-9/eval.scm:
      619:8  3 (_ #(#(#<directory (test-suite test-ports) 7fff9a860460> "/build/reproducible-path/guile-3.0-3.0.10+really3.0.10/ports-test.tmp") #<closed: file 7fff959be8c0>))
      619:8  2 (_ #(#(#(#<directory (test-suite lib) 7fff9a8d80a0> #<variable 7fff9a90a5d0 value: #t>) "SEEK_DATA while in hole" #t #<procedure 7fff95a17c80 at ice-9/eval.scm:330:13 ()>) ("ports.test" "SEEK_DATA while in hole")))
  In ice-9/boot-9.scm:
     260:13  1 (for-each #<procedure 7fff95a17240 at ice-9/eval.scm:333:13 (a)> _)
  In ice-9/eval.scm:
      619:8  0 (_ #(#(#<directory (test-suite lib automake) 7fff9a860820> (fail ("ports.test" "SEEK_DATA while in hole") expected-value 4096 actual-value 10))))

  ice-9/eval.scm:619:8: Throw to key `match-error' with args `("match" "no matching pattern" (fail ("ports.test" "SEEK_DATA while in hole") expected-value 4096 actual-value 10))'.
  Running ports.test
  FAIL: ports.test: SEEK_DATA while in hole - arguments: (expected-value 4096 actual-value 10)

I suspect it's because that buildd is running in an unshare/chroot with
a filesystem that just doesn't support the seek data/hole semantics we
expect, and it sounds like returning the seek offset (here 10) is
allowed (at least by Debian's lseek(2)):

  In the simplest implementation, a filesystem can support the
  operations by making SEEK_HOLE always return the offset of the end of
  the file, and making SEEK_DATA always return offset (i.e., even if the
  location referred to by offset is a hole, it can be considered to
  consist of data that is a sequence of zeros).

How we might want to fix this depends of course on exactly what the test
is trying to verify, but if it's trying to verify that Guile's functions
work properly when the filesystem does support the semantics we expect,
then we could consider limiting the tests to relevant filesystems
(e.g. ext4, ...).

In that case, while I don't know of a simple way to detect the type of
the test filesystem, if it's useful, bup currently uses this to handle
some platforms
(https://codeberg.org/bup/bup/src/branch/main/dev/path-fs):

  #!/usr/bin/env bash

  set -ueo pipefail

  kernel="$(uname -s)"
  case "$kernel" in
      NetBSD)
          fs() { df -G "$1" | sed -En 's/.* ([^ ]*) fstype.*/\1/p'; }
          ;;
      SunOS)
          fs() { df -g "$1" | sed -En 's/.* ([^ ]*) fstype.*/\1/p'; }
          ;;
      *)
          fs() { df -T "$1" | awk 'END{print $2}'; }
  esac

  while test $# -ne 0; do
      fs "$1"
      shift
  done

[1] From https://buildd.debian.org/status/fetch.php?pkg=guile-3.0&arch=ppc64el&ver=3.0.10%2Breally3.0.10-1&stamp=1736918931&raw=0

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Message #8 received at 75590 <at> debbugs.gnu.org (full text, mbox):

From: Mikael Djurfeldt <mikael <at> djurfeldt.com>
To: Rob Browning <rlb <at> defaultvalue.org>
Cc: Mikael Djurfeldt <mikael <at> djurfeldt.com>, 75590 <at> debbugs.gnu.org
Subject: Re: bug#75590: ports.test: "SEEK_DATA while in hole" fails on some
 hosts
Date: Fri, 17 Jan 2025 15:09:35 +0100

[Message part 1 (text/plain, inline)]

We could, in addition to "stat", provide "statfs" (on systems that provide
it).

The test could then first test for the presence of "statfs" and then check
if the filesystem is ext4.

Best regards,
Mikael

On Wed, Jan 15, 2025 at 6:32 PM Rob Browning <rlb <at> defaultvalue.org> wrote:

>
> This test failed on one of the debian buildds[1] like this:
>
>   Running popen.test
>   Backtrace:
>              4 (primitive-load
> "/build/reproducible-path/guile-3.0-3.0.10+really3.0.10/test-suite/tests/ports.test")
>   In ice-9/eval.scm:
>       619:8  3 (_ #(#(#<directory (test-suite test-ports) 7fff9a860460>
> "/build/reproducible-path/guile-3.0-3.0.10+really3.0.10/ports-test.tmp")
> #<closed: file 7fff959be8c0>))
>       619:8  2 (_ #(#(#(#<directory (test-suite lib) 7fff9a8d80a0>
> #<variable 7fff9a90a5d0 value: #t>) "SEEK_DATA while in hole" #t
> #<procedure 7fff95a17c80 at ice-9/eval.scm:330:13 ()>) ("ports.test"
> "SEEK_DATA while in hole")))
>   In ice-9/boot-9.scm:
>      260:13  1 (for-each #<procedure 7fff95a17240 at ice-9/eval.scm:333:13
> (a)> _)
>   In ice-9/eval.scm:
>       619:8  0 (_ #(#(#<directory (test-suite lib automake) 7fff9a860820>
> (fail ("ports.test" "SEEK_DATA while in hole") expected-value 4096
> actual-value 10))))
>
>   ice-9/eval.scm:619:8: Throw to key `match-error' with args `("match" "no
> matching pattern" (fail ("ports.test" "SEEK_DATA while in hole")
> expected-value 4096 actual-value 10))'.
>   Running ports.test
>   FAIL: ports.test: SEEK_DATA while in hole - arguments: (expected-value
> 4096 actual-value 10)
>
> I suspect it's because that buildd is running in an unshare/chroot with
> a filesystem that just doesn't support the seek data/hole semantics we
> expect, and it sounds like returning the seek offset (here 10) is
> allowed (at least by Debian's lseek(2)):
>
>   In the simplest implementation, a filesystem can support the
>   operations by making SEEK_HOLE always return the offset of the end of
>   the file, and making SEEK_DATA always return offset (i.e., even if the
>   location referred to by offset is a hole, it can be considered to
>   consist of data that is a sequence of zeros).
>
> How we might want to fix this depends of course on exactly what the test
> is trying to verify, but if it's trying to verify that Guile's functions
> work properly when the filesystem does support the semantics we expect,
> then we could consider limiting the tests to relevant filesystems
> (e.g. ext4, ...).
>
> In that case, while I don't know of a simple way to detect the type of
> the test filesystem, if it's useful, bup currently uses this to handle
> some platforms
> (https://codeberg.org/bup/bup/src/branch/main/dev/path-fs):
>
>   #!/usr/bin/env bash
>
>   set -ueo pipefail
>
>   kernel="$(uname -s)"
>   case "$kernel" in
>       NetBSD)
>           fs() { df -G "$1" | sed -En 's/.* ([^ ]*) fstype.*/\1/p'; }
>           ;;
>       SunOS)
>           fs() { df -g "$1" | sed -En 's/.* ([^ ]*) fstype.*/\1/p'; }
>           ;;
>       *)
>           fs() { df -T "$1" | awk 'END{print $2}'; }
>   esac
>
>   while test $# -ne 0; do
>       fs "$1"
>       shift
>   done
>
> [1] From
> https://buildd.debian.org/status/fetch.php?pkg=guile-3.0&arch=ppc64el&ver=3.0.10%2Breally3.0.10-1&stamp=1736918931&raw=0
>
> Thanks
> --
> Rob Browning
> rlb @defaultvalue.org and @debian.org
> GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
> GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4
>
>
>
>

[Message part 2 (text/html, inline)]

Message #11 received at 75590 <at> debbugs.gnu.org (full text, mbox):

From: Rob Browning <rlb <at> defaultvalue.org>
To: Mikael Djurfeldt <mikael <at> djurfeldt.com>
Cc: Mikael Djurfeldt <mikael <at> djurfeldt.com>, 75590 <at> debbugs.gnu.org
Subject: Re: bug#75590: ports.test: "SEEK_DATA while in hole" fails on some
 hosts
Date: Sun, 19 Jan 2025 14:14:32 -0600

[Message part 1 (text/plain, inline)]

Mikael Djurfeldt <mikael <at> djurfeldt.com> writes:

> We could, in addition to "stat", provide "statfs" (on systems that provide
> it).
>
> The test could then first test for the presence of "statfs" and then check
> if the filesystem is ext4.

Here's what I've done in Debian for now after double-checking the
test on a few filesystems:

[0001-Skip-ports.test-seek-tests-unless-filesystem-looks-s.patch (text/x-diff, inline)]

From ea1750ad7f27593ee42ca48c74cc5b3e3cdd2d1d Mon Sep 17 00:00:00 2001
From: Rob Browning <rlb <at> defaultvalue.org>
Date: Wed, 15 Jan 2025 12:54:02 -0600
Subject: [PATCH 1/1] Skip ports.test seek tests unless filesystem looks
 suitable

For now, only test on filesystems that we know have the expected
semantics, i.e. btrfs, ext4, and xfs.
---
 test-suite/tests/ports.test | 32 +++++++++++++++++++++-----------
 1 file changed, 21 insertions(+), 11 deletions(-)

diff --git a/test-suite/tests/ports.test b/test-suite/tests/ports.test
index bec5e356c5..baa9edb700 100644
--- a/test-suite/tests/ports.test
+++ b/test-suite/tests/ports.test
@@ -39,9 +39,23 @@
 (define (test-file)
   (data-file-name "ports-test.tmp"))
 
-(define (skip-on-darwin)
-  (when (string-ci=? "darwin" (utsname:sysname (uname)))
-    (throw 'untested)))
+(define (skip-unless-fs-handles-holes-as-expected)
+  ;; For now only allow filesystems that should have the seek hole/data
+  ;; semantics the tests expect.  Filesystems vary both in how they
+  ;; handle sparseness in general (e.g. granularity), how they handle
+  ;; SEEK_DATA and SEEK_HOLE (see lseek(2) for some related info), and
+  ;; even how quickly they reflect changes.  du's output, for example,
+  ;; may not immediately reflect sparseness changes (previously observed
+  ;; on btrfs and zfs).
+  (let* ((p (open-input-pipe "debian/bin/path-fs ."))
+         (fs (read-all p)))
+    (close p)
+    (when (string=? "\n" fs)
+      (error "unexpected output from debian/bin/path-fs" fs))
+    (or (string=? "btrfs\n" fs)
+        (string=? "ext4\n" fs)
+        (string=? "xfs\n" fs)
+        (throw 'untested))))
 
 
 ;;;; Some general utilities for testing ports.
@@ -189,7 +203,6 @@
     (close-port iport))
   (delete-file filename))
 
-;;; Note: Holes are weird on Darwin.
 (let* ((file (test-file))
        (port (open-output-file file)))
   (seek port 4096 SEEK_SET)
@@ -198,15 +211,12 @@
 
   (pass-if-equal "size of sparse file"
       4100
-    ;; XXX: On macOS, APFS does support sparse files, they do not behave
-    ;; like on Linux.  Skip these tests on macOS.
-    (skip-on-darwin)
-
+    (skip-unless-fs-handles-holes-as-expected)
     (stat:size (stat file)))
 
   (pass-if-equal "SEEK_DATA while on data"
       4096
-    (skip-on-darwin)
+    (skip-unless-fs-handles-holes-as-expected)
     (if (defined? 'SEEK_DATA)
         (call-with-input-file file
           (lambda (port)
@@ -219,7 +229,7 @@
 
   (pass-if-equal "SEEK_DATA while in hole"
       4096
-    (skip-on-darwin)
+    (skip-unless-fs-handles-holes-as-expected)
     (if (defined? 'SEEK_DATA)
         (call-with-input-file file
           (lambda (port)
@@ -232,7 +242,7 @@
 
   (pass-if-equal "SEEK_HOLE while in hole"
       10
-    (skip-on-darwin)
+    (skip-unless-fs-handles-holes-as-expected)
     (if (defined? 'SEEK_HOLE)
         (call-with-input-file file
           (lambda (port)
-- 
2.45.2

[Message part 3 (text/plain, inline)]

But happy to help with whatever solution we prefer in guile itself.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Message #14 received at 75590 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Rob Browning <rlb <at> defaultvalue.org>
Cc: Mikael Djurfeldt <mikael <at> djurfeldt.com>, 75590 <at> debbugs.gnu.org
Subject: Re: bug#75590: ports.test: "SEEK_DATA while in hole" fails on some
 hosts
Date: Fri, 28 Feb 2025 21:20:01 +0100

Hi Rob,

Rob Browning <rlb <at> defaultvalue.org> skribis:

> +(define (skip-unless-fs-handles-holes-as-expected)
> +  ;; For now only allow filesystems that should have the seek hole/data
> +  ;; semantics the tests expect.  Filesystems vary both in how they
> +  ;; handle sparseness in general (e.g. granularity), how they handle
> +  ;; SEEK_DATA and SEEK_HOLE (see lseek(2) for some related info), and
> +  ;; even how quickly they reflect changes.  du's output, for example,
> +  ;; may not immediately reflect sparseness changes (previously observed
> +  ;; on btrfs and zfs).
> +  (let* ((p (open-input-pipe "debian/bin/path-fs ."))
> +         (fs (read-all p)))

What does ‘path-fs’ do?  How can we adapt it for use in the test suite?

Thanks for investigating!

Ludo’.

Message #17 received at 75590 <at> debbugs.gnu.org (full text, mbox):

From: Rob Browning <rlb <at> defaultvalue.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Mikael Djurfeldt <mikael <at> djurfeldt.com>, 75590 <at> debbugs.gnu.org
Subject: Re: bug#75590: ports.test: "SEEK_DATA while in hole" fails on some
 hosts
Date: Fri, 28 Feb 2025 16:26:01 -0600

Ludovic Courtès <ludo <at> gnu.org> writes:

> What does ‘path-fs’ do?  How can we adapt it for use in the test suite?

Oh, right, that's here:

  https://salsa.debian.org/rlb/deb-guile/-/blob/deb/guile-3.0/d/sid/master/debian/bin/path-fs?ref_type=heads

via

  https://salsa.debian.org/rlb/deb-guile/-/commit/691c5ab5ca48ce69cce60dabd7b53d30cb600dda

It just reports the fs type for a given path, which doesn't appear to be
(portably) easy.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.