GNU bug report logs - #75405
gzip --synchronous doesn't work with musl

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 75405 in the body.
You can then email your comments to 75405 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Lasse Collin <lasse.collin <at> tukaani.org>
To: bug-gzip <at> gnu.org
Subject: gzip --synchronous doesn't work with musl
Date: Mon, 6 Jan 2025 15:29:22 +0200

[Message part 1 (text/plain, inline)]

On Linux/musl:

    $ echo foo > foo
    $ gzip --synchronous foo

    gzip: foo.gz: Bad file descriptor

From gzip.c:

      dfd = open (dfname, O_SEARCH | O_DIRECTORY);

...

        if ((synchronous
             && ((0 <= dfd && fdatasync (dfd) != 0 && errno != EINVAL)
                 || (fsync (ofd) != 0 && errno != EINVAL)))
            || close (ofd) != 0)
          write_error ();

In musl, O_SEARCH maps to Linux-specific O_PATH which doesn't allow
fsync or fdatasync. Gnulib's fcntl module's docs already warn about a
similar situation with fchmod.

As far as I understand this, O_SEARCH in POSIX is only meant for openat
and such APIs. Then it makes sense that O_SEARCH becomes O_PATH because
O_PATH can open a directory that lacks read permission. This way openat
can be used even for such directories.

Maybe there is no way to sync a directory that only has write and search
permissions. Luckily such a combination is rare.

A short fix is using O_RDONLY when --synchronous has been specified:

      dfd = open (dfname, (synchronous ? O_RDONLY : O_SEARCH)
                          | O_DIRECTORY);

I also tried opening a separate file descriptor for syncing (see the
attachment) but I think the above change is better. However, both have
a weakness that if the directory cannot be opened, --synchronous will
silently skip syncing the directory. It would be safer to show an error
if the directory cannot be opened when --synchronous has been specified.

-- 
Lasse Collin

[dfd_sync.patch (text/x-patch, attachment)]

Message #8 received at 75405 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Lasse Collin <lasse.collin <at> tukaani.org>
Cc: 75405 <at> debbugs.gnu.org
Subject: Re: bug#75405: gzip --synchronous doesn't work with musl
Date: Mon, 6 Jan 2025 10:02:41 -0800

On 2025-01-06 05:29, Lasse Collin wrote:

> In musl, O_SEARCH maps to Linux-specific O_PATH

That is a bug in musl. musl should not define O_SEARCH to O_PATH on 
Linux, because O_PATH is not a valid implementation of O_SEARCH. Or if 
musl wants to do some sort of approximate-but-invalid implementation to 
POSIX, a better approximation is "#define O_SEARCH O_RDONLY", which is 
what Gnulib does in lib/fcntl.in.h. The approximation "#define O_SEARCH 
O_PATH" is more likely to break real-world applications than the 
approximation "#define O_SEARCH O_RDONLY" is.


> As far as I understand this, O_SEARCH in POSIX is only meant for openat
> and such APIs.

No, O_SEARCH is well defined in POSIX to work in plain 'open'. See 
<https://pubs.opengroup.org/onlinepubs/9799919799/functions/open.html>.


Would the following Gnulib patch work around the musl bug?

diff --git a/lib/fcntl.in.h b/lib/fcntl.in.h
index 5f06c4fe10..44d02fb9fd 100644
--- a/lib/fcntl.in.h
+++ b/lib/fcntl.in.h
@@ -369,6 +369,11 @@ _GL_WARN_ON_USE (openat, "openat is not portable - "
 # define O_RSYNC 0
 #endif

+/* musl on GNU/Linux mistakenly has "#define O_SEARCH O_PATH".  */
+#if defined O_SEARCH && defined O_PATH && O_SEARCH == O_PATH
+# undef O_SEARCH
+#endif
+
 #ifndef O_SEARCH
 # define O_SEARCH O_RDONLY /* This is often close enough in older 
systems.  */
 #endif

Message #11 received at 75405 <at> debbugs.gnu.org (full text, mbox):

From: Lasse Collin <lasse.collin <at> tukaani.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 75405 <at> debbugs.gnu.org
Subject: Re: bug#75405: gzip --synchronous doesn't work with musl
Date: Tue, 7 Jan 2025 20:22:56 +0200

On 2025-01-06 Paul Eggert wrote:
> On 2025-01-06 05:29, Lasse Collin wrote:
> > As far as I understand this, O_SEARCH in POSIX is only meant for
> > openat and such APIs.  
> 
> No, O_SEARCH is well defined in POSIX to work in plain 'open'. See 
> <https://pubs.opengroup.org/onlinepubs/9799919799/functions/open.html>.

I had looked at it a few times recently but somehow managed to convince
myself that "search only" means something else than it actually does.
Thanks for steering me back to the right path!

> > In musl, O_SEARCH maps to Linux-specific O_PATH  
> 
> That is a bug in musl. musl should not define O_SEARCH to O_PATH on 
> Linux, because O_PATH is not a valid implementation of O_SEARCH. Or
> if musl wants to do some sort of approximate-but-invalid
> implementation to POSIX, a better approximation is "#define O_SEARCH
> O_RDONLY", which is what Gnulib does in lib/fcntl.in.h. The
> approximation "#define O_SEARCH O_PATH" is more likely to break
> real-world applications than the approximation "#define O_SEARCH
> O_RDONLY" is.

musl's approach has the benefit that O_SEARCH works on dirs that lack
read permission but, now that I have researched it a bit more, I see it
indeed creates a bunch of new issues. For example, if a directory lacks
search permission, opening still succeeds. If one wants to use O_PATH
to create file descriptors for *at functions, one needs to write the
code slightly differently than when using O_SEARCH. So I agree that
O_RDONLY is a better approximation for O_SEARCH in the real world.

However, one thing that isn't an issue in musl is fchmod which is
currently documented in Gnulib's doc/posix-headers/fcntl.texi. fchmod
and a few others were fixed in 2013 by using /proc/self/fd.[1] Perhaps
Gnulib's docs could be updated to remove fchmod and add fsync and
fdatasync. Even if proc-based emulation was added, I don't see how it
could be great if it needs to do open+fsync+close where the open with
O_RDONLY might fail. But I might be missing something (again).

[1] https://git.musl-libc.org/cgit/musl/commit/?id=9ca1f62b0c0d3e50480eb654ac941ff943ce0558

> Would the following Gnulib patch work around the musl bug?

It does, thanks!

Should it check for __linux__ too to be sure that the workaround won't
accidentally apply on some other kernel or libc that has O_PATH? Or
perhaps it's safe to assume that nothing else has O_SEARCH == O_PATH.
For example, FreeBSD has O_PATH but, as expected, it doesn't equal
O_SEARCH.

It could be useful to add a test for gzip --synchronous. "make check"
passed without the Gnulib patch too.

-- 
Lasse Collin

Message #14 received at 75405 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Lasse Collin <lasse.collin <at> tukaani.org>
Cc: 75405 <at> debbugs.gnu.org
Subject: Re: bug#75405: gzip --synchronous doesn't work with musl
Date: Wed, 8 Jan 2025 00:37:12 -0800

On 2025-01-07 10:22, Lasse Collin wrote:

> However, one thing that isn't an issue in musl is fchmod which is
> currently documented in Gnulib's doc/posix-headers/fcntl.texi. fchmod
> and a few others were fixed in 2013 by using /proc/self/fd.[1]

Is that a full fix, though? /proc might not be mounted.


> Gnulib's docs could be updated to remove fchmod and add fsync and
> fdatasync. Even if proc-based emulation was added, I don't see how it
> could be great if it needs to do open+fsync+close where the open with
> O_RDONLY might fail. But I might be missing something (again).
> 
> [1] https://git.musl-libc.org/cgit/musl/commit/?id=9ca1f62b0c0d3e50480eb654ac941ff943ce0558
> 
>> Would the following Gnulib patch work around the musl bug?
> 
> It does, thanks!

OK, I installed it.

> Should it check for __linux__ too to be sure that the workaround won't
> accidentally apply on some other kernel or libc that has O_PATH?

Anything else with O_SEARCH == O_PATH is likely to have similar 
problems. If not, we can cross that bridge when we come to it.


> It could be useful to add a test for gzip --synchronous.

Feel free....

Message #17 received at 75405 <at> debbugs.gnu.org (full text, mbox):

From: Lasse Collin <lasse.collin <at> tukaani.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 75405 <at> debbugs.gnu.org
Subject: Re: bug#75405: gzip --synchronous doesn't work with musl
Date: Wed, 8 Jan 2025 12:50:55 +0200

[Message part 1 (text/plain, inline)]

On 2025-01-08 Paul Eggert wrote:
> On 2025-01-07 10:22, Lasse Collin wrote:
> 
> > However, one thing that isn't an issue in musl is fchmod which is
> > currently documented in Gnulib's doc/posix-headers/fcntl.texi.
> > fchmod and a few others were fixed in 2013 by using
> > /proc/self/fd.[1]  
> 
> Is that a full fix, though? /proc might not be mounted.

No, it requires /proc, but it works most of the time...

> > It could be useful to add a test for gzip --synchronous.  
> 
> Feel free....

Attached. It's based on z-suffix.

-- 
Lasse Collin

[test-synchronous.patch (text/x-patch, attachment)]

Message #20 received at 75405 <at> debbugs.gnu.org (full text, mbox):

From: Lasse Collin <lasse.collin <at> tukaani.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 75405 <at> debbugs.gnu.org
Subject: Re: bug#75405: gzip --synchronous doesn't work with musl
Date: Thu, 9 Jan 2025 09:43:58 +0200

[Message part 1 (text/plain, inline)]

On 2025-01-08 Lasse Collin wrote:
> On 2025-01-08 Paul Eggert wrote:
> > On 2025-01-07 10:22, Lasse Collin wrote:
> > > It could be useful to add a test for gzip --synchronous.    
> > 
> > Feel free....  
> 
> Attached. It's based on z-suffix.

Syncing is the same in compression and decompression modes, so it's
reduntant to test --synchronous with both. Since my previous patch
wasn't committed yet, I attached a patch that is three lines shorter.

-- 
Lasse Collin

[test-synchronous2.patch (text/x-patch, attachment)]

Message #25 received at 75405-done <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Lasse Collin <lasse.collin <at> tukaani.org>
Cc: 75405-done <at> debbugs.gnu.org
Subject: Re: bug#75405: gzip --synchronous doesn't work with musl
Date: Fri, 10 Jan 2025 10:19:32 -0800

Thanks, I installed that test. Marking this gzip bug as done.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.