GNU bug report logs - #77570
Build failures on core packages following daemon changes

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 77570 in the body.
You can then email your comments to 77570 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ada Stevenson <adanskana <at> gmail.com>
To: bug-guix <at> gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: Build failures on core packages following daemon changes
Date: Sun, 6 Apr 2025 06:05:55 +0000

Hi Guix,

Following commit 93474f92886fac8a2e5eb0eb3b388654246d640d (cc'ing Ludo' 
as the author of this commit), I have identified build failures in 
several packages as they have tests that require modifying the build 
tree in some way. These include scons, ruby and go. I'm sure there are 
other packages that are affected by this. We can patch out these tests 
if it is simply untenable to change the daemon in the pursuit of more 
security, but disabling tests is not so favorable in my opinion. I don't 
have any better suggestions regarding this but I think it should be 
bought to attention nonetheless, as these build failures have many 
dependent packages - if these packages need to be rebuilt for some 
reason it will be more pain to fix later than now.
-- 
Warmly,
Ada

Message #8 received at 77570 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Ada Stevenson <adanskana <at> gmail.com>
Cc: 77570 <at> debbugs.gnu.org
Subject: Re: bug#77570: Build failures on core packages following daemon
 changes
Date: Mon, 07 Apr 2025 12:14:50 +0200

Hi Ada,

Ada Stevenson <adanskana <at> gmail.com> skribis:

> Following commit 93474f92886fac8a2e5eb0eb3b388654246d640d (cc'ing
> Ludo' as the author of this commit), I have identified build failures
> in several packages as they have tests that require modifying the
> build tree in some way. These include scons, ruby and go. I'm sure
> there are other packages that are affected by this.

Could you explain how to reproduce the problem?

In particular, are you on Guix System or on a foreign distro?  In the
latter case, is guix-daemon running as an unprivileged user¹ or as root?
What version of guix-daemon is running, as shown by “pgrep -fa
guix-daemon”?

You mention scons, ruby, and go.  Is “guix build scons” enough to
trigger the problem?

Thanks for reporting the problem!

Ludo’.

¹ https://hpc.guix.info/blog/2025/03/build-daemon-drops-its-privileges/

Message #11 received at 77570 <at> debbugs.gnu.org (full text, mbox):

From: Ada Stevenson <adanskana <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77570 <at> debbugs.gnu.org
Subject: Re: bug#77570: Build failures on core packages following daemon
 changes
Date: Mon, 7 Apr 2025 10:20:39 +0000

Hi Ludo',

On 07/04/2025 10:14 am, Ludovic Courtès wrote:
> Hi Ada,
> 
> Ada Stevenson <adanskana <at> gmail.com> skribis:
> 
>> Following commit 93474f92886fac8a2e5eb0eb3b388654246d640d (cc'ing
>> Ludo' as the author of this commit), I have identified build failures
>> in several packages as they have tests that require modifying the
>> build tree in some way. These include scons, ruby and go. I'm sure
>> there are other packages that are affected by this.
> 
> Could you explain how to reproduce the problem?

Simply building the packages from source with tests will result in a 
build failure.

> 
> In particular, are you on Guix System or on a foreign distro?  In the
> latter case, is guix-daemon running as an unprivileged user¹ or as root?
> What version of guix-daemon is running, as shown by “pgrep -fa
> guix-daemon”?

I am running on Guix System, so my daemon is still privileged.

> 
> You mention scons, ruby, and go.  Is “guix build scons” enough to
> trigger the problem?

Forcing a rebuild with no substitutes ie. `guix build --no-substitutes 
--no-grafts --check scons` will result in a failed build.

> 
> Thanks for reporting the problem!

You're welcome! I just noticed this problem as I was building some 
packages from scratch.

> 
> Ludo’.
> 
> ¹ https://hpc.guix.info/blog/2025/03/build-daemon-drops-its-privileges/

-- 
Warmly,
Ada

Message #16 received at 77570 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Ada Stevenson <adanskana <at> gmail.com>
Cc: Reepca Russelstein <reepca <at> russelstein.xyz>, 77570 <at> debbugs.gnu.org
Subject: Re: bug#77570: Build failures on core packages following daemon
 changes
Date: Mon, 07 Apr 2025 16:22:22 +0200

Hello Ada,

(Cc: Reepca, who followed these developments.)

The build discrepancies you report are serious enough that we need to
find a strategy other than re-mounting the root directory MS_RDONLY so
we get EPERM and not EROFS on /.

I think we can go with the solution Reepca outlined in
<https://issues.guix.gnu.org/75810#14-lineno78>: expose
chrootRootDir/top inside the build environment (with permissions 0555
but still user-owned).  I’ll work on it.


Below are details about the failures you reported, Ada, reproduced with
commit 6af680670bf9055b90e6f8b63c4c2ab7b08e7c56 and a recent guix-daemon
where the build environment root is remounted read-only.

For scons, I get:

--8<---------------cut here---------------start------------->8---
ERROR: test_PathIsDirCreate (__main__.PathVariableTestCase)
Test the PathIsDirCreate validator
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/guix-build-scons-4.4.0.drv-0/source/SCons/Variables/PathVariableTests.py", line 129, in test_PathIsDirCreate
    o.validator('X', f, {})
  File "/tmp/guix-build-scons-4.4.0.drv-0/source/SCons/Variables/PathVariable.py", line 105, in PathIsDirCreate
    os.makedirs(val, exist_ok=True)
  File "/gnu/store/nla5820scsv7ibvr3kwpgkm9wr4prf0q-python-3.10.7/lib/python3.10/os.py", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/gnu/store/nla5820scsv7ibvr3kwpgkm9wr4prf0q-python-3.10.7/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/yyy'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/guix-build-scons-4.4.0.drv-0/source/SCons/Variables/PathVariableTests.py", line 133, in test_PathIsDirCreate
    raise Exception("did not catch expected UserError") from e
Exception: did not catch expected UserError
[…]
build of /gnu/store/jwsp6vz0hxriiylfl53wg2m98npwv8iy-scons-4.4.0.drv failed
--8<---------------cut here---------------end--------------->8---

And indeed, the code in question expects EEXIST or EPERM, but not EROFS:

--8<---------------cut here---------------start------------->8---
    def PathIsDirCreate(key, val, env) -> None:
        """Validator to check if Path is a directory,
           creating it if it does not exist."""
        try:
            os.makedirs(val, exist_ok=True)
        except FileExistsError:
            m = 'Path for option %s is a file, not a directory: %s'
            raise SCons.Errors.UserError(m % (key, val))
        except PermissionError:
            m = 'Path for option %s could not be created: %s'
            raise SCons.Errors.UserError(m % (key, val))
--8<---------------cut here---------------end--------------->8---

For Ruby, I get:

--8<---------------cut here---------------start------------->8---
make[1]: Leaving directory '/tmp/guix-build-ruby-3.3.3.drv-0/ruby-3.3.3'
Generating RDoc documentation
uh-oh! RDoc had a problem:
Read-only file system @ dir_s_mkdir - /homeless-shelter

run with --debug for full backtrace
make: *** [uncommon.mk:665: rdoc] Error 1
error: in phase 'build': uncaught exception:
%exception #<&invoke-error program: "make" arguments: ("-j" "4") exit-status: 2 term-signal: #f stop-signal: #f> 
phase `build' failed after 232.1 seconds
command "make" "-j" "4" failed with status 2
build process 6 exited with status 256
note: keeping build directory `/tmp/guix-build-ruby-3.3.3.drv-0'
builder for `/gnu/store/n7zsjg0gm4kz74f2hp6k22iw42zwn5kn-ruby-3.3.3.drv' failed with exit code 1
--8<---------------cut here---------------end--------------->8---

Not sure how it would deal with that unwritable $HOME previously.

However, /gnu/store/jyrmg3qkii4mb51ca1f3j9mv6knicr8a-go-1.23.5.drv
builds fine for me (no test failures).

Ludo’.

Message #19 received at 77570 <at> debbugs.gnu.org (full text, mbox):

From: keinflue <keinflue <at> posteo.net>
To: 77570 <at> debbugs.gnu.org
Subject: bug#77570: Build failures on core packages following daemon changes
Date: Mon, 07 Apr 2025 12:02:32 +0000

Hi everyone,

bug#77548 that I reported is a duplicate of this. After discussing the 
issue in IRC it took me a while to make a report and didn't notice this 
one was already submitted prior.

As I mentioned there, the shepherd package is also affected and at least 
in its case this is because one of the test cases (pid-file.sh) attempts 
to delete a non-existent file /does-not-exist.

The test script expects this to fail with ENOENT, but on current Linux 
kernel with glibc the unlink/unlinkat syscall will fail with EROFS 
instead on a read-only file system. (This was at some point between 
kernel version 3.2 and 3.6 fixed on the Linux side, but seems to have 
regressed after that.)

From what I can tell the other packages have the same or similar issues 
with syscalls reporting EROFS in preference of ENOENT/EACCESS/EPERM.

Message #22 received at 77570 <at> debbugs.gnu.org (full text, mbox):

From: Ada Stevenson <adanskana <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Reepca Russelstein <reepca <at> russelstein.xyz>, 77570 <at> debbugs.gnu.org
Subject: Re: bug#77570: Build failures on core packages following daemon
 changes
Date: Tue, 8 Apr 2025 04:29:42 +0000

Hello Ludo',

On 07/04/2025 2:22 pm, Ludovic Courtès wrote:
> Hello Ada,
> 
> (Cc: Reepca, who followed these developments.)
> 
> The build discrepancies you report are serious enough that we need to
> find a strategy other than re-mounting the root directory MS_RDONLY so
> we get EPERM and not EROFS on /.
> 
> I think we can go with the solution Reepca outlined in
> <https://issues.guix.gnu.org/75810#14-lineno78>: expose
> chrootRootDir/top inside the build environment (with permissions 0555
> but still user-owned).  I’ll work on it.

Thank you for looking into this!

> 
> 
> Below are details about the failures you reported, Ada, reproduced with
> commit 6af680670bf9055b90e6f8b63c4c2ab7b08e7c56 and a recent guix-daemon
> where the build environment root is remounted read-only.
> 
> For scons, I get:
> 
> --8<---------------cut here---------------start------------->8---
> ERROR: test_PathIsDirCreate (__main__.PathVariableTestCase)
> Test the PathIsDirCreate validator
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>    File "/tmp/guix-build-scons-4.4.0.drv-0/source/SCons/Variables/PathVariableTests.py", line 129, in test_PathIsDirCreate
>      o.validator('X', f, {})
>    File "/tmp/guix-build-scons-4.4.0.drv-0/source/SCons/Variables/PathVariable.py", line 105, in PathIsDirCreate
>      os.makedirs(val, exist_ok=True)
>    File "/gnu/store/nla5820scsv7ibvr3kwpgkm9wr4prf0q-python-3.10.7/lib/python3.10/os.py", line 215, in makedirs
>      makedirs(head, exist_ok=exist_ok)
>    File "/gnu/store/nla5820scsv7ibvr3kwpgkm9wr4prf0q-python-3.10.7/lib/python3.10/os.py", line 225, in makedirs
>      mkdir(name, mode)
> OSError: [Errno 30] Read-only file system: '/yyy'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>    File "/tmp/guix-build-scons-4.4.0.drv-0/source/SCons/Variables/PathVariableTests.py", line 133, in test_PathIsDirCreate
>      raise Exception("did not catch expected UserError") from e
> Exception: did not catch expected UserError
> […]
> build of /gnu/store/jwsp6vz0hxriiylfl53wg2m98npwv8iy-scons-4.4.0.drv failed
> --8<---------------cut here---------------end--------------->8---
> 
> And indeed, the code in question expects EEXIST or EPERM, but not EROFS:
> 
> --8<---------------cut here---------------start------------->8---
>      def PathIsDirCreate(key, val, env) -> None:
>          """Validator to check if Path is a directory,
>             creating it if it does not exist."""
>          try:
>              os.makedirs(val, exist_ok=True)
>          except FileExistsError:
>              m = 'Path for option %s is a file, not a directory: %s'
>              raise SCons.Errors.UserError(m % (key, val))
>          except PermissionError:
>              m = 'Path for option %s could not be created: %s'
>              raise SCons.Errors.UserError(m % (key, val))
> --8<---------------cut here---------------end--------------->8---
> 
> For Ruby, I get:
> 
> --8<---------------cut here---------------start------------->8---
> make[1]: Leaving directory '/tmp/guix-build-ruby-3.3.3.drv-0/ruby-3.3.3'
> Generating RDoc documentation
> uh-oh! RDoc had a problem:
> Read-only file system @ dir_s_mkdir - /homeless-shelter
> 
> run with --debug for full backtrace
> make: *** [uncommon.mk:665: rdoc] Error 1
> error: in phase 'build': uncaught exception:
> %exception #<&invoke-error program: "make" arguments: ("-j" "4") exit-status: 2 term-signal: #f stop-signal: #f>
> phase `build' failed after 232.1 seconds
> command "make" "-j" "4" failed with status 2
> build process 6 exited with status 256
> note: keeping build directory `/tmp/guix-build-ruby-3.3.3.drv-0'
> builder for `/gnu/store/n7zsjg0gm4kz74f2hp6k22iw42zwn5kn-ruby-3.3.3.drv' failed with exit code 1
> --8<---------------cut here---------------end--------------->8---
> 
> Not sure how it would deal with that unwritable $HOME previously.
> 
> However, /gnu/store/jyrmg3qkii4mb51ca1f3j9mv6knicr8a-go-1.23.5.drv
> builds fine for me (no test failures).

It appears I was building the full bootstrap chain for go, and it is the 
bootstrap go that is failing. Running this command will produce the 
failing build: `guix build --no-substitutes --no-grafts 
go <at> 1.4-bootstrap-20171003`.

> 
> Ludo’.

-- 
Warmly,
Ada

Message #27 received at 77570 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Ada Stevenson <adanskana <at> gmail.com>, keinflue <keinflue <at> posteo.net>
Cc: Reepca Russelstein <reepca <at> russelstein.xyz>, 77570 <at> debbugs.gnu.org
Subject: Re: bug#77570: Build failures on core packages following daemon
 changes
Date: Tue, 08 Apr 2025 15:58:58 +0200

Hi,

Ludovic Courtès <ludo <at> gnu.org> skribis:

> I think we can go with the solution Reepca outlined in
> <https://issues.guix.gnu.org/75810#14-lineno78>: expose
> chrootRootDir/top inside the build environment (with permissions 0555
> but still user-owned).  I’ll work on it.

I sent a patch:

  https://issues.guix.gnu.org/77642

Feedback welcome!

Ludo’.

Message #32 received at 77570-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> chbouib.org>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: keinflue <keinflue <at> posteo.net>, Ada Stevenson <adanskana <at> gmail.com>,
 Reepca Russelstein <reepca <at> russelstein.xyz>, 77570-done <at> debbugs.gnu.org
Subject: Re: bug#77570: Build failures on core packages following daemon
 changes
Date: Tue, 15 Apr 2025 12:07:44 +0200

Commit 3bedd82ac3c7c69e321d79bed6c77e7e055a2063 updated the ‘guix’
package, fixing this particular issue.

Please report any other problems you find.  It’s particularly valuable
if you’re building most things locally.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.