GNU bug report logs - #77862
guix-daemon run as non-root sets up /etc/group incorrectly in build container

Previous Next

To reply to this bug, email your comments to 77862 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: keinflue <keinflue <at> posteo.net>
To: bug-guix <at> gnu.org
Cc: ludo <at> gnu.org
Subject: guix-daemon run as non-root sets up /etc/group incorrectly in build
 container
Date: Thu, 17 Apr 2025 11:20:47 +0000

When using the new ability of guix-daemon to run as non-root with the 
help of user namespaces, the testsuite of coreutils fails.

This is because the daemon incorrectly uses the host GID instead of the 
guest GID in the build container's /etc/group, which the testsuite uses 
to lookup the group's name via id -gn.

Message #8 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: keinflue <keinflue <at> posteo.net>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Thu, 17 Apr 2025 15:30:38 +0200

[Message part 1 (text/plain, inline)]

Hi,

keinflue <keinflue <at> posteo.net> writes:

> When using the new ability of guix-daemon to run as non-root with the
> help of user namespaces, the testsuite of coreutils fails.

Could you include a build log snippet?  (Also useful to have it inline
so that someone searching for discussions about the bug can find it.)

> This is because the daemon incorrectly uses the host GID instead of
> the guest GID in the build container's /etc/group, which the testsuite
> uses to lookup the group's name via id -gn.

I believe the fix you suggest is this:

[Message part 2 (text/x-patch, inline)]

diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
index 4ee4a1ae5f..a1f39d9a8b 100644
--- a/nix/libstore/build.cc
+++ b/nix/libstore/build.cc
@@ -1854,7 +1854,7 @@ void DerivationGoal::startBuilder()
            view of the system (e.g., "id -gn"). */
         writeFile(chrootRootDir + "/etc/group",
             (format("nixbld:!:%1%:\n")
-                % (buildUser.enabled() ? buildUser.getGID() : getgid())).str());
+                % (buildUser.enabled() ? buildUser.getGID() : guestGID)).str());
 
         /* Create /etc/hosts with localhost entry. */
         if (!fixedOutput)

[Message part 3 (text/plain, inline)]

Correct?

Thanks,
Ludo’.

Message #11 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: keinflue <keinflue <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Thu, 17 Apr 2025 15:36:32 +0000

Here are excerpts from the build log:

> ERROR: tests/chown/separator
> ============================
> 
> ++ initial_cwd_=/tmp/guix-build-coreutils-9.1.drv-0/coreutils-9.1

[...]

> ++ id -u
> + id_u=30001
> + test -n 30001
> ++ id -un
> + id_un=nixbld
> + test -n nixbld
> ++ id -g
> + id_g=30000
> + test -n 30000
> ++ id -gn
> id: cannot find name for group ID 30000
> + id_gn=30000
> + framework_failure_
> + warn_ 'separator.sh: set-up failure: '
> + case $IFS in
> + printf '%s\n' 'separator.sh: set-up failure: '
> separator.sh: set-up failure:
> + test 9 = 2
> + printf '%s\n' 'separator.sh: set-up failure: '
> + sed 1q
> + Exit 99
> + set +e
> + exit 99
> + exit 99
> + remove_tmp_
> + __st=99
> + cleanup_
> + :
> + test '' = yes
> + cd /tmp/guix-build-coreutils-9.1.drv-0/coreutils-9.1
> + chmod -R u+rwx 
> /tmp/guix-build-coreutils-9.1.drv-0/coreutils-9.1/gt-separator.sh.Fk4W
> + rm -rf 
> /tmp/guix-build-coreutils-9.1.drv-0/coreutils-9.1/gt-separator.sh.Fk4W
> + exit 99
> ERROR tests/chown/separator.sh (exit status: 99)

[...]

> error: in phase 'check': uncaught exception:
> srfi-34 #<condition &invoke-error [program: "make" arguments: ("check" 
> "-j" "16") exit-status: 2 term-signal: #f stop-signal: #f] 2df6100> >
> phase `check' failed after 15.2 seconds
> command "make" "check" "-j" "16" failed with status 2
> build process 2 exited with status 256

Yes, I believe the patch as suggested is correct (with my limited 
understanding given that the lines above were changed in the same way).

Unfortunately I made a mistake and accidentally lost the container in 
which I tried this, so I can not verify right now whether the patch 
actually resolves the issue.

It might take me a day or two to restore it.

This happened either during or shortly after bootstrap builds, so I 
don't know whether this was the final coreutils package or one from 
commencement.scm.

Best,
keinflue

Message #14 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: keinflue <keinflue <at> posteo.net>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Thu, 17 Apr 2025 18:51:49 +0200

keinflue <keinflue <at> posteo.net> writes:

> Here are excerpts from the build log:

Thanks.

> Unfortunately I made a mistake and accidentally lost the container in
> which I tried this, so I can not verify right now whether the patch
> actually resolves the issue.
>
> It might take me a day or two to restore it.

No worries, I’ll wait for your feedback.

> This happened either during or shortly after bootstrap builds, so I
> don't know whether this was the final coreutils package or one from
> commencement.scm.

OK.

If you have a setup for full rebuilds (no substitutes) running in a
container, I’m curious to learn more about it!

Ludo’.

Message #19 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: keinflue <keinflue <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Sat, 19 Apr 2025 11:18:51 +0000

I can confirm that the patch resolves the particular failing test.

However I overlooked that there are other failing tests:

> FAIL: tests/chgrp/default-no-deref.sh
> FAIL: tests/chgrp/no-x.sh
> FAIL: tests/chgrp/posix-H.sh
> FAIL: tests/chgrp/recurse.sh
> FAIL: tests/chgrp/basic.sh

Here is an example of the failures:

> + require_membership_in_two_groups_
> + test 0 = 0
> + groups='30000 65534'
> + case "$groups" in
> + require_local_dir_
> + require_mount_list_
> + local 'mount_list_fail=cannot read table of mounted file systems'
> + df --local
> + grep -F 'cannot read table of mounted file systems'
> + is_local_dir_ .
> + test 1 = 1
> + df --local .
> + set _ 30000 65534
> + shift
> + g2=65534
> + mkdir d
> + touch f
> + ln -s ../f d/s
> ++ stat --printf=%g f
> + g_init=30000
> + chgrp -R 65534 d
> chgrp: changing group of 'd/s': Invalid argument
> chgrp: changing group of 'd': Invalid argument
> + fail=1
> ++ stat --printf=%g f
> + test 30000 = 30000
> + Exit 1
> + set +e
> + exit 1
> + exit 1
> + remove_tmp_
> + __st=1
> + cleanup_
> + :
> + test '' = yes
> + cd /tmp/guix-build-coreutils-9.1.drv-0/coreutils-9.1
> + chmod -R u+rwx 
> /tmp/guix-build-coreutils-9.1.drv-0/coreutils-9.1/gt-default-no-deref.sh.AEHe
> + rm -rf 
> /tmp/guix-build-coreutils-9.1.drv-0/coreutils-9.1/gt-default-no-deref.sh.AEHe
> + exit 1
> FAIL tests/chgrp/default-no-deref.sh (exit status: 1)

I think this happens if the user running guix-daemon has supplementary 
groups. These are not mapped via /proc/gid_map in the build container 
and therefore are reported as the overflow gid (65534) by getgroups.

The test cases assume that they can change ownership to this additional 
group but that is not permitted on the overflow gid.

I think supplementary groups should be dropped in the user namespace for 
the build container to make the behavior reproducible. Unfortunately 
this may be impossible if the parent namespace has set 
/proc/[...]/setgroups to "deny".

Best,
keinflue

On 17.04.2025 18:51, Ludovic Courtès wrote:
> keinflue <keinflue <at> posteo.net> writes:
> 
>> Here are excerpts from the build log:
> 
> Thanks.
> 
>> Unfortunately I made a mistake and accidentally lost the container in
>> which I tried this, so I can not verify right now whether the patch
>> actually resolves the issue.
>> 
>> It might take me a day or two to restore it.
> 
> No worries, I’ll wait for your feedback.
> 
>> This happened either during or shortly after bootstrap builds, so I
>> don't know whether this was the final coreutils package or one from
>> commencement.scm.
> 
> OK.
> 
> If you have a setup for full rebuilds (no substitutes) running in a
> container, I’m curious to learn more about it!
> 
> Ludo’.

Message #22 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: keinflue <keinflue <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Sat, 19 Apr 2025 11:48:39 +0000

On 17.04.2025 18:51, Ludovic Courtès wrote:
> keinflue <keinflue <at> posteo.net> writes:
> 
>> Here are excerpts from the build log:
> 
> Thanks.
> 
>> Unfortunately I made a mistake and accidentally lost the container in
>> which I tried this, so I can not verify right now whether the patch
>> actually resolves the issue.
>> 
>> It might take me a day or two to restore it.
> 
> No worries, I’ll wait for your feedback.
> 
>> This happened either during or shortly after bootstrap builds, so I
>> don't know whether this was the final coreutils package or one from
>> commencement.scm.
> 
> OK.
> 
> If you have a setup for full rebuilds (no substitutes) running in a
> container, I’m curious to learn more about it!

I basically just used "guix shell -CN -D guix" plus some extra packages 
and shares. Inside the container I built and ran guix from git with 
--with-store-dir and NIX_STORE set to a different path than /gnu/store. 
Initially I forgot to add a share for /var which is why I unfortunately 
broke the container once I existed it.

> Ludo’.

Message #25 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: keinflue <keinflue <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Sat, 19 Apr 2025 14:36:59 +0000

I just realized that there is also need for special handling of 
/dev/kvm. When running with privileges it was possible to add the build 
users to the kvm group to get access to /dev/kvm in the build container.

To have this continue to work in unprivileged mode, the kvm group (or I 
guess more specifically the group of /dev/kvm) should be mapped in the 
user namespace and presumably should not be dropped from supplementary 
groups.

There is also /dev/tty in the container which has unmapped group 
ownership, although that doesn't prevent access to it.

Alternatively I guess all supplementary groups could be preserved, but 
they should then also all be mapped into the user namespace. Then the 
user would have to drop supplementary groups manually (if they are able 
to) before running guix-daemon if they do not want any of them to 
propagate into the build environment and can't create a new user 
specifically for that purpose (e.g. because they do not have root access 
on the machine).

On 19.04.2025 13:18, keinflue wrote:
> I can confirm that the patch resolves the particular failing test.
> 
> However I overlooked that there are other failing tests:
> 
>> FAIL: tests/chgrp/default-no-deref.sh
>> FAIL: tests/chgrp/no-x.sh
>> FAIL: tests/chgrp/posix-H.sh
>> FAIL: tests/chgrp/recurse.sh
>> FAIL: tests/chgrp/basic.sh
> 
> Here is an example of the failures:
> 
>> + require_membership_in_two_groups_
>> + test 0 = 0
>> + groups='30000 65534'
>> + case "$groups" in
>> + require_local_dir_
>> + require_mount_list_
>> + local 'mount_list_fail=cannot read table of mounted file systems'
>> + df --local
>> + grep -F 'cannot read table of mounted file systems'
>> + is_local_dir_ .
>> + test 1 = 1
>> + df --local .
>> + set _ 30000 65534
>> + shift
>> + g2=65534
>> + mkdir d
>> + touch f
>> + ln -s ../f d/s
>> ++ stat --printf=%g f
>> + g_init=30000
>> + chgrp -R 65534 d
>> chgrp: changing group of 'd/s': Invalid argument
>> chgrp: changing group of 'd': Invalid argument
>> + fail=1
>> ++ stat --printf=%g f
>> + test 30000 = 30000
>> + Exit 1
>> + set +e
>> + exit 1
>> + exit 1
>> + remove_tmp_
>> + __st=1
>> + cleanup_
>> + :
>> + test '' = yes
>> + cd /tmp/guix-build-coreutils-9.1.drv-0/coreutils-9.1
>> + chmod -R u+rwx 
>> /tmp/guix-build-coreutils-9.1.drv-0/coreutils-9.1/gt-default-no-deref.sh.AEHe
>> + rm -rf 
>> /tmp/guix-build-coreutils-9.1.drv-0/coreutils-9.1/gt-default-no-deref.sh.AEHe
>> + exit 1
>> FAIL tests/chgrp/default-no-deref.sh (exit status: 1)
> 
> I think this happens if the user running guix-daemon has supplementary
> groups. These are not mapped via /proc/gid_map in the build container
> and therefore are reported as the overflow gid (65534) by getgroups.
> 
> The test cases assume that they can change ownership to this
> additional group but that is not permitted on the overflow gid.
> 
> I think supplementary groups should be dropped in the user namespace
> for the build container to make the behavior reproducible.
> Unfortunately this may be impossible if the parent namespace has set
> /proc/[...]/setgroups to "deny".
> 
> Best,
> keinflue
> 
> On 17.04.2025 18:51, Ludovic Courtès wrote:
>> keinflue <keinflue <at> posteo.net> writes:
>> 
>>> Here are excerpts from the build log:
>> 
>> Thanks.
>> 
>>> Unfortunately I made a mistake and accidentally lost the container in
>>> which I tried this, so I can not verify right now whether the patch
>>> actually resolves the issue.
>>> 
>>> It might take me a day or two to restore it.
>> 
>> No worries, I’ll wait for your feedback.
>> 
>>> This happened either during or shortly after bootstrap builds, so I
>>> don't know whether this was the final coreutils package or one from
>>> commencement.scm.
>> 
>> OK.
>> 
>> If you have a setup for full rebuilds (no substitutes) running in a
>> container, I’m curious to learn more about it!
>> 
>> Ludo’.

Message #28 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: keinflue <keinflue <at> posteo.net>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Fri, 25 Apr 2025 20:39:29 +0200

Hi,

I committed the /etc/group fix in
0d3bc50b0cffeae05beb12d0c270c6599186c0d7 together with a test.

keinflue <keinflue <at> posteo.net> writes:

> I think this happens if the user running guix-daemon has supplementary
> groups. These are not mapped via /proc/gid_map in the build container
> and therefore are reported as the overflow gid (65534) by getgroups.
>
> The test cases assume that they can change ownership to this
> additional group but that is not permitted on the overflow gid.
>
> I think supplementary groups should be dropped in the user namespace
> for the build container to make the behavior
> reproducible. Unfortunately this may be impossible if the parent
> namespace has set /proc/[...]/setgroups to "deny".

I came up with this test:

--8<---------------cut here---------------start------------->8---
(use-modules (guix)
             (gcrypt hash)
             (gnu packages bootstrap))

(computed-file "kvm-access"
               #~(begin
                   (pk '#$(gettimeofday))
                   (let ((st (stat "/dev/kvm")))
                     (pk '/dev/kvm st)
                     (pk '/dev/kvm:owner (stat:uid st) (stat:gid st))
                     (pk 'getgroups (getgroups))
                     ;; XXX: When running the daemon as root, /dev/kvm is
                     ;; owned by UID 0, which has no entry in /etc/passwd.
                     ;; (pk 'kvm-user (getpwuid (stat:uid st)))
                     ;; xxx: /etc/group never contained an entry to the "kvm"
                     ;; group so the thing below always failed.
                     ;; (pk 'kvm-group (getgrgid (stat:gid st)))
                     )
                   (when (open-fdes "/dev/kvm" O_RDWR)
                     (mkdir #$output)))
               #:guile %bootstrap-guile)
--8<---------------cut here---------------end--------------->8---

Privileged:

--8<---------------cut here---------------start------------->8---
$ guix build -f ~/src/guix-debugging/dev-kvm-access.scm
substitute: looking for substitutes on 'http://192.168.1.48:8123'...   0.0%guix substitute: warning: 192.168.1.48: connection failed: Connection timed out
substitute: 
substitute: looking for substitutes on 'https://ci.guix.gnu.org'... 100.0%
substitute: looking for substitutes on 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: looking for substitutes on 'https://guix.bordeaux.inria.fr'... 100.0%
The following derivation will be built:
  /gnu/store/vc5p6bfrzr7khgp9jha8h6kplixcl5h6-kvm-access.drv
substitute: looking for substitutes on 'http://192.168.1.48:8123'...   0.0%
building /gnu/store/vc5p6bfrzr7khgp9jha8h6kplixcl5h6-kvm-access.drv...

;;; ((1745606160 . 233876))

;;; (/dev/kvm #(6 483 8624 1 0 984 2792 0 1745359386 1745359386 1745359386 4096 0 char-special 432 382791307 382791307 1745359386))

;;; (/dev/kvm:owner 0 984)

;;; (getgroups #(984 30000))
successfully built /gnu/store/vc5p6bfrzr7khgp9jha8h6kplixcl5h6-kvm-access.drv
/gnu/store/36fin1iw2fh9066jg0y2fjd78j9wyjwp-kvm-access
--8<---------------cut here---------------end--------------->8---

Unprivileged:

--8<---------------cut here---------------start------------->8---
$ ./test-env guix build -f ~/src/guix-debugging/dev-kvm-access.scm
accepted connection from pid 2591, user ludo
accepted connection from pid 2601, user ludo
substitute: guix substitute: warning: ACL for archive imports seems to be uninitialized, substitutes may be unavailable
substitute: guix substitute: warning: authentication and authorization of substitutes disabled!
The following derivation will be built:
  /home/ludo/src/guix/test-tmp/store/5p4qn8d3bgnj60a2kwpliiwk81bvrcjp-kvm-access.drv
substitute: guix substitute: warning: authentication and authorization of substitutes disabled!
building /home/ludo/src/guix/test-tmp/store/5p4qn8d3bgnj60a2kwpliiwk81bvrcjp-kvm-access.drv...

;;; ((1745606200 . 636919))

;;; (/dev/kvm #(6 483 8624 1 65534 65534 2792 0 1745359386 1745359386 1745359386 4096 0 char-special 432 382791307 382791307 1745359386))

;;; (/dev/kvm:owner 65534 65534)

;;; (getgroups #(65534 65534 65534 65534 65534 65534 65534 30000 65534))
successfully built /home/ludo/src/guix/test-tmp/store/5p4qn8d3bgnj60a2kwpliiwk81bvrcjp-kvm-access.drv
/home/ludo/src/guix/test-tmp/store/ffh8zaw279dgdsh6q54mlldh4nikxiqp-kvm-access
--8<---------------cut here---------------end--------------->8---

In both cases, /dev/kvm is accessible.

In both cases, only the primary group has an entry in /etc/group;
supplementary groups are lacking.

So:

  1. I don’t think we need to map the “kvm” UID/GID into the user
     namespace;

  2. I’m confused as to what makes the Coreutils test suite fail.

It would still be good to drop any supplementary group other than “kvm”
though.

WDYT?

Thanks,
Ludo’.

Message #31 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: keinflue <keinflue <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Sat, 26 Apr 2025 00:23:01 +0000

On 25.04.2025 20:39, Ludovic Courtès wrote:
> Hi,
> 
> I committed the /etc/group fix in
> 0d3bc50b0cffeae05beb12d0c270c6599186c0d7 together with a test.
> 
> keinflue <keinflue <at> posteo.net> writes:
> 
>> I think this happens if the user running guix-daemon has supplementary
>> groups. These are not mapped via /proc/gid_map in the build container
>> and therefore are reported as the overflow gid (65534) by getgroups.
>> 
>> The test cases assume that they can change ownership to this
>> additional group but that is not permitted on the overflow gid.
>> 
>> I think supplementary groups should be dropped in the user namespace
>> for the build container to make the behavior
>> reproducible. Unfortunately this may be impossible if the parent
>> namespace has set /proc/[...]/setgroups to "deny".
> 
> I came up with this test:
> 
> --8<---------------cut here---------------start------------->8---
> (use-modules (guix)
>              (gcrypt hash)
>              (gnu packages bootstrap))
> 
> (computed-file "kvm-access"
>                #~(begin
>                    (pk '#$(gettimeofday))
>                    (let ((st (stat "/dev/kvm")))
>                      (pk '/dev/kvm st)
>                      (pk '/dev/kvm:owner (stat:uid st) (stat:gid st))
>                      (pk 'getgroups (getgroups))
>                      ;; XXX: When running the daemon as root, /dev/kvm 
> is
>                      ;; owned by UID 0, which has no entry in 
> /etc/passwd.
>                      ;; (pk 'kvm-user (getpwuid (stat:uid st)))
>                      ;; xxx: /etc/group never contained an entry to the 
> "kvm"
>                      ;; group so the thing below always failed.
>                      ;; (pk 'kvm-group (getgrgid (stat:gid st)))
>                      )
>                    (when (open-fdes "/dev/kvm" O_RDWR)
>                      (mkdir #$output)))
>                #:guile %bootstrap-guile)
> --8<---------------cut here---------------end--------------->8---
> 
> Privileged:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix build -f ~/src/guix-debugging/dev-kvm-access.scm
> substitute: looking for substitutes on 'http://192.168.1.48:8123'...
> 0.0%guix substitute: warning: 192.168.1.48: connection failed:
> Connection timed out
> substitute:
> substitute: looking for substitutes on 'https://ci.guix.gnu.org'... 
> 100.0%
> substitute: looking for substitutes on 
> 'https://bordeaux.guix.gnu.org'... 100.0%
> substitute: looking for substitutes on
> 'https://guix.bordeaux.inria.fr'... 100.0%
> The following derivation will be built:
>   /gnu/store/vc5p6bfrzr7khgp9jha8h6kplixcl5h6-kvm-access.drv
> substitute: looking for substitutes on 'http://192.168.1.48:8123'...   
> 0.0%
> building /gnu/store/vc5p6bfrzr7khgp9jha8h6kplixcl5h6-kvm-access.drv...
> 
> ;;; ((1745606160 . 233876))
> 
> ;;; (/dev/kvm #(6 483 8624 1 0 984 2792 0 1745359386 1745359386
> 1745359386 4096 0 char-special 432 382791307 382791307 1745359386))
> 
> ;;; (/dev/kvm:owner 0 984)
> 
> ;;; (getgroups #(984 30000))
> successfully built 
> /gnu/store/vc5p6bfrzr7khgp9jha8h6kplixcl5h6-kvm-access.drv
> /gnu/store/36fin1iw2fh9066jg0y2fjd78j9wyjwp-kvm-access
> --8<---------------cut here---------------end--------------->8---
> 
> Unprivileged:
> 
> --8<---------------cut here---------------start------------->8---
> $ ./test-env guix build -f ~/src/guix-debugging/dev-kvm-access.scm
> accepted connection from pid 2591, user ludo
> accepted connection from pid 2601, user ludo
> substitute: guix substitute: warning: ACL for archive imports seems to
> be uninitialized, substitutes may be unavailable
> substitute: guix substitute: warning: authentication and authorization
> of substitutes disabled!
> The following derivation will be built:
> 
> /home/ludo/src/guix/test-tmp/store/5p4qn8d3bgnj60a2kwpliiwk81bvrcjp-kvm-access.drv
> substitute: guix substitute: warning: authentication and authorization
> of substitutes disabled!
> building
> /home/ludo/src/guix/test-tmp/store/5p4qn8d3bgnj60a2kwpliiwk81bvrcjp-kvm-access.drv...
> 
> ;;; ((1745606200 . 636919))
> 
> ;;; (/dev/kvm #(6 483 8624 1 65534 65534 2792 0 1745359386 1745359386
> 1745359386 4096 0 char-special 432 382791307 382791307 1745359386))
> 
> ;;; (/dev/kvm:owner 65534 65534)
> 
> ;;; (getgroups #(65534 65534 65534 65534 65534 65534 65534 30000 
> 65534))
> successfully built
> /home/ludo/src/guix/test-tmp/store/5p4qn8d3bgnj60a2kwpliiwk81bvrcjp-kvm-access.drv
> /home/ludo/src/guix/test-tmp/store/ffh8zaw279dgdsh6q54mlldh4nikxiqp-kvm-access
> --8<---------------cut here---------------end--------------->8---
> 
> In both cases, /dev/kvm is accessible.
> 
> In both cases, only the primary group has an entry in /etc/group;
> supplementary groups are lacking.
> 
> So:
> 
>   1. I don’t think we need to map the “kvm” UID/GID into the user
>      namespace;

For the purpose of the passive permission checks that is not necessary, 
yes. There are no uids or gids being translated between the user 
namespaces. However if all supplementary groups would be dropped, that 
would include the kvm group and then this test will fail to access 
/dev/kvm. That was the problem I saw with that first suggestion.

>   2. I’m confused as to what makes the Coreutils test suite fail.

The result from getgroups includes both the primary gid 30000 and a 
supplementary gid 65534 (where the repeated 65534 are the overflow gid 
produced by viewing supplementary gids that aren't mapped into the user 
namespace via /proc/[pid]/gid_map).
Coreutils sees this and so assumes that it can do the equivalent of

touch testfile
chgrp 65534 testfile

to create a file owned by group 30000 initially and to then change group 
ownership of that file to 65534. Normally an unprivileged user is 
allowed to change group ownership of files they own between groups that 
they are member of, so this would always succeed outside a user 
namespace context.

However, any uid/gid used inside the user namespace is translated back 
to the host namespace via the uid/gid_map before permission checks. But 
in this case because 65534 doesn't map back to any gid in the host 
namespace, the syscall will fail.

If there is no supplementary group reported by getgroups at all, then 
coreutils just skips the test and it is ok again. Probably the coreutils 
test case should remove any gid reported by getgroups that is equal to 
the overflow gid before making that decision.

Dropping all supplementary groups from the build process (after unshare 
and before writing "deny" to /proc/pid/setgroups) would make it so that 
this test case is always skipped by having getgroups only report 30000, 
however that would also drop the kvm group as mentioned above and is 
also not permitted in all environments (e.g. when the parent namespace 
already set /proc/[pid]/setgroups to "deny").

So I think that instead either all supplementary groups of the user or 
at least the kvm group specifically needs to be mapped via 
/proc/[pid]/gid_map. When doing so getgroups would report 30000 and 984 
(assuming identity gid map for 984) in your test case above and the 
coreutils test case would work again, because

chgrp 984 testfile

would then succeed with 984 mapping back to the host namespace to a 
supplementary group of the process.

From a point of reproducibility and information leakage into the build 
container I think however that it would be preferable to not retain 
supplementary groups if possible. In contrast to the privileged build 
with a distinct build user that the can be given desired supplementary 
groups at will, the unprivileged environment may be one where the 
supplementary groups of the user running the daemon can't easily be 
changed to what is supposed to be seen in the build environment.

The contents of /etc/group are not relevant for this test case failure, 
they are never consulted.

But a few other asides (for which I don't necessarily think anything 
should be changed):

- I also noticed that the build container /etc/group is written with 
65534 assumed as overflow gid. I am not sure whether anyone actually 
does this, but the overflow uid/gid are technically configurable (and 
retrievable) via sysctl entries (/proc/sys/kernel/overflow(uid|gid)). 
65534 is just the default value.

- I also noticed that the operating-system defaults do not write an 
entry for the overflow gid to /etc/group (while they do for the overflow 
uid to /etc/passwd). I think such an entry should exist by default as 
well. The entry for /etc/passwd also assumes the default overflow uid of 
65534. This isn't only relevant for a user namespace context, but also 
file systems that can't map the whole range of Linux uids/gids.

> It would still be good to drop any supplementary group other than “kvm”
> though.
> 
> WDYT?
> 
> Thanks,
> Ludo’.

Message #34 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: keinflue <keinflue <at> posteo.net>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Sat, 26 Apr 2025 10:50:51 +0200

Hi,

keinflue <keinflue <at> posteo.net> writes:

>>   2. I’m confused as to what makes the Coreutils test suite fail.
>
> The result from getgroups includes both the primary gid 30000 and a
> supplementary gid 65534 (where the repeated 65534 are the overflow gid
> produced by viewing supplementary gids that aren't mapped into the
> user namespace via /proc/[pid]/gid_map).
> Coreutils sees this and so assumes that it can do the equivalent of
>
> touch testfile
> chgrp 65534 testfile
>
> to create a file owned by group 30000 initially and to then change
> group ownership of that file to 65534. Normally an unprivileged user
> is allowed to change group ownership of files they own between groups
> that they are member of, so this would always succeed outside a user
> namespace context.
>
> However, any uid/gid used inside the user namespace is translated back
> to the host namespace via the uid/gid_map before permission
> checks. But in this case because 65534 doesn't map back to any gid in
> the host namespace, the syscall will fail.

Oh right, got it.

> If there is no supplementary group reported by getgroups at all, then
> coreutils just skips the test and it is ok again. Probably the
> coreutils test case should remove any gid reported by getgroups that
> is equal to the overflow gid before making that decision.
>
> Dropping all supplementary groups from the build process (after
> unshare and before writing "deny" to /proc/pid/setgroups) would make
> it so that this test case is always skipped by having getgroups only
> report 30000, however that would also drop the kvm group as mentioned
> above and is also not permitted in all environments (e.g. when the
> parent namespace already set /proc/[pid]/setgroups to "deny").
>
> So I think that instead either all supplementary groups of the user or
> at least the kvm group specifically needs to be mapped via
> /proc/[pid]/gid_map. When doing so getgroups would report 30000 and
> 984 (assuming identity gid map for 984) in your test case above and
> the coreutils test case would work again, because
>
> chgrp 984 testfile
>
> would then succeed with 984 mapping back to the host namespace to a
> supplementary group of the process.

Having reread user_namespaces(7), I don’t think we can change the set of
supplementary groups at all: that effectively requires root privileges.

So the best we can do is map the “kvm” group inside the user namespace.

> From a point of reproducibility and information leakage into the build
> container I think however that it would be preferable to not retain
> supplementary groups if possible. In contrast to the privileged build
> with a distinct build user that the can be given desired supplementary
> groups at will, the unprivileged environment may be one where the
> supplementary groups of the user running the daemon can't easily be
> changed to what is supposed to be seen in the build environment.

I agree, though in practice, the daemon will usually run under a
dedicated user anyway: this is what ‘guix-install.sh’ does on other
distros, and this is what happens with (privileged? #t) on Guix System.
In these cases, there’d be no observable difference.

The observable difference (namely getgroups(2) returning a list of
unmapped GIDs) would be when people run the daemon as their own user,
which is currently inconvenient.

> - I also noticed that the build container /etc/group is written with
>   65534 assumed as overflow gid. I am not sure whether anyone actually
>   does this, but the overflow uid/gid are technically configurable
>   (and retrievable) via sysctl entries
>   (/proc/sys/kernel/overflow(uid|gid)). 65534 is just the default
>  value.

Note that the “nobody” UID in /etc/passwd dates back to before the
unprivileged daemon implementation.  It just happens to match the
default overflow UID, but I agree we should use the right one here.

> - I also noticed that the operating-system defaults do not write an
>   entry for the overflow gid to /etc/group (while they do for the
>   overflow uid to /etc/passwd). I think such an entry should exist by
>   default as well. The entry for /etc/passwd also assumes the default
>   overflow uid of 65534. This isn't only relevant for a user namespace
>   context, but also file systems that can't map the whole range of
>  Linux uids/gids.

I’m not sure this needs to be changed because it’s not all that
different from the preexisting situation where “kvm” would not have an
entry in /etc/group.

Thanks,
Ludo’.

Message #37 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: keinflue <keinflue <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Sat, 26 Apr 2025 11:08:57 +0000

Hi,

On 26.04.2025 10:50, Ludovic Courtès wrote:

>> If there is no supplementary group reported by getgroups at all, then
>> coreutils just skips the test and it is ok again. Probably the
>> coreutils test case should remove any gid reported by getgroups that
>> is equal to the overflow gid before making that decision.
>> 
>> Dropping all supplementary groups from the build process (after
>> unshare and before writing "deny" to /proc/pid/setgroups) would make
>> it so that this test case is always skipped by having getgroups only
>> report 30000, however that would also drop the kvm group as mentioned
>> above and is also not permitted in all environments (e.g. when the
>> parent namespace already set /proc/[pid]/setgroups to "deny").
>> 
>> So I think that instead either all supplementary groups of the user or
>> at least the kvm group specifically needs to be mapped via
>> /proc/[pid]/gid_map. When doing so getgroups would report 30000 and
>> 984 (assuming identity gid map for 984) in your test case above and
>> the coreutils test case would work again, because
>> 
>> chgrp 984 testfile
>> 
>> would then succeed with 984 mapping back to the host namespace to a
>> supplementary group of the process.
> 
> Having reread user_namespaces(7), I don’t think we can change the set 
> of
> supplementary groups at all: that effectively requires root privileges.
> 
> So the best we can do is map the “kvm” group inside the user namespace.

I also had another look and I missed that effectively CAP_SETGID is 
required in the _parent_ namespace in order to use setgroups (because 
otherwise writing "deny" to /proc/[pid]/setgroups is essentially 
forced).

But the same seems to also be required to map more than the own 
effective uid/gid of the process into the namespace.

Mapping more uids/gids is otherwise usually handled by a setuid utility 
(newuidmap, newgidmap) I think.

So I guess neither solution of dropping or mapping supplementary groups 
will work completely unprivileged and the only solution is to modify or 
disable the coreutils test case.

And mapping only kvm wouldn't be sufficient either, unless only that 
specific group would be supported. All supplementary groups of the 
process must be mapped for the test case to succeed (or at least one of 
them, I haven't checked by which rule the test cases chooses the gid for 
the test scenario from among the supplementary gids).

Best,
keinflue

Message #40 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: keinflue <keinflue <at> posteo.net>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Fri, 02 May 2025 17:38:59 +0200

[Message part 1 (text/plain, inline)]

Hello,

keinflue <keinflue <at> posteo.net> writes:

> I also had another look and I missed that effectively CAP_SETGID is
> required in the _parent_ namespace in order to use setgroups (because
> otherwise writing "deny" to /proc/[pid]/setgroups is essentially
> forced).
>
> But the same seems to also be required to map more than the own
> effective uid/gid of the process into the namespace.

Right, user_namespaces(7) makes it clear:

 •  The data written to uid_map (gid_map) must consist of a sin‐
    gle  line  that maps the writing process's effective user ID
    (group ID) in the parent user namespace to a user ID  (group
    ID) in the user namespace.

> So I guess neither solution of dropping or mapping supplementary
> groups will work completely unprivileged and the only solution is to
> modify or disable the coreutils test case.

Yes, I came to this conclusion as well.

I believe the attached Coreutils patch should fix that (yet to be
tested).  Would be worth reporting upstream as well because in a way
it’s a failure of the test framework.

Thanks,
Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/init.cfg b/init.cfg
index 856aa2ee7..e19ec5a31 100644
--- a/init.cfg
+++ b/init.cfg
@@ -488,7 +488,12 @@ require_membership_in_two_groups_()
 {
   test $# = 0 || framework_failure_
 
-  groups=${COREUTILS_GROUPS-$( (id -G || /usr/xpg4/bin/id -G) 2>/dev/null)}
+  # Always pretend this user account is not a member of any
+  # supplementary group.  This avoids wrong expectations from tests
+  # when the supplementary group is the overflow GID as is the case
+  # when 'guix-daemon' runs as an unprivileged user that is part of
+  # supplementary groups such as 'kvm'.
+  groups=
   case "$groups" in
     *' '*) ;;
     *) skip_ 'requires membership in two groups

Message #43 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: keinflue <keinflue <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Sat, 03 May 2025 02:16:40 +0000

[Message part 1 (text/plain, inline)]

Hi,

On 02.05.2025 17:38, Ludovic Courtès wrote:
> Hello,
> 
> keinflue <keinflue <at> posteo.net> writes:
> 
>> I also had another look and I missed that effectively CAP_SETGID is
>> required in the _parent_ namespace in order to use setgroups (because
>> otherwise writing "deny" to /proc/[pid]/setgroups is essentially
>> forced).
>> 
>> But the same seems to also be required to map more than the own
>> effective uid/gid of the process into the namespace.
> 
> Right, user_namespaces(7) makes it clear:
> 
>  •  The data written to uid_map (gid_map) must consist of a sin‐
>     gle  line  that maps the writing process's effective user ID
>     (group ID) in the parent user namespace to a user ID  (group
>     ID) in the user namespace.
> 
>> So I guess neither solution of dropping or mapping supplementary
>> groups will work completely unprivileged and the only solution is to
>> modify or disable the coreutils test case.
> 
> Yes, I came to this conclusion as well.
> 
> I believe the attached Coreutils patch should fix that (yet to be
> tested).  Would be worth reporting upstream as well because in a way
> it’s a failure of the test framework.

I tried to test the patch:

I had trouble with the normal origin patches mechanism. The first 
failing coreutils package is coreutils-final from commencement.scm and I 
didn't manage to properly replace that package without a dependency on 
the unpatched package remaining. Instead I wrote an equivalent 
substitute* in a post-unpack phase.

The test cases mentioned earlier now succeeded, but another testsuite 
for gnulib (also as part of coreutils) failed afterwards. The failure 
cause is the same, except this time written in C sources. I also fixed 
that via substitute* clauses. See attached patch. This unfortunately 
causes rebuilds of the coreutils-mesboot (and coreutils-boot0) packages 
as well, although those do not perform the tests.

It seems that now the system build proceeds much further (still 
running).

I will see whether I can report the issue(s) upstream to coreutils and 
gnulib. I noticed that in coreutils 9.2 (guix is currently 9.1) a 
similar fix was applied to handle special gids on MacOS. Unfortunately 
the default Linux overflow gid is not included in that list. In any 
case, the patch needs to be adjusted for newer coreutils versions.

> 
> Thanks,
> Ludo’.

[coreutils.diff (text/x-patch, attachment)]

Message #46 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: keinflue <keinflue <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Sat, 03 May 2025 11:00:28 +0000

 > It seems that now the system build proceeds much further (still 
running).

Unfortunately the python package also fails with equivalent test 
failures. It also has another failure mode where it expects a syscall to 
change ownership to the overflow uid to result in EPERM, while it will 
produce EINVAL (which happens even if there are no supplementary 
groups). Should I post the details here or open a new issue?

> I will see whether I can report the issue(s) upstream to coreutils and
> gnulib. I noticed that in coreutils 9.2 (guix is currently 9.1) a
> similar fix was applied to handle special gids on MacOS. Unfortunately
> the default Linux overflow gid is not included in that list. In any
> case, the patch needs to be adjusted for newer coreutils versions.

coreutils already responded and fixed the issue 
(https://debbugs.gnu.org/cgi/bugreport.cgi?bug=78225).

I still have to report to gnulib, but wanted to try building the 
standalone gnulib package first, which caused me to trip over the python 
issues.

>> 
>> Thanks,
>> Ludo’.

Message #49 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: keinflue <keinflue <at> posteo.net>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Sat, 03 May 2025 18:14:25 +0200

Hi,

keinflue <keinflue <at> posteo.net> writes:

> Unfortunately the python package also fails with equivalent test
> failures. It also has another failure mode where it expects a syscall
> to change ownership to the overflow uid to result in EPERM, while it
> will produce EINVAL (which happens even if there are no supplementary
> groups). Should I post the details here or open a new issue?

I think you can post it here.  Perhaps we should eventually keep all the
issues in this category together in a text file somewhere, with log
excerpts: that would allow us to better assess the packages affected by
this difference between the privileged and the unprivileged daemon is.

>> I will see whether I can report the issue(s) upstream to coreutils and
>> gnulib. I noticed that in coreutils 9.2 (guix is currently 9.1) a
>> similar fix was applied to handle special gids on MacOS. Unfortunately
>> the default Linux overflow gid is not included in that list. In any
>> case, the patch needs to be adjusted for newer coreutils versions.
>
> coreutils already responded and fixed the issue
> (https://debbugs.gnu.org/cgi/bugreport.cgi?bug=78225).

That was fast!

> I still have to report to gnulib, but wanted to try building the
> standalone gnulib package first, which caused me to trip over the
> python issues.

Alright.

Thanks a lot for this very important work.

I wonder if we should set up a separate Cuirass instance or something
building everything with the unprivileged daemon.

Thanks,
Ludo’.

Message #52 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: keinflue <keinflue <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Sat, 03 May 2025 19:05:34 +0000

On 03.05.2025 18:14, Ludovic Courtès wrote:
> Hi,
> 
> keinflue <keinflue <at> posteo.net> writes:
> 
>> Unfortunately the python package also fails with equivalent test
>> failures. It also has another failure mode where it expects a syscall
>> to change ownership to the overflow uid to result in EPERM, while it
>> will produce EINVAL (which happens even if there are no supplementary
>> groups). Should I post the details here or open a new issue?
> 
> I think you can post it here.  Perhaps we should eventually keep all 
> the
> issues in this category together in a text file somewhere, with log
> excerpts: that would allow us to better assess the packages affected by
> this difference between the privileged and the unprivileged daemon is.

It seems that the "chown to overflowgid" issue is somewhat widespread. I 
also see the testsuite for go (bootstrap) failing in the same way. I'd 
guess most implementations of "chown" system call wrappers in various 
languages will have test cases like this that fail to anticipate user 
namespaces. I will let my system build keep running a bit longer and 
will then post the list of packages I found with log excerpts here.

> 
> I wonder if we should set up a separate Cuirass instance or something
> building everything with the unprivileged daemon.

That would probably help since I am going to only test the packages that 
I am using myself in order to evaluate switching to the unprivileged 
guix-daemon. I don't have the resources to do more.

> Thanks,
> Ludo’.

Message #55 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: keinflue <keinflue <at> posteo.net>
Cc: 77862 <at> debbugs.gnu.org, Reepca Russelstein <reepca <at> russelstein.xyz>
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Mon, 19 May 2025 15:37:20 +0200

[Message part 1 (text/plain, inline)]

Hello,

(Cc: Reepca.)

keinflue <keinflue <at> posteo.net> writes:

> It seems that the "chown to overflowgid" issue is somewhat
> widespread. I also see the testsuite for go (bootstrap) failing in the
> same way. I'd guess most implementations of "chown" system call
> wrappers in various languages will have test cases like this that fail
> to anticipate user namespaces. I will let my system build keep running
> a bit longer and will then post the list of packages I found with log
> excerpts here.

I think it would be best to support chown-to-supplementary-group even
with the unprivileged daemon (specifically for the case where
guix-daemon runs as a dedicated user, and that this user has one
supplementary group, kvm).

The attached patch tries to do that, by calling out to ‘newuidmap’, and
under the assumption that /etc/subgid allows mapping the ‘kvm’ group.

It does the job (a build process can chown to ‘kvm’), but I couldn’t get
the GID mapping preserved across the ‘unshare’ call (the call that is
made to “lock” mounts), hence the “#if 0” there.

The problem is that when we call ‘unshare’, the ‘newgidmap’ setuid
binary is not longer accessible because we’re already in a chroot, so it
seems that we cannot preserve the GID map.

Thoughts?

Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
index a1f39d9a8b..83754b06bd 100644
--- a/nix/libstore/build.cc
+++ b/nix/libstore/build.cc
@@ -13,6 +13,7 @@
 #include <map>
 #include <sstream>
 #include <algorithm>
+#include <iostream>
 
 #include <limits.h>
 #include <time.h>
@@ -23,6 +24,7 @@
 #include <sys/utsname.h>
 #include <fcntl.h>
 #include <unistd.h>
+#include <grp.h>
 #include <errno.h>
 #include <stdio.h>
 #include <cstring>
@@ -1635,15 +1637,77 @@ static const gid_t guestGID = 30000;
 /* Initialize the user namespace of CHILD.  */
 static void initializeUserNamespace(pid_t child,
 				    uid_t hostUID = getuid(),
-				    gid_t hostGID = getgid())
+				    gid_t hostGID = getgid(),
+				    const std::vector<std::pair<gid_t, gid_t>> extraGIDs = {},
+				    bool haveCapSetGID = false)
 {
     writeFile("/proc/" + std::to_string(child) + "/uid_map",
 	      (format("%d %d 1") % guestUID % hostUID).str());
 
-    writeFile("/proc/" + std::to_string(child) + "/setgroups", "deny");
+    if (!haveCapSetGID && !extraGIDs.empty()) {
+	try {
+	    Strings args = {
+		std::to_string(child),
+		std::to_string(guestGID), std::to_string(hostGID), "1"
+	    };
+	    for (auto &pair: extraGIDs) {
+		args.push_back(std::to_string(pair.second));
+		args.push_back(std::to_string(pair.first));
+		args.push_back("1");
+	    }
+	    runProgram("newgidmap", true, args);
+	    printMsg(lvlChatty,
+		     format("mapped %1% extra GIDs into namespace of PID %2%")
+		     % extraGIDs.size() % child);
+	    return;
+	} catch (const ExecError &e) {
+	    ignoreException();
+	}
+	std::cerr << "here i am\n";
+    }
 
-    writeFile("/proc/" + std::to_string(child) + "/gid_map",
-	      (format("%d %d 1") % guestGID % hostGID).str());
+    if (!haveCapSetGID)
+	writeFile("/proc/" + std::to_string(child) + "/setgroups", "deny");
+
+    auto content = (format("%d %d 1\n") % guestGID % hostGID).str();
+    if (haveCapSetGID) {
+	for (auto &mapping: extraGIDs) {
+	    content += (format("%d %d 1\n") % mapping.second % mapping.first).str();
+	}
+    }
+    writeFile("/proc/" + std::to_string(child) + "/gid_map", content);
+}
+
+/* Return the ID of the "kvm" group or -1 if it does not exist or is not part
+   of the current users supplementary groups.  */
+static gid_t kvmGID()
+{
+    struct group *kvm = getgrnam("kvm");
+    if (kvm == NULL) return -1;
+
+    size_t max = 64;
+    gid_t groups[max];
+    int count = getgroups(max, groups);
+    if (count < 0) return -1;
+
+    for (int i = 0; i < count; i++) {
+	if (groups[i] == kvm->gr_gid) return kvm->gr_gid;
+    }
+
+    return -1;
+}
+
+static gid_t guestKVMGID = 40000;
+
+static std::vector<std::pair<gid_t, gid_t>> kvmGIDMapping()
+{
+    gid_t kvm = kvmGID();
+    if (kvm == -1)
+	return {};
+    else {
+	std::pair<gid_t, gid_t> mapping(kvm, guestKVMGID);
+	return { mapping };
+    }
 }
 
 void DerivationGoal::startBuilder()
@@ -2016,8 +2080,9 @@ void DerivationGoal::startBuilder()
 	readiness.readSide.close();
 	if ((flags & CLONE_NEWUSER) != 0) {
 	     /* Initialize the UID/GID mapping of the child process.  */
-	     initializeUserNamespace(pid);
-	     writeFull(readiness.writeSide, (unsigned char*)"go\n", 3);
+	    auto extraGIDs = kvmGIDMapping();
+	    initializeUserNamespace(pid, getuid(), getgid(), extraGIDs);
+	    writeFull(readiness.writeSide, (unsigned char*)"go\n", 3);
 	}
 	readiness.writeSide.close();
     } else
@@ -2269,10 +2334,12 @@ void DerivationGoal::runChild()
 		auto uid = getuid();
 		auto gid = getgid();
 
+#if 0						  // FIXME!
 		if (unshare(CLONE_NEWNS | CLONE_NEWUSER) == -1)
 		    throw SysError(format("creating new user and mount namespaces"));
 
-		initializeUserNamespace(getpid(), uid, gid);
+		auto extraGIDs = { std::pair<gid_t, gid_t>(guestKVMGID, guestKVMGID) };
+		initializeUserNamespace(getpid(), uid, gid, extraGIDs, true);
 
 		/* Check that mounts within the build environment are "locked"
 		   together and cannot be separated from within the build
@@ -2282,6 +2349,7 @@ void DerivationGoal::runChild()
 		   check that this is what we get.  */
 		int ret = umount(tmpDirInSandbox.c_str());
 		assert(ret == -1 && errno == EINVAL);
+#endif
 	    }
         }
 #endif
diff --git a/nix/libutil/util.cc b/nix/libutil/util.cc
index 56f116046c..b2c9b9f639 100644
--- a/nix/libutil/util.cc
+++ b/nix/libutil/util.cc
@@ -1081,9 +1081,11 @@ string runProgram(Path program, bool searchPath, const Strings & args)
 
     /* Wait for the child to finish. */
     int status = pid.wait(true);
-    if (!statusOk(status))
+    std::cerr << "status not ok: " << status << "\n";
+    if (!statusOk(status)) {
         throw ExecError(format("program `%1%' %2%")
             % program % statusToString(status));
+    }
 
     return result;
 }

Message #58 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: Reepca Russelstein <reepca <at> russelstein.xyz>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: keinflue <keinflue <at> posteo.net>, 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Mon, 19 May 2025 20:09:12 -0500

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hello,
>
> (Cc: Reepca.)
>
> keinflue <keinflue <at> posteo.net> writes:
>
>> It seems that the "chown to overflowgid" issue is somewhat
>> widespread. I also see the testsuite for go (bootstrap) failing in the
>> same way. I'd guess most implementations of "chown" system call
>> wrappers in various languages will have test cases like this that fail
>> to anticipate user namespaces. I will let my system build keep running
>> a bit longer and will then post the list of packages I found with log
>> excerpts here.
>
> I think it would be best to support chown-to-supplementary-group even
> with the unprivileged daemon (specifically for the case where
> guix-daemon runs as a dedicated user, and that this user has one
> supplementary group, kvm).
>
> The attached patch tries to do that, by calling out to ‘newuidmap’, and
> under the assumption that /etc/subgid allows mapping the ‘kvm’ group.
>
> It does the job (a build process can chown to ‘kvm’), but I couldn’t get
> the GID mapping preserved across the ‘unshare’ call (the call that is
> made to “lock” mounts), hence the “#if 0” there.
>
> The problem is that when we call ‘unshare’, the ‘newgidmap’ setuid
> binary is not longer accessible because we’re already in a chroot, so it
> seems that we cannot preserve the GID map.

... and even if we had the setuid binary accessible (for example via a
saved file descriptor that could be used with execveat, or a
bind-mount), it wouldn't be of any use at this point because (man
user_namespaces):

"if either the user or the group ID of the file has no mapping inside
the namespace, the set-user-ID (set-group- ID) bit is silently ignored:
the new program is executed, but the process's effective user (group) ID
is left unchanged."

Naturally, uid 0 isn't going to be mapped!  In fact, more generally,
newuidmap and newgidmap can't ever be used from within an uninitialized
user namespace, since by definition uid 0 isn't yet mapped in it.

So it falls to the parent process to do the initialization - that is, it
now has to do the initialization twice.  Of course, it's going to need
some way of knowing when the second user namespace has been created, and
the child is going to need some way of knowing when it's been
initialized, so we'll need to either use two pipes or switch to using a
socketpair.

Of some concern also is the ominous statement in "man newgidmap" that
"Note that newgidmap may be used only once for a given process."  I have
no idea how or why it would enforce this, and I'm going to assume for
now that what is actually meant is that "a given user namespace's gid
mapping cannot be written more than once", which is just a restatement
of what "man user_namespaces" says.



It's too bad that the user namespaces implementation doesn't allow
unprivileged users to map their own supplementary groups.  I can't think
of any reason not to - a user can already switch their effective gid to
any supplementary gid they have by creating and executing a setgid
program for that gid.  Are there any operations that require both a
capability /and/ a (mapped) egid check to pass?

It wouldn't surprise me if the ultimate reason is that the kernel devs
wanted to reuse code for uid_map and gid_map, and that it's easier to
verify one line than try to verify an arbitrary set of gids, including
ones in potentially-large ranges.  It's just an unfortunate consequence
that this means that all unprivileged user namespaces have to carry
gibberish supplementary groups around that they can "use" but not
comprehend.

But as long as /etc/subgid is configured to allow each user to map all
of the groups they are a member of, it can at least be worked around.

- reepca

[signature.asc (application/pgp-signature, inline)]

Message #61 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Reepca Russelstein <reepca <at> russelstein.xyz>
Cc: keinflue <keinflue <at> posteo.net>, 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Mon, 02 Jun 2025 23:06:08 +0200

Hey Reepca,

Reepca Russelstein <reepca <at> russelstein.xyz> writes:

> Ludovic Courtès <ludo <at> gnu.org> writes:

[...]

>> The attached patch tries to do that, by calling out to ‘newuidmap’, and
>> under the assumption that /etc/subgid allows mapping the ‘kvm’ group.
>>
>> It does the job (a build process can chown to ‘kvm’), but I couldn’t get
>> the GID mapping preserved across the ‘unshare’ call (the call that is
>> made to “lock” mounts), hence the “#if 0” there.
>>
>> The problem is that when we call ‘unshare’, the ‘newgidmap’ setuid
>> binary is not longer accessible because we’re already in a chroot, so it
>> seems that we cannot preserve the GID map.
>
> ... and even if we had the setuid binary accessible (for example via a
> saved file descriptor that could be used with execveat, or a
> bind-mount), it wouldn't be of any use at this point because (man
> user_namespaces):
>
> "if either the user or the group ID of the file has no mapping inside
> the namespace, the set-user-ID (set-group- ID) bit is silently ignored:
> the new program is executed, but the process's effective user (group) ID
> is left unchanged."
>
> Naturally, uid 0 isn't going to be mapped!  In fact, more generally,
> newuidmap and newgidmap can't ever be used from within an uninitialized
> user namespace, since by definition uid 0 isn't yet mapped in it.
>
> So it falls to the parent process to do the initialization - that is, it
> now has to do the initialization twice.  Of course, it's going to need
> some way of knowing when the second user namespace has been created, and
> the child is going to need some way of knowing when it's been
> initialized, so we'll need to either use two pipes or switch to using a
> socketpair.

That makes sense, thanks for explaining!

However, I think part of the rules still don’t fit in my head.  Here’s
what I have now (parent is 6929):

--8<---------------cut here---------------start------------->8---
6929  openat(AT_FDCWD, "/proc/6938/uid_map", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 17
6929  write(17, "30001 1000 1", 12)     = 12
6929  close(17)                         = 0
6929  openat(AT_FDCWD, "/proc/6938/setgroups", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 17
6929  write(17, "deny", 4)              = 4
6929  close(17)                         = 0
6929  openat(AT_FDCWD, "/proc/6938/gid_map", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 17
6929  write(17, "30000 998 1", 11)      = 11
6929  close(17)                         = 0
[…]
6938  unshare(CLONE_NEWNS|CLONE_NEWUSER) = 0
6938  write(20, "reinit\n", 7)          = 7
6929  <... read resumed>"reinit\n", 20) = 7
6938  read(17,  <unfinished ...>
6929  write(4, "gmlo\0\0\0\0+\0\0\0\0\0\0\0|   reinitializing UID/GID mapping of 6938\n\0\0\0\0\0", 64) = 64
6929  getgid()                          = 998
6929  getuid()                          = 1000
6929  openat(AT_FDCWD, "/proc/6938/uid_map", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 17
6929  write(17, "30001 1000 1", 12)     = -1 EPERM (Operation not permitted)
--8<---------------cut here---------------end--------------->8---

Why oh why would we get EPERM the second time?

These restrictions appear to be respected:

   •  The data written to uid_map (gid_map) must consist of a sin‐
      gle  line  that maps the writing process's effective user ID
      (group ID) in the parent user namespace to a user ID  (group
      ID) in the user namespace.

   •  The  writing process must have the same effective user ID as
      the process that created the user namespace.

Or is the user namespace of the parent (6929) not the same as “the
parent user namespace”?

Thanks for your patience.  :-)

Ludo’.

PS: The more I use it, the less I can stand this user namespace soup
    presented as an “API”.

Message #64 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: Reepca Russelstein <reepca <at> russelstein.xyz>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: keinflue <keinflue <at> posteo.net>, 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Tue, 03 Jun 2025 08:05:21 -0500

[Message part 1 (text/plain, inline)]

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hey Reepca,
>
> Reepca Russelstein <reepca <at> russelstein.xyz> writes:
>
>> Ludovic Courtès <ludo <at> gnu.org> writes:
>
> [...]
>
>>> The attached patch tries to do that, by calling out to ‘newuidmap’, and
>>> under the assumption that /etc/subgid allows mapping the ‘kvm’ group.
>>>
>>> It does the job (a build process can chown to ‘kvm’), but I couldn’t get
>>> the GID mapping preserved across the ‘unshare’ call (the call that is
>>> made to “lock” mounts), hence the “#if 0” there.
>>>
>>> The problem is that when we call ‘unshare’, the ‘newgidmap’ setuid
>>> binary is not longer accessible because we’re already in a chroot, so it
>>> seems that we cannot preserve the GID map.
>>
>> ... and even if we had the setuid binary accessible (for example via a
>> saved file descriptor that could be used with execveat, or a
>> bind-mount), it wouldn't be of any use at this point because (man
>> user_namespaces):
>>
>> "if either the user or the group ID of the file has no mapping inside
>> the namespace, the set-user-ID (set-group- ID) bit is silently ignored:
>> the new program is executed, but the process's effective user (group) ID
>> is left unchanged."
>>
>> Naturally, uid 0 isn't going to be mapped!  In fact, more generally,
>> newuidmap and newgidmap can't ever be used from within an uninitialized
>> user namespace, since by definition uid 0 isn't yet mapped in it.
>>
>> So it falls to the parent process to do the initialization - that is, it
>> now has to do the initialization twice.  Of course, it's going to need
>> some way of knowing when the second user namespace has been created, and
>> the child is going to need some way of knowing when it's been
>> initialized, so we'll need to either use two pipes or switch to using a
>> socketpair.
>
> That makes sense, thanks for explaining!
>
> However, I think part of the rules still don’t fit in my head.  Here’s
> what I have now (parent is 6929):
>
> 6929  openat(AT_FDCWD, "/proc/6938/uid_map", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 17
> 6929  write(17, "30001 1000 1", 12)     = 12
> 6929  close(17)                         = 0
> 6929  openat(AT_FDCWD, "/proc/6938/setgroups", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 17
> 6929  write(17, "deny", 4)              = 4
> 6929  close(17)                         = 0
> 6929  openat(AT_FDCWD, "/proc/6938/gid_map", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 17
> 6929  write(17, "30000 998 1", 11)      = 11
> 6929  close(17)                         = 0
> […]
> 6938  unshare(CLONE_NEWNS|CLONE_NEWUSER) = 0
> 6938  write(20, "reinit\n", 7)          = 7
> 6929  <... read resumed>"reinit\n", 20) = 7
> 6938  read(17,  <unfinished ...>
> 6929  write(4, "gmlo\0\0\0\0+\0\0\0\0\0\0\0|   reinitializing UID/GID mapping of 6938\n\0\0\0\0\0", 64) = 64
> 6929  getgid()                          = 998
> 6929  getuid()                          = 1000
> 6929  openat(AT_FDCWD, "/proc/6938/uid_map", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 17
> 6929  write(17, "30001 1000 1", 12)     = -1 EPERM (Operation not permitted)
>
> Why oh why would we get EPERM the second time?
>
> These restrictions appear to be respected:
>
>    •  The data written to uid_map (gid_map) must consist of a sin‐
>       gle  line  that maps the writing process's effective user ID
>       (group ID) in the parent user namespace to a user ID  (group
>       ID) in the user namespace.
>
>    •  The  writing process must have the same effective user ID as
>       the process that created the user namespace.
>
> Or is the user namespace of the parent (6929) not the same as “the
> parent user namespace”?

I'm afraid that is indeed what we're running in to: "the parent user
namespace" is the one created by clone.  And because the parent user
namespace is not the root user namespace, we couldn't run newgidmap from
within it, even if we still had a process around that was in it.

However, at this point, we actually don't need to use newgidmap, because
we have CAP_SETGID in the current user namespace... but in order to use
the "no restrictions" version of writing to gid_map, it's necessary for
the writing process to have CAP_SETGID in the *parent* user namespace
(in hindsight, this is what the actual problem was with the original
patch - it specified "true" for haveCapSetGID, so it shouldn't have been
using newgidmap anyway).  This necessarily requires that the writing
process is not in the user namespace being initialized, and because it
must either be in the namespace being initialized or its parent user
namespace, that means it must be in the parent user namespace.  Of
course, at this point, no processes exist in the parent user namespace.

So if you'll bear with the extreme awkwardness, we could fork a helper
process immediately prior to calling unshare, which, upon receiving a
notification, will initialize the parent process's user namespace.  Note
that the naming here is going to be inverted for process ancestry and
user namespace ancestry: the child process is in the parent user
namespace, and the parent process is in the child user namespace.

> Thanks for your patience.  :-)
>
> Ludo’.
>
> PS: The more I use it, the less I can stand this user namespace soup
>     presented as an “API”.

It certainly has a lot of clauses.  I will say, I did some reading of
the GNU Mach manual and was awed by how much simpler things could be if
every system call dealing with processes just took an explicit task port
argument like it does there.  The requirement that so many things
associated with processes can only be manipulated indirectly, and
usually only by the process itself, has caused no end of troubles, and
as the number of process-associated attributes in Linux continues to
grow, the interactions will likely only get more complicated.

It's not the kind of job security that gives any satisfaction.

- reepca

[signature.asc (application/pgp-signature, inline)]

Message #67 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Reepca Russelstein <reepca <at> russelstein.xyz>
Cc: keinflue <keinflue <at> posteo.net>, 77862 <at> debbugs.gnu.org
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Fri, 06 Jun 2025 17:38:49 +0200

Hello!

Reepca Russelstein <reepca <at> russelstein.xyz> writes:

> So if you'll bear with the extreme awkwardness, we could fork a helper
> process immediately prior to calling unshare, which, upon receiving a
> notification, will initialize the parent process's user namespace.  Note
> that the naming here is going to be inverted for process ancestry and
> user namespace ancestry: the child process is in the parent user
> namespace, and the parent process is in the child user namespace.

User namespaces seem to be an infinite supply of awkwardness!

I pushed a branch that implements those changes and actually works:

  https://codeberg.org/guix/guix/pulls/452

I marked it as WIP because I’m still in the process of updating the
‘guix’ package so I can actually run all the guix-daemon system tests
and there may be some adjustments to be made, such as ensuring that
‘newgidmap’ is found both on Guix System and on Debian.

Next step would be to run the test suites of Coreutils, Go, and Python
as keinflue did but I don’t have a good setup for that.

Thanks,
Ludo’.

Message #70 received at 77862 <at> debbugs.gnu.org (full text, mbox):

From: keinflue <keinflue <at> posteo.net>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 77862 <at> debbugs.gnu.org, Reepca Russelstein <reepca <at> russelstein.xyz>
Subject: Re: guix-daemon run as non-root sets up /etc/group incorrectly in
 build container
Date: Fri, 06 Jun 2025 20:34:34 +0000

Hi,

I have been busy the last weeks. I might be able to continue looking at 
this next week. I can however report that my system build did not 
encounter any other packages with the same problem, other than the ones 
I already mentioned. However, the build also didn't fully succeed for 
other reasons. I learned that my approach to use a guix shell container 
with a modified store name to rebuild everything in a completely 
separate guix-daemon and store instance does not work that well. Some 
things simply don't work independently of the unprivileged daemon.

Best,
keinflue

On 06.06.2025 17:38, Ludovic Courtès wrote:
> Hello!
> 
> Reepca Russelstein <reepca <at> russelstein.xyz> writes:
> 
>> So if you'll bear with the extreme awkwardness, we could fork a helper
>> process immediately prior to calling unshare, which, upon receiving a
>> notification, will initialize the parent process's user namespace.  
>> Note
>> that the naming here is going to be inverted for process ancestry and
>> user namespace ancestry: the child process is in the parent user
>> namespace, and the parent process is in the child user namespace.
> 
> User namespaces seem to be an infinite supply of awkwardness!
> 
> I pushed a branch that implements those changes and actually works:
> 
>   https://codeberg.org/guix/guix/pulls/452
> 
> I marked it as WIP because I’m still in the process of updating the
> ‘guix’ package so I can actually run all the guix-daemon system tests
> and there may be some adjustments to be made, such as ensuring that
> ‘newgidmap’ is found both on Guix System and on Debian.
> 
> Next step would be to run the test suites of Coreutils, Go, and Python
> as keinflue did but I don’t have a good setup for that.
> 
> Thanks,
> Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.