GNU bug report logs - #56677
[PATCH 0/2] environment: Add --emulate-fhs option.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 56677 in the body.
You can then email your comments to 56677 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: John Kehayias <john.kehayias <at> protonmail.com>
To: Guix-patches <guix-patches <at> gnu.org>
Subject: [PATCH 0/2] environment: Add --emulate-fhs option.
Date: Thu, 21 Jul 2022 04:14:19 +0000

Hello Guix,

As discussed on guix-devel here (please see for more detailed discussion and design aims): https://lists.gnu.org/r/guix-devel/2022-07/msg00161.html this is a patch to add an FHS (Filesystem Hierarchy Standard) emulation option for environments.

The overall goal is to mimic typical GNU/Linux distributions in following FHS (/bin, /etc, and /usr in particular) as well as a glibc that reads a global /etc/ld.so.cache and PATH with /bin, and so on. The idea is that following instructions for setting up a development environment, building software, running something, and so on in "typical" Linux environments, should "just work" with 'guix shell --container --emulate-fhs ...', provided the right inputs and other options are set.

For testers, this can be used by using pre-inst-env (outside of the pure shell used to build a local guix) to run guix shell with this patch. Please see the mailing list discussion for particular examples as well.

For review, in particular:

1. On the mailing list there was discussion about the necessity or not of glibc-for-fhs (added in the first patch). I find this useful and a big piece of making this FHS option work, but open to discussion or if it should be a further option.

2. Right now I used a script written to the containers /tmp/fhs.sh to generate the ld cache, supplement $PATH (somewhat optional, but I found useful for less tinkering), and finally launch the given command or shell. I found that when not providing a command the prompt for /bin/sh is not the same as when not using --emulate-fhs. So I'm not sure if this is the correct way to launch the default /bin/sh if no command is given. Open to ideas of a better way to implement these actions for a container start up as well.

3. This is my first time touching a guix script and the documentation, so please do check the commit message and guix.texi.

4. I decided to link the second level FHS directories, like /usr/bin, as well as optional ones like /lib64 (or /lib32), to the top level /bin, /lib, and so on. These could just be bind mounted to profile/bin and so on as well, but again tried to mimic an FHS distribution like Arch where the files only live in one place. While perhaps making the code a little more involved, I hope this makes the container look tidier.

I may be forgetting other elements in the implementation decisions I made, but I have been testing these patches along the way and have gotten good usage of them. Please test further too!

Thanks,
John

Message #8 received at 56677 <at> debbugs.gnu.org (full text, mbox):

From: John Kehayias <john.kehayias <at> protonmail.com>
To: "56677 <at> debbugs.gnu.org" <56677 <at> debbugs.gnu.org>
Subject: [PATCH 1/2] gnu: Add glibc-for-fhs.
Date: Thu, 21 Jul 2022 04:19:16 +0000

[Message part 1 (text/plain, inline)]

Empty Message

[0001-gnu-Add-glibc-for-fhs.patch (text/x-patch, attachment)]

Message #11 received at 56677 <at> debbugs.gnu.org (full text, mbox):

From: John Kehayias <john.kehayias <at> protonmail.com>
To: "56677 <at> debbugs.gnu.org" <56677 <at> debbugs.gnu.org>
Subject: [PATCH 2/2] environment: Add '--emulate-fhs'.
Date: Thu, 21 Jul 2022 04:19:45 +0000

[Message part 1 (text/plain, inline)]

Empty Message

[0002-environment-Add-emulate-fhs.patch (text/x-patch, attachment)]

Message #14 received at 56677 <at> debbugs.gnu.org (full text, mbox):

From: John Kehayias <john.kehayias <at> protonmail.com>
To: "56677 <at> debbugs.gnu.org" <56677 <at> debbugs.gnu.org>
Subject: Re: [PATCH 2/2 v2] environment: Add '--emulate-fhs'.
Date: Tue, 02 Aug 2022 14:27:10 +0000

[Message part 1 (text/plain, inline)]

Here is a tiny update to this patch; I noticed after I sent it originally that there was a formatting typo in the guix.texi changes. Update attached.

Has anyone had a chance to try out the patch or take a look at it?

I do use this in some daily work, for a project that uses Python's poetry shell and the Playwright libraries (uses browser binaries it downloads to do web automation). Here is the command I use, where since I need to use browsers in the container just getting the development inputs for ungoogled-chromium is a nice shortcut. The share/expose is to get all graphical/hardware acceleration working.

~/path/to/guix-source/pre-inst-env guix shell -C -F -N poetry coreutils gcc:lib -D ungoogled-chromium --share=$HOME/temphome=$HOME --preserve='^DISPLAY$' --preserve='^XAUTHORITY$' --share=$XAUTHORITY -- "poetry shell"

All works great and is very handy for doing this on my Guix machine.

[0002-environment-Add-emulate-fhs.patch (text/x-patch, attachment)]

Message #17 received at 56677 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: John Kehayias <john.kehayias <at> protonmail.com>
Cc: 56677 <at> debbugs.gnu.org
Subject: Re: bug#56677: [PATCH 0/2] environment: Add --emulate-fhs option.
Date: Thu, 04 Aug 2022 12:36:06 +0200

Hello,

John Kehayias <john.kehayias <at> protonmail.com> skribis:

> As discussed on guix-devel here (please see for more detailed discussion and design aims): https://lists.gnu.org/r/guix-devel/2022-07/msg00161.html this is a patch to add an FHS (Filesystem Hierarchy Standard) emulation option for environments.

Wo0t!

> 1. On the mailing list there was discussion about the necessity or not of glibc-for-fhs (added in the first patch). I find this useful and a big piece of making this FHS option work, but open to discussion or if it should be a further option.

I would prefer to keep complexity as low as possible, and thus not have
this glibc variant.

Now, I don’t know for this use case how much it matters that libc honors
/etc/ld.so.cache.  Intuitively, like I wrote on guix-devel, I’d think
ld.so.cache doesn’t matter, but you encountered counterexamples.

So I guess that if in practice presence of /etc/ld.so.cache *and* having
glibc honor it is necessary often enough, we can do that.

It seems that ‘glibc-for-fhs’ is merely added to the environment though,
and not actually used?

+                                      ;; For an FHS-container, add the
+                                      ;; (hidden) package glibc-for-fhs which
+                                      ;; uses the global cache at
+                                      ;; /etc/ld.so.cache.
+                                      (if emulate-fhs?
+                                          (alist-cons 'expression
+                                                      '(ad-hoc-package
+                                                        "(@@ (gnu packages base) glibc-for-fhs)")
+                                                      opts)
+                                          opts)))

Or rather it’s only used when running ‘ldconfig’, right?

> 2. Right now I used a script written to the containers /tmp/fhs.sh to generate the ld cache, supplement $PATH (somewhat optional, but I found useful for less tinkering), and finally launch the given command or shell. I found that when not providing a command the prompt for /bin/sh is not the same as when not using --emulate-fhs. So I'm not sure if this is the correct way to launch the default /bin/sh if no command is given. Open to ideas of a better way to implement these actions for a container start up as well.
>
> 3. This is my first time touching a guix script and the documentation, so please do check the commit message and guix.texi.
>
> 4. I decided to link the second level FHS directories, like /usr/bin, as well as optional ones like /lib64 (or /lib32), to the top level /bin, /lib, and so on. These could just be bind mounted to profile/bin and so on as well, but again tried to mimic an FHS distribution like Arch where the files only live in one place. While perhaps making the code a little more involved, I hope this makes the container look tidier.
>
> I may be forgetting other elements in the implementation decisions I made, but I have been testing these patches along the way and have gotten good usage of them. Please test further too!
 
At first sight, I find it pretty cool!  I would have two grievances:

  1. Can we make the implementation more orthogonal and less entangled
     in the already-long ‘launch-environment/container’?

     Maybe that can be accomplished by moving all the code conditional
     on ‘emulate-fhs?’ out of the way in a separate procedure, and
     possibly by adding a generic hook in ‘launch-environment/container’
     that would call said procedure.

  2. Please add tests.  You can probably augment
     ‘tests/guix-environment-container.sh’ for that.  Let us know if
     you’re not sure how to do that.

Thanks for all the work, and sorry for the delay: it seems to be summer
time for many of us.  :-)

Ludo’.

Message #20 received at 56677 <at> debbugs.gnu.org (full text, mbox):

From: John Kehayias <john.kehayias <at> protonmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 56677 <at> debbugs.gnu.org
Subject: Re: bug#56677: [PATCH 0/2] environment: Add --emulate-fhs option.
Date: Wed, 17 Aug 2022 21:43:08 +0000

Hello,

Took me a while to respond as well, summer meaning more time to hack on things and then also let them get away...

------- Original Message -------
On Thursday, August 4th, 2022 at 6:36 AM, Ludovic Courtès <ludo <at> gnu.org> wrote:
> 
> Hello,
> 
> John Kehayias john.kehayias <at> protonmail.com skribis:
> 
> > As discussed on guix-devel here (please see for more detailed discussion and design aims): https://lists.gnu.org/r/guix-devel/2022-07/msg00161.html this is a patch to add an FHS (Filesystem Hierarchy Standard) emulation option for environments.
> 
> Wo0t!
> 
> > 1. On the mailing list there was discussion about the necessity or not of glibc-for-fhs (added in the first patch). I find this useful and a big piece of making this FHS option work, but open to discussion or if it should be a further option.
> 
> I would prefer to keep complexity as low as possible, and thus not have
> this glibc variant.
> 
> Now, I don’t know for this use case how much it matters that libc honors
> /etc/ld.so.cache. Intuitively, like I wrote on guix-devel, I’d think
> ld.so.cache doesn’t matter, but you encountered counterexamples.
> 
> So I guess that if in practice presence of /etc/ld.so.cache and having
> glibc honor it is necessary often enough, we can do that.
> 

Right, though as I said, happy to hear of alternatives or what other use cases come up. This seems rather robust to any "usual" assumptions though.

> It seems that ‘glibc-for-fhs’ is merely added to the environment though,
> and not actually used?
> 

Well, it is added to the environment which here means the glibc-for-fhs lib directory ends up in the container's global /lib. This may be useful for anything expecting a more "typical" glibc to be found in the typical location. I can't say I know the particulars here, other than binaries and an example of other nested containers (used in non-free software, but the containers are bwrap and friends) expecting glibc to default to a global ld cache. Again, there may be other workarounds or ways to reduce this, but for now I followed the "emulate" part of the flag :)

> + ;; For an FHS-container, add the
> + ;; (hidden) package glibc-for-fhs which
> + ;; uses the global cache at
> + ;; /etc/ld.so.cache.
> + (if emulate-fhs?
> + (alist-cons 'expression
> + '(ad-hoc-package
> + "(@@ (gnu packages base) glibc-for-fhs)")
> + opts)
> + opts)))
> 
> Or rather it’s only used when running ‘ldconfig’, right?
> 

Yes, since that is the glibc in the container. Though actually generating a cache shouldn't matter, right? Guix's glibc will do that as well, as the only patch removed is the one that changed where glibc reads the ld cache from. So the cache could be generated with Guix's glibc, but then likely won't be read otherwise? Sorry, a bit out of what I know for all the details; I'm mostly at "this is what is often expected" and "this makes it work."

> > 2. Right now I used a script written to the containers /tmp/fhs.sh to generate the ld cache, supplement $PATH (somewhat optional, but I found useful for less tinkering), and finally launch the given command or shell. I found that when not providing a command the prompt for /bin/sh is not the same as when not using --emulate-fhs. So I'm not sure if this is the correct way to launch the default /bin/sh if no command is given. Open to ideas of a better way to implement these actions for a container start up as well.
> > 
> > 3. This is my first time touching a guix script and the documentation, so please do check the commit message and guix.texi.
> > 
> > 4. I decided to link the second level FHS directories, like /usr/bin, as well as optional ones like /lib64 (or /lib32), to the top level /bin, /lib, and so on. These could just be bind mounted to profile/bin and so on as well, but again tried to mimic an FHS distribution like Arch where the files only live in one place. While perhaps making the code a little more involved, I hope this makes the container look tidier.
> > 
> > I may be forgetting other elements in the implementation decisions I made, but I have been testing these patches along the way and have gotten good usage of them. Please test further too!
> 
> At first sight, I find it pretty cool! I would have two grievances:
> 

Thanks! It is serving me in some day-to-day work nicely.

> 1. Can we make the implementation more orthogonal and less entangled
> in the already-long ‘launch-environment/container’?
> 
> Maybe that can be accomplished by moving all the code conditional
> on ‘emulate-fhs?’ out of the way in a separate procedure, and
> possibly by adding a generic hook in ‘launch-environment/container’
> that would call said procedure.
> 

Sure, this sounds like a good idea. I can certainly separate out the FHS setup to a separate function and call it. But I'm not sure what you mean by a "generic hook" here. Do you mean that launch-environment/container would have as an argument say a list of functions it would call?

> 2. Please add tests. You can probably augment
> ‘tests/guix-environment-container.sh’ for that. Let us know if
> you’re not sure how to do that.
> 

Thanks, definitely forgot about that. In looking at that, I've just ran it with "./pre-inst-env sh tests/guix-environment-container.sh" and see that the exit code is 0. Is that the correct way to run these?

Secondly, I'm trying to think of what tests to add. I could of course run the same tests already, but with the --emulate-fhs option, to check that there are no regressions. Other than that, maybe checking that e.g. there's /etc/ld.so.cache, /lib, and so on?

> Thanks for all the work, and sorry for the delay: it seems to be summer
> time for many of us. :-)
> 
> Ludo’.

No worries, summer is a good time to get away, or dig in :)

John

Message #23 received at 56677 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: John Kehayias <john.kehayias <at> protonmail.com>
Cc: 56677 <at> debbugs.gnu.org
Subject: Re: bug#56677: [PATCH 0/2] environment: Add --emulate-fhs option.
Date: Thu, 08 Sep 2022 22:58:48 +0200

Howdy,

John Kehayias <john.kehayias <at> protonmail.com> skribis:

>> I would prefer to keep complexity as low as possible, and thus not have
>> this glibc variant.
>> 
>> Now, I don’t know for this use case how much it matters that libc honors
>> /etc/ld.so.cache. Intuitively, like I wrote on guix-devel, I’d think
>> ld.so.cache doesn’t matter, but you encountered counterexamples.
>> 
>> So I guess that if in practice presence of /etc/ld.so.cache and having
>> glibc honor it is necessary often enough, we can do that.
>> 
>
> Right, though as I said, happy to hear of alternatives or what other use cases come up. This seems rather robust to any "usual" assumptions though.
>
>> It seems that ‘glibc-for-fhs’ is merely added to the environment though,
>> and not actually used?
>> 
>
> Well, it is added to the environment which here means the glibc-for-fhs lib directory ends up in the container's global /lib. This may be useful for anything expecting a more "typical" glibc to be found in the typical location. I can't say I know the particulars here, other than binaries and an example of other nested containers (used in non-free software, but the containers are bwrap and friends) expecting glibc to default to a global ld cache. Again, there may be other workarounds or ways to reduce this, but for now I followed the "emulate" part of the flag :)

Oh I got it; that /lib/libc.so *is* used, but only by binaries that were
built on an FHS distro and that you’d bring in (that’s the whole point,
I guess).  It’s not used by Guix packages.

>> 1. Can we make the implementation more orthogonal and less entangled
>> in the already-long ‘launch-environment/container’?
>> 
>> Maybe that can be accomplished by moving all the code conditional
>> on ‘emulate-fhs?’ out of the way in a separate procedure, and
>> possibly by adding a generic hook in ‘launch-environment/container’
>> that would call said procedure.
>> 
>
> Sure, this sounds like a good idea. I can certainly separate out the FHS setup to a separate function and call it. But I'm not sure what you mean by a "generic hook" here. Do you mean that launch-environment/container would have as an argument say a list of functions it would call?

Yes, or an argument with a single procedure to call at a specific
point.  That would default to a no-op.

>> 2. Please add tests. You can probably augment
>> ‘tests/guix-environment-container.sh’ for that. Let us know if
>> you’re not sure how to do that.
>> 
>
> Thanks, definitely forgot about that. In looking at that, I've just ran it with "./pre-inst-env sh tests/guix-environment-container.sh" and see that the exit code is 0. Is that the correct way to run these?

The correct way is:

  make check TESTS=tests/guix-environment-container.sh

Compared to what you wrote, it uses ./test-env (which spawns a daemon
that uses the local store, not /gnu/sore) and sets a bunch of
environment variables.

See
<https://guix.gnu.org/manual/devel/en/html_node/Running-the-Test-Suite.html>.

> Secondly, I'm trying to think of what tests to add. I could of course run the same tests already, but with the --emulate-fhs option, to check that there are no regressions. Other than that, maybe checking that e.g. there's /etc/ld.so.cache, /lib, and so on?

Right, at least you’d want to check for these files/directories.

Note that since the test relies on ‘glibc-for-fhs’, it cannot be done
the “normal way” (that is, using the local store rather than /gnu/store)
because it would end up building the world.

The solution here is to use /gnu/store, if available, and to otherwise
skip the test (return 77).  See ‘tests/guix-pack-relocatable.sh’ up to
line 40 on how to do that.

HTH!

Ludo’.

Message #26 received at 56677 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: John Kehayias <john.kehayias <at> protonmail.com>
Cc: 56677 <at> debbugs.gnu.org
Subject: Re: bug#56677: [PATCH 0/2] environment: Add --emulate-fhs option.
Date: Thu, 08 Sep 2022 22:58:59 +0200

Howdy,

John Kehayias <john.kehayias <at> protonmail.com> skribis:

>> I would prefer to keep complexity as low as possible, and thus not have
>> this glibc variant.
>> 
>> Now, I don’t know for this use case how much it matters that libc honors
>> /etc/ld.so.cache. Intuitively, like I wrote on guix-devel, I’d think
>> ld.so.cache doesn’t matter, but you encountered counterexamples.
>> 
>> So I guess that if in practice presence of /etc/ld.so.cache and having
>> glibc honor it is necessary often enough, we can do that.
>> 
>
> Right, though as I said, happy to hear of alternatives or what other use cases come up. This seems rather robust to any "usual" assumptions though.
>
>> It seems that ‘glibc-for-fhs’ is merely added to the environment though,
>> and not actually used?
>> 
>
> Well, it is added to the environment which here means the glibc-for-fhs lib directory ends up in the container's global /lib. This may be useful for anything expecting a more "typical" glibc to be found in the typical location. I can't say I know the particulars here, other than binaries and an example of other nested containers (used in non-free software, but the containers are bwrap and friends) expecting glibc to default to a global ld cache. Again, there may be other workarounds or ways to reduce this, but for now I followed the "emulate" part of the flag :)

Oh I got it; that /lib/libc.so *is* used, but only by binaries that were
built on an FHS distro and that you’d bring in (that’s the whole point,
I guess).  It’s not used by Guix packages.

>> 1. Can we make the implementation more orthogonal and less entangled
>> in the already-long ‘launch-environment/container’?
>> 
>> Maybe that can be accomplished by moving all the code conditional
>> on ‘emulate-fhs?’ out of the way in a separate procedure, and
>> possibly by adding a generic hook in ‘launch-environment/container’
>> that would call said procedure.
>> 
>
> Sure, this sounds like a good idea. I can certainly separate out the FHS setup to a separate function and call it. But I'm not sure what you mean by a "generic hook" here. Do you mean that launch-environment/container would have as an argument say a list of functions it would call?

Yes, or an argument with a single procedure to call at a specific
point.  That would default to a no-op.

>> 2. Please add tests. You can probably augment
>> ‘tests/guix-environment-container.sh’ for that. Let us know if
>> you’re not sure how to do that.
>> 
>
> Thanks, definitely forgot about that. In looking at that, I've just ran it with "./pre-inst-env sh tests/guix-environment-container.sh" and see that the exit code is 0. Is that the correct way to run these?

The correct way is:

  make check TESTS=tests/guix-environment-container.sh

Compared to what you wrote, it uses ./test-env (which spawns a daemon
that uses the local store, not /gnu/sore) and sets a bunch of
environment variables.

See
<https://guix.gnu.org/manual/devel/en/html_node/Running-the-Test-Suite.html>.

> Secondly, I'm trying to think of what tests to add. I could of course run the same tests already, but with the --emulate-fhs option, to check that there are no regressions. Other than that, maybe checking that e.g. there's /etc/ld.so.cache, /lib, and so on?

Right, at least you’d want to check for these files/directories.

Note that since the test relies on ‘glibc-for-fhs’, it cannot be done
the “normal way” (that is, using the local store rather than /gnu/store)
because it would end up building the world.

The solution here is to use /gnu/store, if available, and to otherwise
skip the test (return 77).  See ‘tests/guix-pack-relocatable.sh’ up to
line 40 on how to do that.

HTH!

Ludo’.

Message #29 received at 56677 <at> debbugs.gnu.org (full text, mbox):

From: John Kehayias <john.kehayias <at> protonmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 56677 <at> debbugs.gnu.org
Subject: Re: bug#56677: [PATCH v2 1/2] environment: Add --emulate-fhs option.
Date: Tue, 04 Oct 2022 18:32:44 +0000

[Message part 1 (text/plain, inline)]

Hi Ludo’!

Apologies for the long delay! I've done some more rewriting than expected, so let me outline some changes in the v2 patch (attached, while path 1/2 for glibc-for-fhs remains unchanged).

On Thu, Sep 08, 2022 at 10:58 PM, Ludovic Courtès wrote:

> Howdy,
>
> John Kehayias <john.kehayias <at> protonmail.com> skribis:
>
[snip]
>>> 1. Can we make the implementation more orthogonal and less entangled
>>> in the already-long ‘launch-environment/container’?
>>>
>>> Maybe that can be accomplished by moving all the code conditional
>>> on ‘emulate-fhs?’ out of the way in a separate procedure, and
>>> possibly by adding a generic hook in ‘launch-environment/container’
>>> that would call said procedure.
>>>
>>
>> Sure, this sounds like a good idea. I can certainly separate out the FHS setup to a
>> separate function and call it. But I'm not sure what you mean by a "generic hook" here.
>> Do you mean that launch-environment/container would have as an argument say a list of
>> functions it would call?
>
> Yes, or an argument with a single procedure to call at a specific
> point.  That would default to a no-op.
>

I've done some rearranging to address this and along the way fixed the command passing to the container to work the same as without --emulate-fhs (previously I hacked together a startup script that didn't properly capture the command given by the user).

Unfortunately, since some of the needed directories are bind mounted and others can be linked after creating filesystems, not all of the FHS directory setup can be done in the same place. To be explicit, bind mounts for all filesystems are set up in the early mlet of launch-environment/container, before being able to call a procedure to do other setup needed here.

The end result:

launch-environment/container now takes a key 'setup-hook' which defaults to #f. For now this only handles a single list with the function name and arguments (could extend this to be more general, but didn't have any use cases for it off hand). For the FHS container this is set, by the entry point guix-environment*, to the new function setup-fhs which sets up symlinks and ld.so.conf. Handling arguments was necessary as setup-fhs needs the profile used in the container.

So then launch-environment also gets a new 'emulate-fhs?' option so it can run some additional setup before running the command given by the user. What was the (hacky) /tmp/fhs.sh is now a setenv and invoke, before the final execlp for the given command.

A little more complicated perhaps, but I think functionally separates the different steps (bind mounts, symlinks and file creation, and running in the container) at least.

>>> 2. Please add tests. You can probably augment
>>> ‘tests/guix-environment-container.sh’ for that. Let us know if
>>> you’re not sure how to do that.
>>>
>>
>> Thanks, definitely forgot about that. In looking at that, I've just ran it with
>> "./pre-inst-env sh tests/guix-environment-container.sh" and see that the exit code is 0.
>> Is that the correct way to run these?
>
> The correct way is:
>
>   make check TESTS=tests/guix-environment-container.sh
>
> Compared to what you wrote, it uses ./test-env (which spawns a daemon
> that uses the local store, not /gnu/sore) and sets a bunch of
> environment variables.
>
> See
> <https://guix.gnu.org/manual/devel/en/html_node/Running-the-Test-Suite.html>.
>

Thanks, missed that somehow.

>> Secondly, I'm trying to think of what tests to add. I could of course run the same tests
>> already, but with the --emulate-fhs option, to check that there are no regressions.
>> Other than that, maybe checking that e.g. there's /etc/ld.so.cache, /lib, and so on?
>
> Right, at least you’d want to check for these files/directories.
>
> Note that since the test relies on ‘glibc-for-fhs’, it cannot be done
> the “normal way” (that is, using the local store rather than /gnu/store)
> because it would end up building the world.
>
> The solution here is to use /gnu/store, if available, and to otherwise
> skip the test (return 77).  See ‘tests/guix-pack-relocatable.sh’ up to
> line 40 on how to do that.
>
> HTH!
>

Very helpful, thanks!

I added in two tests to the end of tests/guix-environment-container.sh. One checks for the FHS file structure in the container and the other tries to read the ld cache (using 'ldconfig -p').

If we wanted to run all the non-fhs tests with --emulate-fhs, then maybe we'd want to make it so the FHS specific tests live in a new file and guix-environment-container.sh can be called in a way to enable that option? (A quick guess would be to just set an alias so that guix environment is always called with --emulate-fhs, but not sure if that works in the test environment.)

Wasn't sure if all that is necessary so I just went with the FHS-specific tests for now. I checked that they pass for me.

The commit changelog has gotten a bit more complicated with these changes, hopefully I got everything in there.

Thanks for your help on this and I'll make sure any future revisions are more timely!
John

[0002-environment-Add-emulate-fhs-v2.patch (text/x-patch, attachment)]

Message #32 received at 56677 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: John Kehayias <john.kehayias <at> protonmail.com>
Cc: 56677 <at> debbugs.gnu.org
Subject: Re: bug#56677: [PATCH 0/2] environment: Add --emulate-fhs option.
Date: Thu, 13 Oct 2022 09:37:41 +0200

[Message part 1 (text/plain, inline)]

Hi John,

John Kehayias <john.kehayias <at> protonmail.com> skribis:

> From de7ae9b203a65e1ec1b1429ca4076f3f91b1ae33 Mon Sep 17 00:00:00 2001
> From: John Kehayias <john.kehayias <at> protonmail.com>
> Date: Wed, 20 Jul 2022 23:46:45 -0400
> Subject: [PATCH] environment: Add '--emulate-fhs'.
>
> * guix/scripts/environment.scm (show-environment-options-help, %options): Add
> '--emulate-fhs'.
> (setup-fhs): New procedure.  Setup for the Filesystem Hierarchy Standard (FHS)
> container.  Defines and uses FHS-SYMLINKS and LINK-CONTENTS to create FHS
> expected directories and creates /etc/ld.so.conf.
> (launch-environment): Add 'emulate-fhs?' key and implement it to set $PATH and
> generate /etc/ld.so.cache before calling COMMAND.
> (launch-environment/container): Add 'emulate-fhs?' and 'setup-hook' keys and
> implement them.  Define and use FHS-MAPPINGS, to set up additional bind mounts
> in the container to follow FHS expectations.
> (guix-environment*): Add glibc-for-fhs to the container packages when
> 'emulate-fhs?' key is in OPTS.
> * doc/guix.texi (Invoking guix shell): Document '--emulate-fhs'.
> (Invoking guix environment): Document '--emulate-fhs'.
> * tests/guix-environment-container.sh: Add tests for '--emulate-fhs'.

This new version of the patch looks great to me!

I applied it and took the liberty to make the changes below; let me know
if anything’s amiss.

Pushed together with a news entry:

  2c11e857af news: Add entry for 'guix shell --emulate-fhs'.
  c7ba5f38b8 environment: Add '--emulate-fhs'.
  3d1d29e440 gnu: Add glibc-for-fhs.

Thanks a lot for your work and for your patience!

Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/doc/guix.texi b/doc/guix.texi
index 43dfdd64f2..8803353f24 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -6196,21 +6196,26 @@ directory:
 guix shell --container --expose=$HOME=/exchange guile -- guile
 @end example
 
+@cindex file system hierarchy standard (FHS)
+@cindex FHS (file system hierarchy standard)
 @item --emulate-fhs
-@item -F
-For containers, emulate a Filesystem Hierarchy Standard (FHS)
-configuration within the container, see
-@uref{https://refspecs.linuxfoundation.org/fhs.shtml, the official
-specification}.  As Guix deviates from the FHS specification, this
+@itemx -F
+When used with @option{--container}, emulate a
+@uref{https://refspecs.linuxfoundation.org/fhs.shtml, Filesystem
+Hierarchy Standard (FHS)} configuration within the container, providing
+@file{/bin}, @file{/lib}, and other directories and files specified by
+the FHS.
+
+As Guix deviates from the FHS specification, this
 option sets up the container to more closely mimic that of other
 GNU/Linux distributions.  This is useful for reproducing other
 development environments, testing, and using programs which expect the
 FHS specification to be followed.  With this option, the container will
-include a version of @code{glibc} which will read
-@code{/etc/ld.so.cache} within the container for the shared library
-cache (contrary to @code{glibc} in regular Guix usage) and set up the
-expected FHS directories: @code{/bin}, @code{/etc}, @code{/lib}, and
-@code{/usr} from the container's profile.
+include a version of glibc that will read
+@file{/etc/ld.so.cache} within the container for the shared library
+cache (contrary to glibc in regular Guix usage) and set up the
+expected FHS directories: @file{/bin}, @file{/etc}, @file{/lib}, and
+@file{/usr} from the container's profile.
 
 @item --rebuild-cache
 @cindex caching, of profiles
diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index b566057b41..2a9f2f34fc 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -933,15 +933,15 @@ (define (linker-script? file)
 ;; in FHS containers.
 (define-public glibc-for-fhs
   (hidden-package
-   (package
-     (inherit glibc)
+   (package/inherit glibc
      (name "glibc-for-fhs")
      (source (origin (inherit (package-source glibc))
                      ;; Remove Guix's patch to read ld.so.cache from /gnu/store
                      ;; directories, re-enabling the default /etc/ld.so.cache
                      ;; behavior.
-                     (patches (delete (car (search-patches "glibc-dl-cache.patch"))
-                                      (origin-patches (package-source glibc)))))))))
+                     (patches
+                      (delete (search-patch "glibc-dl-cache.patch")
+                              (origin-patches (package-source glibc)))))))))
 
 ;; Below are old libc versions, which we use mostly to build locale data in
 ;; the old format (which the new libc cannot cope with.)
diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index f5b417457d..cf99760859 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -475,10 +475,10 @@ (define* (launch-environment command profile manifest
      (catch 'system-error
        (lambda ()
          (when emulate-fhs?
-           ;; When running in a container with EMULATE-FHS?, supplement $PATH
+           ;; When running in a container with EMULATE-FHS?, override $PATH
            ;; (optional, but to better match FHS expectations), and generate
            ;; /etc/ld.so.cache.
-           (setenv "PATH" "/bin:/usr/bin:/sbin:/usr/sbin:$PATH")
+           (setenv "PATH" "/bin:/usr/bin:/sbin:/usr/sbin")
            (invoke "ldconfig" "-X"))
          (apply execlp program program args))
        (lambda _
@@ -687,12 +687,15 @@ (define* (launch-environment/container #:key command bash user user-mappings
 list of file system mappings, contains the user-specified host file systems to
 mount inside the container.  If USER is not #f, each target of USER-MAPPINGS
 will be re-written relative to '/home/USER', and USER will be used for the
-passwd entry.  When EMULATE-FHS?, set up the container to follow the
-Filesystem Hierarchy Standard and provide a glibc that reads the cache from
-/etc/ld.so.cache.  SETUP-HOOK is an additional setup procedure to be called, as a
-list with the function name and arguments, currently only used with the
-EMULATE-FHS? option.  LINK-PROFILE? creates a symbolic link from
-~/.guix-profile to the environment profile.
+passwd entry.
+
+When EMULATE-FHS?, set up the container to follow the Filesystem Hierarchy
+Standard and provide a glibc that reads the cache from /etc/ld.so.cache.
+SETUP-HOOK is an additional setup procedure to be called, currently only used
+with the EMULATE-FHS? option.
+
+LINK-PROFILE? creates a symbolic link from ~/.guix-profile to the
+environment profile.
 
 Preserve environment variables whose name matches the one of the regexps in
 WHILE-LIST."
@@ -801,11 +804,9 @@ (define fhs-mappings
             (mkdir-p home-dir)
             (setenv "HOME" home-dir)
 
-            ;; Call an additional setup procedure, if provided.  Currently
-            ;; this is only used with the EMULATE-FHS? option, but could be
-            ;; expanded to a general list of functions to be called.
-            (if setup-hook
-                (apply (car setup-hook) (cdr setup-hook)))
+            ;; Call an additional setup procedure, if provided.
+            (when setup-hook
+              (setup-hook profile))
 
             ;; If requested, link $GUIX_ENVIRONMENT to $HOME/.guix-profile;
             ;; this allows programs expecting that path to continue working as
@@ -1113,9 +1114,9 @@ (define manifest
                                                     #:network? network?
                                                     #:map-cwd? (not no-cwd?)
                                                     #:emulate-fhs? emulate-fhs?
-                                                    #:setup-hook (if emulate-fhs?
-                                                                     (list setup-fhs profile)
-                                                                     #f))))
+                                                    #:setup-hook
+                                                    (and emulate-fhs?
+                                                         setup-fhs))))
 
                    (else
                     (return
diff --git a/tests/guix-environment-container.sh b/tests/guix-environment-container.sh
index 31e409420f..898c57f41b 100644
--- a/tests/guix-environment-container.sh
+++ b/tests/guix-environment-container.sh
@@ -230,4 +230,4 @@ guix environment -C --emulate-fhs --ad-hoc --bootstrap guile-bootstrap \
 
 # Test that the ld cache was generated and can be successfully read.
 guix environment -C --emulate-fhs --ad-hoc --bootstrap guile-bootstrap \
-     -- guile -c '(exit (execlp "ldconfig" "-p"))'
+     -- guile -c '(execlp "ldconfig" "ldconfig" "-p")'

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.