GNU bug report logs - #53355
guix shell --check: confusing error message

Previous Next

To reply to this bug, email your comments to 53355 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: bug-guix <at> gnu.org
Subject: guix shell --check: confusing error message
Date: Tue, 18 Jan 2022 19:29:00 -0800

[Message part 1 (text/plain, inline)]

Hi,

I've grown so used to using "guix environment," I thought I'd try out
"guix shell."  It looks pretty neat!  It's good to try to improve the
CLI.

However, when I tried "guix shell," I quickly observed this confusing
behavior:

--8<---------------cut here---------------start------------->8---
[130] marusich <at> suzaku:~/guix-master
$ guix shell --container --check -D guix
guix shell: checking the environment variables visible from shell '/bin/bash'...
guix shell: warning: variable 'PKG_CONFIG_PATH' is missing from shell environment
hint: One or more environment variables have a different value in the shell than
the one we set.  This means that you may find yourself running code in an
environment different from the one you asked Guix to prepare.

This usually indicates that your shell startup files are unexpectedly
modifying those environment variables.  For example, if you are using Bash,
make sure that environment variables are set or modified in
`~/.bash_profile' and _not_ in `~/.bashrc'.  For more information on Bash
startup files, run:

     info "(bash) Bash Startup Files"

Alternatively, you can avoid the problem by passing the `--container' or
`-C' option.  That will give you a fully isolated environment running in a
"container", immune to the issue described above.

[1] marusich <at> suzaku:~/guix-master
$ env | grep PKG_CONF
[1] marusich <at> suzaku:~/guix-master
$ guix shell --check -D guix
guix shell: checking the environment variables visible from shell '/bin/bash'...
guix shell: warning: variable 'PKG_CONFIG_PATH' is missing from shell environment
hint: One or more environment variables have a different value in the shell than
the one we set.  This means that you may find yourself running code in an
environment different from the one you asked Guix to prepare.

This usually indicates that your shell startup files are unexpectedly
modifying those environment variables.  For example, if you are using Bash,
make sure that environment variables are set or modified in
`~/.bash_profile' and _not_ in `~/.bashrc'.  For more information on Bash
startup files, run:

     info "(bash) Bash Startup Files"

Alternatively, you can avoid the problem by passing the `--container' or
`-C' option.  That will give you a fully isolated environment running in a
"container", immune to the issue described above.

[1] marusich <at> suzaku:~/guix-master
$ guix shell -D guix
[0] [env] marusich <at> suzaku:~/guix-master
$ env | grep PKG
PKG_CONFIG_PATH=/gnu/store/qr79b2m6cfdj8ar7g0psqg4hglm6djfm-profile/lib/pkgconfig
[0] [env] marusich <at> suzaku:~/guix-master
$
exit
[0] marusich <at> suzaku:~/guix-master
$ guix shell --container -D guix
marusich <at> suzaku ~/guix-master [env]$ env | grep PKG
PKG_CONFIG_PATH=/gnu/store/qr79b2m6cfdj8ar7g0psqg4hglm6djfm-profile/lib/pkgconfig
marusich <at> suzaku ~/guix-master [env]$
--8<---------------cut here---------------end--------------->8---

I found the following things to be confusing:

(1) The error message claims that PKG_CONFIG_PATH is "missing from shell
environment."  However, it seems to be present when I run "env".

(2) It says I can avoid the problem by passing the `--container' option,
but even when I do that, the problem seems to persist.  If that is
expected behavior, then perhaps the wording should be changed to
something less certain, such as "you might be able to avoid the
problem".  It does not seem to be the case that I can avoid the problem
by passing the `--container' option in this case.

What's really going on here?  It's good to be able to look at this
feature with the eyes of a newbie, since I'm very used to using "guix
environment", but "guix shell" is totally new to me.  I thought it would
be a good opportunity to provide feedback.

-- 
Chris

PGP: https://savannah.gnu.org/people/viewgpg.php?user_id=106836

[signature.asc (application/pgp-signature, inline)]

Message #8 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: 53355 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Mon, 24 Jan 2022 15:35:52 +0100

[Message part 1 (text/plain, inline)]

Hi Chris,

Chris Marusich <cmmarusich <at> gmail.com> skribis:

> [130] marusich <at> suzaku:~/guix-master
> $ guix shell --container --check -D guix
> guix shell: checking the environment variables visible from shell '/bin/bash'...
> guix shell: warning: variable 'PKG_CONFIG_PATH' is missing from shell environment

[...]

> I found the following things to be confusing:
>
> (1) The error message claims that PKG_CONFIG_PATH is "missing from shell
> environment."  However, it seems to be present when I run "env".
>
> (2) It says I can avoid the problem by passing the `--container' option,
> but even when I do that, the problem seems to persist.  If that is
> expected behavior, then perhaps the wording should be changed to
> something less certain, such as "you might be able to avoid the
> problem".  It does not seem to be the case that I can avoid the problem
> by passing the `--container' option in this case.

What’s confusing is that ‘--check’ does the same job whether or not
‘--container’ is passed: it checks the behavior of your shell *outside*
a container.

I think ‘--check’ should just do nothing when ‘--container’ is used,
possibly emitting a warning saying it’s not doing anything (patch
below).

Now, the diagnostic is hopefully correct if you use, say, ‘--pure’
instead of ‘--container’.  Could you check whether this is the case?

Thanks,
Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
index 510cee727f..ec071402f4 100644
--- a/guix/scripts/environment.scm
+++ b/guix/scripts/environment.scm
@@ -1,6 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2014, 2015, 2018 David Thompson <davet <at> gnu.org>
-;;; Copyright © 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2015-2022 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2018 Mike Gerwitz <mtg <at> gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -975,7 +975,10 @@ (define manifest
 
                   (mwhen (assoc-ref opts 'check?)
                     (return
-                     (validate-child-shell-environment profile manifest)))
+                     (if container?
+                         (warning (G_ "'--check' is unnecessary \
+when using '--container'; doing nothing~%"))
+                         (validate-child-shell-environment profile manifest))))
 
                   (cond
                    ((assoc-ref opts 'search-paths)

Message #11 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 53355 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Mon, 24 Jan 2022 16:55:13 -0800

[Message part 1 (text/plain, inline)]

Hi Ludo,

Thank you for the response!

Ludovic Courtès <ludo <at> gnu.org> writes:

> What’s confusing is that ‘--check’ does the same job whether or not
> ‘--container’ is passed: it checks the behavior of your shell *outside*
> a container.
>
> I think ‘--check’ should just do nothing when ‘--container’ is used,
> possibly emitting a warning saying it’s not doing anything (patch
> below).
>
> Now, the diagnostic is hopefully correct if you use, say, ‘--pure’
> instead of ‘--container’.  Could you check whether this is the case?

That sounds reasonable.  I tested your patch.  It now correctly emits a
warning when both --container and --check are provided.  However, I now
see that the issue occurs even when --container is omitted.  So it seems
like something else might be going on.

Below, I'll provide details of what I did to test your patch.

In one test, I committed your change locally and did "guix pull" to
install the patched Guix into ~/tmpguixprofile.  I then tried using it:

--8<---------------cut here---------------start------------->8---
[0] marusich <at> suzaku:~/guix-master
$ env -i bash
[0] marusich <at> suzaku:/home/marusich/guix-master
$ activate-profile ~/tmpguixprofile
[0] marusich <at> suzaku:/home/marusich/guix-master
$ which guix
/home/marusich/tmpguixprofile/bin/guix
--8<---------------cut here---------------end--------------->8---

First, I tried without --pure or --container.  Below, you can see that
it claims LIBRARY_PATH is missing, but it does not seem to be missing:

--8<---------------cut here---------------start------------->8---
[0] marusich <at> suzaku:/home/marusich/guix-master
$ guix shell --check -D guix
guix shell: checking the environment variables visible from shell '/bin/sh'...
guix shell: warning: variable 'LIBRARY_PATH' is missing from shell environment
hint: One or more environment variables have a different value in the shell than
the one we set.  This means that you may find yourself running code in an
environment different from the one you asked Guix to prepare.

This usually indicates that your shell startup files are unexpectedly
modifying those environment variables.  For example, if you are using Bash,
make sure that environment variables are set or modified in
`~/.bash_profile' and _not_ in `~/.bashrc'.  For more information on Bash
startup files, run:

     info "(bash) Bash Startup Files"

Alternatively, you can avoid the problem by passing the `--container' or
`-C' option.  That will give you a fully isolated environment running in a
"container", immune to the issue described above.

[1] marusich <at> suzaku:/home/marusich/guix-master
$ env | grep LIBRARY_PATH
[1] marusich <at> suzaku:/home/marusich/guix-master
$ guix shell -D guix
[0] \u@\H:\w\n$ env | grep LIBRARY_PATH
LIBRARY_PATH=/gnu/store/hvcq6yjfjjc7060pq09zm1rj02mivg4h-profile/lib
--8<---------------cut here---------------end--------------->8---

Next, I tried with --container - you can see it works as intended:

--8<---------------cut here---------------start------------->8---
[0] marusich <at> suzaku:/home/marusich/guix-master
$ guix shell --container --check -D guix
guix shell: warning: '--check' is unnecessary when using '--container'; doing nothing
--8<---------------cut here---------------end--------------->8---

Next, I tried with --pure and --check - once again, it claims
LIBRARY_PATH is missing, even though it does not seem to be missing:

--8<---------------cut here---------------start------------->8---
[0] marusich <at> suzaku:/home/marusich/guix-master
$ guix shell --pure --check -D guix
guix shell: checking the environment variables visible from shell '/bin/sh'...
guix shell: warning: variable 'LIBRARY_PATH' is missing from shell environment
hint: One or more environment variables have a different value in the shell than
the one we set.  This means that you may find yourself running code in an
environment different from the one you asked Guix to prepare.

This usually indicates that your shell startup files are unexpectedly
modifying those environment variables.  For example, if you are using Bash,
make sure that environment variables are set or modified in
`~/.bash_profile' and _not_ in `~/.bashrc'.  For more information on Bash
startup files, run:

     info "(bash) Bash Startup Files"

Alternatively, you can avoid the problem by passing the `--container' or
`-C' option.  That will give you a fully isolated environment running in a
"container", immune to the issue described above.

[1] marusich <at> suzaku:/home/marusich/guix-master
$ env | grep LIBRARY_PATH
[1] marusich <at> suzaku:/home/marusich/guix-master
$ guix shell --pure -D guix
$ env | grep LIBRARY_PATH
LIBRARY_PATH=/gnu/store/hvcq6yjfjjc7060pq09zm1rj02mivg4h-profile/lib
--8<---------------cut here---------------end--------------->8---

A similar error message occurs if I try the same steps from a checkout
using pre-inst-env.  However, in that case the offending environment
variable is different (PKG_CONFIG_PATH in this case):

--8<---------------cut here---------------start------------->8---
[0] marusich <at> suzaku:~/guix-master
$ guix environment guix
[0] [env] marusich <at> suzaku:~/guix-master
$ ./pre-inst-env guix shell --check -D guix -- bash -c 'echo in env, PKG_CONFIG_PATH="$PKG_CONFIG_PATH"'
guix shell: checking the environment variables visible from shell '/bin/bash'...
guix shell: warning: variable 'PKG_CONFIG_PATH' is missing from shell environment
hint: One or more environment variables have a different value in the shell than
the one we set.  This means that you may find yourself running code in an
environment different from the one you asked Guix to prepare.

This usually indicates that your shell startup files are unexpectedly
modifying those environment variables.  For example, if you are using Bash,
make sure that environment variables are set or modified in
`~/.bash_profile' and _not_ in `~/.bashrc'.  For more information on Bash
startup files, run:

     info "(bash) Bash Startup Files"

Alternatively, you can avoid the problem by passing the `--container' or
`-C' option.  That will give you a fully isolated environment running in a
"container", immune to the issue described above.

[1] [env] marusich <at> suzaku:~/guix-master
$ ./pre-inst-env guix shell --check --pure -D guix -- bash -c 'echo in env, PKG_CONFIG_PATH="$PKG_CONFIG_PATH"'
guix shell: checking the environment variables visible from shell '/bin/bash'...
guix shell: warning: variable 'PKG_CONFIG_PATH' is missing from shell environment
hint: One or more environment variables have a different value in the shell than
the one we set.  This means that you may find yourself running code in an
environment different from the one you asked Guix to prepare.

This usually indicates that your shell startup files are unexpectedly
modifying those environment variables.  For example, if you are using Bash,
make sure that environment variables are set or modified in
`~/.bash_profile' and _not_ in `~/.bashrc'.  For more information on Bash
startup files, run:

     info "(bash) Bash Startup Files"

Alternatively, you can avoid the problem by passing the `--container' or
`-C' option.  That will give you a fully isolated environment running in a
"container", immune to the issue described above.

[1] [env] marusich <at> suzaku:~/guix-master
$ ./pre-inst-env guix shell --check --container -D guix -- bash -c 'echo in env, PKG_CONFIG_PATH="$PKG_CONFIG_PATH"'
guix shell: warning: '--check' is unnecessary when using '--container'; doing nothing
in env, PKG_CONFIG_PATH=/gnu/store/hvcq6yjfjjc7060pq09zm1rj02mivg4h-profile/lib/pkgconfig
[0] [env] marusich <at> suzaku:~/guix-master
$ ./pre-inst-env guix shell -D guix -- bash -c 'echo in env, PKG_CONFIG_PATH="$PKG_CONFIG_PATH"'
in env, PKG_CONFIG_PATH=/gnu/store/hvcq6yjfjjc7060pq09zm1rj02mivg4h-profile/lib/pkgconfig:/gnu/store/hvcq6yjfjjc7060pq09zm1rj02mivg4h-profile/lib/pkgconfig
[0] [env] marusich <at> suzaku:~/guix-master
$ ./pre-inst-env guix shell --pure -D guix -- bash -c 'echo in env, PKG_CONFIG_PATH="$PKG_CONFIG_PATH"'
in env, PKG_CONFIG_PATH=/gnu/store/hvcq6yjfjjc7060pq09zm1rj02mivg4h-profile/lib/pkgconfig
[0] [env] marusich <at> suzaku:~/guix-master
$ ./pre-inst-env guix shell --container -D guix -- bash -c 'echo in env, PKG_CONFIG_PATH="$PKG_CONFIG_PATH"'
in env, PKG_CONFIG_PATH=/gnu/store/hvcq6yjfjjc7060pq09zm1rj02mivg4h-profile/lib/pkgconfig
[0] [env] marusich <at> suzaku:~/guix-master
$ echo out of env, PKG_CONFIG_PATH="$PKG_CONFIG_PATH"
out of env, PKG_CONFIG_PATH=/gnu/store/hvcq6yjfjjc7060pq09zm1rj02mivg4h-profile/lib/pkgconfig
[0] [env] marusich <at> suzaku:~/guix-master
$
--8<---------------cut here---------------end--------------->8---

It seems this issue happens regardless of whether I use pre-inst-env or
run Guix from a "guix pull" installation.

-- 
Chris

PGP: https://savannah.gnu.org/people/viewgpg.php?user_id=106836

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: 53355 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Tue, 25 Jan 2022 14:39:46 +0100

Hi Chris,

Chris Marusich <cmmarusich <at> gmail.com> skribis:

> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> What’s confusing is that ‘--check’ does the same job whether or not
>> ‘--container’ is passed: it checks the behavior of your shell *outside*
>> a container.
>>
>> I think ‘--check’ should just do nothing when ‘--container’ is used,
>> possibly emitting a warning saying it’s not doing anything (patch
>> below).
>>
>> Now, the diagnostic is hopefully correct if you use, say, ‘--pure’
>> instead of ‘--container’.  Could you check whether this is the case?
>
> That sounds reasonable.  I tested your patch.  It now correctly emits a
> warning when both --container and --check are provided.

Great, I’ll commit it.

> First, I tried without --pure or --container.  Below, you can see that
> it claims LIBRARY_PATH is missing, but it does not seem to be missing:

[...]

> Next, I tried with --pure and --check - once again, it claims
> LIBRARY_PATH is missing, even though it does not seem to be missing:

It looks like the shell-check machinery is misdiagnosing things, as
Vagrant reported in <https://issues.guix.gnu.org/51466> (is this on
Debian too?).

Could you try the debugging tricks I proposed there?

TIA,
Ludo’.

Message #19 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: 53355 <at> debbugs.gnu.org, 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Tue, 08 Mar 2022 20:07:21 +0100

Hi Chris,

Did you have a chance to look into it?

  https://issues.guix.gnu.org/53355

TIA.  :-)

Ludo’.

Ludovic Courtès <ludo <at> gnu.org> skribis:

> Hi Chris,
>
> Thanks for debugging this!
>
> Chris Marusich <cmmarusich <at> gmail.com> skribis:
>
>> From c3eea81846ae71a246e6b592be74062f4bf26474 Mon Sep 17 00:00:00 2001
>> From: Chris Marusich <cmmarusich <at> gmail.com>
>> Date: Sun, 13 Feb 2022 14:15:14 -0800
>> Subject: [PATCH] environment: Prevent PS1 from clobbering output in 'check'.
>>
>> Fixes: <https://issues.guix.gnu.org/51466>.
>>
>> * guix/scripts/environment.scm (child-shell-environment): In the script
>> executed the child shell, set PS1 to an empty value and then echo three
>> sentinel lines to try to "flush" the original PS1 value before printing the
>> environment variables.  In the parent process, read and discard all lines up
>> to and including the last sentinel line.  After that, read the remaining lines
>> as usual.
>
> [...]
>
>> +    ;; Why print "GUIX_FLUSH" a few times?  We are trying to "flush" the
>> +    ;; original PS1 value to the port so we can read it (and discard it)
>> +    ;; before we start reading the environment variables from the port.  If we
>> +    ;; don't do this, the original PS1 value can sometimes get interleaved
>> +    ;; into the output, which interferes with our parsing logic.  It's a hack,
>> +    ;; but in practice it seems to do the job.  If you know of a more graceful
>> +    ;; solution, please implement it!  See: https://issues.guix.gnu.org/51466
>> +    "PS1=; for i in 1 2 3; do echo GUIX_FLUSH_$i; done; \
>> +env || /usr/bin/env || set; echo GUIX-CHECK-DONE; read x; exit\n")
>
> So you confirm that a single “echo” is not enough, right?
>
> Perhaps we should unroll the ‘for’ loop for portability, to be on the
> safe side.  Initially I tested with Bash, Zsh, and Fish:
>
>   https://issues.guix.gnu.org/51285#0-lineno49
>
> I think Fish has a very non-POSIX syntax, hence the suggestion to avoid
> ‘for’.
>
> I realized that setting PS1 could interfere with the logic below that
> checks for PS1.  And since it doesn’t seem to help, perhaps we can
> remove “PS1=;”?
>
> Thoughts?
>
> Sorry to answer with yet more questions!
>
> Thanks,
> Ludo’.

Message #22 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: 53355 <at> debbugs.gnu.org, 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Fri, 20 May 2022 23:37:01 +0200

Hi Chris,

A friendly ping.  :-)

Ludo’.

Ludovic Courtès <ludo <at> gnu.org> skribis:

> Hi Chris,
>
> Did you have a chance to look into it?
>
>   https://issues.guix.gnu.org/53355
>
> TIA.  :-)
>
> Ludo’.
>
> Ludovic Courtès <ludo <at> gnu.org> skribis:
>
>> Hi Chris,
>>
>> Thanks for debugging this!
>>
>> Chris Marusich <cmmarusich <at> gmail.com> skribis:
>>
>>> From c3eea81846ae71a246e6b592be74062f4bf26474 Mon Sep 17 00:00:00 2001
>>> From: Chris Marusich <cmmarusich <at> gmail.com>
>>> Date: Sun, 13 Feb 2022 14:15:14 -0800
>>> Subject: [PATCH] environment: Prevent PS1 from clobbering output in 'check'.
>>>
>>> Fixes: <https://issues.guix.gnu.org/51466>.
>>>
>>> * guix/scripts/environment.scm (child-shell-environment): In the script
>>> executed the child shell, set PS1 to an empty value and then echo three
>>> sentinel lines to try to "flush" the original PS1 value before printing the
>>> environment variables.  In the parent process, read and discard all lines up
>>> to and including the last sentinel line.  After that, read the remaining lines
>>> as usual.
>>
>> [...]
>>
>>> +    ;; Why print "GUIX_FLUSH" a few times?  We are trying to "flush" the
>>> +    ;; original PS1 value to the port so we can read it (and discard it)
>>> +    ;; before we start reading the environment variables from the port.  If we
>>> +    ;; don't do this, the original PS1 value can sometimes get interleaved
>>> +    ;; into the output, which interferes with our parsing logic.  It's a hack,
>>> +    ;; but in practice it seems to do the job.  If you know of a more graceful
>>> +    ;; solution, please implement it!  See: https://issues.guix.gnu.org/51466
>>> +    "PS1=; for i in 1 2 3; do echo GUIX_FLUSH_$i; done; \
>>> +env || /usr/bin/env || set; echo GUIX-CHECK-DONE; read x; exit\n")
>>
>> So you confirm that a single “echo” is not enough, right?
>>
>> Perhaps we should unroll the ‘for’ loop for portability, to be on the
>> safe side.  Initially I tested with Bash, Zsh, and Fish:
>>
>>   https://issues.guix.gnu.org/51285#0-lineno49
>>
>> I think Fish has a very non-POSIX syntax, hence the suggestion to avoid
>> ‘for’.
>>
>> I realized that setting PS1 could interfere with the logic below that
>> checks for PS1.  And since it doesn’t seem to help, perhaps we can
>> remove “PS1=;”?
>>
>> Thoughts?
>>
>> Sorry to answer with yet more questions!
>>
>> Thanks,
>> Ludo’.

Message #25 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 53355 <at> debbugs.gnu.org, 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Mon, 23 May 2022 21:42:36 -0700

[Message part 1 (text/plain, inline)]

Hi Ludo,

Ludovic Courtès <ludo <at> gnu.org> writes:

> So you confirm that a single “echo” is not enough, right?

I didn't test one specifically. It might work with just one, but it did
work with three.  If we want to proceed with the "echo" approach, let me
know and I'll test just one echo to see if that is reliable enough.

> Perhaps we should unroll the ‘for’ loop for portability, to be on the
> safe side.  Initially I tested with Bash, Zsh, and Fish:
>
>   https://issues.guix.gnu.org/51285#0-lineno49
>
> I think Fish has a very non-POSIX syntax, hence the suggestion to avoid
> ‘for’.

I see.  Yes, I'll do that if we decide to go with the echo-based
approach.

> I realized that setting PS1 could interfere with the logic below that
> checks for PS1.  And since it doesn’t seem to help, perhaps we can
> remove “PS1=;”?

I recall that I tried removing PS1, and I still had trouble.  I believe
it was because even if we unset PS1 as the very first command we do, the
original prompt is still printed.  Foreign distros usually set PS1 to
something, and whatever that is will be printed before we have a chance
to input any commands.  It's hard to avoid that in general.

> Thoughts?

One alternative method I tried successfully in a variety of shells was
to use shell redirection (see attached).  I like this approach.
However, this will only work in shells that support redirection.  I
recall testing with bash, ash (busybox's shell), dash, zsh, fish, ksh,
and csh.  I recall that only csh failed, since it doesn't support
redirection.

I personally like the attached patch better than what I proposed
earlier.  The earlier patch just echoes a few times.  Presumably, it
only works because the PS1 prompt is likely (but not guaranteed) to be
emitted before the last of the echo commands finishes printing.  I'd
rather just control the desired output and ignore PS1 entirely, and that
is what the attached patch accomplishes using FDs.  However, if support
for shells without redirection is a requirement, then maybe the original
hack (echo a few times) is OK, or perhaps we need something else.

How would you like to proceed?  Is it OK to rely on shell redirection?

-- 
Chris

PGP: https://savannah.gnu.org/people/viewgpg.php?user_id=106836

[0001-environment-Prevent-PS1-from-clobbering-output-in-ch.patch (text/x-patch, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #28 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: 53355 <at> debbugs.gnu.org, 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Mon, 13 Jun 2022 12:03:28 +0200

Hi Chris,

Chris Marusich <cmmarusich <at> gmail.com> skribis:

> One alternative method I tried successfully in a variety of shells was
> to use shell redirection (see attached).  I like this approach.
> However, this will only work in shells that support redirection.  I
> recall testing with bash, ash (busybox's shell), dash, zsh, fish, ksh,
> and csh.  I recall that only csh failed, since it doesn't support
> redirection.

That’s a good success list in my view; not being a POSIX shell, (t)csh
was already excluded from the list in the original submission I think:
<https://issues.guix.gnu.org/51285>.

> I personally like the attached patch better than what I proposed
> earlier.  The earlier patch just echoes a few times.  Presumably, it
> only works because the PS1 prompt is likely (but not guaranteed) to be
> emitted before the last of the echo commands finishes printing.  I'd
> rather just control the desired output and ignore PS1 entirely, and that
> is what the attached patch accomplishes using FDs.  However, if support
> for shells without redirection is a requirement, then maybe the original
> hack (echo a few times) is OK, or perhaps we need something else.
>
> How would you like to proceed?  Is it OK to rely on shell redirection?

Yeah, I think so.  This new approach looks more robust.

> From 9a1cef589abf01b61e22656f44c76441f4c50ffd Mon Sep 17 00:00:00 2001
> From: Chris Marusich <cmmarusich <at> gmail.com>
> Date: Fri, 11 Mar 2022 00:20:12 -0800
> Subject: [PATCH] environment: Prevent PS1 from clobbering output in 'check'.
>
> Fixes: <https://issues.guix.gnu.org/51466>.
>
> * guix/scripts/environment.scm (child-shell-environment) [shell-pipe]
> [shell-pipe-in, shell-pipe-out]: New local variables.
> [script]: Redirect the stdout of each command to the file descriptor of the
> shell-pipe-out port.
> [lines]: In the child, close shell-pipe-in before starting the shell.  In the
> parent, close shell-pipe-out before sending the script to the shell.  Read
> lines from shell-pipe-in, not port, so that the shell's PS1 prompt cannot
> clobber the lines.  Close shell-pipe-in just before waiting on the child.

LGTM, please push!

Ludo’.

Message #31 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 53355 <at> debbugs.gnu.org, 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Sun, 19 Jun 2022 13:40:50 -0700

[Message part 1 (text/plain, inline)]

Hi Ludo,

Thank you for the review!

Ludovic Courtès <ludo <at> gnu.org> writes:

> LGTM, please push!

Before pushing, I did some more tests to make sure it was still working.
When I did this, I noticed that read-line was no longer returning
strings that end in "\r".  This prevents child-shell-environment from
behaving correctly, since it incorrectly assumes that all the lines end
in "\r", stripping it off unconditionally.  In the past, I'm sure
read-line was returning strings that end in "\r".  I don't know what
changed, but I've attached a second patch that fixes this issue, also.

Unless you have more feedback, I'll go ahead and push both patches to
master in a few days.

-- 
Chris

PGP: https://savannah.gnu.org/people/viewgpg.php?user_id=106836

[0001-environment-Don-t-assume-that-lines-have-a-trailing-.patch (text/x-patch, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #34 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: 53355 <at> debbugs.gnu.org, 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Mon, 20 Jun 2022 09:34:33 +0200

Hi,

Chris Marusich <cmmarusich <at> gmail.com> skribis:

> Before pushing, I did some more tests to make sure it was still working.
> When I did this, I noticed that read-line was no longer returning
> strings that end in "\r".  This prevents child-shell-environment from
> behaving correctly, since it incorrectly assumes that all the lines end
> in "\r", stripping it off unconditionally.  In the past, I'm sure
> read-line was returning strings that end in "\r".  I don't know what
> changed, but I've attached a second patch that fixes this issue, also.

Weird, not sure what could have changed.

> Unless you have more feedback, I'll go ahead and push both patches to
> master in a few days.

Great, thank you!

Ludo’.

Message #37 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: bokr <at> bokr.com
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 53355 <at> debbugs.gnu.org,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#51466: bug#53355: guix shell --check: confusing error message
Date: Mon, 20 Jun 2022 12:12:10 +0200

Hi Chris,

Did you observe this behaviour inside a git repo directory?
I wonder if this git security thing could be relevant:
    https://lwn.net/Articles/892755/
It makes also me wonder about readline completion stuff
possibly interacting. Isn't that implemented with readline?

I have had some mystery bash parsing errors, and I noticed
    set|less
shows a heck of a lot of functions defined that I don't
remember seeing in the past. 
Anyway, shouldn't stuff like that have better hygiene than just prefixed
_underscore ? Or maybe set|less doesn't show all that on your system?

Disclaimer: I played a lot of games trying to make stuff conditional
at login, where I  renamed .bash_profile and .bashrc (e.g. .my_bashrc)
which brought .profile into play, and I messed with the downstream
of that to source some .my_'s conditionally, so I've go a fragile mess right now ;/

Anyway, did you determine why things changed in the first place?
Or will this be a whack-a-mole game with future weirdnesses? :)

Semms like IWBN to have interfaces governed by contracts :)

Best,
Bengt Richter

On +2022-06-19 13:40:50 -0700, Chris Marusich wrote:
> Hi Ludo,
> 
> Thank you for the review!
> 
> Ludovic Courtès <ludo <at> gnu.org> writes:
> 
> > LGTM, please push!
> 
> Before pushing, I did some more tests to make sure it was still working.
> When I did this, I noticed that read-line was no longer returning
> strings that end in "\r".  This prevents child-shell-environment from
> behaving correctly, since it incorrectly assumes that all the lines end
> in "\r", stripping it off unconditionally.  In the past, I'm sure
> read-line was returning strings that end in "\r".  I don't know what
> changed, but I've attached a second patch that fixes this issue, also.
> 
> Unless you have more feedback, I'll go ahead and push both patches to
> master in a few days.
> 
> -- 
> Chris
> 
> PGP: https://savannah.gnu.org/people/viewgpg.php?user_id=106836

> From c4fee9e63f8cb694de86ae46bd1e2e4c692eb6f6 Mon Sep 17 00:00:00 2001
> From: Chris Marusich <cmmarusich <at> gmail.com>
> Date: Sun, 19 Jun 2022 13:16:04 -0700
> Subject: [PATCH] environment: Don't assume that lines have a trailing "\r".
> 
> I've noticed that the child-shell-environment procedure is misbehaving on my
> computer because the lines returned by read-line do not have a trailing "\r".
> In the past, I recall that such lines did in fact have a trailing "\r".  I'm
> not sure why it changed, but it seems prudent to just rewrite this code to
> tolerate both cases, since it seems that both cases can happen.
> 
> * guix/scripts/environment.scm (child-shell-environment) [lines]: Instead of
> checking if the line exactly matches "GUIX_CHECK_DONE\r"; check if the line
> begins with "GUIX_CHECK_DONE".  Instead of always stripping the trailing
> character from the line, only do it if the line has a trailing "\r".
> ---
>  guix/scripts/environment.scm | 13 +++++++++----
>  1 file changed, 9 insertions(+), 4 deletions(-)
> 
> diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm
> index f0cb341aab..1fb4f5b7c6 100644
> --- a/guix/scripts/environment.scm
> +++ b/guix/scripts/environment.scm
> @@ -462,13 +462,18 @@ (define lines
>                            ;; prompt from getting mixed into what we read.
>                            (match (read-line shell-pipe-in)
>                              ((? eof-object?) (reverse lines))
> -                            ("GUIX-CHECK-DONE\r"
> +                            ((? (lambda (line)
> +                                  ;; The line might or might not have a trailing \r.
> +                                  (string-prefix? "GUIX-CHECK-DONE" line)))
>                               (display "done\n" port)
>                               (reverse lines))
>                              (line
> -                             ;; Drop the '\r' from LINE.
> -                             (loop (cons (string-drop-right line 1)
> -                                         lines))))))))
> +                             ;; Strip the trailing '\r' from LINE if present.
> +                             (let ((stripped-line
> +                                    (if (string-suffix? "\r" line)
> +                                        (string-drop-right line 1)
> +                                        line)))
> +                               (loop (cons stripped-line lines)))))))))
>           (close-port port)
>           (close-port shell-pipe-in)
>           (waitpid pid)
> -- 
> 2.34.0
> 

Message #40 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Bengt Richter <bokr <at> bokr.com>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 53355 <at> debbugs.gnu.org,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: bug#51466: bug#53355: guix shell --check: confusing
 error message
Date: Mon, 20 Jun 2022 19:56:56 +0200

Sorry to reply to myself, but forgot to illustrate.

On +2022-06-20 12:12:10 +0200, bokr <at> bokr.com wrote:
> Hi Chris,
[...]
> 
> I have had some mystery bash parsing errors, and I noticed
>     set|less
> shows a heck of a lot of functions defined that I don't
> remember seeing in the past. 
> Anyway, shouldn't stuff like that have better hygiene than just prefixed
> _underscore ? Or maybe set|less doesn't show all that on your system?
>

There are a couple functions without prefixed underscore too,
which invoke some underscore-prefixed ones that look too trusting
of their arguments if you ask me: can someone declare these safe?

    I think I can grok quote () ...
(escape single quotes and enclose result in single quotes, trusting bash state)
But what if I want to define my own function quote?? How would I know I was
overriding this? I really don't like my programming space occupied by unknowns :-(

--8<---------------cut here---------------start------------->8---
quote () 
{ 
    local quoted=${1//\'/\'\\\'\'};
    printf "'%s'" "$quoted"
}
--8<---------------cut here---------------end--------------->8---

    but this one below will take more time than I want to spend on code
I'm not intentionally going to use, and which invites name clashes
in my command name space :-(

--8<---------------cut here---------------start------------->8---
quote_readline () 
{ 
    local quoted;
    _quote_readline_by_ref "$1" ret;
    printf %s "$ret"
}
--8<---------------cut here---------------end--------------->8---

    where the above calls this:

--8<---------------cut here---------------start------------->8---
_quote_readline_by_ref () 
{ 
    if [ -z "$1" ]; then
        printf -v $2 %s "$1";
    else
        if [[ $1 == \'* ]]; then
            printf -v $2 %s "${1:1}";
        else
            if [[ $1 == ~* ]]; then
                printf -v $2 ~%q "${1:1}";
            else
                printf -v $2 %q "$1";
            fi;
        fi;
    fi;
    [[ ${!2} == \$* ]] && eval $2=${!2}
}
--8<---------------cut here---------------end--------------->8---

HTH somehow.
--
Regards,
Bengt Richter

Message #43 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Bengt Richter <bokr <at> bokr.com>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 53355 <at> debbugs.gnu.org,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: bug#51466: bug#53355: guix shell --check: confusing
 error message
Date: Tue, 21 Jun 2022 01:27:49 +0200

Sorry again, but I found the source:

tl;dr: These functions are defined in
    /usr/share/bash-completion/bash_completion
which looks awful kludgey to me, (however clever :)

There is a reference to
    http://lists.gnu.org/archive/html/bug-bash/2009-03/msg00155.html
in the header comments for
    _quote_readline_by_ref () 

I also found
    https://askubuntu.com/questions/571544/bash-tab-completion-bash-unexpected-eof-while-looking-for-matching-bash
My bet is there is at least one bug active now.

Completion is really nice when it works, but IMO they certainly
shouldn't use a name like unadorned ``quote'' in their implementation.

And I think it would be prettier in scheme :)
Lots could be prettier if bash could be extended with scheme.

I'm about out of time to chase this, but I expect to bump into it again ;/
HTH.
--
Regards,
Bengt Richter

On +2022-06-20 19:56:56 +0200, Bengt Richter wrote:
> Sorry to reply to myself, but forgot to illustrate.
> 
> On +2022-06-20 12:12:10 +0200, bokr <at> bokr.com wrote:
> > Hi Chris,
> [...]
> > 
> > I have had some mystery bash parsing errors, and I noticed
> >     set|less
> > shows a heck of a lot of functions defined that I don't
> > remember seeing in the past. 
> > Anyway, shouldn't stuff like that have better hygiene than just prefixed
> > _underscore ? Or maybe set|less doesn't show all that on your system?
> >
> 
> There are a couple functions without prefixed underscore too,
> which invoke some underscore-prefixed ones that look too trusting
> of their arguments if you ask me: can someone declare these safe?
> 
>     I think I can grok quote () ...
> (escape single quotes and enclose result in single quotes, trusting bash state)
> But what if I want to define my own function quote?? How would I know I was
> overriding this? I really don't like my programming space occupied by unknowns :-(
> 
> --8<---------------cut here---------------start------------->8---
> quote () 
> { 
>     local quoted=${1//\'/\'\\\'\'};
>     printf "'%s'" "$quoted"
> }
> --8<---------------cut here---------------end--------------->8---
> 
>     but this one below will take more time than I want to spend on code
> I'm not intentionally going to use, and which invites name clashes
> in my command name space :-(
> 
> --8<---------------cut here---------------start------------->8---
> quote_readline () 
> { 
>     local quoted;
>     _quote_readline_by_ref "$1" ret;
>     printf %s "$ret"
> }
> --8<---------------cut here---------------end--------------->8---
> 
>     where the above calls this:
> 
> --8<---------------cut here---------------start------------->8---
> _quote_readline_by_ref () 
> { 
>     if [ -z "$1" ]; then
>         printf -v $2 %s "$1";
>     else
>         if [[ $1 == \'* ]]; then
>             printf -v $2 %s "${1:1}";
>         else
>             if [[ $1 == ~* ]]; then
>                 printf -v $2 ~%q "${1:1}";
>             else
>                 printf -v $2 %q "$1";
>             fi;
>         fi;
>     fi;
>     [[ ${!2} == \$* ]] && eval $2=${!2}
> }
> --8<---------------cut here---------------end--------------->8---
> 
> HTH somehow.
> --
> Regards,
> Bengt Richter
> 
> 
> 

Message #46 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Thiago Jung Bauermann <bauermann <at> kolabnow.com>
To: Bengt Richter <bokr <at> bokr.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 Chris Marusich <cmmarusich <at> gmail.com>, 53355 <at> debbugs.gnu.org,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#51466: bug#53355: bug#51466: bug#53355: guix shell --check:
 confusing error message
Date: Tue, 21 Jun 2022 01:00:48 -0300

Hello,

Bengt Richter <bokr <at> bokr.com> writes:

> Lots could be prettier if bash could be extended with scheme.

Today is your lucky day. :-)

$ guix show guile-bash | recsel -p name,synopsis
name: guile-bash
synopsis: Extend Bash using Guile  

-- 
Thanks
Thiago

Message #49 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 53355 <at> debbugs.gnu.org, 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Sat, 25 Jun 2022 02:07:50 -0700

[Message part 1 (text/plain, inline)]

Hi Ludo & Everyone,

Chris Marusich <cmmarusich <at> gmail.com> writes:

> Is it OK to rely on shell redirection?

It turns out that it is probably not OK to rely on shell redirection in
this case, after all.  For example, "dash does not support multi-digit
file descriptors":

https://bugs.launchpad.net/ubuntu/+source/dash/+bug/249620

Indeed, the patch I proposed earlier to rely on shell redirection caused
a command like

./pre-inst-env env SHELL=/gnu/store/nm0hccsphymxi8c24xmg6ixm9vcf25xb-dash-0.5.11.5/bin/dash guix shell --check --container -D guix

to hang.  It hangs because the FD Guile chooses to create and embed in
the script is 19 (on my machine, at least).  A redirection like
"env >&19" causes dash to error out, so no environment information gets
sent back to the parent process.  The same issue seemed to occur for the
ksh from our oksh package.

To resolve this, I changed the code so that it just writes to a
temporary file.  This is simple and more robust.  With the attached
patch, I was able to use a command like the one above to verify that
"guix environment --check" works correctly for Guix-built bash, dash,
ksh, fish, zsh, and ash.  I also verified that it works for Fedora's
/bin/sh and /bin/bash.

What do you think of this file-based approach?  Supporting many
different shells is pretty tricky, but I think this patch does a good
enough job.

-- 
Chris

PGP: https://savannah.gnu.org/people/viewgpg.php?user_id=106836

[0001-environment-Prevent-PS1-from-clobbering-output-in-ch.patch (text/x-patch, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #52 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: Chris Marusich <cmmarusich <at> gmail.com>, Ludovic
 Courtès <ludo <at> gnu.org>
Cc: 53355 <at> debbugs.gnu.org, 51466 <at> debbugs.gnu.org
Subject: Re: bug#51466: bug#53355: guix shell --check: confusing error message
Date: Sat, 25 Jun 2022 11:37:46 +0200

[Message part 1 (text/plain, inline)]

Chris Marusich schreef op za 25-06-2022 om 02:07 [-0700]:
> It turns out that it is probably not OK to rely on shell redirection
> in
> this case, after all.  For example, "dash does not support multi-
> digit
> file descriptors":
> 
> https://bugs.launchpad.net/ubuntu/+source/dash/+bug/249620

I consider temporary files to be more fragile -- you have to take care
of file permissions, removing the file afterwards even after an
interrupt with C-c, deleting the temporary file can fail, there might
be an out-of-space error, in case of file system corruption things
might be remounted read-only, some other program could read, write or
delete the file ..., so I think it would be best to just fix the bug in
dash instead.

Greetings,
Maxime.

[signature.asc (application/pgp-signature, inline)]

Message #55 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Chris Marusich <cmmarusich <at> gmail.com>
To: Maxime Devos <maximedevos <at> telenet.be>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 53355 <at> debbugs.gnu.org,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Sat, 25 Jun 2022 09:52:26 -0700

[Message part 1 (text/plain, inline)]

Hi Maxime,

Maxime Devos <maximedevos <at> telenet.be> writes:

> Chris Marusich schreef op za 25-06-2022 om 02:07 [-0700]:
>> It turns out that it is probably not OK to rely on shell redirection
>> in
>> this case, after all.  For example, "dash does not support multi-
>> digit
>> file descriptors":
>> 
>> https://bugs.launchpad.net/ubuntu/+source/dash/+bug/249620
>
> I consider temporary files to be more fragile -- you have to take care
> of file permissions, removing the file afterwards even after an
> interrupt with C-c, deleting the temporary file can fail, there might
> be an out-of-space error, in case of file system corruption things
> might be remounted read-only, some other program could read, write or
> delete the file ..., so I think it would be best to just fix the bug in
> dash instead.

Yes, I agree those are good reasons to avoid a temporary file if we can.
To that end, do you know if we can somehow force Guile to use a specific
file descriptor for the pipe?  In the patch I wrote earlier, which uses
redirection, the problem was that I could not control Guile's choice of
file descriptors.  Guile chose file descriptor 19 for one end of the
pipe, and I don't know how to make it use anything else.  If we can
arrange for Guile to consistently use file descriptor 7, for example,
then probably it would work in all the shell I've tested.

I wonder if maybe I can just duplicate the file descriptor?  I don't
know; if for example Guile reserves all the file descriptors below 10
for other uses, it might be hard.

What do you think?  Is there a way to do it?

-- 
Chris

PGP: https://savannah.gnu.org/people/viewgpg.php?user_id=106836

[signature.asc (application/pgp-signature, inline)]

Message #58 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 53355 <at> debbugs.gnu.org,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Sat, 25 Jun 2022 19:40:48 +0200

[Message part 1 (text/plain, inline)]

Chris Marusich schreef op za 25-06-2022 om 09:52 [-0700]:
> Yes, I agree those are good reasons to avoid a temporary file if we
> can.
> To that end, do you know if we can somehow force Guile to use a
> specific
> file descriptor for the pipe?  In the patch I wrote earlier, which
> uses
> redirection, the problem was that I could not control Guile's choice
> of
> file descriptors.  Guile chose file descriptor 19 for one end of the
> pipe, and I don't know how to make it use anything else.  If we can
> arrange for Guile to consistently use file descriptor 7, for example,
> then probably it would work in all the shell I've tested.
> 
> I wonder if maybe I can just duplicate the file descriptor?  I don't
> know; if for example Guile reserves all the file descriptors below 10
> for other uses, it might be hard.
> 

Have a look at ‘(guile)Ports and File Descriptors’.  It has lots of
procedures for duplicating and renumbering.  That's fragile though, you
might accidentally overwrite an fd that's being used for something
else.

(Normally move->fdes would prevent overwriting things by moving pre-
existing fds out of the way, adjusting ports automatically, but move-
>fdes doesn't know (yet) about the pipe that Guile uses for
finalisation, see maybe:
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=48563>)

I think it would be best to patch the dash appropriately (though fixing
move->fdes would be nice too).

Greetings,
Maximee.

[signature.asc (application/pgp-signature, inline)]

Message #61 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: bokr <at> bokr.com
To: Maxime Devos <maximedevos <at> telenet.be>
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 Chris Marusich <cmmarusich <at> gmail.com>, 53355 <at> debbugs.gnu.org,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Sat, 25 Jun 2022 22:06:46 +0200

On +2022-06-25 19:40:48 +0200, Maxime Devos wrote:
> Chris Marusich schreef op za 25-06-2022 om 09:52 [-0700]:
> > Yes, I agree those are good reasons to avoid a temporary file if we
> > can.
> > To that end, do you know if we can somehow force Guile to use a
> > specific
> > file descriptor for the pipe?  In the patch I wrote earlier, which
> > uses
> > redirection, the problem was that I could not control Guile's choice
> > of
> > file descriptors.  Guile chose file descriptor 19 for one end of the
> > pipe, and I don't know how to make it use anything else.  If we can
> > arrange for Guile to consistently use file descriptor 7, for example,
> > then probably it would work in all the shell I've tested.
> > 
> > I wonder if maybe I can just duplicate the file descriptor?  I don't
> > know; if for example Guile reserves all the file descriptors below 10
> > for other uses, it might be hard.
> > 
> 
> Have a look at ‘(guile)Ports and File Descriptors’.  It has lots of
> procedures for duplicating and renumbering.  That's fragile though, you
> might accidentally overwrite an fd that's being used for something
> else.
> 
> (Normally move->fdes would prevent overwriting things by moving pre-
> existing fds out of the way, adjusting ports automatically, but move-
> >fdes doesn't know (yet) about the pipe that Guile uses for
> finalisation, see maybe:
> <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=48563>)
> 
> I think it would be best to patch the dash appropriately (though fixing
> move->fdes would be nice too).
> 
> Greetings,
> Maximee.

Could this help?:

(from man 2 openat (scroll down a fair bit):
-8<---------------cut here---------------start------------->8---
  There are two main use cases for O_TMPFILE:

  *  Improved tmpfile(3) functionality: race-free creation of temporary files that (1) are automatically
     deleted  when closed; (2) can never be reached via any pathname; (3) are not subject to symlink at‐
     tacks; and (4) do not require the caller to devise unique names.

  *  Creating a file that is initially invisible, which is then populated with data and adjusted to have
     appropriate  filesystem  attributes (fchown(2), fchmod(2), fsetxattr(2), etc.)  before being atomi‐
     cally linked into the filesystem in a fully formed state (using linkat(2) as described above).

  O_TMPFILE requires support by the underlying filesystem; only a subset of  Linux  filesystems  provide
  that  support.   In  the  initial  implementation,  support was provided in the ext2, ext3, ext4, UDF,
  Minix, and shmem filesystems.  Support for other filesystems has subsequently been added  as  follows:
  XFS (Linux 3.15); Btrfs (Linux 3.16); F2FS (Linux 3.16); and ubifs (Linux 4.9)
--8<---------------cut here---------------end--------------->8---

BTW, IIRC, this can be used to create an invisible file that
can be mmap-ed, and the mmap will persist when you delete
the file. Which then can be used as an anonymous buffer
passed to wayland, along with metadate saying what the buffer
contains, e.g. different kinds of rgb or rgba permutations
and encodings, (or anything, which you can tell wayland just
to keep track of for you.

You need a directory for openat, so probably
    XDG_RUNTIME_DIR=/run/user/1000
is suitable if it exists. Worked in my case.

HTH
--
Regards,
Bengt Richter

Message #64 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: bokr <at> bokr.com
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 Chris Marusich <cmmarusich <at> gmail.com>, 53355 <at> debbugs.gnu.org,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Sat, 25 Jun 2022 23:04:10 +0200

[Message part 1 (text/plain, inline)]

bokr <at> bokr.com schreef op za 25-06-2022 om 22:06 [+0200]:
> BTW, IIRC, this can be used to create an invisible file that

Invisible files don't have file names, so they cannot be put in the
tiny shell script:

> +    (format
> +     #f "env >~a || /usr/bin/env >~a || set >~a; \
> +echo GUIX-CHECK-DONE >>~a; exit\n"
> +     temporary-file temporary-file temporary-file temporary-file))

Also, I just noticed that this tiny shell script isn't quoting
anything, so you'll get in trouble if $TMPDIR (or was it $TEMPDIR,
whatever) contains " or ' or \ or whatever.

So to avoid messy and fragile escaping, I'd recommend to work with file
descriptors.

Greetings,
Maxime.

[signature.asc (application/pgp-signature, inline)]

Message #67 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Josselin Poiret <dev <at> jpoiret.xyz>
To: Maxime Devos <maximedevos <at> telenet.be>, Chris Marusich
 <cmmarusich <at> gmail.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 53355 <at> debbugs.gnu.org,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Sun, 26 Jun 2022 12:33:37 +0200

Hello everyone, 

Maxime Devos <maximedevos <at> telenet.be> writes:

> Chris Marusich schreef op za 25-06-2022 om 09:52 [-0700]:
>> [...]
>> the problem was that I could not control Guile's choice
>> of
>> file descriptors.  Guile chose file descriptor 19 for one end of the
>> pipe, and I don't know how to make it use anything else.  If we can
>> arrange for Guile to consistently use file descriptor 7, for example,
>> then probably it would work in all the shell I've tested.
> Have a look at ‘(guile)Ports and File Descriptors’.  It has lots of
> procedures for duplicating and renumbering.  That's fragile though, you
> might accidentally overwrite an fd that's being used for something
> else.

Just my 2¢, from my experience, Guile uses a lot of fds (`guile -q` uses
15).  I'm not sure it would be safe or advisable to move fds, since we
cannot be sure that they're backing ports or not, and if they're not it
would break things.  In general, the Guile ports/fds interface works ok,
but there is a lot of code that is fragile and doesn't handle
edge-cases.

Best,
-- 
Josselin Poiret

Message #70 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: Josselin Poiret <dev <at> jpoiret.xyz>, Chris Marusich <cmmarusich <at> gmail.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 53355 <at> debbugs.gnu.org,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Sun, 26 Jun 2022 15:07:11 +0200

[Message part 1 (text/plain, inline)]

Josselin Poiret schreef op zo 26-06-2022 om 12:33 [+0200]:
> Just my 2¢, from my experience, Guile uses a lot of fds (`guile -q` uses
> 15).  I'm not sure it would be safe or advisable to move fds, since we
> cannot be sure that they're backing ports or not, and if they're not it
> would break things

As mentioned previously, move->fdes looks in the port table to see if
the file descriptor things are being moved too is still in use, and if
so, moves that (fd, port) out of the way first, so should be safe.

That only works if Guile knows about the fd though, and Guile currently
does not know about it's own finalisation pipe, which I think is the
cause of <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=48563>, so
currently still fragile ...

(To be clear, I still recommend just fixing dash ...)

[signature.asc (application/pgp-signature, inline)]

Message #73 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Tobias Geerinckx-Rice <me <at> tobias.gr>
To: bug-guix <at> gnu.org, Maxime Devos <maximedevos <at> telenet.be>,
 Josselin Poiret <dev <at> jpoiret.xyz>, Chris Marusich <cmmarusich <at> gmail.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 53355 <at> debbugs.gnu.org,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Sun, 26 Jun 2022 19:45:24 +0000

On 26 June 2022 13:07:11 UTC, Maxime Devos <maximedevos <at> telenet.be> wrote:
>(To be clear, I still recommend just fixing dash ...)

That's the long-term solution, but Guix needs a work-around regardless.


Kind regards,

T G-R

Sent on the go.  Excuse or enjoy my brevity.

Message #79 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Chris Marusich <cmmarusich <at> gmail.com>
Cc: 53355 <at> debbugs.gnu.org, 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: guix shell --check: confusing error message
Date: Mon, 27 Jun 2022 12:17:08 +0200

Hi Chris,

Chris Marusich <cmmarusich <at> gmail.com> skribis:

> It turns out that it is probably not OK to rely on shell redirection in
> this case, after all.  For example, "dash does not support multi-digit
> file descriptors":
>
> https://bugs.launchpad.net/ubuntu/+source/dash/+bug/249620

Bah.  :-/

[...]

> To resolve this, I changed the code so that it just writes to a
> temporary file.  This is simple and more robust.  With the attached
> patch, I was able to use a command like the one above to verify that
> "guix environment --check" works correctly for Guix-built bash, dash,
> ksh, fish, zsh, and ash.  I also verified that it works for Fedora's
> /bin/sh and /bin/bash.
>
> What do you think of this file-based approach?  Supporting many
> different shells is pretty tricky, but I think this patch does a good
> enough job.

Like Maxime, I’d rather not create a temporary file.

I agree that Dash should be fixed, but in the meantime, we still want
our stuff to work with the broken Dash (it’s the default on
Debian/Ubuntu, isn’t it?).

I don’t have a better idea to offer though…

Ludo’.

Message #82 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: Ludovic Courtès <ludo <at> gnu.org>, Chris Marusich
 <cmmarusich <at> gmail.com>
Cc: 53355 <at> debbugs.gnu.org, 51466 <at> debbugs.gnu.org
Subject: Re: bug#51466: bug#53355: guix shell --check: confusing error message
Date: Mon, 27 Jun 2022 12:34:21 +0200

[Message part 1 (text/plain, inline)]

Ludovic Courtès schreef op ma 27-06-2022 om 12:17 [+0200]:
> I agree that Dash should be fixed, but in the meantime, we still want
> our stuff to work with the broken Dash (it’s the default on
> Debian/Ubuntu, isn’t it?).

If Dash is fixed, then it's non-broken, and we don't have to work with
the broken Dash.  I don't expect fixing it to be more complicated than
the work-arounds in Guile.

> On 26 June 2022 13:07:11 UTC, Maxime Devos <maximedevos <at> telenet.be>
> wrote:
> >(To be clear, I still recommend just fixing dash ...)
> 
> That's the long-term solution, but Guix needs a work-around
> regardless.

Fixing dash seems to me something that could be done in the short term?

Greetings,
Maxime

[signature.asc (application/pgp-signature, inline)]

Message #85 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: bokr <at> bokr.com
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 53355 <at> debbugs.gnu.org, Chris Marusich <cmmarusich <at> gmail.com>,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#51466: bug#53355: guix shell --check: confusing error message
Date: Mon, 27 Jun 2022 13:23:15 +0200

On +2022-06-27 12:17:08 +0200, Ludovic Courtès wrote:
> Hi Chris,
> 
> Chris Marusich <cmmarusich <at> gmail.com> skribis:
> 
> > It turns out that it is probably not OK to rely on shell redirection in
> > this case, after all.  For example, "dash does not support multi-digit
> > file descriptors":
> >
> > https://bugs.launchpad.net/ubuntu/+source/dash/+bug/249620
> 
> Bah.  :-/
> 
> [...]
> 
> > To resolve this, I changed the code so that it just writes to a
> > temporary file.  This is simple and more robust.  With the attached
> > patch, I was able to use a command like the one above to verify that
> > "guix environment --check" works correctly for Guix-built bash, dash,
> > ksh, fish, zsh, and ash.  I also verified that it works for Fedora's
> > /bin/sh and /bin/bash.
> >
> > What do you think of this file-based approach?  Supporting many
> > different shells is pretty tricky, but I think this patch does a good
> > enough job.
> 
> Like Maxime, I’d rather not create a temporary file.
> 
> I agree that Dash should be fixed, but in the meantime, we still want
> our stuff to work with the broken Dash (it’s the default on
> Debian/Ubuntu, isn’t it?).
> 
> I don’t have a better idea to offer though…
> 
> Ludo’.
> 
> 
>
If this is all about capturing an environment as text,
how about

--8<---------------cut here---------------start------------->8---
xargs -0 < /proc/$$/environ
--8<---------------cut here---------------end--------------->8---

(not limited to $$ I would think)
--
Regards,
Bengt Richter

Message #88 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Bengt Richter <bokr <at> bokr.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 53355 <at> debbugs.gnu.org, Chris Marusich <cmmarusich <at> gmail.com>,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: bug#51466: bug#53355: guix shell --check: confusing
 error message
Date: Mon, 27 Jun 2022 16:22:32 +0200

On +2022-06-27 13:23:15 +0200, bokr <at> bokr.com wrote:
> >
> If this is all about capturing an environment as text,
> how about
> 
> --8<---------------cut here---------------start------------->8---
> xargs -0 < /proc/$$/environ
> --8<---------------cut here---------------end--------------->8---
> 
> [...]
>
Actually, why fight shell stuff when guile can read it?
Or are these modules not available in your context?
--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (use-modules (ice-9 textual-ports))
scheme@(guile-user)> (define ep (open-input-file "/proc/12430/environ"))
scheme@(guile-user)> (define es (get-string-all ep))
--8<---------------cut here---------------end--------------->8---
es from above got it all, when I tried manually as above.

FWIW I also did the same with (use-modules (rnrs bytevectors))
and the nulls show up as the expected zeroes.
--
Regards,
Bengt Richter

Message #91 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Maxime Devos <maximedevos <at> telenet.be>
Cc: 53355 <at> debbugs.gnu.org, Chris Marusich <cmmarusich <at> gmail.com>,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#51466: bug#53355: guix shell --check: confusing error message
Date: Tue, 28 Jun 2022 09:45:14 +0200

Maxime Devos <maximedevos <at> telenet.be> skribis:

> Ludovic Courtès schreef op ma 27-06-2022 om 12:17 [+0200]:
>> I agree that Dash should be fixed, but in the meantime, we still want
>> our stuff to work with the broken Dash (it’s the default on
>> Debian/Ubuntu, isn’t it?).
>
> If Dash is fixed, then it's non-broken

Even if a fix goes upstream today, people will still be using a distro
with the broken Dash for years, literally.  That’s why I think we have
to cope with it.

Ludo’.

Message #94 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 53355 <at> debbugs.gnu.org, Chris Marusich <cmmarusich <at> gmail.com>,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#51466: bug#53355: guix shell --check: confusing error message
Date: Tue, 28 Jun 2022 12:38:15 +0200

[Message part 1 (text/plain, inline)]

Ludovic Courtès schreef op di 28-06-2022 om 09:45 [+0200]:
> Maxime Devos <maximedevos <at> telenet.be> skribis:
> 
> > Ludovic Courtès schreef op ma 27-06-2022 om 12:17 [+0200]:
> > > I agree that Dash should be fixed, but in the meantime, we still want
> > > our stuff to work with the broken Dash (it’s the default on
> > > Debian/Ubuntu, isn’t it?).
> > 
> > If Dash is fixed, then it's non-broken
> 
> Even if a fix goes upstream today, people will still be using a distro
> with the broken Dash for years, literally.  That’s why I think we have
> to cope with it.

Then it could be fixed in that distro?  And if the distro intentionally
keeps it broken for years, then that seems more a problem in the distro
than Guix to me.

Greetings,
Maxime.

[signature.asc (application/pgp-signature, inline)]

Message #97 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: "(" <paren <at> disroot.org>
To: "Maxime Devos" <maximedevos <at> telenet.be>,
 Ludovic Courtès <ludo <at> gnu.org>
Cc: 53355 <at> debbugs.gnu.org, Chris Marusich <cmmarusich <at> gmail.com>,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: bug#51466: bug#53355: guix shell --check: confusing
 error message
Date: Tue, 28 Jun 2022 17:57:40 +0100

On Tue Jun 28, 2022 at 11:38 AM BST, Maxime Devos wrote:
> Then it could be fixed in that distro?  And if the distro intentionally
> keeps it broken for years, then that seems more a problem in the distro
> than Guix to me.

I believe Ludo' is referring to LTS distros and other situations where
somebody might not update for a long time.

    -- (

Message #100 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: "(" <paren <at> disroot.org>, Ludovic Courtès <ludo <at> gnu.org>
Cc: 53355 <at> debbugs.gnu.org, Chris Marusich <cmmarusich <at> gmail.com>,
 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: bug#51466: bug#53355: guix shell --check: confusing
 error message
Date: Tue, 28 Jun 2022 19:31:16 +0200

[Message part 1 (text/plain, inline)]

( schreef op di 28-06-2022 om 17:57 [+0100]:
> On Tue Jun 28, 2022 at 11:38 AM BST, Maxime Devos wrote:
> > Then it could be fixed in that distro?  And if the distro intentionally
> > keeps it broken for years, then that seems more a problem in the distro
> > than Guix to me.
> 
> I believe Ludo' is referring to LTS distros and other situations where
> somebody might not update for a long time.
> 
>     -- (

I'm thinking so too, though I want to mention LTS is not just ‘don't
update’ -- the third letter means ‘support’, which covers backporting
bug fixes, though I don't know if the distro would include dash > 9 fds
in this.

Greetings,
Maxime.

[signature.asc (application/pgp-signature, inline)]

Message #103 received at 53355 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: "(" <paren <at> disroot.org>
Cc: 53355 <at> debbugs.gnu.org, Chris Marusich <cmmarusich <at> gmail.com>,
 Maxime Devos <maximedevos <at> telenet.be>, 51466 <at> debbugs.gnu.org
Subject: Re: bug#53355: bug#51466: bug#53355: guix shell --check: confusing
 error message
Date: Mon, 04 Jul 2022 10:11:06 +0200

Hi,

"(" <paren <at> disroot.org> skribis:

> On Tue Jun 28, 2022 at 11:38 AM BST, Maxime Devos wrote:
>> Then it could be fixed in that distro?  And if the distro intentionally
>> keeps it broken for years, then that seems more a problem in the distro
>> than Guix to me.
>
> I believe Ludo' is referring to LTS distros and other situations where
> somebody might not update for a long time.

Yeah.  Whatever the reason, it’s a fact that our users might run Guix on
a system with a broken Dash, which is why I think we have to cope with it.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.