GNU bug report logs - #79413
[PATCH] Fix path and environment in remote Python shell

Previous Next

To reply to this bug, email your comments to 79413 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Liu Hui <liuhui1610 <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: [PATCH] Fix path and environment in remote Python shell
Date: Tue, 09 Sep 2025 17:53:11 +0800

[Message part 1 (text/plain, inline)]

Tags: patch


Options such as python-shell-exec-path and
python-shell-process-environment may not be applied for remote Python
shell:

(1) create a python virtual environment on the remote host:

$ python3 -m venv /tmp/test_venv

(2) emacs -Q and then eval the following code:

  --8<---------------cut here---------------start------------->8---
    (require 'python)
    (require 'tramp)

    (setq remote-dir "/ssh:<remote>:~/")

    ;; use the remote virtual environment by setting
    ;; python-shell-exec-path, python-shell-remote-exec-path, or
    ;; python-shell-virtualenv-root
    (setq python-shell-exec-path (list "/tmp/test_venv/bin"))

    ;; set environment variables for remote Python process
    (setq python-shell-process-environment '("FOO=1"))

    ;; python-shell-process-environment is not applied remotely when
    ;; connection-local tramp-remote-process-environment is used
    (connection-local-set-profile-variables
     'test
     `((tramp-remote-process-environment
        . ,(append
            '("BAR=bar")
            tramp-remote-process-environment))))
    (connection-local-set-profiles
     `(:application tramp :machine ,(file-remote-p remote-dir 'host))
     'test)

    (find-file remote-dir)
  --8<---------------cut here---------------end--------------->8---

(3) M-x run-python

(4) type 'import sys; sys.executable' in the remote python shell

The expected output is '/tmp/test_venv/bin/python', but the actual
output is '/bin/python'.

In emacs 30 and earlier version, the result is correct, with a side
effect that the remote path is changed globally, i.e. (exec-path) in
other remote buffers also contains "/tmp/test_venv/bin".

(5) type 'import os; os.environ.get("FOO")' in the python shell

The result should be '1', but there is no output.


This patch fixes python-shell--tramp-with-environment and adds tests.
The python-tests-remote-path-1 test needs a real ssh remote connection
like REMOTE_TEMPORARY_FILE_DIRECTORY=/ssh:remote:/tmp/.


There is still a corner case: if python-shell-process-environment
contains an environment variable that already exists in the default
process-environment, the environment variable is not applied by tramp
for the remote Python shell. For example, the following test fails:

FOO=1 REMOTE_TEMPORARY_FILE_DIRECTORY=/ssh:remote:/tmp/ \
  make python-tests SELECTOR='"python-tests-remote-env"'

where FOO=1 is in both python-shell-process-environment and default
process-environment. This issue may need to be addressed in tramp.

[0001-Fix-path-and-environment-in-remote-Python-shell.patch (text/patch, attachment)]

Message #8 received at 79413 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Liu Hui <liuhui1610 <at> gmail.com>
Cc: 79413 <at> debbugs.gnu.org
Subject: Re: bug#79413: [PATCH] Fix path and environment in remote Python shell
Date: Tue, 09 Sep 2025 15:01:18 +0200

Liu Hui <liuhui1610 <at> gmail.com> writes:

Hi,

> There is still a corner case: if python-shell-process-environment
> contains an environment variable that already exists in the default
> process-environment, the environment variable is not applied by tramp
> for the remote Python shell. For example, the following test fails:
>
> FOO=1 REMOTE_TEMPORARY_FILE_DIRECTORY=/ssh:remote:/tmp/ \
>   make python-tests SELECTOR='"python-tests-remote-env"'
>
> where FOO=1 is in both python-shell-process-environment and default
> process-environment. This issue may need to be addressed in tramp.

This is a Tramp feature. See this comment in tramp-handle-make-process,
tramp-sh-handle-make-process and tramp-sh-handle-process-file:

--8<---------------cut here---------------start------------->8---
	   ;; We use as environment the difference to toplevel
	   ;; `process-environment'.
--8<---------------cut here---------------end--------------->8---

The reason is, that often a local value of $FOO might be wrong on the
remote host.

Best regards, Michael.

Message #11 received at 79413 <at> debbugs.gnu.org (full text, mbox):

From: Liu Hui <liuhui1610 <at> gmail.com>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: 79413 <at> debbugs.gnu.org
Subject: Re: bug#79413: [PATCH] Fix path and environment in remote Python shell
Date: Wed, 10 Sep 2025 11:45:45 +0800

Michael Albinus <michael.albinus <at> gmx.de> writes:

> Liu Hui <liuhui1610 <at> gmail.com> writes:
>
> Hi,
>
>> There is still a corner case: if python-shell-process-environment
>> contains an environment variable that already exists in the default
>> process-environment, the environment variable is not applied by tramp
>> for the remote Python shell. For example, the following test fails:
>>
>> FOO=1 REMOTE_TEMPORARY_FILE_DIRECTORY=/ssh:remote:/tmp/ \
>>   make python-tests SELECTOR='"python-tests-remote-env"'
>>
>> where FOO=1 is in both python-shell-process-environment and default
>> process-environment. This issue may need to be addressed in tramp.
>
> This is a Tramp feature. See this comment in tramp-handle-make-process,
> tramp-sh-handle-make-process and tramp-sh-handle-process-file:
>
> --8<---------------cut here---------------start------------->8---
> 	   ;; We use as environment the difference to toplevel
> 	   ;; `process-environment'.
> --8<---------------cut here---------------end--------------->8---
>
> The reason is, that often a local value of $FOO might be wrong on the
> remote host.

Thanks for the explanation. Then let-binding process-environment is
not a reliable way if we want to ensure some environment variables,
python-shell-process-environment in this case, are applied to the
remote process.

Could tramp add an option that disables this feature temporarily or
ensures specific environment variables are always applied in
tramp-sh-handle-make-process and tramp-sh-handle-process-file?

Otherwise, we have to continue using
python-shell-tramp-refresh-process-environment, which I think should
be moved from python.el to tramp?

Message #14 received at 79413 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Liu Hui <liuhui1610 <at> gmail.com>
Cc: 79413 <at> debbugs.gnu.org
Subject: Re: bug#79413: [PATCH] Fix path and environment in remote Python shell
Date: Wed, 10 Sep 2025 10:46:48 +0200

Liu Hui <liuhui1610 <at> gmail.com> writes:

Hi,

>> This is a Tramp feature. See this comment in tramp-handle-make-process,
>> tramp-sh-handle-make-process and tramp-sh-handle-process-file:
>>
>> --8<---------------cut here---------------start------------->8---
>> 	   ;; We use as environment the difference to toplevel
>> 	   ;; `process-environment'.
>> --8<---------------cut here---------------end--------------->8---
>>
>> The reason is, that often a local value of $FOO might be wrong on the
>> remote host.
>
> Thanks for the explanation. Then let-binding process-environment is
> not a reliable way if we want to ensure some environment variables,
> python-shell-process-environment in this case, are applied to the
> remote process.
>
> Could tramp add an option that disables this feature temporarily or
> ensures specific environment variables are always applied in
> tramp-sh-handle-make-process and tramp-sh-handle-process-file?

You would run into the same problem why this feature exist. You don't
want to expand local environment variables like DISPLAY, SSH_* or
whatever to the remote host.

You could let-bind and modify tramp-remote-process-environment for
tramp-sh-handle-make-process. Note, that this won't work for direct
async processes, which use tramp-handle-make-process internally. And it
won't work for tramp-sh-handle-process-file.

Alternatively, you could call "env VAR1=VAL1 VAR2=VAL2 ... program ..."
instead of "program ...".

Best regards, Michael.

Message #17 received at 79413 <at> debbugs.gnu.org (full text, mbox):

From: Liu Hui <liuhui1610 <at> gmail.com>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: 79413 <at> debbugs.gnu.org
Subject: Re: bug#79413: [PATCH] Fix path and environment in remote Python shell
Date: Thu, 11 Sep 2025 12:47:13 +0800

Michael Albinus <michael.albinus <at> gmx.de> writes:

> Liu Hui <liuhui1610 <at> gmail.com> writes:
>
> Hi,
>
>>> This is a Tramp feature. See this comment in tramp-handle-make-process,
>>> tramp-sh-handle-make-process and tramp-sh-handle-process-file:
>>>
>>> --8<---------------cut here---------------start------------->8---
>>> 	   ;; We use as environment the difference to toplevel
>>> 	   ;; `process-environment'.
>>> --8<---------------cut here---------------end--------------->8---
>>>
>>> The reason is, that often a local value of $FOO might be wrong on the
>>> remote host.
>>
>> Thanks for the explanation. Then let-binding process-environment is
>> not a reliable way if we want to ensure some environment variables,
>> python-shell-process-environment in this case, are applied to the
>> remote process.
>>
>> Could tramp add an option that disables this feature temporarily or
>> ensures specific environment variables are always applied in
>> tramp-sh-handle-make-process and tramp-sh-handle-process-file?
>
> You would run into the same problem why this feature exist. You don't
> want to expand local environment variables like DISPLAY, SSH_* or
> whatever to the remote host.

I agree local environment variables should not be applied on the
remote host. I meant to propagate specific environment variables (e.g.
"PYTHONPATH=/opt/mypackage") in python-shell-process-environment. The
problem is that "PYTHONPATH=/opt/mypackage" is skipped for remote
processes if it happens to exist in the default process environment
(e.g. it is added in local ~/.profile).

How about the following change? It doesn't change the default behavior
and allows the caller to override tramp-local-environment-variable-p
when necessary.

diff --git a/lisp/net/tramp-sh.el b/lisp/net/tramp-sh.el
index 9d13cdc3a2d..0a454fed69b 100644
--- a/lisp/net/tramp-sh.el
+++ b/lisp/net/tramp-sh.el
@@ -3052,8 +3052,7 @@ tramp-sh-handle-make-process
 	     ;; `process-environment'.
 	     env uenv
 	     (env (dolist (elt (cons prompt process-environment) env)
-		    (or (member
-			 elt (default-toplevel-value 'process-environment))
+		    (or (tramp-local-environment-variable-p elt)
 			(if (string-search "=" elt)
 			    (setq env (append env `(,elt)))
 			  (setq uenv (cons elt uenv))))))
@@ -3288,7 +3287,7 @@ tramp-sh-handle-process-file
 			       (cons program args) " "))
       ;; We use as environment the difference to toplevel `process-environment'.
       (dolist (elt process-environment)
-        (or (member elt (default-toplevel-value 'process-environment))
+        (or (tramp-local-environment-variable-p elt)
             (if (string-search "=" elt)
                 (setq env (append env `(,elt)))
               (setq uenv (cons elt uenv)))))
diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el
index 9bf1b4ae6c3..3ce25a5c663 100644
--- a/lisp/net/tramp.el
+++ b/lisp/net/tramp.el
@@ -5342,6 +5342,12 @@ tramp-direct-async-process-p
 	 (or (not (stringp buffer)) (not (tramp-tramp-file-p buffer)))
 	 (or (not (stringp stderr)) (not (tramp-tramp-file-p stderr))))))

+(defun tramp-local-environment-variable-p (arg)
+  "Return non-nil if ARG exists in default `process-environment'.
+Tramp does not propagate a local environment variable in remote
+processes."
+  (member arg (default-toplevel-value 'process-environment)))
+
 (defun tramp-handle-make-process (&rest args)
   "An alternative `make-process' implementation for Tramp files."
   (tramp-skeleton-make-process args nil nil
@@ -5360,9 +5366,7 @@ tramp-handle-make-process
 	   (env (dolist (elt process-environment env)
 		  (when (and
 			 (string-search "=" elt)
-			 (not
-			  (member
-			   elt (default-toplevel-value 'process-environment))))
+			 (not (tramp-local-environment-variable-p elt)))
 		    (setq env (cons elt env)))))
 	   ;; Add remote path if exists.
 	   (env (if-let* ((sh-file-name-handler-p)


> You could let-bind and modify tramp-remote-process-environment for
> tramp-sh-handle-make-process. Note, that this won't work for direct
> async processes, which use tramp-handle-make-process internally. And it
> won't work for tramp-sh-handle-process-file.
>
> Alternatively, you could call "env VAR1=VAL1 VAR2=VAL2 ... program ..."
> instead of "program ...".

Thanks for the suggestion. The problem is that
python-shell-with-environment is just a wrapper that temporarily sets
path and environment:

(defmacro python-shell-with-environment (&rest body)
   ...
  `(python-shell--with-environment
    (python-shell--calculate-process-environment)
    (lambda () ,@body)))

so we don't know what functions and programs users call.

Message #20 received at 79413 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Liu Hui <liuhui1610 <at> gmail.com>
Cc: 79413 <at> debbugs.gnu.org
Subject: Re: bug#79413: [PATCH] Fix path and environment in remote Python shell
Date: Thu, 11 Sep 2025 16:11:48 +0200

Liu Hui <liuhui1610 <at> gmail.com> writes:

Hi,

> How about the following change? It doesn't change the default behavior
> and allows the caller to override tramp-local-environment-variable-p
> when necessary.

Looks OK. Pls provide a complete patch, including commit
messages. tramp-androidsu-handle-make-process in tramp-android.el must
be changed as well.

Perhaps add also a comment on top of tramp-local-environment-variable-p,
explaining the reason for this function. I'm not decided yet whether we
shall document it in the Tramp manual. If we do, we would need a more
general approach I guess.

Best regards, Michael.

Message #23 received at 79413 <at> debbugs.gnu.org (full text, mbox):

From: Liu Hui <liuhui1610 <at> gmail.com>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: 79413 <at> debbugs.gnu.org
Subject: Re: bug#79413: [PATCH] Fix path and environment in remote Python shell
Date: Fri, 12 Sep 2025 17:58:04 +0800

[Message part 1 (text/plain, inline)]

Michael Albinus <michael.albinus <at> gmx.de> writes:

> Liu Hui <liuhui1610 <at> gmail.com> writes:
>
> Hi,
>
>> How about the following change? It doesn't change the default behavior
>> and allows the caller to override tramp-local-environment-variable-p
>> when necessary.
>
> Looks OK. Pls provide a complete patch, including commit
> messages. tramp-androidsu-handle-make-process in tramp-android.el must
> be changed as well.
>
> Perhaps add also a comment on top of tramp-local-environment-variable-p,
> explaining the reason for this function. I'm not decided yet whether we
> shall document it in the Tramp manual. If we do, we would need a more
> general approach I guess.

Patch is attached. Thanks.

[0001-Tramp-Refactor-environment-variable-filtering-to-a-s.patch (text/x-diff, attachment)]

Message #26 received at 79413 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Liu Hui <liuhui1610 <at> gmail.com>
Cc: 79413 <at> debbugs.gnu.org
Subject: Re: bug#79413: [PATCH] Fix path and environment in remote Python shell
Date: Fri, 12 Sep 2025 13:26:40 +0200

Liu Hui <liuhui1610 <at> gmail.com> writes:

Hi,

>> Looks OK. Pls provide a complete patch, including commit
>> messages. tramp-androidsu-handle-make-process in tramp-android.el must
>> be changed as well.
>>
>> Perhaps add also a comment on top of tramp-local-environment-variable-p,
>> explaining the reason for this function. I'm not decided yet whether we
>> shall document it in the Tramp manual. If we do, we would need a more
>> general approach I guess.
>
> Patch is attached. Thanks.

Thanks, I've pushed it to the repositories.

Best regards, Michael.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.