Subject: [PATCH] Fix path and environment in remote Python shell

From: Liu Hui <[email-masked]>
Date: Tue, 09 Sep 2025 17:53:11 +0800

Tags: patch

Options such as python-shell-exec-path and
python-shell-process-environment may not be applied for remote Python
shell:

(1) create a python virtual environment on the remote host:

$ python3 -m venv /tmp/test_venv

(2) emacs -Q and then eval the following code:

  --8<---------------cut here---------------start------------->8---
    (require 'python)
    (require 'tramp)

    (setq remote-dir "/ssh:<remote>:~/")

    ;; use the remote virtual environment by setting
    ;; python-shell-exec-path, python-shell-remote-exec-path, or
    ;; python-shell-virtualenv-root
    (setq python-shell-exec-path (list "/tmp/test_venv/bin"))

    ;; set environment variables for remote Python process
    (setq python-shell-process-environment '("FOO=1"))

    ;; python-shell-process-environment is not applied remotely when
    ;; connection-local tramp-remote-process-environment is used
    (connection-local-set-profile-variables
     'test
     `((tramp-remote-process-environment
        . ,(append
            '("BAR=bar")
            tramp-remote-process-environment))))
    (connection-local-set-profiles
     `(:application tramp :machine ,(file-remote-p remote-dir 'host))
     'test)

    (find-file remote-dir)
  --8<---------------cut here---------------end--------------->8---

(3) M-x run-python

(4) type 'import sys; sys.executable' in the remote python shell

The expected output is '/tmp/test_venv/bin/python', but the actual
output is '/bin/python'.

In emacs 30 and earlier version, the result is correct, with a side
effect that the remote path is changed globally, i.e. (exec-path) in
other remote buffers also contains "/tmp/test_venv/bin".

(5) type 'import os; os.environ.get("FOO")' in the python shell

The result should be '1', but there is no output.

This patch fixes python-shell--tramp-with-environment and adds tests.
The python-tests-remote-path-1 test needs a real ssh remote connection
like REMOTE_TEMPORARY_FILE_DIRECTORY=/ssh:remote:/tmp/.

There is still a corner case: if python-shell-process-environment
contains an environment variable that already exists in the default
process-environment, the environment variable is not applied by tramp
for the remote Python shell. For example, the following test fails:

FOO=1 REMOTE_TEMPORARY_FILE_DIRECTORY=/ssh:remote:/tmp/ \
  make python-tests SELECTOR='"python-tests-remote-env"'

where FOO=1 is in both python-shell-process-environment and default
process-environment. This issue may need to be addressed in tramp.

>From aa42cfba252e6fc2c451c7404c9b409250dcd09c Mon Sep 17 00:00:00 2001
From: Liu Hui <[email-masked]>
Date: Tue, 9 Sep 2025 17:37:50 +0800
Subject: [PATCH] Fix path and environment in remote Python shell

* lisp/progmodes/python.el
(python-shell-tramp-refresh-remote-path)
(python-shell-tramp-refresh-process-environment): Obsolete.
(python-shell--tramp-with-environment): Fix remote exec-path and
process-environment.
* test/lisp/progmodes/python-tests.el
(python-tests-remote-enabled, python-tests-remote-environment-1)
(python-tests-remote-environment-2, python-tests-remote-path-1):
Add new function and tests.
(python--tests-process-env-eql, python-shell-with-environment-2):
Update test.
---
 lisp/progmodes/python.el            | 46 ++++++++-------
 test/lisp/progmodes/python-tests.el | 92 ++++++++++++++++++++++++++---
 2 files changed, 110 insertions(+), 28 deletions(-)

diff --git a/lisp/progmodes/python.el b/lisp/progmodes/python.el
index e5cc3b0078b..c9c751dbf24 100644
--- a/lisp/progmodes/python.el
+++ b/lisp/progmodes/python.el
@@ -3062,6 +3062,7 @@ python-shell-calculate-exec-path
 
 (defun python-shell-tramp-refresh-remote-path (vec paths)
   "Update VEC's remote-path giving PATHS priority."
+  (declare (obsolete nil "31.1")) ; Not used.
   (cl-assert (featurep 'tramp))
   (declare-function tramp-set-remote-path "tramp-sh")
   (declare-function tramp-set-connection-property "tramp-cache")
@@ -3076,6 +3077,7 @@ python-shell-tramp-refresh-remote-path
 
 (defun python-shell-tramp-refresh-process-environment (vec env)
   "Update VEC's process environment with ENV."
+  (declare (obsolete nil "31.1")) ; Not used.
   (cl-assert (featurep 'tramp))
   (defvar tramp-end-of-heredoc)
   (defvar tramp-end-of-output)
@@ -3139,28 +3141,30 @@ python-shell--with-environment
         (funcall bodyfun)))))
 
 (defun python-shell--tramp-with-environment (vec extraenv bodyfun)
-  (defvar tramp-remote-process-environment)
   (declare-function tramp-get-connection-process "tramp" (vec))
-  (let* ((tramp-remote-process-environment
-          (append extraenv tramp-remote-process-environment)))
-    (when (tramp-get-connection-process vec)
-      ;; For already existing connections, the new exec path must
-      ;; be re-set, otherwise it won't take effect.  One example
-      ;; of such case is when remote dir-locals are read and
-      ;; *then* subprocesses are triggered within the same
-      ;; connection.
-      (python-shell-tramp-refresh-remote-path
-       ;; FIXME: This is still Python-specific.
-       vec (python-shell-calculate-exec-path))
-      ;; The `tramp-remote-process-environment' variable is only
-      ;; effective when the started process is an interactive
-      ;; shell, otherwise (like in the case of processes started
-      ;; with `process-file') the environment is not changed.
-      ;; This makes environment modifications effective
-      ;; unconditionally.
-      (python-shell-tramp-refresh-process-environment
-       vec tramp-remote-process-environment))
-    (funcall bodyfun)))
+  (declare-function tramp-get-remote-path "tramp-sh" (vec))
+  (eval-when-compile (require 'tramp-cache))
+  ;; FIXME: If extraenv is in the global default `process-environment',
+  ;; it will not be set for the remote Python process, because TRAMP
+  ;; only applies environment variables not present in the global
+  ;; default value of `process-environment'.
+  (let ((process-environment
+         (append (cl-loop for x in extraenv
+                          unless (member x process-environment)
+                          collect x)
+                 process-environment)))
+    ;; FIXME: This is still Python-specific.
+    (if-let* ((path (python-shell-calculate-exec-path))
+              (key (tramp-get-connection-process vec)))
+        ;; Do not override global remote path.
+        (with-tramp-saved-connection-property key "remote-path"
+          (let ((r-path (tramp-get-remote-path vec)))
+            (if r-path
+                (python-shell--add-to-path-with-priority r-path path)
+              (setq r-path path))
+            (tramp-set-connection-property key "remote-path" r-path))
+          (funcall bodyfun))
+      (funcall bodyfun))))
 
 (defvar python-shell--prompt-calculated-input-regexp nil
   "Calculated input prompt regexp for inferior python shell.
diff --git a/test/lisp/progmodes/python-tests.el b/test/lisp/progmodes/python-tests.el
index b9130da495d..698665edcc2 100644
--- a/test/lisp/progmodes/python-tests.el
+++ b/test/lisp/progmodes/python-tests.el
@@ -23,6 +23,7 @@
 
 (require 'cl-extra)
 (require 'ert)
+(require 'tramp)
 (require 'ert-x)
 (require 'python)
 
@@ -31,6 +32,18 @@
 (require 'hideshow)
 (require 'tramp-sh)
 
+(defun python-tests-remote-enabled (&optional method)
+  "Whether remote file access is enabled via METHOD."
+  (ignore-errors
+    (and
+     (file-remote-p ert-remote-temporary-file-directory)
+     (file-directory-p ert-remote-temporary-file-directory)
+     (file-writable-p ert-remote-temporary-file-directory)
+     (if method
+         (with-parsed-tramp-file-name
+             ert-remote-temporary-file-directory v
+           (equal method v-method))
+       t))))
 
 (defmacro python-tests-with-temp-buffer (contents &rest body)
   "Create a `python-mode' enabled temp buffer with CONTENTS.
@@ -4095,10 +4108,6 @@ python--tests-process-env-canonical
         (setenv x nil)))
     process-environment))
 
-(defun python--tests-process-env-eql (pe1 pe2)
-  (equal (python--tests-process-env-canonical pe1)
-         (python--tests-process-env-canonical pe2)))
-
 (ert-deftest python-shell-with-environment-2 ()
   "Test environment with remote `default-directory'."
   (let* ((default-directory "/ssh::/example/dir/")
@@ -4116,9 +4125,7 @@ python-shell-with-environment-2
              (append (python-shell--calculate-process-environment)
                      tramp-remote-process-environment)))
         (should (not (getenv "PYTHONHOME")))
-        (should (string= (getenv "VIRTUAL_ENV") "/env"))
-        (should (python--tests-process-env-eql
-                 tramp-remote-process-environment process-environment))))
+        (should (string= (getenv "VIRTUAL_ENV") "/env"))))
     (should (equal tramp-remote-process-environment
                    original-process-environment))))
 
@@ -4139,6 +4146,77 @@ python-shell-with-environment-3
                      process-environment))))))
     (should (equal single-call nested-call))))
 
+(ert-deftest python-tests-remote-environment-1 ()
+  "Test `python-shell-process-environment' in remote environment."
+  (skip-unless (python-tests-remote-enabled))
+  (let ((default-directory ert-remote-temporary-file-directory)
+        (python-shell-process-environment '("FOO=1"))
+        (tramp-allow-unsafe-temporary-files t))
+    (python-tests-with-temp-buffer-with-shell-interpreter
+     nil
+     ""
+     (should (equal (python-shell-send-string-no-output
+                     "import os; os.environ.get('FOO')")
+                    "'1'")))))
+
+(ert-deftest python-tests-remote-environment-2 ()
+  "Test `python-shell-process-environment' and
+`tramp-remote-process-environment' in remote environment."
+  (skip-unless (python-tests-remote-enabled))
+  (let ((default-directory ert-remote-temporary-file-directory)
+        (python-shell-process-environment '("FOO=1"))
+        (tramp-allow-unsafe-temporary-files t)
+        (clpa connection-local-profile-alist)
+	(clca connection-local-criteria-alist))
+    (unwind-protect
+        (progn
+          (connection-local-set-profile-variables
+           'test
+           `((tramp-remote-process-environment
+              . ,(append
+                  '("BAR=1")
+                  tramp-remote-process-environment))))
+          (connection-local-set-profiles
+           `(:application tramp :machine ,(file-remote-p default-directory 'host))
+           'test)
+          (python-tests-with-temp-buffer-with-shell-interpreter
+           nil
+           ""
+           (should (equal (python-shell-send-string-no-output
+                           "import os; os.environ.get('BAR')")
+                          "'1'"))
+           (should (equal (python-shell-send-string-no-output
+                           "os.environ.get('FOO')")
+                          "'1'"))))
+      ;; Cleanup.
+      (custom-set-variables
+       `(connection-local-profile-alist ',clpa now)
+       `(connection-local-criteria-alist ',clca now)))))
+
+(ert-deftest python-tests-remote-path-1 ()
+  "Test remote path for python shell."
+  (skip-unless (python-tests-remote-enabled "ssh"))
+  (let* ((default-directory ert-remote-temporary-file-directory)
+         (tramp-remote-path (append '(tramp-own-remote-path) tramp-remote-path))
+         (tramp-allow-unsafe-temporary-files t)
+         (python-abs-path (executable-find python-shell-interpreter t)))
+    (skip-unless python-abs-path)
+    (let* ((temporary-file-directory default-directory)
+           (tmp-dir (make-temp-file "python-test" t))
+           (python-shell-interpreter (file-name-nondirectory python-abs-path))
+           (link (expand-file-name python-shell-interpreter tmp-dir))
+           (python-shell-exec-path (list (file-local-name tmp-dir))))
+      (unwind-protect
+          (progn
+            (make-symbolic-link
+             (concat (file-remote-p default-directory) python-abs-path) link)
+            (python-tests-with-temp-buffer-with-shell
+             ""
+             (should (equal (format "'%s'" (file-local-name link))
+                            (python-shell-send-string-no-output
+                             "import sys; sys.executable")))))
+        (delete-directory tmp-dir t)))))
+
 (ert-deftest python-shell-make-comint-1 ()
   "Check comint creation for global shell buffer."
   ;; The interpreter can get killed too quickly to allow it to clean
-- 
2.25.1