GNU bug report logs - #68241
[PATCH] gnu: python: Fix sitecustomize.py

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 68241 in the body.
You can then email your comments to 68241 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: guix-patches <at> gnu.org
Subject: [PATCH] gnu: python: Fix sitecustomize.py
Date: Thu, 04 Jan 2024 13:44:19 +0100

* gnu/packages/aux-files/python/sitecustomize.py: normalize sys.prefix
  to deal with situations where it contains "../"

This happens in particular when running Python from a Singularity image
created by Guix. See https://issues.guix.gnu.org/53258.

Change-Id: Ibfe13d7c2a14beaa199f599e64bc0b7bfb500fe8
---
 gnu/packages/aux-files/python/sitecustomize.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/packages/aux-files/python/sitecustomize.py b/gnu/packages/aux-files/python/sitecustomize.py
index e2348e0356..82ea91104d 100644
--- a/gnu/packages/aux-files/python/sitecustomize.py
+++ b/gnu/packages/aux-files/python/sitecustomize.py
@@ -37,7 +37,7 @@ import sys
 major_minor = '{}.{}'.format(*sys.version_info)
 site_packages_prefix = os.path.join(
     'lib', 'python' + major_minor, 'site-packages')
-python_site = os.path.join(sys.prefix, site_packages_prefix)
+python_site = os.path.normpath(os.path.join(sys.prefix, site_packages_prefix))
 
 try:
     all_sites_raw = os.environ['GUIX_PYTHONPATH'].split(os.path.pathsep)

base-commit: 7b0863f07a113caef26fea13909bd97d250b629e
-- 
2.41.0

Message #8 received at 68241 <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: 68241 <at> debbugs.gnu.org
Subject: Re: [PATCH] gnu: python: Fix sitecustomize.py
Date: Sat, 06 Jan 2024 10:32:00 +0100

Konrad Hinsen <konrad.hinsen <at> fastmail.net> writes:

> * gnu/packages/aux-files/python/sitecustomize.py: normalize sys.prefix
>   to deal with situations where it contains "../"

This patch leads to massive rebuilds for non-trivial environments. I
will submit an alternative that patches Python via a graft.

Cheers,
  Konrad.

Message #11 received at 68241 <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: 68241 <at> debbugs.gnu.org
Subject: [PATCH] gnu: python: Fix sitecustomize.py
Date: Sat, 06 Jan 2024 10:37:23 +0100

* gnu/packages/aux-files/python/sitecustomize.py: normalize sys.prefix
  to deal with situations where it contains "../"

This happens in particular when running Python from a Singularity image
created by Guix. See https://issues.guix.gnu.org/53258.

This is a variant of https://issues.guix.gnu.org/68241 that implements the
fix as a graft rather than as a patch to the Python package. Since many
packages in Guix depend on Python, the basic patch leads to enormous
build times. A graft should be sufficient since the fix has no impact
on Python being used within Guix.

Change-Id: I3b07f4edf474a12fcabfbf90f37c4c0223278330
---
 .../aux-files/python/sitecustomize-fixed.py   | 65 +++++++++++++++++++
 gnu/packages/python.scm                       | 22 ++++++-
 2 files changed, 85 insertions(+), 2 deletions(-)
 create mode 100644 gnu/packages/aux-files/python/sitecustomize-fixed.py

diff --git a/gnu/packages/aux-files/python/sitecustomize-fixed.py b/gnu/packages/aux-files/python/sitecustomize-fixed.py
new file mode 100644
index 0000000000..82ea91104d
--- /dev/null
+++ b/gnu/packages/aux-files/python/sitecustomize-fixed.py
@@ -0,0 +1,65 @@
+# -*- coding: utf-8 -*-
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2021 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import site
+import sys
+
+# Commentary:
+#
+# Site-specific customization for Guix.
+#
+# The program below honors the GUIX_PYTHONPATH environment variable to
+# discover Python packages.  File names appearing in this variable that match
+# a predefined versioned installation prefix are added to the sys.path.  To be
+# considered, a Python package must be installed under the
+# 'lib/pythonX.Y/site-packages' directory, where X and Y are the major and
+# minor version numbers of the Python interpreter.
+#
+# Code:
+
+major_minor = '{}.{}'.format(*sys.version_info)
+site_packages_prefix = os.path.join(
+    'lib', 'python' + major_minor, 'site-packages')
+python_site = os.path.normpath(os.path.join(sys.prefix, site_packages_prefix))
+
+try:
+    all_sites_raw = os.environ['GUIX_PYTHONPATH'].split(os.path.pathsep)
+except KeyError:
+    all_sites_raw = []
+# Normalize paths, otherwise a trailing slash would cause it to not match.
+all_sites_norm = [os.path.normpath(p) for p in all_sites_raw]
+matching_sites = [p for p in all_sites_norm
+                  if p.endswith(site_packages_prefix)]
+
+if matching_sites:
+    # Deduplicate the entries, append them to sys.path, and handle any
+    # .pth files they contain.
+    for s in matching_sites:
+        site.addsitedir(s)
+
+    # Move the entries that were appended to sys.path in front of
+    # Python's own site-packages directory.  This enables Guix
+    # packages to override Python's bundled packages, such as 'pip'.
+    python_site_index = sys.path.index(python_site)
+    new_site_start_index = sys.path.index(matching_sites[0])
+    if python_site_index < new_site_start_index:
+        sys.path = (sys.path[:python_site_index]
+                    + sys.path[new_site_start_index:]
+                    + sys.path[python_site_index:new_site_start_index])
diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 51d5f598d7..3cc3c08360 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -414,7 +414,16 @@ (define-public python-2.7
 packages; exception-based error handling; and very high level dynamic
 data types.")
     (properties '((cpe-name . "python")))
-    (license license:psfl)))
+    (license license:psfl)
+    (replacement python-2.7-fixed)))
+
+(define-public python-2.7-fixed
+  (package
+    (inherit python-2.7)
+    (native-inputs
+     (modify-inputs (package-native-inputs python-2.7)
+       (replace "sitecustomize.py"
+         (local-file (search-auxiliary-file "python/sitecustomize-fixed.py")))))))
 
 ;; Current 2.x version.
 (define-public python-2 python-2.7)
@@ -594,7 +603,16 @@ (define-public python-3.10
            ;; Python 3.9.
            (search-path-specification
             (variable "PYTHONTZPATH")
-            (files (list "share/zoneinfo")))))))
+            (files (list "share/zoneinfo")))))
+    (replacement python-3.10-fixed)))
+
+(define-public python-3.10-fixed
+  (package
+    (inherit python-3.10)
+    (native-inputs
+     (modify-inputs (package-native-inputs python-3.10)
+       (replace "sitecustomize.py"
+         (local-file (search-auxiliary-file "python/sitecustomize-fixed.py")))))))
 
 ;; Current 3.x version.
 (define-public python-3 python-3.10)

base-commit: 7b0863f07a113caef26fea13909bd97d250b629e
-- 
2.41.0

Message #14 received at 68241 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
Cc: Josselin Poiret <dev <at> jpoiret.xyz>, 68241 <at> debbugs.gnu.org
Subject: Re: [bug#68241] [PATCH] gnu: python: Fix sitecustomize.py
Date: Mon, 08 Jan 2024 09:57:23 +0100

Hi Konrad!

Konrad Hinsen <konrad.hinsen <at> fastmail.net> skribis:

> * gnu/packages/aux-files/python/sitecustomize.py: normalize sys.prefix
>   to deal with situations where it contains "../"
>
> This happens in particular when running Python from a Singularity image
> created by Guix. See https://issues.guix.gnu.org/53258.
>
> Change-Id: Ibfe13d7c2a14beaa199f599e64bc0b7bfb500fe8
> ---
>  gnu/packages/aux-files/python/sitecustomize.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/gnu/packages/aux-files/python/sitecustomize.py b/gnu/packages/aux-files/python/sitecustomize.py
> index e2348e0356..82ea91104d 100644
> --- a/gnu/packages/aux-files/python/sitecustomize.py
> +++ b/gnu/packages/aux-files/python/sitecustomize.py
> @@ -37,7 +37,7 @@ import sys
>  major_minor = '{}.{}'.format(*sys.version_info)
>  site_packages_prefix = os.path.join(
>      'lib', 'python' + major_minor, 'site-packages')
> -python_site = os.path.join(sys.prefix, site_packages_prefix)
> +python_site = os.path.normpath(os.path.join(sys.prefix, site_packages_prefix))

IIRC Josselin had a patch for <https://issues.guix.gnu.org/53258>, but I
can’t find it.  Josselin?

Also, do I get it right that the patch you sent fixes more than #53258?

Thanks!

Ludo’.

Message #17 received at 68241 <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Cc: Josselin Poiret <dev <at> jpoiret.xyz>, 68241 <at> debbugs.gnu.org
Subject: Re: [bug#68241] [PATCH] gnu: python: Fix sitecustomize.py
Date: Mon, 08 Jan 2024 10:14:14 +0100

Hi Ludo,

> Also, do I get it right that the patch you sent fixes more than #53258?

#53258 is two things: a weird error message, and a failure to run valid
Python code, the two being possibly related.

My patch fixes the error message. The Python example that Marek gave
already works fine in today's Guix, so it was probably fixed by
something else as a side effect.

Could the bug that causes the error message also mess up valid Python
code? In theory, yes, but I don't have an example at hand.

Cheers,
  Konrad.

Message #20 received at 68241 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
Cc: Ludovic Courtès <ludovic.courtes <at> inria.fr>,
 Josselin Poiret <dev <at> jpoiret.xyz>, 68241 <at> debbugs.gnu.org
Subject: Re: [bug#68241] [PATCH] gnu: python: Fix sitecustomize.py
Date: Sat, 20 Jan 2024 23:22:21 -0500

Hi Konrad,

Konrad Hinsen <konrad.hinsen <at> fastmail.net> writes:

> Hi Ludo,
>
>> Also, do I get it right that the patch you sent fixes more than #53258?
>
> #53258 is two things: a weird error message, and a failure to run valid
> Python code, the two being possibly related.
>
> My patch fixes the error message. The Python example that Marek gave
> already works fine in today's Guix, so it was probably fixed by
> something else as a side effect.

Oh?  And you tested using the exact same reproducer, using a relocatable
pack?

-- 
Thanks,
Maxim

Message #23 received at 68241 <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: 68241 <at> debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: Ludovic Courtès <ludovic.courtes <at> inria.fr>,
 Josselin Poiret <dev <at> jpoiret.xyz>
Subject: Re: [bug#68241] [PATCH] gnu: python: Fix sitecustomize.py
Date: Sun, 21 Jan 2024 12:11:10 +0100

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

>> My patch fixes the error message. The Python example that Marek gave
>> already works fine in today's Guix, so it was probably fixed by
>> something else as a side effect.
>
> Oh?  And you tested using the exact same reproducer, using a relocatable
> pack?

Yes, I tested it in a Singularity container.

Cheers,
  konrad.

Message #28 received at 68241-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
Cc: Ludovic Courtès <ludovic.courtes <at> inria.fr>,
 Josselin Poiret <dev <at> jpoiret.xyz>, 68241-done <at> debbugs.gnu.org
Subject: Re: [bug#68241] [PATCH] gnu: python: Fix sitecustomize.py
Date: Sun, 21 Jan 2024 13:29:20 -0500

Hi Konrad,

Konrad Hinsen <konrad.hinsen <at> fastmail.net> writes:

> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:
>
>>> My patch fixes the error message. The Python example that Marek gave
>>> already works fine in today's Guix, so it was probably fixed by
>>> something else as a side effect.
>>
>> Oh?  And you tested using the exact same reproducer, using a relocatable
>> pack?
>
> Yes, I tested it in a Singularity container.

Great, it's now on core-updates.

-- 
Thanks,
Maxim

Message #31 received at 68241-done <at> debbugs.gnu.org (full text, mbox):

From: Konrad Hinsen <konrad.hinsen <at> fastmail.net>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: Ludovic Courtès <ludovic.courtes <at> inria.fr>,
 Josselin Poiret <dev <at> jpoiret.xyz>, 68241-done <at> debbugs.gnu.org
Subject: Re: [bug#68241] [PATCH] gnu: python: Fix sitecustomize.py
Date: Mon, 22 Jan 2024 08:15:56 +0100

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> writes:

> Great, it's now on core-updates.

Thanks!

Konrad.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.