GNU bug report logs - #68394
[PATCH python-team] gnu: python: Make the build reproducible.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 68394 in the body.
You can then email your comments to 68394 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Tomas Volf <~@wolfsden.cz>
To: guix-patches <at> gnu.org
Cc: Tomas Volf <~@wolfsden.cz>
Subject: [PATCH python-team] gnu: python: Make the build reproducible.
Date: Fri, 12 Jan 2024 01:33:16 +0100

While python build was reproducible on a single machine, once multiple
file systems entered the picture, it was no longer true.  My local builds on
BTRFS differed from build on ext4 done in a virtual machine.

The distutils library present in current python is sensitive to file system
ordering.  The solution is the same opensuse used, sorting the list of files.

With this patch, build on my machine (BTRFS) and in a guix system vm (ext4)
produce the same store item.

More info: https://bugzilla.opensuse.org/show_bug.cgi?id=1049186

* gnu/packages/python.scm (python-3.10)[arguments]<#:phases>: Add
'patch-distutils phase.

Change-Id: I0273dc0f8511a7acdcc2b462a26cc29a9756c801
---
 gnu/packages/python.scm | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 51d5f598d7..319a917b4b 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -515,6 +515,19 @@ (define-public python-3.10
                        (substitute* "Makefile.pre.in"
                          (("-j0") "-j1")))))
                  '())
+           (add-after 'unpack 'patch-distutils
+             (lambda _
+               ;; Ensure byte_compile produces the same output regardless
+               ;; filesystem ordering.  For more information see:
+               ;; https://bugzilla.opensuse.org/show_bug.cgi?id=1049186
+               (let* ((file "Lib/distutils/util.py")
+                      (old-content (call-with-input-file file get-string-all)))
+                 (substitute* file
+                   (("^        for file in py_files:\n$")
+                    "        for file in sorted(py_files):\n"))
+                 (if (string=? old-content
+                               (call-with-input-file file get-string-all))
+                     (error "substitute did nothing, phase requires an update")))))
            (add-after 'unpack 'remove-windows-binaries
              (lambda _
                ;; Delete .exe from embedded .whl (zip) files

base-commit: 5c0f77f4241c9beac0c82deae946bfdc70b49ff0
-- 
2.41.0

Message #8 received at 68394 <at> debbugs.gnu.org (full text, mbox):

From: Lars-Dominik Braun <lars <at> 6xq.net>
To: Tomas Volf <~@wolfsden.cz>
Cc: Munyoki Kilyungi <me <at> bonfacemunyoki.com>, 68394 <at> debbugs.gnu.org,
 jgart <jgart <at> dismail.de>, Marius Bakke <marius <at> gnu.org>
Subject: Re: [bug#68394] [PATCH python-team] gnu: python: Make the build
 reproducible.
Date: Fri, 12 Jan 2024 08:39:20 +0100

Hi,

> +                 (substitute* file
> +                   (("^        for file in py_files:\n$")
> +                    "        for file in sorted(py_files):\n"))

as far as I understand this change was rejected from upstream Python,
see https://github.com/python/cpython/pull/8057, and instead this is
the accepted (and merged) solution:
https://github.com/python/cpython/pull/8226
That patch is also used by Debian.

Cheers,
Lars

Message #11 received at 68394 <at> debbugs.gnu.org (full text, mbox):

From: Tomas Volf <~@wolfsden.cz>
To: 68394 <at> debbugs.gnu.org
Cc: Tomas Volf <~@wolfsden.cz>
Subject: [PATCH python-team v2] gnu: python: Make the build reproducible.
Date: Fri, 12 Jan 2024 17:11:29 +0100

While python build was reproducible on a single machine, once multiple
file systems entered the picture, it was no longer true.  My local builds on
BTRFS differed from build on ext4 done in a virtual machine.

The solution adopted by the upstream (and debian) was cherry-picked.  With
this patch, build on my machine (BTRFS) and in a guix system vm (ext4) produce
the same store item.

More info: https://github.com/python/cpython/pull/8226

* gnu/packages/python.scm (python-3.10)[source]: Apply reproducibility patch.

Change-Id: I0273dc0f8511a7acdcc2b462a26cc29a9756c801
---
Use patch directly from the upstream.

 gnu/packages/python.scm | 21 +++++++++++++++------
 1 file changed, 15 insertions(+), 6 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 51d5f598d7..c92ac720a5 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -428,12 +428,21 @@ (define-public python-3.10
               (method url-fetch)
               (uri (string-append "https://www.python.org/ftp/python/"
                                   version "/Python-" version ".tar.xz"))
-              (patches (search-patches
-                        "python-3-arm-alignment.patch"
-                        "python-3-deterministic-build-info.patch"
-                        "python-3-fix-tests.patch"
-                        "python-3-hurd-configure.patch"
-                        "python-3-search-paths.patch"))
+              (patches
+               (cons*
+                ;; https://github.com/python/cpython/pull/8226
+                (origin
+                  (method url-fetch)
+                  (uri "https://github.com/python/cpython/commit/6c8ea7c1dacd42f3ba00440231ec0e6b1a38300d.patch")
+                  (sha256
+                   (base32
+                    "13llngsyskp4c9j8lwqqpwp7h07mxai734zk1i387z8g261jk46v")))
+                (search-patches
+                 "python-3-arm-alignment.patch"
+                 "python-3-deterministic-build-info.patch"
+                 "python-3-fix-tests.patch"
+                 "python-3-hurd-configure.patch"
+                 "python-3-search-paths.patch")))
               (sha256
                (base32
                 "0j6wvh2ad5jjq5n7sjmj1k66mh6lipabavchc3rb4vsinwaq9vbf"))

base-commit: 5c0f77f4241c9beac0c82deae946bfdc70b49ff0
-- 
2.41.0

Message #16 received at 68394-done <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Tomas Volf <~@wolfsden.cz>
Cc: Munyoki Kilyungi <me <at> bonfacemunyoki.com>, jgart <jgart <at> dismail.de>,
 68394-done <at> debbugs.gnu.org, Lars-Dominik Braun <lars <at> 6xq.net>,
 Marius Bakke <marius <at> gnu.org>
Subject: Re: bug#68394: [PATCH python-team] gnu: python: Make the build
 reproducible.
Date: Tue, 23 Jan 2024 09:12:14 -0500

Hi Tomas,

Tomas Volf <~@wolfsden.cz> writes:

> While python build was reproducible on a single machine, once multiple
> file systems entered the picture, it was no longer true.  My local builds on
> BTRFS differed from build on ext4 done in a virtual machine.
>
> The solution adopted by the upstream (and debian) was cherry-picked.  With
> this patch, build on my machine (BTRFS) and in a guix system vm (ext4) produce
> the same store item.
>
> More info: https://github.com/python/cpython/pull/8226
>
> * gnu/packages/python.scm (python-3.10)[source]: Apply reproducibility patch.
>
> Change-Id: I0273dc0f8511a7acdcc2b462a26cc29a9756c801

I've pushed a variant of this on to core-updates in commit e84519a949
("gnu: python: Make the build reproducible."), thank you!

> ---
> Use patch directly from the upstream.
>
>  gnu/packages/python.scm | 21 +++++++++++++++------
>  1 file changed, 15 insertions(+), 6 deletions(-)
>
> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
> index 51d5f598d7..c92ac720a5 100644
> --- a/gnu/packages/python.scm
> +++ b/gnu/packages/python.scm
> @@ -428,12 +428,21 @@ (define-public python-3.10
>                (method url-fetch)
>                (uri (string-append "https://www.python.org/ftp/python/"
>                                    version "/Python-" version ".tar.xz"))
> -              (patches (search-patches
> -                        "python-3-arm-alignment.patch"
> -                        "python-3-deterministic-build-info.patch"
> -                        "python-3-fix-tests.patch"
> -                        "python-3-hurd-configure.patch"
> -                        "python-3-search-paths.patch"))
> +              (patches
> +               (cons*
> +                ;; https://github.com/python/cpython/pull/8226
> +                (origin
> +                  (method url-fetch)
> +                  (uri "https://github.com/python/cpython/commit/6c8ea7c1dacd42f3ba00440231ec0e6b1a38300d.patch")
> +                  (sha256
> +                   (base32
> +                    "13llngsyskp4c9j8lwqqpwp7h07mxai734zk1i387z8g261jk46v")))

I've opted to keep the patch local, which has been discussed as
preferred in the past (I think for reliability -- they don't disappear).

-- 
Thanks,
Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.