GNU bug report logs - #22533
Non-determinism in python-3 ".pyc" bytecode

Previous Next

Full log

View this message in rfc822 format

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Gábor Boskovits <boskovits <at> gmail.com>
Cc: Marius Bakke <mbakke <at> fastmail.com>, 22533 <at> debbugs.gnu.org
Subject: bug#22533: Python bytecode reproducibility
Date: Sun, 04 Mar 2018 20:18:23 +0100

[Message part 1 (text/plain, inline)]

I have applied this patch locally:

[1.diff (text/x-patch, inline)]

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 5f701701a..0d1ecc3c6 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -359,8 +359,42 @@ data types.")
                               "Lib/ctypes/test/test_win32.py" ; fails on aarch64
                               "Lib/test/test_fcntl.py")) ; fails on aarch64
                   #t))))
-    (arguments (substitute-keyword-arguments (package-arguments python-2)
-                 ((#:tests? _) #t)))
+    (arguments
+     (substitute-keyword-arguments (package-arguments python-2)
+       ((#:tests? _) #t)
+       ((#:phases phases)
+        `(modify-phases ,phases
+           (add-after 'unpack 'patch-timestamp-for-pyc-files
+             (lambda _
+               ;; We set DETERMINISTIC_BUILD to only override the mtime when
+               ;; building with Guix, lest we break auto-compilation in
+               ;; environments.
+               (setenv "DETERMINISTIC_BUILD" "1")
+               (substitute* "Lib/py_compile.py"
+                 (("source_stats\\['mtime'\\]")
+                  "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"))
+
+               ;; Use deterministic hashes for strings, bytes, and datetime
+               ;; objects.
+               (setenv "PYTHONHASHSEED" "0")
+
+               ;; Reset mtime when validating bytecode header.
+               (substitute* "Lib/importlib/_bootstrap_external.py"
+                 (("source_mtime = int\\(source_stats\\['mtime'\\]\\)")
+                  "source_mtime = 1"))
+               #t))
+           (add-after 'unpack 'disable-timestamp-tests
+             (lambda _
+               (substitute* "Lib/test/test_importlib/source/test_file_loader.py"
+                 (("test_bad_marshal")
+                  "disable_test_bad_marshal")
+                 (("test_no_marshal")
+                  "disable_test_no_marshal")
+                 (("test_non_code_marshal")
+                  "disable_test_non_code_marshal"))
+               #t))
+           (add-before 'check 'allow-non-deterministic-compilation
+             (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t))))))
     (native-search-paths
      (list (search-path-specification
             (variable "PYTHONPATH")

[Message part 3 (text/plain, inline)]

It allows me to build python-six and python-sip reproducibly.  It does
not fix problems with Python 2, and I haven’t yet tested if it causes
any new problems.

It’s a little worrying that I had to disable three more tests that I
think shouldn’t have failed.

What do you think?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.