GNU bug report logs - #22533
Non-determinism in python-3 ".pyc" bytecode

Previous Next

Full log

Message #67 received at 22533 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: Gábor Boskovits <boskovits <at> gmail.com>
Cc: 22533 <at> debbugs.gnu.org
Subject: Re: bug#22533: Python bytecode reproducibility
Date: Mon, 05 Mar 2018 23:06:51 +0100

Ricardo Wurmus <rekado <at> elephly.net> writes:

> Ricardo Wurmus <rekado <at> elephly.net> writes:
>
>> I have applied this patch locally:
>>
>> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
>> index 5f701701a..0d1ecc3c6 100644
>> --- a/gnu/packages/python.scm
>> +++ b/gnu/packages/python.scm
>> @@ -359,8 +359,42 @@ data types.")
>>                                "Lib/ctypes/test/test_win32.py" ; fails on aarch64
>>                                "Lib/test/test_fcntl.py")) ; fails on aarch64
>>                    #t))))
>> -    (arguments (substitute-keyword-arguments (package-arguments python-2)
>> -                 ((#:tests? _) #t)))
>> +    (arguments
>> +     (substitute-keyword-arguments (package-arguments python-2)
>> +       ((#:tests? _) #t)
>> +       ((#:phases phases)
>> +        `(modify-phases ,phases
>> +           (add-after 'unpack 'patch-timestamp-for-pyc-files
>> +             (lambda _
>> +               ;; We set DETERMINISTIC_BUILD to only override the mtime when
>> +               ;; building with Guix, lest we break auto-compilation in
>> +               ;; environments.
>> +               (setenv "DETERMINISTIC_BUILD" "1")
>> +               (substitute* "Lib/py_compile.py"
>> +                 (("source_stats\\['mtime'\\]")
>> +                  "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])"))
>> +
>> +               ;; Use deterministic hashes for strings, bytes, and datetime
>> +               ;; objects.
>> +               (setenv "PYTHONHASHSEED" "0")
>> +
>> +               ;; Reset mtime when validating bytecode header.
>> +               (substitute* "Lib/importlib/_bootstrap_external.py"
>> +                 (("source_mtime = int\\(source_stats\\['mtime'\\]\\)")
>> +                  "source_mtime = 1"))
>> +               #t))
>> +           (add-after 'unpack 'disable-timestamp-tests
>> +             (lambda _
>> +               (substitute* "Lib/test/test_importlib/source/test_file_loader.py"
>> +                 (("test_bad_marshal")
>> +                  "disable_test_bad_marshal")
>> +                 (("test_no_marshal")
>> +                  "disable_test_no_marshal")
>> +                 (("test_non_code_marshal")
>> +                  "disable_test_non_code_marshal"))
>> +               #t))
>> +           (add-before 'check 'allow-non-deterministic-compilation
>> +             (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t))))))
>>      (native-search-paths
>>       (list (search-path-specification
>>              (variable "PYTHONPATH")
>>
>> It allows me to build python-six and python-sip reproducibly.  It does
>> not fix problems with Python 2, and I haven’t yet tested if it causes
>> any new problems.

I should also note that Python 3 itself still contains pyc files with
timestamps.  This could be the reason why in Nix all pyc files are
rebuilt (more than once).

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.