GNU bug report logs - #22533
Non-determinism in python-3 ".pyc" bytecode

Previous Next

Full log

Message #8 received at 22533 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: 22533 <at> debbugs.gnu.org
Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode
Date: Tue, 2 Feb 2016 03:54:39 -0500

On Tue, Feb 02, 2016 at 12:15:44AM -0500, Leo Famulari wrote:
> While preparing a package for borg [0], I found that the built output
> was not reproducible. The problem is that the bytecode compiler [1] for
> Python 3.4.3 (our current version) encodes the mtime of the
> corresponding Python source file in the output. This is described in
> PEP-3147 [2], and the responsible Python code is referenced below [3].
> 
> I tested a few of our existing python-3 packages: python-ccm,
> python-pysam, and python-scripttest all exhibit the same problem.
> 
> We fixed this in python-2 with the patch
> python-2.7-source-date-epoch.patch, but I don't know how to write this
> patch for python-3.

mark_weaver suggested setting the timestamps of the source files before
building. I think this is a better option if it doesn't break anything.
It would allow the bytecode "staleness" check to work as expected while
keeping the output consistent.

> 
> Can somebody write this patch?
> 
> I asked about this on #debian-reproducible and they said that it wasn't
> an issue for Debian since they don't ship bytecode, but instead generate
> it at install time. Of course, that doesn't really apply to Guix.
> 
> I used diffoscope-34 to inspect the build outputs to find this, and you
> can see the report here:
> https://famulari.name/misc/7c55c9e97f668234ddea50299d986f14/borg-diffoscope-report.html
> 
> It's first demonstrated in the file
> ...-borg-0.30.0/lib/python3.4/site-packages/__pycache__/site.cpython-34.pyc.
> 
> The first 2 bytes are the "magic numbers" described in PEP-3147, which
> specify the version of the bytecode format. The next 2 bytes are the
> problematic timestamp, as described in the PEP-3147.
> 
> [0]
> http://borgbackup.github.io/
> 
> [1]
> https://docs.python.org/3/library/py_compile.html
> 
> [2]
> https://www.python.org/dev/peps/pep-3147/
> 
> [3] Check out the Guix git commit 4efc8eb27502c, and from there:
> $ tar xf $(./pre-inst-env guix build --source python-3)
> $ sed -n 139,140p Python-3.4.3/Lib/py_compile.py
>     bytecode = importlib._bootstrap._code_to_bytecode(
>             code, source_stats['mtime'], source_stats['size'])
> 
> 
> 

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.