From unknown Tue Jun 17 20:12:39 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#22533 <22533@debbugs.gnu.org> To: bug#22533 <22533@debbugs.gnu.org> Subject: Status: Non-determinism in python-3 ".pyc" bytecode Reply-To: bug#22533 <22533@debbugs.gnu.org> Date: Wed, 18 Jun 2025 03:12:39 +0000 retitle 22533 Non-determinism in python-3 ".pyc" bytecode reassign 22533 guix submitter 22533 Leo Famulari severity 22533 important thanks From debbugs-submit-bounces@debbugs.gnu.org Tue Feb 02 00:16:03 2016 Received: (at submit) by debbugs.gnu.org; 2 Feb 2016 05:16:03 +0000 Received: from localhost ([127.0.0.1]:55601 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aQTJn-0006fy-F4 for submit@debbugs.gnu.org; Tue, 02 Feb 2016 00:16:03 -0500 Received: from eggs.gnu.org ([208.118.235.92]:54384) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aQTJm-0006fl-3x for submit@debbugs.gnu.org; Tue, 02 Feb 2016 00:15:58 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aQTJf-0004vt-Ow for submit@debbugs.gnu.org; Tue, 02 Feb 2016 00:15:52 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=5.0 tests=BAYES_05,T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:46320) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aQTJf-0004vp-M9 for submit@debbugs.gnu.org; Tue, 02 Feb 2016 00:15:51 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45355) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aQTJe-0008R3-HA for bug-guix@gnu.org; Tue, 02 Feb 2016 00:15:51 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aQTJa-0004ve-GU for bug-guix@gnu.org; Tue, 02 Feb 2016 00:15:50 -0500 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:43960) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aQTJZ-0004va-TZ for bug-guix@gnu.org; Tue, 02 Feb 2016 00:15:46 -0500 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 7051C2008E; Tue, 2 Feb 2016 00:15:45 -0500 (EST) Received: from frontend1 ([10.202.2.160]) by compute1.internal (MEProxy); Tue, 02 Feb 2016 00:15:45 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=famulari.name; h= content-type:date:from:message-id:mime-version:subject:to :x-sasl-enc:x-sasl-enc; s=mesmtp; bh=shM1vEUSai5utsd2gGa5cQtc4Fg =; b=XrqYcbRelhKGYhDSkSy4x/b9cl3Q5ka0J3IQ6NTZK5DhylzHGbzSfNiTY3T pztXI4byn0520DZFwG9w6CNvO/lU9FIO9SXzo+Q5fDqzlQzmlXq0EpmJKBz23xyV 9UnEVTGLQ4nHprEqT636c3OaiG7GSi34TXOcS/odY0q+r1Hg= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=sh M1vEUSai5utsd2gGa5cQtc4Fg=; b=pnfI2paP0e5ceG3bZRr/w3dfJLOP67dTkc GesfihhlJ40LbK++Qq97OuLoxkYgCMaxvpVK75Onx2MHN6s+yKSntz++10HfCJNK ONeMPeEaRlhNu7mu7bW3n5lWAWmc9Yj2TAt0zVNFljxtTFe75oelPhahwZwEWXR1 foYdqX/3c= X-Sasl-enc: BKrZe55VqbINpIbpEwCRGdYTAFb+aV2/Bn6yKfxLsfDS 1454390145 Received: from localhost (c-69-249-5-231.hsd1.pa.comcast.net [69.249.5.231]) by mail.messagingengine.com (Postfix) with ESMTPA id 1C742C0001A for ; Tue, 2 Feb 2016 00:15:45 -0500 (EST) Date: Tue, 2 Feb 2016 00:15:44 -0500 From: Leo Famulari To: bug-guix@gnu.org Subject: Non-determinism in python-3 ".pyc" bytecode Message-ID: <20160202051544.GA11744@jasmine> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.1 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.9 (/) While preparing a package for borg [0], I found that the built output was not reproducible. The problem is that the bytecode compiler [1] for Python 3.4.3 (our current version) encodes the mtime of the corresponding Python source file in the output. This is described in PEP-3147 [2], and the responsible Python code is referenced below [3]. I tested a few of our existing python-3 packages: python-ccm, python-pysam, and python-scripttest all exhibit the same problem. We fixed this in python-2 with the patch python-2.7-source-date-epoch.patch, but I don't know how to write this patch for python-3. Can somebody write this patch? I asked about this on #debian-reproducible and they said that it wasn't an issue for Debian since they don't ship bytecode, but instead generate it at install time. Of course, that doesn't really apply to Guix. I used diffoscope-34 to inspect the build outputs to find this, and you can see the report here: https://famulari.name/misc/7c55c9e97f668234ddea50299d986f14/borg-diffoscope-report.html It's first demonstrated in the file ...-borg-0.30.0/lib/python3.4/site-packages/__pycache__/site.cpython-34.pyc. The first 2 bytes are the "magic numbers" described in PEP-3147, which specify the version of the bytecode format. The next 2 bytes are the problematic timestamp, as described in the PEP-3147. [0] http://borgbackup.github.io/ [1] https://docs.python.org/3/library/py_compile.html [2] https://www.python.org/dev/peps/pep-3147/ [3] Check out the Guix git commit 4efc8eb27502c, and from there: $ tar xf $(./pre-inst-env guix build --source python-3) $ sed -n 139,140p Python-3.4.3/Lib/py_compile.py bytecode = importlib._bootstrap._code_to_bytecode( code, source_stats['mtime'], source_stats['size']) From debbugs-submit-bounces@debbugs.gnu.org Tue Feb 02 03:54:42 2016 Received: (at 22533) by debbugs.gnu.org; 2 Feb 2016 08:54:42 +0000 Received: from localhost ([127.0.0.1]:55658 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aQWjS-0003VB-B9 for submit@debbugs.gnu.org; Tue, 02 Feb 2016 03:54:42 -0500 Received: from out4-smtp.messagingengine.com ([66.111.4.28]:50073) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aQWjQ-0003V3-Hb for 22533@debbugs.gnu.org; Tue, 02 Feb 2016 03:54:41 -0500 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id C6AD9205DE; Tue, 2 Feb 2016 03:54:39 -0500 (EST) Received: from frontend2 ([10.202.2.161]) by compute1.internal (MEProxy); Tue, 02 Feb 2016 03:54:39 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=famulari.name; h= content-type:date:from:in-reply-to:message-id:mime-version :references:subject:to:x-sasl-enc:x-sasl-enc; s=mesmtp; bh=L2HKc E4TzcUeANp+yNRUPyNkbcE=; b=UawYEHZNfIztpeUFwt6K065k9JH28l1hsTwTU eoUHzMcZ2m1lEmSa3GVffKZcoLcQ4wlgazm/SYZsl6eI+aRjqPgMYd3pLMMeruy1 jXflT2j2lRC+sgzCXTTs/OayQ1Qh+VAScc9+G7uQ4ekxCcEKZzwH59pH+RlTVKEU u7zPSQ= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-sasl-enc :x-sasl-enc; s=smtpout; bh=L2HKcE4TzcUeANp+yNRUPyNkbcE=; b=Kq4JM eDYYkWRsr6ePZdAEDjMPMPI0YmVhLIJ28NEodi25Tj45ckKi8jEkLhg9HIRjglN9 sL4pvGAWWQsvd4J9hzgh2EWBtwB+nsWLLCC8kO2mMffm0oxJJw5d5fW4jS9qAr+Z +J247JrWBfmIuvHGooQd3t4C4djDlUhRyfHt0A= X-Sasl-enc: d3pgw1jdNMW+kL3KrW51ivi7grGZ8EDPVmwrIzr+Hz8z 1454403279 Received: from localhost (c-69-249-5-231.hsd1.pa.comcast.net [69.249.5.231]) by mail.messagingengine.com (Postfix) with ESMTPA id 7D74F68010F for <22533@debbugs.gnu.org>; Tue, 2 Feb 2016 03:54:39 -0500 (EST) Date: Tue, 2 Feb 2016 03:54:39 -0500 From: Leo Famulari To: 22533@debbugs.gnu.org Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode Message-ID: <20160202085439.GA14802@jasmine> References: <20160202051544.GA11744@jasmine> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160202051544.GA11744@jasmine> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 22533 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) On Tue, Feb 02, 2016 at 12:15:44AM -0500, Leo Famulari wrote: > While preparing a package for borg [0], I found that the built output > was not reproducible. The problem is that the bytecode compiler [1] for > Python 3.4.3 (our current version) encodes the mtime of the > corresponding Python source file in the output. This is described in > PEP-3147 [2], and the responsible Python code is referenced below [3]. > > I tested a few of our existing python-3 packages: python-ccm, > python-pysam, and python-scripttest all exhibit the same problem. > > We fixed this in python-2 with the patch > python-2.7-source-date-epoch.patch, but I don't know how to write this > patch for python-3. mark_weaver suggested setting the timestamps of the source files before building. I think this is a better option if it doesn't break anything. It would allow the bytecode "staleness" check to work as expected while keeping the output consistent. > > Can somebody write this patch? > > I asked about this on #debian-reproducible and they said that it wasn't > an issue for Debian since they don't ship bytecode, but instead generate > it at install time. Of course, that doesn't really apply to Guix. > > I used diffoscope-34 to inspect the build outputs to find this, and you > can see the report here: > https://famulari.name/misc/7c55c9e97f668234ddea50299d986f14/borg-diffoscope-report.html > > It's first demonstrated in the file > ...-borg-0.30.0/lib/python3.4/site-packages/__pycache__/site.cpython-34.pyc. > > The first 2 bytes are the "magic numbers" described in PEP-3147, which > specify the version of the bytecode format. The next 2 bytes are the > problematic timestamp, as described in the PEP-3147. > > [0] > http://borgbackup.github.io/ > > [1] > https://docs.python.org/3/library/py_compile.html > > [2] > https://www.python.org/dev/peps/pep-3147/ > > [3] Check out the Guix git commit 4efc8eb27502c, and from there: > $ tar xf $(./pre-inst-env guix build --source python-3) > $ sed -n 139,140p Python-3.4.3/Lib/py_compile.py > bytecode = importlib._bootstrap._code_to_bytecode( > code, source_stats['mtime'], source_stats['size']) > > > From debbugs-submit-bounces@debbugs.gnu.org Tue Feb 02 15:41:33 2016 Received: (at 22533) by debbugs.gnu.org; 2 Feb 2016 20:41:33 +0000 Received: from localhost ([127.0.0.1]:57143 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aQhlV-0004Ki-IL for submit@debbugs.gnu.org; Tue, 02 Feb 2016 15:41:33 -0500 Received: from eggs.gnu.org ([208.118.235.92]:56669) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aQhlT-0004KW-T6 for 22533@debbugs.gnu.org; Tue, 02 Feb 2016 15:41:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aQhlK-0000bN-N1 for 22533@debbugs.gnu.org; Tue, 02 Feb 2016 15:41:26 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-2.3 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:59431) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aQhlK-0000bJ-IY; Tue, 02 Feb 2016 15:41:22 -0500 Received: from reverse-83.fdn.fr ([80.67.176.83]:48172 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from ) id 1aQhlJ-0006gH-W0; Tue, 02 Feb 2016 15:41:22 -0500 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Leo Famulari Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode References: <20160202051544.GA11744@jasmine> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 14 =?utf-8?Q?Pluvi=C3=B4se?= an 224 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x3D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Tue, 02 Feb 2016 21:41:19 +0100 In-Reply-To: <20160202051544.GA11744@jasmine> (Leo Famulari's message of "Tue, 2 Feb 2016 00:15:44 -0500") Message-ID: <87powezvyo.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.5 (-----) X-Debbugs-Envelope-To: 22533 Cc: 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.5 (-----) --=-=-= Content-Type: text/plain Leo Famulari skribis: > We fixed this in python-2 with the patch > python-2.7-source-date-epoch.patch, but I don't know how to write this > patch for python-3. I would imagine something like this (untested): --=-=-= Content-Type: text/x-patch Content-Disposition: inline --- Python-3.4.3/Lib/importlib/_bootstrap.py 2016-02-02 21:38:48.655809055 +0100 +++ Python-3.4.3/Lib/importlib/_bootstrap.py.new 2016-02-02 21:38:43.659769251 +0100 @@ -667,7 +667,10 @@ def _code_to_bytecode(code, mtime=0, sou """Compile a code object into bytecode for writing out to a byte-compiled file.""" data = bytearray(MAGIC_NUMBER) - data.extend(_w_long(mtime)) + if 'SOURCE_DATE_EPOCH' in _os.environ: + data.extend(_w_long(string.atoi(_os.environ['SOURCE_DATE_EPOCH']))) + else: + data.extend(_w_long(mtime)) data.extend(_w_long(source_size)) data.extend(marshal.dumps(code)) return data --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Could you give it a try and refine as needed? :-) > I asked about this on #debian-reproducible and they said that it wasn't > an issue for Debian since they don't ship bytecode, but instead generate > it at install time. Of course, that doesn't really apply to Guix. I=E2=80=99d recommend trying #reproducible-builds on OFTC, which is more generic. Also, in some cases, it=E2=80=99s useful to look at , which contains notes about non-reproducible packages (currently partly Debian-specific, but we need to lobby to make it more generic. ;-)) Thanks, Ludo=E2=80=99. --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Feb 04 18:17:19 2016 Received: (at 22533) by debbugs.gnu.org; 4 Feb 2016 23:17:20 +0000 Received: from localhost ([127.0.0.1]:60729 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aRT9L-0007bc-J2 for submit@debbugs.gnu.org; Thu, 04 Feb 2016 18:17:19 -0500 Received: from out3-smtp.messagingengine.com ([66.111.4.27]:57913) by debbugs.gnu.org with esmtp (Exim 4.84) (envelope-from ) id 1aRT9J-0007bS-7c for 22533@debbugs.gnu.org; Thu, 04 Feb 2016 18:17:18 -0500 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id C323420720; Thu, 4 Feb 2016 18:17:16 -0500 (EST) Received: from frontend2 ([10.202.2.161]) by compute6.internal (MEProxy); Thu, 04 Feb 2016 18:17:16 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=famulari.name; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-sasl-enc :x-sasl-enc; s=mesmtp; bh=iYp5UYcGOiilUrakLqJxWPAohj8=; b=oe7+1Z O5RjR9NTqA0P7TfJjw/pi10gHcELe2OEFVbvY4RrKYtfvHndlA85RaLlOK7EZsBH OPrXXb1Ov81O+PfLc5RhcI+uknWV++KOtKLnOQqqy+2yF4QRxrzEon1IBejazi+I HMR0mi3dDJCz5hRQx0XBMlNI9jr44m/evCP2w= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=iYp5UYcGOiilUra kLqJxWPAohj8=; b=ge/i+pom4MHNYNooRgoCfsDAd1lsXhxw3+najyXn++4hgyn 5KUa6l94RWZmsPDr1HDC/oe2ND80zc3ttah5XbEGTbuAz36h5yh3CpId+ASMHHDg asmt3nGOiZteoWOicuw/fVui0xPZ97c8scqEr3sIBp/1EX3rFv9d438YR5FY= X-Sasl-enc: KHVm3y2NVYUuKBXBK2QYdNubSfyUeNru07B6t6kXhHk0 1454627834 Received: from localhost (c-73-233-160-251.hsd1.pa.comcast.net [73.233.160.251]) by mail.messagingengine.com (Postfix) with ESMTPA id EF279680109; Thu, 4 Feb 2016 18:17:14 -0500 (EST) Date: Thu, 4 Feb 2016 18:17:08 -0500 From: Leo Famulari To: Ludovic =?iso-8859-1?Q?Court=E8s?= Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode Message-ID: <20160204231708.GA1297@jasmine> References: <20160202051544.GA11744@jasmine> <87powezvyo.fsf@gnu.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="LQksG6bCIzRHxTLp" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <87powezvyo.fsf@gnu.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 22533 Cc: 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --LQksG6bCIzRHxTLp Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit On Tue, Feb 02, 2016 at 09:41:19PM +0100, Ludovic Courtès wrote: > Could you give it a try and refine as needed? :-) I altered your example as shown in the attached patch. It causes some tests related to timestamps to fail, so I disabled them in a very crude way. The final patch should address those tests more carefully. But, the patch doesn't seem to have the desired effect so I'm asking for help! Here is how I tested the patch: I build python-3 with it, and then `export SOURCE_DATE_EPOCH=1` and enter the resulting Python shell. I manually define the '_w_long' function used by the patched function. Then: print (_w_long(locale.atoi(os.getenv('SOURCE_DATE_EPOCH')))) b'\x01\x00\x00\x00' But, when I leave the Python shell and issue `python3 -m compileall helloworld.py`, the timestamps are present in the compiled bytecode. I can watch the clock "tick" by doing this repeatedly: $ touch helloworld.py && rm -r __pycache__ && \ python3 -m compileall helloworld.py && \ hexdump __pycache__/helloworld.cpython-34.pyc | head -n1 I'm not much of a Python programmer, so I'm stumped. --LQksG6bCIzRHxTLp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-SOURCE_DATE_EPOCH.patch" >From d34a71e4ec4501cb53acd3e15633bc1a05665be9 Mon Sep 17 00:00:00 2001 Message-Id: From: Leo Famulari Date: Wed, 3 Feb 2016 20:44:02 -0500 Subject: [PATCH 1/1] SOURCE_DATE_EPOCH --- .../patches/python-3.4.3-source-date-epoch.patch | 21 +++++++++++++++++++++ gnu/packages/python.scm | 14 +++++++++++++- 2 files changed, 34 insertions(+), 1 deletion(-) create mode 100644 gnu/packages/patches/python-3.4.3-source-date-epoch.patch diff --git a/gnu/packages/patches/python-3.4.3-source-date-epoch.patch b/gnu/packages/patches/python-3.4.3-source-date-epoch.patch new file mode 100644 index 0000000..403b2df --- /dev/null +++ b/gnu/packages/patches/python-3.4.3-source-date-epoch.patch @@ -0,0 +1,21 @@ +diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py +index 5b91c05..a87d178 100644 +--- Lib/importlib/_bootstrap.py ++++ Lib/importlib/_bootstrap.py +@@ -666,8 +666,15 @@ def _compile_bytecode(data, name=None, bytecode_path=None, source_path=None): + def _code_to_bytecode(code, mtime=0, source_size=0): + """Compile a code object into bytecode for writing out to a byte-compiled + file.""" ++ """os and locale are required for the SOURCE_DATE_EPOCH ++ deterministic timestamp conditional.""" ++ import os ++ import locale + data = bytearray(MAGIC_NUMBER) +- data.extend(_w_long(mtime)) ++ if os.getenv('SOURCE_DATE_EPOCH'): ++ data.extend(_w_long(locale.atoi(os.getenv('SOURCE_DATE_EPOCH')))) ++ else: ++ data.extend(_w_long(mtime)) + data.extend(_w_long(source_size)) + data.extend(marshal.dumps(code)) + return data diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 48f65b5..cd366f5 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -173,6 +173,17 @@ ;; gnu-build-system.scm. (setenv "SOURCE_DATE_EPOCH" "1") #t)) + (add-before 'configure 'disable-timestamp-tests + (lambda _ + ;; Filter for existing files, since this only affects + ;; Python-3 if the SOURCE_DATE_EPOCH patch is applied. + (substitute* (filter file-exists? + '("Lib/test/test_importlib/test_abc.py")) + (("test_code_bad_timestamp") "disable_test_code_bad_timestamp")) + (substitute* (filter file-exists? + '("Lib/test/test_importlib/source/test_file_loader.py")) + (("test_old_timestamp") "disable_test_old_timestamp")) + )) (add-before 'configure 'do-not-record-configure-flags (lambda* (#:key configure-flags #:allow-other-keys) ;; Remove configure flags from the installed '_sysconfigdata.py' @@ -268,7 +279,8 @@ data types.") ;; XXX Try removing this patch for python > 3.4.3 "python-disable-ssl-test.patch" "python-3-deterministic-build-info.patch" - "python-3-search-paths.patch"))) + "python-3-search-paths.patch" + "python-3.4.3-source-date-epoch.patch"))) (patch-flags '("-p0")) (sha256 (base32 -- 2.6.3 --LQksG6bCIzRHxTLp-- From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 25 04:46:54 2016 Received: (at control) by debbugs.gnu.org; 25 Mar 2016 08:46:54 +0000 Received: from localhost ([127.0.0.1]:36826 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ajNOP-0008KC-RO for submit@debbugs.gnu.org; Fri, 25 Mar 2016 04:46:54 -0400 Received: from eggs.gnu.org ([208.118.235.92]:60580) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ajNOO-0008F4-2i for control@debbugs.gnu.org; Fri, 25 Mar 2016 04:46:52 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ajNOF-0004H5-VR for control@debbugs.gnu.org; Fri, 25 Mar 2016 04:46:47 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=5.0 tests=BAYES_00,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:40756) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ajNOF-0004Gx-SE for control@debbugs.gnu.org; Fri, 25 Mar 2016 04:46:43 -0400 Received: from pluto.bordeaux.inria.fr ([193.50.110.57]:32814 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from ) id 1ajNOF-0007rC-99 for control@debbugs.gnu.org; Fri, 25 Mar 2016 04:46:43 -0400 Date: Fri, 25 Mar 2016 09:46:41 +0100 Message-Id: <87egazt05q.fsf@gnu.org> To: control@debbugs.gnu.org From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) Subject: control message for bug #22533 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) severity 22533 important From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 29 19:12:03 2016 Received: (at submit) by debbugs.gnu.org; 29 Mar 2016 23:12:03 +0000 Received: from localhost ([127.0.0.1]:44227 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1al2nr-0007Ph-3K for submit@debbugs.gnu.org; Tue, 29 Mar 2016 19:12:03 -0400 Received: from eggs.gnu.org ([208.118.235.92]:59955) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1al2np-0007PC-N2 for submit@debbugs.gnu.org; Tue, 29 Mar 2016 19:12:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1al2nj-00089t-Mb for submit@debbugs.gnu.org; Tue, 29 Mar 2016 19:11:56 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM, T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:44252) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1al2nj-00089o-Ja for submit@debbugs.gnu.org; Tue, 29 Mar 2016 19:11:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50918) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1al2ni-0003na-Ex for bug-guix@gnu.org; Tue, 29 Mar 2016 19:11:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1al2nf-00087S-9N for bug-guix@gnu.org; Tue, 29 Mar 2016 19:11:54 -0400 Received: from mail-wm0-x22c.google.com ([2a00:1450:400c:c09::22c]:34838) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1al2nf-00087K-2H for bug-guix@gnu.org; Tue, 29 Mar 2016 19:11:51 -0400 Received: by mail-wm0-x22c.google.com with SMTP id 191so68952326wmq.0 for ; Tue, 29 Mar 2016 16:11:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to; bh=AX9Ihvgu9ENVK87/c0r0d2P9X2fkRIm/y3bIKNnsCZg=; b=WKK1bW2i2YHmBnrwqgm0p5Ce8CF+7FJuOS8Khd3e0brGyG080QdEYpNRLpFn1gOdo4 Hgw8G+Z3QD30N0e6rPAU5ZHSFagK0g3OexY4TFkr3i654Pqh/XPcyeLZwSMBDr8I2aXc DYAjytG1BHIqAIhMtE0NdwepDwOS260ycII6K8VxV0bkknxKD7LSsnHDShDgsDdziW0N 2Ju6VNhyeEbA+2JETOI8VYnHlV0Hx5gp/2x0rR64JWWoCm/a5b1ylHW0VejHubkG/qEn 7Bl3K/GC52YXBx3g2n20F9F+xFrWINSOucXqBjtxb3ViM8RKNg/bkGZix4cTaUcppGVf uzSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to; bh=AX9Ihvgu9ENVK87/c0r0d2P9X2fkRIm/y3bIKNnsCZg=; b=TzFKiuA7TUuiZ8TVNqI2ke+1JTo6O6O5bLyPjoMutz/2bIc//40KxgylJi0U/vr19p G/e6KzxltjfDKh+e1MUFfC1WuWut9Mx52kJMrVUiCbzV2/tIFxR9fpH9xVOX+XOqQ4LO fsiXWGt2S1hEvX+naF9mcl2oblp7rQshbracODGny8LUakLrZYbnEr07y2gt5EzKmEhm y0IgkMokV6I3TcwIxOc+ninYUWj7wJiCeMTqN5ekMq+CiMldqfpwI/aUAWaYj9z4MF1n neCXGk8DZbwGI5r38vOVKZtjXk/lr8AhKSi+zGfLCiHdMqo843SoGtIZCiaU9GSf+ZqT 4w1w== X-Gm-Message-State: AD7BkJLuxUYHRm1e7IDIsuHrvPSjJBo3XayE9EO8x8nAY14N0HKAUfmo7BNWIkGeOYW7bg== X-Received: by 10.28.105.5 with SMTP id e5mr6324824wmc.87.1459293110268; Tue, 29 Mar 2016 16:11:50 -0700 (PDT) Received: from [192.168.0.10] (tal33-3-82-233-82-24.fbx.proxad.net. [82.233.82.24]) by smtp.googlemail.com with ESMTPSA id w184sm1407375wmb.1.2016.03.29.16.11.48 for (version=TLSv1/SSLv3 cipher=OTHER); Tue, 29 Mar 2016 16:11:49 -0700 (PDT) Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode To: bug-guix@gnu.org References: <20160202051544.GA11744@jasmine> <87powezvyo.fsf@gnu.org> <20160204231708.GA1297@jasmine> From: Cyril Roelandt Message-ID: <56FB0BB3.5040303@gmail.com> Date: Wed, 30 Mar 2016 01:11:47 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Icedove/38.7.0 MIME-Version: 1.0 In-Reply-To: <20160204231708.GA1297@jasmine> Content-Type: multipart/mixed; boundary="------------060100070105040008030705" X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.0 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.0 (----) This is a multi-part message in MIME format. --------------060100070105040008030705 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit Here is a version of the patch that works with the upstream Python, but that I cannot get to work with our Guix recipe. Could you test it and tell me what you think? I intend to push this to CPython. Cyril. --------------060100070105040008030705 Content-Type: text/x-diff; name="upstream.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="upstream.patch" diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py index c4ee41a..d9885c9 100644 --- Lib/importlib/_bootstrap.py +++ Lib/importlib/_bootstrap.py @@ -1443,7 +1443,8 @@ class SourceLoader(_LoaderBasics): Implementing this method allows the loader to read bytecode file= s. Raises IOError when the path cannot be handled. """ - return {'mtime': self.path_mtime(path)} + return {'mtime': float(_os.environ.get(b'SOURCE_DATE_EPOCH', + st.st_mtime))} =20 def _cache_bytecode(self, source_path, cache_path, data): """Optional method which writes data (bytes) to a file path (a s= tr). @@ -1580,7 +1581,10 @@ class SourceFileLoader(FileLoader, SourceLoader): def path_stats(self, path): """Return the metadata for the path.""" st =3D _path_stat(path) - return {'mtime': st.st_mtime, 'size': st.st_size} + return { + 'mtime': float(_os.environ.get(b'SOURCE_DATE_EPOCH', st.st_= mtime)), + 'size': st.st_size + } =20 def _cache_bytecode(self, source_path, bytecode_path, data): # Adapt between the two APIs --------------060100070105040008030705-- From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 29 19:13:34 2016 Received: (at 22533) by debbugs.gnu.org; 29 Mar 2016 23:13:34 +0000 Received: from localhost ([127.0.0.1]:44231 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1al2pK-0007S2-EW for submit@debbugs.gnu.org; Tue, 29 Mar 2016 19:13:34 -0400 Received: from mail-wm0-f67.google.com ([74.125.82.67]:35064) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1al2pI-0007Rm-9e for 22533@debbugs.gnu.org; Tue, 29 Mar 2016 19:13:32 -0400 Received: by mail-wm0-f67.google.com with SMTP id 139so9978491wmn.2 for <22533@debbugs.gnu.org>; Tue, 29 Mar 2016 16:13:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to; bh=M1Ax1OHxMsLgZexYUGLjaG0RM6s8H16JvP5YAtS1aqQ=; b=h3p8J2xhER/34cYAc2ISTF625k9rOFtKonYgYugCZ1O9ms1FuApPtiGel36k3dwajd aFkBtUSlCuo943M7l7+BIwdDtX9NJ4uCL3h25DoL57BbzGaew3dXk/5r8kCREc091sDs c9RjPCI2dcdH4gKbKeQKbYvDnJoTJFHOKtbkhLPYMdxgGnO1pHUqDV2hxKOSG9WCRp9B hTaONnOJploR6PmTGcjfQA23jfqIWhKGgj/KTCizzYxrsQWk17Tjad3LY9mdCFAWI1c8 uvjFgkPKAkdvGMiyzs68RXP3IzdmB6th60JDhi+UMQT2emTh95zHi8Xb9FXYWrblWEXt xqiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to; bh=M1Ax1OHxMsLgZexYUGLjaG0RM6s8H16JvP5YAtS1aqQ=; b=f21zgn9PHeAl3z465x5U1+u1hieYLedqIYT8tMz9Sp6TGVCPaEi0jyKLjOlgz3Ydp7 6st4omZ+7RPJUL6f3StJFkwQuFOEmy/xbQ9eCivL/8gb/OZWmdqNcIRvaDo506i2Uh9B ifxKZhU7OZW3MmGr62hfh9WA4IddHgMfto4HhVS3NRZGzLjelPRz9C3FcIlVxZEsB8KW skrlg3v+TekJUGLTJfRyhQKQnfmRjpZwhTEwBe++3lRMZ2MY9GoWtjyGszj+w4KfyEFh pweW2H7jXAjl9tgCPn0mdn1gQe30+9q1y7wlSe6HWbLe1k02mAHLs4ccOtkEb+7JWsmf Fieg== X-Gm-Message-State: AD7BkJKLkFUd76ou2pEEaQoJSacYkRMG+PtjvYckSnfH2UXl3C6ARvO+M5sD2nVvFw4klw== X-Received: by 10.194.9.201 with SMTP id c9mr5824078wjb.47.1459293206714; Tue, 29 Mar 2016 16:13:26 -0700 (PDT) Received: from [192.168.0.10] (tal33-3-82-233-82-24.fbx.proxad.net. [82.233.82.24]) by smtp.googlemail.com with ESMTPSA id w184sm1411901wmb.1.2016.03.29.16.13.25 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 29 Mar 2016 16:13:26 -0700 (PDT) Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode To: Leo Famulari , =?UTF-8?Q?Ludovic_Court=c3=a8s?= References: <20160202051544.GA11744@jasmine> <87powezvyo.fsf@gnu.org> <20160204231708.GA1297@jasmine> From: Cyril Roelandt Message-ID: <56FB0C14.9010909@gmail.com> Date: Wed, 30 Mar 2016 01:13:24 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Icedove/38.7.0 MIME-Version: 1.0 In-Reply-To: <20160204231708.GA1297@jasmine> Content-Type: multipart/mixed; boundary="------------030500010504060804020309" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 22533 Cc: 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) This is a multi-part message in MIME format. --------------030500010504060804020309 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Here is a version of the patch that works with the upstream Python, but that I cannot get to work with our Guix recipe. Could you test it and tell me what you think? I intend to push this to CPython. Cyril. --------------030500010504060804020309 Content-Type: text/x-diff; name="upstream.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="upstream.patch" diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py index c4ee41a..d9885c9 100644 --- Lib/importlib/_bootstrap.py +++ Lib/importlib/_bootstrap.py @@ -1443,7 +1443,8 @@ class SourceLoader(_LoaderBasics): Implementing this method allows the loader to read bytecode files. Raises IOError when the path cannot be handled. """ - return {'mtime': self.path_mtime(path)} + return {'mtime': float(_os.environ.get(b'SOURCE_DATE_EPOCH', + st.st_mtime))} def _cache_bytecode(self, source_path, cache_path, data): """Optional method which writes data (bytes) to a file path (a str). @@ -1580,7 +1581,10 @@ class SourceFileLoader(FileLoader, SourceLoader): def path_stats(self, path): """Return the metadata for the path.""" st = _path_stat(path) - return {'mtime': st.st_mtime, 'size': st.st_size} + return { + 'mtime': float(_os.environ.get(b'SOURCE_DATE_EPOCH', st.st_mtime)), + 'size': st.st_size + } def _cache_bytecode(self, source_path, bytecode_path, data): # Adapt between the two APIs --------------030500010504060804020309-- From debbugs-submit-bounces@debbugs.gnu.org Wed Apr 06 04:30:13 2016 Received: (at 22533) by debbugs.gnu.org; 6 Apr 2016 08:30:14 +0000 Received: from localhost ([127.0.0.1]:50955 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1aniqr-0007jD-ID for submit@debbugs.gnu.org; Wed, 06 Apr 2016 04:30:13 -0400 Received: from eggs.gnu.org ([208.118.235.92]:49331) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1aniqq-0007j1-P3 for 22533@debbugs.gnu.org; Wed, 06 Apr 2016 04:30:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aniqf-0005wV-Dz for 22533@debbugs.gnu.org; Wed, 06 Apr 2016 04:30:07 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.2 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:36475) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aniqf-0005wR-9f; Wed, 06 Apr 2016 04:30:01 -0400 Received: from reverse-83.fdn.fr ([80.67.176.83]:58896 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from ) id 1aniqe-0002c3-Bs; Wed, 06 Apr 2016 04:30:01 -0400 From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Cyril Roelandt Subject: Re: bug#22533: Non-determinism in python-3 ".pyc" bytecode References: <20160202051544.GA11744@jasmine> <87powezvyo.fsf@gnu.org> <20160204231708.GA1297@jasmine> <56FB0C14.9010909@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 18 Germinal an 224 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x3D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Wed, 06 Apr 2016 10:29:57 +0200 In-Reply-To: <56FB0C14.9010909@gmail.com> (Cyril Roelandt's message of "Wed, 30 Mar 2016 01:13:24 +0200") Message-ID: <87oa9nuol6.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -6.0 (------) X-Debbugs-Envelope-To: 22533 Cc: 22533@debbugs.gnu.org, Leo Famulari X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -6.0 (------) --=-=-= Content-Type: text/plain Cyril Roelandt skribis: > Here is a version of the patch that works with the upstream Python, but > that I cannot get to work with our Guix recipe. At first sight the patch LGTM. How does it not work for you? :-) I applied this: --=-=-= Content-Type: text/x-patch Content-Disposition: inline diff --git a/gnu/packages/patches/python-3-deterministic-build-info.patch b/gnu/packages/patches/python-3-deterministic-build-info.patch index 22c372a..bdf9f20 100644 --- a/gnu/packages/patches/python-3-deterministic-build-info.patch +++ b/gnu/packages/patches/python-3-deterministic-build-info.patch @@ -15,3 +15,28 @@ We cannot pass it in CPPFLAGS due to whitespace in the DATE string. #ifndef DATE #ifdef __DATE__ #define DATE __DATE__ + +--- Lib/importlib/_bootstrap.py ++++ Lib/importlib/_bootstrap.py +@@ -1443,7 +1443,8 @@ class SourceLoader(_LoaderBasics): + Implementing this method allows the loader to read bytecode files. + Raises IOError when the path cannot be handled. + """ +- return {'mtime': self.path_mtime(path)} ++ return {'mtime': float(_os.environ.get(b'SOURCE_DATE_EPOCH', ++ st.st_mtime))} + + def _cache_bytecode(self, source_path, cache_path, data): + """Optional method which writes data (bytes) to a file path (a str). +@@ -1580,7 +1581,10 @@ class SourceFileLoader(FileLoader, SourceLoader): + def path_stats(self, path): + """Return the metadata for the path.""" + st = _path_stat(path) +- return {'mtime': st.st_mtime, 'size': st.st_size} ++ return { ++ 'mtime': float(_os.environ.get(b'SOURCE_DATE_EPOCH', st.st_mtime)), ++ 'size': st.st_size ++ } + + def _cache_bytecode(self, source_path, bytecode_path, data): + # Adapt between the two APIs --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable =E2=80=A6 and that leads to these test failures: --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix build python@3 --rounds=3D2 -K [...] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D FAIL: test_bad_marshal (test.test_importlib.source.test_file_loader.Source_= SourceLoaderBadBytecodeTestPEP302) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/util.py", line 22, in wrapper to_return =3D fxn(*args, **kwargs) File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 452, in test_bad_marshal self._test_bad_marshal() File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 342, in _test_bad_marshal self.import_(file_path, '_temp') AssertionError: EOFError not raised =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D FAIL: test_no_marshal (test.test_importlib.source.test_file_loader.Source_S= ourceLoaderBadBytecodeTestPEP302) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/util.py", line 22, in wrapper to_return =3D fxn(*args, **kwargs) File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 441, in test_no_marshal self._test_no_marshal() File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 322, in _test_no_marshal self.import_(file_path, '_temp') AssertionError: EOFError not raised =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D FAIL: test_non_code_marshal (test.test_importlib.source.test_file_loader.So= urce_SourceLoaderBadBytecodeTestPEP302) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/util.py", line 22, in wrapper to_return =3D fxn(*args, **kwargs) File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 445, in test_non_code_marshal self._test_non_code_marshal() File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 331, in _test_non_code_marsh= al self.import_(file_path, '_temp') AssertionError: ImportError not raised =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D FAIL: test_old_timestamp (test.test_importlib.source.test_file_loader.Sourc= e_SourceLoaderBadBytecodeTestPEP302) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/util.py", line 22, in wrapper to_return =3D fxn(*args, **kwargs) File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 471, in test_old_timestamp self.assertEqual(bytecode_file.read(4), source_timestamp) AssertionError: b'\x01\x00\x00\x00' !=3D b'\x7f\xc7\x04W' =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D FAIL: test_bad_marshal (test.test_importlib.source.test_file_loader.Source_= SourceLoaderBadBytecodeTestPEP451) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/util.py", line 22, in wrapper to_return =3D fxn(*args, **kwargs) File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 452, in test_bad_marshal self._test_bad_marshal() File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 342, in _test_bad_marshal self.import_(file_path, '_temp') AssertionError: EOFError not raised =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D FAIL: test_no_marshal (test.test_importlib.source.test_file_loader.Source_S= ourceLoaderBadBytecodeTestPEP451) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/util.py", line 22, in wrapper to_return =3D fxn(*args, **kwargs) File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 441, in test_no_marshal self._test_no_marshal() File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 322, in _test_no_marshal self.import_(file_path, '_temp') AssertionError: EOFError not raised =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D FAIL: test_non_code_marshal (test.test_importlib.source.test_file_loader.So= urce_SourceLoaderBadBytecodeTestPEP451) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/util.py", line 22, in wrapper to_return =3D fxn(*args, **kwargs) File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 445, in test_non_code_marshal self._test_non_code_marshal() File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 331, in _test_non_code_marsh= al self.import_(file_path, '_temp') AssertionError: ImportError not raised =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D FAIL: test_old_timestamp (test.test_importlib.source.test_file_loader.Sourc= e_SourceLoaderBadBytecodeTestPEP451) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/util.py", line 22, in wrapper to_return =3D fxn(*args, **kwargs) File "/tmp/guix-build-python-minimal-3.4.3.drv-0/Python-3.4.3/Lib/test/te= st_importlib/source/test_file_loader.py", line 471, in test_old_timestamp self.assertEqual(bytecode_file.read(4), source_timestamp) AssertionError: b'\x01\x00\x00\x00' !=3D b'\x7f\xc7\x04W' ---------------------------------------------------------------------- Ran 951 tests in 1.102s FAILED (failures=3D8, skipped=3D19, expected failures=3D1) Makefile:958: recipe for target 'test' failed --8<---------------cut here---------------end--------------->8--- =E2=80=98test_old_timestamp=E2=80=99 clearly needs to be adjusted to accoun= t for the change. The others have to do with the bytecode loader, so it=E2=80=99s probably a similar story. Could you look into it? Perhaps you tested with SOURCE_DATE_EPOCH unset? Thanks for working on this, it=E2=80=99s an important bug to fix! Ludo=E2=80=99. --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri May 26 09:41:43 2017 Received: (at 22533) by debbugs.gnu.org; 26 May 2017 13:41:43 +0000 Received: from localhost ([127.0.0.1]:38792 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dEFUt-0004Yd-Ei for submit@debbugs.gnu.org; Fri, 26 May 2017 09:41:43 -0400 Received: from out1-smtp.messagingengine.com ([66.111.4.25]:44977) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dEFUs-0004YV-9W for 22533@debbugs.gnu.org; Fri, 26 May 2017 09:41:42 -0400 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 0625420ACD for <22533@debbugs.gnu.org>; Fri, 26 May 2017 09:41:42 -0400 (EDT) Received: from frontend1 ([10.202.2.160]) by compute5.internal (MEProxy); Fri, 26 May 2017 09:41:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.com; h= content-type:date:from:message-id:mime-version:subject:to :x-me-sender:x-me-sender:x-sasl-enc:x-sasl-enc; s=fm1; bh=IgEIln MVsjdekXQQEGJTQocPmCnOdXnZj5zyWtVp3fc=; b=syW7NjdYJlH9fHp71cOmFZ 5k71JDJOHbCn4GrylJCkvF14O8IRbZwv7/BaSeQj6T8EaOdfGZq2a1GhFjKr9XCB h8Vqu6kMiVn0Hau7/OTCZdtJN1qhP2VAEHx1Bj5mOaeyJObN2CauJ/PU/PUdfZHE uWaMszsPYOKum54W5s/M+HWQa/ig8YDtHYIEzo9JjrTSMAdqgyQF6oXoPf7YNMMA KA6yeBRsm40Ac8j86pIC18sq+qM+OiTlXuyZiOkLcSQ7j3xkb7WY8nj8MXyOPBGE 8BU76oWc/1Kd3UXyTPIcpIUV3JSYk5nFyUd/3PwS1O1L5EI2UUnZKMneck6prT4g == DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:subject:to:x-me-sender:x-me-sender:x-sasl-enc :x-sasl-enc; s=fm1; bh=IgEIlnMVsjdekXQQEGJTQocPmCnOdXnZj5zyWtVp3 fc=; b=fiw48lCW31Mjc4Fp9B7tLDPeCAUvwErUM0hYb6zThkp0B1PE6RNTW1OhI RHwyL8MFZx1X6WbmS2aSMFyu399e5e/am72FkMKZQ4CNRgLuhL6smA/JllU35GhV HZWNnypeYDedQZdn/XII8bdUX9dr+1YArogHCiVAr7HJtHfe10gRqrMzIg3P3bbN lryVhs6jhx0A+PaSxbf5VUkN6ovaw/FleHC+PQM9qTHad0+RurEP6E32e/Tzewbh ENdFHgdL+2fhCOwpZOhrMoFtoGKt92g758OVHmFFSg7mKz30VlO+ddIQLZ3pcIXs t5uEfYL7zp2OxADKE6Az6kJEC5uyg== X-ME-Sender: X-Sasl-enc: p4C/GXJvdi49ufXlsjww5mid40l0d3y1VpUdiiWj8u+D 1495806101 Received: from localhost (unknown [188.113.81.93]) by mail.messagingengine.com (Postfix) with ESMTPA id 876797E84B for <22533@debbugs.gnu.org>; Fri, 26 May 2017 09:41:41 -0400 (EDT) From: Marius Bakke To: 22533@debbugs.gnu.org Subject: Python bytecode reproducibility User-Agent: Notmuch/0.24.1 (https://notmuchmail.org) Emacs/25.2.1 (x86_64-unknown-linux-gnu) Date: Fri, 26 May 2017 15:41:39 +0200 Message-ID: <87bmqfu44s.fsf@fastmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 22533 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --=-=-= Content-Type: text/plain Hello! I stumbled across this bug after re-discovering that Python bytecode is not reproducible (through "glib"). Just sharing some notes.. Nix recently made an effort to fix this. AFAICT the ".pyc" files are still a problem, but at least they got the interpreters building reproducibly: https://github.com/NixOS/nixpkgs/issues/22570 https://github.com/NixOS/nixpkgs/pull/22585 It would be great to revive this longstanding bug! *walks away slowly before anyone notices* --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAlkoMJQACgkQoqBt8qM6 VPoLTQgAsNXfMc86cXSiDAJaJCSTbdUEgeEBCEQDFCqcZA8eTxaosuWwjJy3J9lV 9D+nGYols54UeCypEGgfrrrgB49s9XnSuZNlbkMyvPYzASpHmVpx2OWp6NBiLUf3 Rn9qwMTwqchqZNoRJ2F8MGAbklavS7bqruiI9c/38zA/Sg4aET42vDEOZiB3DDIW bwKFKrZs/BqWSfJXLj9nqzMsbjzn00FzPsOmdxx0rVv07lDdOAUWL0DCm2FbL2OD NhFNt8pW1+0Y6CmYkln+cwmk4XJraSKvH3+QaEHNEBFmL8FuEKixZ0DTRK+bWdzf HUB5JYA2299APLld+UFM9qMaMYA5sQ== =QJ16 -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 03 17:37:51 2018 Received: (at 22533) by debbugs.gnu.org; 3 Mar 2018 22:37:51 +0000 Received: from localhost ([127.0.0.1]:43287 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esFmp-0007cL-Hi for submit@debbugs.gnu.org; Sat, 03 Mar 2018 17:37:51 -0500 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21130) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esFmn-0007cC-N9 for 22533@debbugs.gnu.org; Sat, 03 Mar 2018 17:37:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520116652; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; l=1246; bh=pbfIpUnK4KJw4E9/30piOmYcIWs0PH3xUbo7vmxLWj4=; b=ifWAJwe5pepJzPXxL2C0rCZrbHg0EeVa4F16dooQn2PFPMlMxjI482ze6wtakoox KD8Zgt2TR6+3DFznMBGn982HbdiBagMxanUqOWp1xl81sPgILH5bArIjn01FTymKooy fAR/EXnClSgwFqdhE5XcvetavO+9tCEKp0FmN2vs= Received: from localhost (port-92-200-66-188.dynamic.qsc.de [92.200.66.188]) by mx.zohomail.com with SMTPS id 1520116652148789.4105675915988; Sat, 3 Mar 2018 14:37:32 -0800 (PST) References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> User-agent: mu4e 1.0; emacs 25.3.1 From: Ricardo Wurmus To: Marius Bakke Subject: Re: bug#22533: Python bytecode reproducibility In-reply-to: <87bmqfu44s.fsf@fastmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Sat, 03 Mar 2018 23:37:29 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID: <87606c23bq.fsf@elephly.net> X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Hi Guix, Marius Bakke writes: > It would be great to revive this longstanding bug! Indeed. Here=E2=80=99s another attempt. As far as I understand, the timestamp in t= he pyc files only affects the header. Up until Python 3.6 (incl) the header looks like this: magic | timestamp | size Since Python 3.7 the header may either contain a timestamp or a hash: magic | 00000000000000000000000000000000 | timestamp | size magic | 00000000000000000000000000000001 | hash | size This means we likely won=E2=80=99t have this problem any more with Python 3= .7. For Python 3.6 I guess we could add a final build phase that overwrites the timestamp in the *binary*. This needs to happen before any of the compiled files are wrapped up in a wheel. Should we just wait for Python 3.7 which is expected to be released in June 2018? We=E2=80=99d still have to deal with this problem in Python 2, though. Is it a bad idea to override the timestamps in the generated binaries? I think that we could avoid the recency check then, which was an obstacle to resetting the timestamps of the source files. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 04 04:21:25 2018 Received: (at 22533) by debbugs.gnu.org; 4 Mar 2018 09:21:25 +0000 Received: from localhost ([127.0.0.1]:43489 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esPpd-0007mu-ID for submit@debbugs.gnu.org; Sun, 04 Mar 2018 04:21:25 -0500 Received: from mail-io0-f178.google.com ([209.85.223.178]:33087) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esPpb-0007mg-32 for 22533@debbugs.gnu.org; Sun, 04 Mar 2018 04:21:23 -0500 Received: by mail-io0-f178.google.com with SMTP id f1so14899711iob.0 for <22533@debbugs.gnu.org>; Sun, 04 Mar 2018 01:21:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=qlpGl8u0zY4e7E7tyVG7IOPyJ+IUQTATj0uodQIyF5o=; b=c7I91Azd3jhrjRXTw3UcJ58d/81JKPFTj/TeKFkEjy+SSql9BTNiEyZZqgCvQFOldF PsPBFc5/klu2l1KAEU/znqVCiGKkYyRcZRWMtei8z0RO09DopLTd3IxrrpQjBH7mSk8w NDFu5FhVc+35G4qFZagdFW+h1Q4HVZOUlGSHvKN8GcSTyCPrGotYSOKQLgEEpD0XuYq0 mu+cvIMzPO4CI6gP11WmH8QKl0U3VFoRsNEyT3lLqf/3hJF2ilyKRUmGzxwLi0Z27JLS Ntvt9hPsxlYQK6ohuQQWld+gySUB5fv9d3kVhgRTLANEGiGkko+v5rCZuUkyzb4PGx9+ OCTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=qlpGl8u0zY4e7E7tyVG7IOPyJ+IUQTATj0uodQIyF5o=; b=pqj2mWIILPCUsG4rEvmjg34FVWFXNiGob0Am3ru+dNrAWICjxxjBdzfEQAYunthoS9 p0ughU/dqKxw8RjMi7X2BnJ68uTF3rc2TejWB/iWIV2kvNLDX5JNmgpm6Pe0KLLxXzIH BUGMqxIJcWMm59mjXmz8jU5SdAZ3Flh9M634ZreLRf1Kzz+y8KzP/Kd825dvWlzLqlgd ehwRKTcOy3hoy6Urw/+Kr4rowPWklzlqmfzPFS+sRNtwEORC91iHt4qbQ1sRhDzDiEpE GQsfJ6dklwLxifYYFJtQUH3krckuH/XqMFBP9XGrfFCnnraznpScQQKXzhVGeBwMMGSo GdmQ== X-Gm-Message-State: APf1xPCGpf+oxvVnaQDnu5AO8auxDMNH77+8Tgi2dfSlcCXCWj7y65Eu 4lIXp+tLfMZGvk1Yl8n/4rnMfvmvqQiDypGQHg== X-Google-Smtp-Source: AG47ELsGHlRL86MAim1b83X0nL6jjCQZFEeL7fSFJf/8ZAOnUA5a8u1uGsIGI9jDpYSfHpgKAIPMHRT0hEkNdz1arVY= X-Received: by 10.107.53.231 with SMTP id k100mr14361685ioo.292.1520155277529; Sun, 04 Mar 2018 01:21:17 -0800 (PST) MIME-Version: 1.0 Received: by 10.79.157.15 with HTTP; Sun, 4 Mar 2018 01:21:17 -0800 (PST) In-Reply-To: <87606c23bq.fsf@elephly.net> References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= Date: Sun, 4 Mar 2018 10:21:17 +0100 Message-ID: Subject: Re: bug#22533: Python bytecode reproducibility To: Ricardo Wurmus Content-Type: multipart/alternative; boundary="001a1144a158521715056692bbd8" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: Marius Bakke , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) --001a1144a158521715056692bbd8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable 2018-03-03 23:37 GMT+01:00 Ricardo Wurmus : > Hi Guix, > > Marius Bakke writes: > > > It would be great to revive this longstanding bug! > > Indeed. > > Here=E2=80=99s another attempt. As far as I understand, the timestamp in= the > pyc files only affects the header. > > Up until Python 3.6 (incl) the header looks like this: > > magic | timestamp | size > > Since Python 3.7 the header may either contain a timestamp or a hash: > > magic | 00000000000000000000000000000000 | timestamp | size > magic | 00000000000000000000000000000001 | hash | size > > This means we likely won=E2=80=99t have this problem any more with Python= 3.7. > For Python 3.6 I guess we could add a final build phase that overwrites > the timestamp in the *binary*. This needs to happen before any of the > compiled files are wrapped up in a wheel. > > Should we just wait for Python 3.7 which is expected to be released in > June 2018? We=E2=80=99d still have to deal with this problem in Python 2= , > though. > > Is it a bad idea to override the timestamps in the generated binaries? > I think that we could avoid the recency check then, which was an > obstacle to resetting the timestamps of the source files. -- > Ricardo > > GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC > https://elephly.net > > Nix had this issue, it seems they have a python 3.5 solution, which should be easy to adopt: https://github.com/NixOS/nixpkgs/issues/22570. WDYT? --001a1144a158521715056692bbd8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
2018= -03-03 23:37 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:=
Hi Guix,

Marius Bakke <mbakke@fastmail.com= > writes:

> It would be great to revive this longstanding bug!

Indeed.

Here=E2=80=99s another attempt.=C2=A0 As far as I understand, the timestamp= in the
pyc files only affects the header.

Up until Python 3.6 (incl) the header looks like this:

=C2=A0 magic | timestamp | size

Since Python 3.7 the header may either contain a timestamp or a hash:

=C2=A0 magic | 00000000000000000000000000000000 | timestamp | size
=C2=A0 magic | 00000000000000000000000000000001 | hash=C2=A0 =C2=A0 = =C2=A0 | size

This means we likely won=E2=80=99t have this problem any more with Python 3= .7.
For Python 3.6 I guess we could add a final build phase that overwrites
the timestamp in the *binary*.=C2=A0 This needs to happen before any of the=
compiled files are wrapped up in a wheel.

Should we just wait for Python 3.7 which is expected to be released in
June 2018?=C2=A0 We=E2=80=99d still have to deal with this problem in Pytho= n 2,
though.

Is it a bad idea to override the timestamps in the generated binaries?
I think that we could avoid the recency check then, which was an
obstacle to resetting the timestamps of the source files.
--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC
https:= //elephly.net


Nix had this issue, it seems they have= a python 3.5 solution, which
WDYT?

--001a1144a158521715056692bbd8-- From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 04 07:46:37 2018 Received: (at 22533) by debbugs.gnu.org; 4 Mar 2018 12:46:37 +0000 Received: from localhost ([127.0.0.1]:43524 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esT2D-00065G-0O for submit@debbugs.gnu.org; Sun, 04 Mar 2018 07:46:37 -0500 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21088) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esT2A-000658-LT for 22533@debbugs.gnu.org; Sun, 04 Mar 2018 07:46:35 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520167571; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; l=2427; bh=3cE1rro51XNl5hK6+8kx7xMSfqrfmzWIAD0eMhv8tOU=; b=a+ze8Rck0dRuF9iIhnJJP15V/l68lWgWtnyN9tVqsusEq+n7L6+w1jGhIAd2WSZf uyCioB3c5Z4Bi2fxpsw9epIqL6aXouLreTerE5k8qUn02WorpBoGyucc+t0e90bdfvn lxodLH9ET30xx4jhA/elRFvtE9qnGKp+WhkyeYjM= Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by mx.zohomail.com with SMTPS id 1520167571020712.8951878439287; Sun, 4 Mar 2018 04:46:11 -0800 (PST) References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> User-agent: mu4e 1.0; emacs 25.3.1 From: Ricardo Wurmus To: =?utf-8?Q?G=C3=A1bor?= Boskovits Subject: Re: bug#22533: Python bytecode reproducibility In-reply-to: Date: Sun, 04 Mar 2018 13:46:07 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID: <874llw101c.fsf@elephly.net> X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: Marius Bakke , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Hi G=C3=A1bor, > Nix had this issue, it seems they have a python 3.5 solution, which > should be easy to adopt: https://github.com/NixOS/nixpkgs/issues/22570. > WDYT? Here=E2=80=99s the patch for Nix: https://patch-diff.githubusercontent.com/raw/NixOS/nixpkgs/pull/22585.dif= f Here are the relevant changes to the Python packages: * Python 3.4 substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']" "= (1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])" substituteInPlace "Lib/importlib/_bootstrap.py" --replace "source_mtime = =3D int(source_stats['mtime'])" "source_mtime =3D 1" * Python 3.5 substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']" "= (1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])" substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace "sourc= e_mtime =3D int(st['mtime'])" "source_mtime =3D 1" * Python 3.6 substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']" "= (1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])" substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace "sourc= e_mtime =3D int(st['mtime'])" "source_mtime =3D 1" For all packages they set these environment variables: - set PYTHONHASHSEED=3D0 (for hashes of str, bytes and datetime objects) - set DETERMINISTIC_BUILD; for conditional patching of the timestamp for package builds. The timestamp is not patched in ad-hoc environments, because that would mess with Python=E2=80=99s ability to determine whether to compile source files. They also rebuild all bytecode (with the exception of lib2to3 because it is Python 2 code) three times, once for each optimization level. --8<---------------cut here---------------start------------->8--- + # Determinism: rebuild all bytecode + # We exclude lib2to3 because that's Python 2 code which fails + # We rebuild three times, once for each optimization level + find $out -name "*.py" | $out/bin/python -m compileall -q -f -x "lib2t= o3" -i - + find $out -name "*.py" | $out/bin/python -O -m compileall -q -f -x "li= b2to3" -i - + find $out -name "*.py" | $out/bin/python -OO -m compileall -q -f -x "l= ib2to3" -i - --8<---------------cut here---------------end--------------->8--- -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 04 10:31:08 2018 Received: (at 22533) by debbugs.gnu.org; 4 Mar 2018 15:31:08 +0000 Received: from localhost ([127.0.0.1]:44569 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esVbP-0003gA-Pr for submit@debbugs.gnu.org; Sun, 04 Mar 2018 10:31:08 -0500 Received: from mail-it0-f52.google.com ([209.85.214.52]:35835) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esVbO-0003ff-1Q for 22533@debbugs.gnu.org; Sun, 04 Mar 2018 10:31:06 -0500 Received: by mail-it0-f52.google.com with SMTP id v194so7093024itb.0 for <22533@debbugs.gnu.org>; Sun, 04 Mar 2018 07:31:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=Aq4zBcYuWi7LuDL2UGQrwZQXb38/y6JTcb2jFeNWMls=; b=MwW5cdBdshn2QueZU46mm+RrfCnxfDQuhZ5phWAEEU87CmFIob0KM9t4p5Ed5PBPHv w1JWHalFkqu7WW3wlrhUcdu1A3j/XzQIb0JtaVXxu9oi9w3K+2w5vF+3ei0TvtiInayE P6coZHeT1C32K8DUMUv/5kk3TUtI1/xTn3JdKWVN0cvHlc3t5b9Ic+iiWK9gmOp2zoxr 2kzHZ816KoB7Xb1/KsEr3LvGX49cp9XF8IJWxCYjTQMby13IuU+3eBQG8icRk4LTtRR1 vA+a9WGmLgPedLxCvSAKoHAXQN926UhTdEQCi34r7Z7DDAcq9DUCpuEvVYUUr8dSx5ct CCxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=Aq4zBcYuWi7LuDL2UGQrwZQXb38/y6JTcb2jFeNWMls=; b=WOso0Zqyfto/OP/kIbS/vFPbRqz2oGrg+0ipmkcl9LfgLpHf2bSelw13QsZubUIOW7 ZTFSR9gwJ1tB93AMMeJw6CIazQ8zIoab7zhnXTxn+JY0plQcYvam6qfemXEigPIsaJjL qNT5WR3sNQKnHY5GullSvHiQqexUfJa39xk7N1TxbN5SQ5UEMfgQPyqY/v8APstwYBsK 77fjnAMEKXnU2j1TnbOimUjDLdB7WAwe7N1GVaiODRydlRL4WCiRB9dZz4b6rQ08oQyr XpHgBrhAAXqI0xN/VQ5FlQAgmGaPEst0A8SamDKWTl9PHz7G+qaBhQ1y8GdK65Vyh6XV s5Ew== X-Gm-Message-State: AElRT7GdnhrNa6MKBHAIubmYiZgqEZSxOu2KmgDC+KhaPLRPFDzcmUWA iVuDZxAyglpHUCCXfqJ6YqjBZmRA3PRVeQVKwQ== X-Google-Smtp-Source: AG47ELuf6FOGWnyCgn8UK0ONnKTAbWc8N4p0ydbFOJYpERGUv7rRWxWcLvHeDAbiYWVVGBP1QmZ7+lg+TtzYh2LpfmE= X-Received: by 10.36.118.211 with SMTP id z202mr10913044itb.104.1520177460333; Sun, 04 Mar 2018 07:31:00 -0800 (PST) MIME-Version: 1.0 Received: by 10.79.157.15 with HTTP; Sun, 4 Mar 2018 07:30:59 -0800 (PST) In-Reply-To: <874llw101c.fsf@elephly.net> References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= Date: Sun, 4 Mar 2018 16:30:59 +0100 Message-ID: Subject: Re: bug#22533: Python bytecode reproducibility To: Ricardo Wurmus Content-Type: multipart/alternative; boundary="001a1144123684cf89056697e53c" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: Marius Bakke , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) --001a1144123684cf89056697e53c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable 2018-03-04 13:46 GMT+01:00 Ricardo Wurmus : > > Hi G=C3=A1bor, > > > Nix had this issue, it seems they have a python 3.5 solution, which > > should be easy to adopt: https://github.com/NixOS/nixpkgs/issues/22570. > > WDYT? > > Here=E2=80=99s the patch for Nix: > > https://patch-diff.githubusercontent.com/raw/ > NixOS/nixpkgs/pull/22585.diff > > Here are the relevant changes to the Python packages: > > * Python 3.4 > > substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']" > "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])" > substituteInPlace "Lib/importlib/_bootstrap.py" --replace "source_mtime > =3D int(source_stats['mtime'])" "source_mtime =3D 1" > > * Python 3.5 > > substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']" > "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])" > substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace > "source_mtime =3D int(st['mtime'])" "source_mtime =3D 1" > > * Python 3.6 > substituteInPlace "Lib/py_compile.py" --replace "source_stats['mtime']" > "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])" > substituteInPlace "Lib/importlib/_bootstrap_external.py" --replace > "source_mtime =3D int(st['mtime'])" "source_mtime =3D 1" > > > Nice, thanks for the summary. Can we adopt this as is? Do we need the 3.4 and 3.5 fix or the 3.6 one is enough? > For all packages they set these environment variables: > > - set PYTHONHASHSEED=3D0 (for hashes of str, bytes and datetime objects= ) > > - set DETERMINISTIC_BUILD; for conditional patching of the timestamp > for package builds. The timestamp is not patched in ad-hoc > environments, because that would mess with Python=E2=80=99s ability t= o > determine whether to compile source files. > > Should we set these in python-build-system? What about python booststrap? I guess we use gnu-build-system there, so bootstrap packages might need to set these explicitly? > They also rebuild all bytecode (with the exception of lib2to3 because it > is Python 2 code) three times, once for each optimization level. > > --8<---------------cut here---------------start------------->8--- > + # Determinism: rebuild all bytecode > + # We exclude lib2to3 because that's Python 2 code which fails > + # We rebuild three times, once for each optimization level > + find $out -name "*.py" | $out/bin/python -m compileall -q -f -x > "lib2to3" -i - > + find $out -name "*.py" | $out/bin/python -O -m compileall -q -f -x > "lib2to3" -i - > + find $out -name "*.py" | $out/bin/python -OO -m compileall -q -f -x > "lib2to3" -i - > --8<---------------cut here---------------end--------------->8--- > > Do we also have to do this, or should we settle with one optimization level? Which one? > -- > Ricardo > > GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC > https://elephly.net > > > --001a1144123684cf89056697e53c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
2018= -03-04 13:46 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:=

Hi G=C3=A1bor,

> Nix had this issue, it seems they have a python 3.5 solution, which > should be easy to adopt: https://github.com/NixOS/= nixpkgs/issues/22570.
> WDYT?

Here=E2=80=99s the patch for Nix:

=C2=A0 https://patch-diff.= githubusercontent.com/raw/NixOS/nixpkgs/pull/22585.diff

Here are the relevant changes to the Python packages:

* Python 3.4

=C2=A0 substituteInPlace "Lib/py_compile.py" --replace "sour= ce_stats['mtime']" "(1 if 'DETERMINISTIC_BUILD' i= n os.environ else source_stats['mtime'])"
=C2=A0 substituteInPlace "Lib/importlib/_bootstrap.py" --replace = "source_mtime =3D int(source_stats['mtime'])" "sourc= e_mtime =3D 1"

* Python 3.5

=C2=A0 substituteInPlace "Lib/py_compile.py" --replace "sour= ce_stats['mtime']" "(1 if 'DETERMINISTIC_BUILD' i= n os.environ else source_stats['mtime'])"
=C2=A0 substituteInPlace "Lib/importlib/_bootstrap_external.py&qu= ot; --replace "source_mtime =3D int(st['mtime'])" "s= ource_mtime =3D 1"

* Python 3.6
=C2=A0 substituteInPlace "Lib/py_compile.py" --replace "sour= ce_stats['mtime']" "(1 if 'DETERMINISTIC_BUILD' i= n os.environ else source_stats['mtime'])"
=C2=A0 substituteInPlace "Lib/importlib/_bootstrap_external.py&qu= ot; --replace "source_mtime =3D int(st['mtime'])" "s= ource_mtime =3D 1"



Nice, thanks for the summary.
Can we adopt this as is?
Do we need the 3.4 and 3.5 fix or the = 3.6 one is enough?=C2=A0
=C2=A0
For all packages they set these environment variables:

=C2=A0 - set PYTHONHASHSEED=3D0 (for hashes of str, bytes and datetime obje= cts)

=C2=A0 - set DETERMINISTIC_BUILD; for conditional patching of the timestamp=
=C2=A0 =C2=A0 for package builds.=C2=A0 The timestamp is not patched in ad-= hoc
=C2=A0 =C2=A0 environments, because that would mess with Python=E2=80=99s a= bility to
=C2=A0 =C2=A0 determine whether to compile source files.


Should we set these in python-build-sy= stem? What about python booststrap?
I guess we use gnu-build-syst= em there, so bootstrap packages might need to
set these explicitl= y?
=C2=A0
They also rebuild all bytecode (with the exception of lib2to3 because it is Python 2 code) three times, once for each optimization level.

--8<---------------cut here---------------start------------->8--= -
+=C2=A0 =C2=A0 # Determinism: rebuild all bytecode
+=C2=A0 =C2=A0 # We exclude lib2to3 because that's Python 2 code which = fails
+=C2=A0 =C2=A0 # We rebuild three times, once for each optimization level +=C2=A0 =C2=A0 find $out -name "*.py" | $out/bin/python -m compil= eall -q -f -x "lib2to3" -i -
+=C2=A0 =C2=A0 find $out -name "*.py" | $out/bin/python -O -m com= pileall -q -f -x "lib2to3" -i -
+=C2=A0 =C2=A0 find $out -name "*.py" | $out/bin/python -OO -m co= mpileall -q -f -x "lib2to3" -i -
--8<---------------cut here---------------end--------------->8--= -

<= br>
Do we also have to do this, or should we settle with one opti= mization level? Which one?
=C2=A0
--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC
https:= //elephly.net



--001a1144123684cf89056697e53c-- From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 04 14:18:58 2018 Received: (at 22533) by debbugs.gnu.org; 4 Mar 2018 19:18:58 +0000 Received: from localhost ([127.0.0.1]:44710 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esZ9u-0004Tc-6A for submit@debbugs.gnu.org; Sun, 04 Mar 2018 14:18:58 -0500 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21078) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esZ9s-0004TS-Fs for 22533@debbugs.gnu.org; Sun, 04 Mar 2018 14:18:57 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520191107; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Message-ID; l=2456; bh=H11D0GMfJkaKSkwL+A7ge/nCnoQjs64ZBB1kEmd5inI=; b=CW2KPW/uN3M+Mrw0RutnIYqb2HqRT/CGti5D2WS31+zjwr/KU5dp0DfmjY9g44q/ EBLSsFnx9AoXl6uX2HA3eTgEKlx8+6SPVaeppM48YeRTT8mG/wBzoGqXA0Biw5ce8jg Vk8CCQSyk8Xi2l+a/NzOVFC/uyyS4Z343Djsxueg= Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by mx.zohomail.com with SMTPS id 152019110705415.552663867062392; Sun, 4 Mar 2018 11:18:27 -0800 (PST) References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> User-agent: mu4e 1.0; emacs 25.3.1 From: Ricardo Wurmus To: =?utf-8?Q?G=C3=A1bor?= Boskovits Subject: Re: bug#22533: Python bytecode reproducibility In-reply-to: <874llw101c.fsf@elephly.net> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Sun, 04 Mar 2018 20:18:23 +0100 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Message-ID: <871sgz1wg0.fsf@elephly.net> X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: Marius Bakke , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) --=-=-= Content-Type: text/plain I have applied this patch locally: --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=1.diff diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 5f701701a..0d1ecc3c6 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -359,8 +359,42 @@ data types.") "Lib/ctypes/test/test_win32.py" ; fails on aarch64 "Lib/test/test_fcntl.py")) ; fails on aarch64 #t)))) - (arguments (substitute-keyword-arguments (package-arguments python-2) - ((#:tests? _) #t))) + (arguments + (substitute-keyword-arguments (package-arguments python-2) + ((#:tests? _) #t) + ((#:phases phases) + `(modify-phases ,phases + (add-after 'unpack 'patch-timestamp-for-pyc-files + (lambda _ + ;; We set DETERMINISTIC_BUILD to only override the mtime when + ;; building with Guix, lest we break auto-compilation in + ;; environments. + (setenv "DETERMINISTIC_BUILD" "1") + (substitute* "Lib/py_compile.py" + (("source_stats\\['mtime'\\]") + "(1 if 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime'])")) + + ;; Use deterministic hashes for strings, bytes, and datetime + ;; objects. + (setenv "PYTHONHASHSEED" "0") + + ;; Reset mtime when validating bytecode header. + (substitute* "Lib/importlib/_bootstrap_external.py" + (("source_mtime = int\\(source_stats\\['mtime'\\]\\)") + "source_mtime = 1")) + #t)) + (add-after 'unpack 'disable-timestamp-tests + (lambda _ + (substitute* "Lib/test/test_importlib/source/test_file_loader.py" + (("test_bad_marshal") + "disable_test_bad_marshal") + (("test_no_marshal") + "disable_test_no_marshal") + (("test_non_code_marshal") + "disable_test_non_code_marshal")) + #t)) + (add-before 'check 'allow-non-deterministic-compilation + (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t)))))) (native-search-paths (list (search-path-specification (variable "PYTHONPATH") --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable It allows me to build python-six and python-sip reproducibly. It does not fix problems with Python 2, and I haven=E2=80=99t yet tested if it caus= es any new problems. It=E2=80=99s a little worrying that I had to disable three more tests that = I think shouldn=E2=80=99t have failed. What do you think? -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 04 19:02:54 2018 Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 00:02:54 +0000 Received: from localhost ([127.0.0.1]:44860 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esdag-0002mc-4e for submit@debbugs.gnu.org; Sun, 04 Mar 2018 19:02:54 -0500 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21094) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esdae-0002mR-02 for 22533@debbugs.gnu.org; Sun, 04 Mar 2018 19:02:52 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520208139; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; l=4597; bh=aCWG4IL3DL3qcLo895frCums7f4YqqqNjRsmPhgPbIk=; b=eCRUCw7beM2pVDlTfUgv0z3LfspCTt+d5/i+2kfPDXzGLWpJrGs6C9iaSPD8l7xc WJUBIbDMweHFcI97TftNPZMOPL0Akc0LH8pap5u2JDs55QZSFCPzB20wa6vjOmMmlYz I3uB6WiWc/EK0FUuYMkNNUQZxdIU2l9i4+FP3Q88= Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by mx.zohomail.com with SMTPS id 1520208138979261.13922832052685; Sun, 4 Mar 2018 16:02:18 -0800 (PST) References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> User-agent: mu4e 1.0; emacs 25.3.1 From: Ricardo Wurmus To: =?utf-8?Q?G=C3=A1bor?= Boskovits Subject: Re: bug#22533: Python bytecode reproducibility In-reply-to: <871sgz1wg0.fsf@elephly.net> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 05 Mar 2018 01:02:15 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID: <87y3j7z8xk.fsf@elephly.net> X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: Marius Bakke , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Ricardo Wurmus writes: > I have applied this patch locally: > > diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm > index 5f701701a..0d1ecc3c6 100644 > --- a/gnu/packages/python.scm > +++ b/gnu/packages/python.scm > @@ -359,8 +359,42 @@ data types.") > "Lib/ctypes/test/test_win32.py" ; fails on= aarch64 > "Lib/test/test_fcntl.py")) ; fails on aarc= h64 > #t)))) > - (arguments (substitute-keyword-arguments (package-arguments python-2= ) > - ((#:tests? _) #t))) > + (arguments > + (substitute-keyword-arguments (package-arguments python-2) > + ((#:tests? _) #t) > + ((#:phases phases) > + `(modify-phases ,phases > + (add-after 'unpack 'patch-timestamp-for-pyc-files > + (lambda _ > + ;; We set DETERMINISTIC_BUILD to only override the mtime = when > + ;; building with Guix, lest we break auto-compilation in > + ;; environments. > + (setenv "DETERMINISTIC_BUILD" "1") > + (substitute* "Lib/py_compile.py" > + (("source_stats\\['mtime'\\]") > + "(1 if 'DETERMINISTIC_BUILD' in os.environ else source= _stats['mtime'])")) > + > + ;; Use deterministic hashes for strings, bytes, and datet= ime > + ;; objects. > + (setenv "PYTHONHASHSEED" "0") > + > + ;; Reset mtime when validating bytecode header. > + (substitute* "Lib/importlib/_bootstrap_external.py" > + (("source_mtime =3D int\\(source_stats\\['mtime'\\]\\)"= ) > + "source_mtime =3D 1")) > + #t)) > + (add-after 'unpack 'disable-timestamp-tests > + (lambda _ > + (substitute* "Lib/test/test_importlib/source/test_file_lo= ader.py" > + (("test_bad_marshal") > + "disable_test_bad_marshal") > + (("test_no_marshal") > + "disable_test_no_marshal") > + (("test_non_code_marshal") > + "disable_test_non_code_marshal")) > + #t)) > + (add-before 'check 'allow-non-deterministic-compilation > + (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t)))))) > (native-search-paths > (list (search-path-specification > (variable "PYTHONPATH") > > It allows me to build python-six and python-sip reproducibly. It does > not fix problems with Python 2, and I haven=E2=80=99t yet tested if it ca= uses > any new problems. I tested importing modules in an ad-hoc environment =E2=80=94 no problems. Unfortunately, this doesn=E2=80=99t fix all reproducibility problems with n= umpy: --8<---------------cut here---------------start------------->8--- Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.= 0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config__.= cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy= -1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config__.= cpython-36.pyc differ Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.= 0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_comman= d.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-num= py-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_comm= and.cpython-36.pyc differ Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.= 0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_info= .cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-nump= y-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_inf= o.cpython-36.pyc differ Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.= 0-check/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36= .pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/li= b/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc diffe= r Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.= 0-check/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.py= c and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/p= ython3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc differ Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.= 0-check/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache__/ut= ils.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache_= _/utils.cpython-36.pyc differ --8<---------------cut here---------------end--------------->8--- But the successes with simpler Python packages are promising. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 04 19:05:12 2018 Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 00:05:12 +0000 Received: from localhost ([127.0.0.1]:44865 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esdct-0002qA-Lv for submit@debbugs.gnu.org; Sun, 04 Mar 2018 19:05:11 -0500 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21063) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esdcs-0002q1-Df for 22533@debbugs.gnu.org; Sun, 04 Mar 2018 19:05:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520208306; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; l=3773; bh=kDf+kAvi1x68kUrwTEkVh+TvJ7yDjSovBp2TKCmYxos=; b=YsfyrUe0VuqLT6iT3ZoUuXITVlpAuv3jnIKjLK9vWzfwW6UltdaozBM4S2eMsXmA 1dIkSr/0eIUW1Bah+cOPtF3KIEWe7b5JQyfIADmrZo7e1hUK4lFL8SZmHlfyKnKz2nW bj5aOHVIllneMalmEbCtkkyTPgtJnoOhg9QtatSc= Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by mx.zohomail.com with SMTPS id 1520208306760731.7313417155125; Sun, 4 Mar 2018 16:05:06 -0800 (PST) References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87y3j7z8xk.fsf@elephly.net> User-agent: mu4e 1.0; emacs 25.3.1 From: Ricardo Wurmus To: =?utf-8?Q?G=C3=A1bor?= Boskovits Subject: Re: bug#22533: Python bytecode reproducibility In-reply-to: <87y3j7z8xk.fsf@elephly.net> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 05 Mar 2018 01:05:04 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID: <87woyrz8sv.fsf@elephly.net> X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: Marius Bakke , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Ricardo Wurmus writes: > Unfortunately, this doesn=E2=80=99t fix all reproducibility problems with= numpy: > > --8<---------------cut here---------------start------------->8--- > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.1= 4.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config_= _.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-num= py-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config_= _.cpython-36.pyc differ > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.1= 4.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_comm= and.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_co= mmand.cpython-36.pyc differ > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.1= 4.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_in= fo.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-nu= mpy-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_i= nfo.cpython-36.pyc differ > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.1= 4.0-check/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-= 36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/= lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc dif= fer > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.1= 4.0-check/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.= pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib= /python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc differ > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.1= 4.0-check/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache__/= utils.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python= -numpy-1.14.0/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycach= e__/utils.cpython-36.pyc differ > --8<---------------cut here---------------end--------------->8--- Here=E2=80=99s what diffoscope says: --8<---------------cut here---------------start------------->8--- diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0{= -check,}/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.p= yc --- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check/l= ib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc +++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/pyt= hon3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc @@ -1,8 +1,8 @@ -00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000 3......Z&....... +00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000 3......Z&....... 00000010: 0000 0000 0000 0000 0001 0000 0040 0000 .............@.. 00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400 .s ...d.Z.d.Z.d. 00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502 Z.d.Z.d.Z.e.s.e. 00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30 Z.d.S.).z.1.14.0 00000050: da28 3639 3134 6262 3431 6630 6662 3363 .(6914bb41f0fb3c 00000060: 3162 6135 3030 6261 6534 6537 6436 3731 1ba500bae4e7d671 00000070: 6461 3935 3336 3738 3666 544e 2905 da0d da9536786fTN)... --8<---------------cut here---------------end--------------->8--- In other words: this is the timestamp field of the pyc file. Maybe this can be avoided by setting DETERMINISTIC_BUILD in the python-build-system? -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 05 04:25:43 2018 Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 09:25:43 +0000 Received: from localhost ([127.0.0.1]:45095 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esmNL-0008DG-5G for submit@debbugs.gnu.org; Mon, 05 Mar 2018 04:25:43 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:35672) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esmNJ-0008D7-T3 for 22533@debbugs.gnu.org; Mon, 05 Mar 2018 04:25:42 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 68AE011DF9; Mon, 5 Mar 2018 10:25:41 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rcWp7iJK2RVr; Mon, 5 Mar 2018 10:25:40 +0100 (CET) Received: from ribbon (unknown [193.50.110.134]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 89CDC10CB1; Mon, 5 Mar 2018 10:25:40 +0100 (CET) From: ludo@gnu.org (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Ricardo Wurmus Subject: Re: bug#22533: Python bytecode reproducibility References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 15 =?utf-8?Q?Vent=C3=B4se?= an 226 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 05 Mar 2018 10:25:40 +0100 In-Reply-To: <87606c23bq.fsf@elephly.net> (Ricardo Wurmus's message of "Sat, 03 Mar 2018 23:37:29 +0100") Message-ID: <87h8pudgbv.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 22533 Cc: Marius Bakke , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) Hello! Ricardo Wurmus skribis: > Is it a bad idea to override the timestamps in the generated binaries? > I think that we could avoid the recency check then, which was an > obstacle to resetting the timestamps of the source files. I think it=E2=80=99s good if we can fix Python itself to honor SOURCE_DATE_= EPOCH for its timestamps, but it=E2=80=99s also OK to patch timestamps in generat= ed binaries. We do that already in gzip headers, with =E2=80=98reset-gzip-timestamp=E2= =80=99. Thanks for tackling this! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 05 10:36:39 2018 Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 15:36:39 +0000 Received: from localhost ([127.0.0.1]:46097 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1essAJ-0002CO-4S for submit@debbugs.gnu.org; Mon, 05 Mar 2018 10:36:39 -0500 Received: from mail-it0-f48.google.com ([209.85.214.48]:56037) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1essAH-0002CC-Dx for 22533@debbugs.gnu.org; Mon, 05 Mar 2018 10:36:37 -0500 Received: by mail-it0-f48.google.com with SMTP id n7so9855370ita.5 for <22533@debbugs.gnu.org>; Mon, 05 Mar 2018 07:36:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=d5VLzDe3uM/OmHCvMJ956GZEhIsgT4wg7Hpf19yyIUE=; b=SWbWkyzUX6vQLYt97HzOM2BORD4LZ/TdIb+nISkqCx97pNnKrq3IqhmoHtWdIf0PJD C9Il4SlPbYOxa0uUldeSlYMG5QHG96QYUVpqmxCHloMPLSRHlvu4hSmM30IanrM/hMqT oJJtp81HcebiYm5NWStm6iJkYUQjLLlQbqxbC6MOvMRnUut4u5Ow7+IlOx+CNUG2P6VZ XPw0VJ+bPzU+OWxd2WyKSMgOSYisJPEcniFTNusiic/bUFc7Qx+i+YZvMusORgmPJxVY Cpnoe4CdMTUoFe/qqxaNJ5Ul5Vd+eqbCi4tK1ktGP/PfnklXc6hcSMY+qtgYqHiIIxTS 6g+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=d5VLzDe3uM/OmHCvMJ956GZEhIsgT4wg7Hpf19yyIUE=; b=mLmpQRaAWAiVgYEB+4vPQZ/PFphmq4XRdHHvdlMTyI4WMQt3Kn3BIj0sEEYsQNq1ko RNtYlaRudoxD38yOAWb2/FHKx3EV09QM52ornt/vv9jMQgoyh9GGlLuCqgxjYoLT+uTm rupZdCkhqLF3323De/D4gOvc5HKWTvhNzysymMOmCD+mnpALGlFRTIB65OoCxPyQAV32 eFzVdnIlv6nco9CXfM7RoWTxYHSQi5GPb/+POgq6nXiX/yUmr4hwLzDCYE978YWPFHib Xkh6qxeYE7IQGJXdCzDVXkVABvkY+GdPzROmdKJvQQu7hqSKepT6GVnso0Vf2s+Dtbls 3kLQ== X-Gm-Message-State: AElRT7GSv0qS9xdiMmWdPm6ehtqbZCNVwmH1QxkLdf+EJd+AUKrPZcsv 6+vQKb9mcoibLLZkg9v/pLTBjYHZMrKSrePNpw== X-Google-Smtp-Source: AG47ELtuw/rfXX37ClbgxqTFx7FXHNdTAb1aujP0JNmK7nfsfiIGcpvrJx5Sy4Fe2c6Rd7sSCM/cfmiXF6GfFcd4M7o= X-Received: by 10.36.118.211 with SMTP id z202mr14819343itb.104.1520264191744; Mon, 05 Mar 2018 07:36:31 -0800 (PST) MIME-Version: 1.0 Received: by 10.79.157.15 with HTTP; Mon, 5 Mar 2018 07:36:31 -0800 (PST) In-Reply-To: <87woyrz8sv.fsf@elephly.net> References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87y3j7z8xk.fsf@elephly.net> <87woyrz8sv.fsf@elephly.net> From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= Date: Mon, 5 Mar 2018 16:36:31 +0100 Message-ID: Subject: Re: bug#22533: Python bytecode reproducibility To: Ricardo Wurmus Content-Type: multipart/alternative; boundary="001a114412361d206f0566ac1712" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: Marius Bakke , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) --001a114412361d206f0566ac1712 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable 2018-03-05 1:05 GMT+01:00 Ricardo Wurmus : > > Ricardo Wurmus writes: > > > Unfortunately, this doesn=E2=80=99t fix all reproducibility problems wi= th numpy: > > > > --8<---------------cut here---------------start------------->8--- > > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/ > numpy/distutils/__pycache__/__config__.cpython-36.pyc and /gnu/store/ > kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/distutils/__pycache__/__config__.cpython-36= .pyc > differ > > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/ > numpy/distutils/__pycache__/exec_command.cpython-36.pyc and /gnu/store/ > kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/distutils/__pycache__/exec_command.cpython-= 36.pyc > differ > > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/ > numpy/distutils/__pycache__/system_info.cpython-36.pyc and /gnu/store/ > kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/distutils/__pycache__/system_info.cpython-3= 6.pyc > differ > > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/ > numpy/__pycache__/__config__.cpython-36.pyc and /gnu/store/ > kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc diffe= r > > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/ > numpy/__pycache__/version.cpython-36.pyc and /gnu/store/ > kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc differ > > Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0-check/lib/python3.6/site-packages/ > numpy/testing/nose_tools/__pycache__/utils.cpython-36.pyc and /gnu/store/ > kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/testing/nose_tools/__pycache__/utils.cpytho= n-36.pyc > differ > > --8<---------------cut here---------------end--------------->8--- > > Here=E2=80=99s what diffoscope says: > > --8<---------------cut here---------------start------------->8--- > diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dw > cc-python-numpy-1.14.0{-check,}/lib/python3.6/site-packages/ > numpy/__pycache__/version.cpython-36.pyc > --- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check= / > lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc > +++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/ > python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc > @@ -1,8 +1,8 @@ > -00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000 3......Z&....... > +00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000 3......Z&....... > 00000010: 0000 0000 0000 0000 0001 0000 0040 0000 .............@.. > 00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400 .s ...d.Z.d.Z.d. > 00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502 Z.d.Z.d.Z.e.s.e. > 00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30 Z.d.S.).z.1.14.0 > 00000050: da28 3639 3134 6262 3431 6630 6662 3363 .(6914bb41f0fb3c > 00000060: 3162 6135 3030 6261 6534 6537 6436 3731 1ba500bae4e7d671 > 00000070: 6461 3935 3336 3738 3666 544e 2905 da0d da9536786fTN)... > --8<---------------cut here---------------end--------------->8--- > > In other words: this is the timestamp field of the pyc file. > > Maybe this can be avoided by setting DETERMINISTIC_BUILD in the > python-build-system? > > It seems that the deterministic build patch already landed upstream https://github.com/python/cpython/pull/5200, so we might consider applying the upstream patches. WDYT? > -- > Ricardo > > GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC > https://elephly.net > > > --001a114412361d206f0566ac1712 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
2018= -03-05 1:05 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:<= br>

Ricardo Wurmus <rekado@elephly.net= > writes:

> Unfortunately, this doesn=E2=80=99t fix all reproducibility problems w= ith numpy:
>
> --8<---------------cut here---------------start-------------&g= t;8---
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pyt= hon-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutil= s/__pycache__/__config__.cpython-36.pyc and /gnu/store/kd06ql8fyn= lydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-pac= kages/numpy/distutils/__pycache__/__config__.cpython-36.pyc diffe= r
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pyt= hon-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutil= s/__pycache__/exec_command.cpython-36.pyc and /gnu/store/kd06ql8f= ynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-p= ackages/numpy/distutils/__pycache__/exec_command.cpython-36.pyc d= iffer
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pyt= hon-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutil= s/__pycache__/system_info.cpython-36.pyc and /gnu/store/kd06ql8fy= nlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-pa= ckages/numpy/distutils/__pycache__/system_info.cpython-36.pyc dif= fer
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pyt= hon-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/__pycach= e__/__config__.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnw= k2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/nump= y/__pycache__/__config__.cpython-36.pyc differ
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pyt= hon-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/__pycach= e__/version.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2l= h0778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/<= wbr>__pycache__/version.cpython-36.pyc differ
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pyt= hon-numpy-1.14.0-check/lib/python3.6/site-packages/numpy/testing/= nose_tools/__pycache__/utils.cpython-36.pyc and /gnu/store/kd06ql= 8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site= -packages/numpy/testing/nose_tools/__pycache__/utils.cpython-36.p= yc differ
> --8<---------------cut here---------------end---------------&g= t;8---

Here=E2=80=99s what diffoscope says:

--8<---------------cut here---------------start------------->8--= -
diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-num= py-1.14.0{-check,}/lib/python3.6/site-packages/numpy/__pycache__/= version.cpython-36.pyc
--- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14= .0-check/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc
+++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14= .0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-= 36.pyc
@@ -1,8 +1,8 @@
-00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000=C2=A0 3......Z&.....= ..
+00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000=C2=A0 3......Z&.....= ..
=C2=A000000010: 0000 0000 0000 0000 0001 0000 0040 0000=C2=A0 .............= @..
=C2=A000000020: 0073 2000 0000 6400 5a00 6400 5a01 6400=C2=A0 .s ...d.Z.d.Z= .d.
=C2=A000000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502=C2=A0 Z.d.Z.d.Z.e.s= .e.
=C2=A000000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30=C2=A0 Z.d.S.).z.1.1= 4.0
=C2=A000000050: da28 3639 3134 6262 3431 6630 6662 3363=C2=A0 .(6914bb41f0f= b3c
=C2=A000000060: 3162 6135 3030 6261 6534 6537 6436 3731=C2=A0 1ba500bae4e7d= 671
=C2=A000000070: 6461 3935 3336 3738 3666 544e 2905 da0d=C2=A0 da9536786fTN)= ...
--8<---------------cut here---------------end--------------->8--= -

In other words: this is the timestamp field of the pyc file.

Maybe this can be avoided by setting DETERMINISTIC_BUILD in the
python-build-system?


It seems that the deterministic build patch alrea= dy landed upstream
applying the upstream patches. WDYT?
=C2=A0
--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC
https:= //elephly.net



--001a114412361d206f0566ac1712-- From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 05 15:33:10 2018 Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 20:33:10 +0000 Received: from localhost ([127.0.0.1]:46274 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eswnG-0004tN-Bs for submit@debbugs.gnu.org; Mon, 05 Mar 2018 15:33:10 -0500 Received: from mail-it0-f48.google.com ([209.85.214.48]:51868) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eswnE-0004tB-Oa for 22533@debbugs.gnu.org; Mon, 05 Mar 2018 15:33:09 -0500 Received: by mail-it0-f48.google.com with SMTP id u66so11000435ith.1 for <22533@debbugs.gnu.org>; Mon, 05 Mar 2018 12:33:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=kGMBdOxSMOgl1QRj6NBLebjmjr4kq6K0XUYrt7eCyy4=; b=Q9QYVe0tCl7sapJdS78aIgj9KQPXBXvtLEZdSPXBQjBQ0heUHfjphQ0TARyQr8Q6UI PTI+c5qUa/eob20XpTJ/ds+gI2IzuyUqWdobiJYHD32s9y0wQB/V7dmG2rgqW2W7WDi2 jt0CuEvsPY4FnbUu9aTubSAfHr0Zqn2f/VooHj8kVvOVD0MC+M2YFl9bgTdfRBe3Y/vj ZztAh507ZaQhLGNUCjUGFMOSZrZt2AtBhClYRtebgWq2coO4oiZ4XLE/TrrimfRemCzu zEibm7dSIVsn3+jZriMGaj00HkhY9dXvO56I1lk3P2akc4JIkF9zYTtutAnmT5AFIJUE K7KQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=kGMBdOxSMOgl1QRj6NBLebjmjr4kq6K0XUYrt7eCyy4=; b=WHIF1ssqMSCSLAAUq1dkyGnFlaw4EhKKq+NXuDHT9QpbkV+5oRH81HJUyfWLc9CJJO 93CNQxhhxP+XOMuCIMhW28cFpGZSXuy+nSfGusEtNtp6GaocrJs3c/VQpANLfviuElrq A8c08HZEhKYcvARdVu2uuZaVtvqRwEz5kQ/0ut1jRIyMnqiFIfWUCcoirFQAt5SRSa2t LOYDnr0VhL5QFiId/AHHhyiG9J5//nJ21l4T7VkoZtMCe9lLHPEDlCl7/JH1XtZX/ss4 6HOQzTHlCRQiZMgLj/KllSItSIjGDwN0ZnI2ozhibZ+wv0hx5sH5nUiqMT8AGVuk5JNq gzVA== X-Gm-Message-State: AElRT7EeWF6tPJHS2F4Ctmzlur0OjgqAIMwyJRTNB0o+FLHAMlz96279 28lghXZndk3Q/rurtoG8CfKzEnvz5mB7Zk6yzQ== X-Google-Smtp-Source: AG47ELtv5f9DaB5Ul7DIaECVs4kdR7HGbheHwMTP8jg4w9bU1QuJwUyR0SpLe6exXvi+Zjp9evPXRhsRrc+znPy4cXA= X-Received: by 10.36.2.200 with SMTP id 191mr7590270itu.108.1520281982894; Mon, 05 Mar 2018 12:33:02 -0800 (PST) MIME-Version: 1.0 Received: by 10.79.157.15 with HTTP; Mon, 5 Mar 2018 12:33:02 -0800 (PST) In-Reply-To: References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87y3j7z8xk.fsf@elephly.net> <87woyrz8sv.fsf@elephly.net> From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= Date: Mon, 5 Mar 2018 21:33:02 +0100 Message-ID: Subject: Re: bug#22533: Python bytecode reproducibility To: Ricardo Wurmus Content-Type: multipart/alternative; boundary="001a11446d988c8c0f0566b03bdf" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: Marius Bakke , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) --001a11446d988c8c0f0566b03bdf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable 2018-03-05 16:36 GMT+01:00 G=C3=A1bor Boskovits : > 2018-03-05 1:05 GMT+01:00 Ricardo Wurmus : > >> >> Ricardo Wurmus writes: >> >> > Unfortunately, this doesn=E2=80=99t fix all reproducibility problems w= ith numpy: >> > >> > --8<---------------cut here---------------start------------->8--- >> > Binary files /gnu/store/kd06ql8fynlydymzhhn >> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >> packages/numpy/distutils/__pycache__/__config__.cpython-36.pyc and >> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/distutils/__ >> pycache__/__config__.cpython-36.pyc differ >> > Binary files /gnu/store/kd06ql8fynlydymzhhn >> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >> packages/numpy/distutils/__pycache__/exec_command.cpython-36.pyc and >> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/distutils/__ >> pycache__/exec_command.cpython-36.pyc differ >> > Binary files /gnu/store/kd06ql8fynlydymzhhn >> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >> packages/numpy/distutils/__pycache__/system_info.cpython-36.pyc and >> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/distutils/__ >> pycache__/system_info.cpython-36.pyc differ >> > Binary files /gnu/store/kd06ql8fynlydymzhhn >> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >> packages/numpy/__pycache__/__config__.cpython-36.pyc and >> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36= .pyc >> differ >> > Binary files /gnu/store/kd06ql8fynlydymzhhn >> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >> packages/numpy/__pycache__/version.cpython-36.pyc and >> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.py= c >> differ >> > Binary files /gnu/store/kd06ql8fynlydymzhhn >> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >> packages/numpy/testing/nose_tools/__pycache__/utils.cpython-36.pyc and >> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/testing/nose_ >> tools/__pycache__/utils.cpython-36.pyc differ >> > --8<---------------cut here---------------end--------------->8--- >> >> Here=E2=80=99s what diffoscope says: >> >> --8<---------------cut here---------------start------------->8--- >> diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0{-check,}/lib/python3.6/site-packages/numpy/__pycache_ >> _/version.cpython-36.pyc >> --- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0-check/lib/python3.6/site-packages/numpy/__pycache__/ >> version.cpython-36.pyc >> +++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.py= c >> @@ -1,8 +1,8 @@ >> -00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000 3......Z&....... >> +00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000 3......Z&....... >> 00000010: 0000 0000 0000 0000 0001 0000 0040 0000 .............@.. >> 00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400 .s ...d.Z.d.Z.d. >> 00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502 Z.d.Z.d.Z.e.s.e. >> 00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30 Z.d.S.).z.1.14.0 >> 00000050: da28 3639 3134 6262 3431 6630 6662 3363 .(6914bb41f0fb3c >> 00000060: 3162 6135 3030 6261 6534 6537 6436 3731 1ba500bae4e7d671 >> 00000070: 6461 3935 3336 3738 3666 544e 2905 da0d da9536786fTN)... >> --8<---------------cut here---------------end--------------->8--- >> >> In other words: this is the timestamp field of the pyc file. >> >> Maybe this can be avoided by setting DETERMINISTIC_BUILD in the >> python-build-system? >> >> > It seems that the deterministic build patch already landed upstream > https://github.com/python/cpython/pull/5200, so we might consider > applying the upstream patches. WDYT? > And also this: https://github.com/python/cpython/pull/4575. I'm now having a look at this approach. However this second one seems quite invasive... > > >> -- >> Ricardo >> >> GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC >> https://elephly.net >> >> >> > --001a11446d988c8c0f0566b03bdf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
2018= -03-05 16:36 GMT+01:00 G=C3=A1bor Boskovits <boskovits@gmail.com&g= t;:
2018-03-05 1:05 GMT+01:00 Ricardo Wurmus &l= t;rekado@elephly.ne= t>:

Ricardo Wurmus <= rekado@elephly.net> writes:

> Unfortunately, this doesn=E2=80=99t fix all reproducibility problems w= ith numpy:
>
> --8<---------------cut here---------------start-------------&g= t;8---
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutils/__<= wbr>pycache__/__config__.cpython-36.pyc and /gnu/store/kd06ql8fynlydym= zhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-pac= kages/numpy/distutils/__pycache__/__config__.cpython-36.pyc diffe= r
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutils/__<= wbr>pycache__/exec_command.cpython-36.pyc and /gnu/store/kd06ql8fynlyd= ymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-p= ackages/numpy/distutils/__pycache__/exec_command.cpython-36.pyc d= iffer
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0-check/lib/python3.6/site-packages/numpy/distutils/__<= wbr>pycache__/system_info.cpython-36.pyc and /gnu/store/kd06ql8fynlydy= mzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-pa= ckages/numpy/distutils/__pycache__/system_info.cpython-36.pyc dif= fer
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0-check/lib/python3.6/site-packages/numpy/__pycache__/_= _config__.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0= 778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/__p= ycache__/__config__.cpython-36.pyc differ
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0-check/lib/python3.6/site-packages/numpy/__pycache__/<= wbr>version.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778= dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/__pyca= che__/version.cpython-36.pyc differ
> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0-check/lib/python3.6/site-packages/numpy/testing/nose_= tools/__pycache__/utils.cpython-36.pyc and /gnu/store/kd06ql8fynl= ydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache__/utils.cpython-36.p= yc differ
> --8<---------------cut here---------------end---------------&g= t;8---

Here=E2=80=99s what diffoscope says:

--8<---------------cut here---------------start------------->8--= -
diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.= 14.0{-check,}/lib/python3.6/site-packages/numpy/__pycache__/= version.cpython-36.pyc
--- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14= .0-check/lib/python3.6/site-packages/numpy/__pycache__/version.cp= ython-36.pyc
+++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14= .0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-= 36.pyc
@@ -1,8 +1,8 @@
-00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000=C2=A0 3......Z&.....= ..
+00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000=C2=A0 3......Z&.....= ..
=C2=A000000010: 0000 0000 0000 0000 0001 0000 0040 0000=C2=A0 .............= @..
=C2=A000000020: 0073 2000 0000 6400 5a00 6400 5a01 6400=C2=A0 .s ...d.Z.d.Z= .d.
=C2=A000000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502=C2=A0 Z.d.Z.d.Z.e.s= .e.
=C2=A000000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30=C2=A0 Z.d.S.).z.1.1= 4.0
=C2=A000000050: da28 3639 3134 6262 3431 6630 6662 3363=C2=A0 .(6914bb41f0f= b3c
=C2=A000000060: 3162 6135 3030 6261 6534 6537 6436 3731=C2=A0 1ba500bae4e7d= 671
=C2=A000000070: 6461 3935 3336 3738 3666 544e 2905 da0d=C2=A0 da9536786fTN)= ...
--8<---------------cut here---------------end--------------->8--= -

In other words: this is the timestamp field of the pyc file.

Maybe this can be avoided by setting DETERMINISTIC_BUILD in the
python-build-system?


=
It seems that the deterministic build patch already landed= upstream
applying the upstream patches. WDYT?
=

I'm now having a look at this approach. However this second one
=
seems quite invasive...
=C2=A0
=C2=A0
--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC
https:= //elephly.net




--001a11446d988c8c0f0566b03bdf-- From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 05 16:47:03 2018 Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 21:47:03 +0000 Received: from localhost ([127.0.0.1]:46349 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esxwk-0000AZ-T4 for submit@debbugs.gnu.org; Mon, 05 Mar 2018 16:47:03 -0500 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21029) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esxwi-0000AA-NZ for 22533@debbugs.gnu.org; Mon, 05 Mar 2018 16:47:01 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520286401; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; l=4779; bh=rB8EHrJXIoTMq5ngQ+i7eNck+g8O/1ezFUf9KIMWCDA=; b=a+Ml+GCo5V8AVbbCIvU+dsLwmEBVlLU3GqR+XAMlLx/XHg45Y4Xq6dJyH4qJTRPe hdXnHE+RNETAoOvfLkODM2ZzI0fAguZwDvUkdDjGganobSI7dJhl54czV+aCXfWBUXJ CAAEHXhRAmSZSmpx12KyAOWOaqhroJ269qR5iKok= Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by mx.zohomail.com with SMTPS id 1520286400945738.2980576769127; Mon, 5 Mar 2018 13:46:40 -0800 (PST) References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87y3j7z8xk.fsf@elephly.net> <87woyrz8sv.fsf@elephly.net> User-agent: mu4e 1.0; emacs 25.3.1 From: Ricardo Wurmus To: =?utf-8?Q?G=C3=A1bor?= Boskovits Subject: Re: bug#22533: Python bytecode reproducibility In-reply-to: X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 05 Mar 2018 22:46:38 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID: <87zi3mqjpd.fsf@elephly.net> X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: Marius Bakke , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) G=C3=A1bor Boskovits writes: > 2018-03-05 16:36 GMT+01:00 G=C3=A1bor Boskovits : > >> 2018-03-05 1:05 GMT+01:00 Ricardo Wurmus : >> >>> >>> Ricardo Wurmus writes: >>> >>> > Unfortunately, this doesn=E2=80=99t fix all reproducibility problems = with numpy: >>> > >>> > --8<---------------cut here---------------start------------->8--- >>> > Binary files /gnu/store/kd06ql8fynlydymzhhn >>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >>> packages/numpy/distutils/__pycache__/__config__.cpython-36.pyc and >>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/distutils/__ >>> pycache__/__config__.cpython-36.pyc differ >>> > Binary files /gnu/store/kd06ql8fynlydymzhhn >>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >>> packages/numpy/distutils/__pycache__/exec_command.cpython-36.pyc and >>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/distutils/__ >>> pycache__/exec_command.cpython-36.pyc differ >>> > Binary files /gnu/store/kd06ql8fynlydymzhhn >>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >>> packages/numpy/distutils/__pycache__/system_info.cpython-36.pyc and >>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/distutils/__ >>> pycache__/system_info.cpython-36.pyc differ >>> > Binary files /gnu/store/kd06ql8fynlydymzhhn >>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >>> packages/numpy/__pycache__/__config__.cpython-36.pyc and >>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-3= 6.pyc >>> differ >>> > Binary files /gnu/store/kd06ql8fynlydymzhhn >>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >>> packages/numpy/__pycache__/version.cpython-36.pyc and >>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.p= yc >>> differ >>> > Binary files /gnu/store/kd06ql8fynlydymzhhn >>> wk2lh0778dwcc-python-numpy-1.14.0-check/lib/python3.6/site- >>> packages/numpy/testing/nose_tools/__pycache__/utils.cpython-36.pyc and >>> /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/testing/nose_ >>> tools/__pycache__/utils.cpython-36.pyc differ >>> > --8<---------------cut here---------------end--------------->8--- >>> >>> Here=E2=80=99s what diffoscope says: >>> >>> --8<---------------cut here---------------start------------->8--- >>> diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0{-check,}/lib/python3.6/site-packages/numpy/__pycache_ >>> _/version.cpython-36.pyc >>> --- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0-check/lib/python3.6/site-packages/numpy/__pycache__/ >>> version.cpython-36.pyc >>> +++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1. >>> 14.0/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.p= yc >>> @@ -1,8 +1,8 @@ >>> -00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000 3......Z&....... >>> +00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000 3......Z&....... >>> 00000010: 0000 0000 0000 0000 0001 0000 0040 0000 .............@.. >>> 00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400 .s ...d.Z.d.Z.d. >>> 00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502 Z.d.Z.d.Z.e.s.e. >>> 00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30 Z.d.S.).z.1.14.0 >>> 00000050: da28 3639 3134 6262 3431 6630 6662 3363 .(6914bb41f0fb3c >>> 00000060: 3162 6135 3030 6261 6534 6537 6436 3731 1ba500bae4e7d671 >>> 00000070: 6461 3935 3336 3738 3666 544e 2905 da0d da9536786fTN)... >>> --8<---------------cut here---------------end--------------->8--- >>> >>> In other words: this is the timestamp field of the pyc file. >>> >>> Maybe this can be avoided by setting DETERMINISTIC_BUILD in the >>> python-build-system? >>> >>> >> It seems that the deterministic build patch already landed upstream >> https://github.com/python/cpython/pull/5200, so we might consider >> applying the upstream patches. WDYT? >> > > And also this: https://github.com/python/cpython/pull/4575. > I'm now having a look at this approach. However this second one > seems quite invasive... These patches are for what will become Python 3.7. Python 3.6 does not have support for =E2=80=9Cinvalidation_mode=E2=80=9D, so at least the first= patch would not work for us. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 05 17:02:58 2018 Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 22:02:58 +0000 Received: from localhost ([127.0.0.1]:46353 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esyCA-0000XL-HF for submit@debbugs.gnu.org; Mon, 05 Mar 2018 17:02:58 -0500 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21061) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esyC9-0000XC-4M for 22533@debbugs.gnu.org; Mon, 05 Mar 2018 17:02:57 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520287352; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; l=3905; bh=izzqYGxRAQxble/mdBDQcoQPP4NELOzBPwMIa3dvQMA=; b=dhx2ynvJibiXvvgqGLqnDeFGkLRmkYw0JG37YONnfhecqCqHZnL9+7PTnZy+RlL/ a7lEor4V/uAaWlp/ICRGaD4NJq+K79f9IJunJOar2hPeu17F0jVPClS70xo6BdG0zTn nxK5wjdrv1EVuMPaW7+U1i305bm5nV3TOTkjjk+Q= Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by mx.zohomail.com with SMTPS id 1520287352075495.14100502512724; Mon, 5 Mar 2018 14:02:32 -0800 (PST) References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87y3j7z8xk.fsf@elephly.net> <87woyrz8sv.fsf@elephly.net> User-agent: mu4e 1.0; emacs 25.3.1 From: Ricardo Wurmus To: =?utf-8?Q?G=C3=A1bor?= Boskovits Subject: Re: bug#22533: Python bytecode reproducibility In-reply-to: <87woyrz8sv.fsf@elephly.net> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 05 Mar 2018 23:02:29 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID: <87y3j6qiyy.fsf@elephly.net> X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Ricardo Wurmus writes: > Ricardo Wurmus writes: > >> Unfortunately, this doesn=E2=80=99t fix all reproducibility problems wit= h numpy: >> >> --8<---------------cut here---------------start------------->8--- >> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.= 14.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config= __.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-nu= mpy-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/__config= __.cpython-36.pyc differ >> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.= 14.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_com= mand.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-= numpy-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/exec_c= ommand.cpython-36.pyc differ >> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.= 14.0-check/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_i= nfo.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-n= umpy-1.14.0/lib/python3.6/site-packages/numpy/distutils/__pycache__/system_= info.cpython-36.pyc differ >> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.= 14.0-check/lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython= -36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0= /lib/python3.6/site-packages/numpy/__pycache__/__config__.cpython-36.pyc di= ffer >> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.= 14.0-check/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36= .pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/li= b/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc differ >> Binary files /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.= 14.0-check/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache__= /utils.cpython-36.pyc and /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-pytho= n-numpy-1.14.0/lib/python3.6/site-packages/numpy/testing/nose_tools/__pycac= he__/utils.cpython-36.pyc differ >> --8<---------------cut here---------------end--------------->8--- > > Here=E2=80=99s what diffoscope says: > > --8<---------------cut here---------------start------------->8--- > diffoscope /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.= 0{-check,}/lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36= .pyc > --- /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0-check= /lib/python3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc > +++ /gnu/store/kd06ql8fynlydymzhhnwk2lh0778dwcc-python-numpy-1.14.0/lib/p= ython3.6/site-packages/numpy/__pycache__/version.cpython-36.pyc > @@ -1,8 +1,8 @@ > -00000000: 330d 0d0a fa87 9c5a 2601 0000 e300 0000 3......Z&....... > +00000000: 330d 0d0a c485 9c5a 2601 0000 e300 0000 3......Z&....... > 00000010: 0000 0000 0000 0000 0001 0000 0040 0000 .............@.. > 00000020: 0073 2000 0000 6400 5a00 6400 5a01 6400 .s ...d.Z.d.Z.d. > 00000030: 5a02 6401 5a03 6402 5a04 6504 731c 6502 Z.d.Z.d.Z.e.s.e. > 00000040: 5a01 6403 5300 2904 7a06 312e 3134 2e30 Z.d.S.).z.1.14.0 > 00000050: da28 3639 3134 6262 3431 6630 6662 3363 .(6914bb41f0fb3c > 00000060: 3162 6135 3030 6261 6534 6537 6436 3731 1ba500bae4e7d671 > 00000070: 6461 3935 3336 3738 3666 544e 2905 da0d da9536786fTN)... > --8<---------------cut here---------------end--------------->8--- > > In other words: this is the timestamp field of the pyc file. > > Maybe this can be avoided by setting DETERMINISTIC_BUILD in the > python-build-system? It cannot. So, something=E2=80=99s still missing from my patch. Does anyone see what = might be missing? --=20 Ricardo From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 05 17:07:04 2018 Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 22:07:04 +0000 Received: from localhost ([127.0.0.1]:46362 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esyG8-0000eC-73 for submit@debbugs.gnu.org; Mon, 05 Mar 2018 17:07:04 -0500 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21047) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1esyG5-0000dk-0C for 22533@debbugs.gnu.org; Mon, 05 Mar 2018 17:07:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520287614; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; l=2528; bh=woesyRo1lU0C4gmgneg2GpwmkgX7D+3pq8U95JycRAk=; b=H7ViicHgBqaQBxGWK/uJdA2EHe7MfVUfj2RlhJh7UWl84Rycs9X5AfStisUdpVZ0 HZTnV0WBFL//Bgq7crITF5lWO8kPc1U7jH3cQmLVgN9vSSRmQyWzddd0sLYmMtkq8cu 54PM8ETMap8ZnauU6rPElbSTIG9CF+D//O9lgZbw= Received: from localhost (port-92-200-95-79.dynamic.qsc.de [92.200.95.79]) by mx.zohomail.com with SMTPS id 1520287614401592.5012332278376; Mon, 5 Mar 2018 14:06:54 -0800 (PST) References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87y3j7z8xk.fsf@elephly.net> User-agent: mu4e 1.0; emacs 25.3.1 From: Ricardo Wurmus To: =?utf-8?Q?G=C3=A1bor?= Boskovits Subject: Re: bug#22533: Python bytecode reproducibility In-reply-to: <87y3j7z8xk.fsf@elephly.net> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 05 Mar 2018 23:06:51 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID: <87woyqqiro.fsf@elephly.net> X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Ricardo Wurmus writes: > Ricardo Wurmus writes: > >> I have applied this patch locally: >> >> diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm >> index 5f701701a..0d1ecc3c6 100644 >> --- a/gnu/packages/python.scm >> +++ b/gnu/packages/python.scm >> @@ -359,8 +359,42 @@ data types.") >> "Lib/ctypes/test/test_win32.py" ; fails o= n aarch64 >> "Lib/test/test_fcntl.py")) ; fails on aar= ch64 >> #t)))) >> - (arguments (substitute-keyword-arguments (package-arguments python-= 2) >> - ((#:tests? _) #t))) >> + (arguments >> + (substitute-keyword-arguments (package-arguments python-2) >> + ((#:tests? _) #t) >> + ((#:phases phases) >> + `(modify-phases ,phases >> + (add-after 'unpack 'patch-timestamp-for-pyc-files >> + (lambda _ >> + ;; We set DETERMINISTIC_BUILD to only override the mtime= when >> + ;; building with Guix, lest we break auto-compilation in >> + ;; environments. >> + (setenv "DETERMINISTIC_BUILD" "1") >> + (substitute* "Lib/py_compile.py" >> + (("source_stats\\['mtime'\\]") >> + "(1 if 'DETERMINISTIC_BUILD' in os.environ else sourc= e_stats['mtime'])")) >> + >> + ;; Use deterministic hashes for strings, bytes, and date= time >> + ;; objects. >> + (setenv "PYTHONHASHSEED" "0") >> + >> + ;; Reset mtime when validating bytecode header. >> + (substitute* "Lib/importlib/_bootstrap_external.py" >> + (("source_mtime =3D int\\(source_stats\\['mtime'\\]\\)= ") >> + "source_mtime =3D 1")) >> + #t)) >> + (add-after 'unpack 'disable-timestamp-tests >> + (lambda _ >> + (substitute* "Lib/test/test_importlib/source/test_file_l= oader.py" >> + (("test_bad_marshal") >> + "disable_test_bad_marshal") >> + (("test_no_marshal") >> + "disable_test_no_marshal") >> + (("test_non_code_marshal") >> + "disable_test_non_code_marshal")) >> + #t)) >> + (add-before 'check 'allow-non-deterministic-compilation >> + (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t)))))) >> (native-search-paths >> (list (search-path-specification >> (variable "PYTHONPATH") >> >> It allows me to build python-six and python-sip reproducibly. It does >> not fix problems with Python 2, and I haven=E2=80=99t yet tested if it c= auses >> any new problems. I should also note that Python 3 itself still contains pyc files with timestamps. This could be the reason why in Nix all pyc files are rebuilt (more than once). -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net From debbugs-submit-bounces@debbugs.gnu.org Mon Mar 05 18:21:33 2018 Received: (at 22533) by debbugs.gnu.org; 5 Mar 2018 23:21:33 +0000 Received: from localhost ([127.0.0.1]:46426 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eszQ6-0004Ly-Ko for submit@debbugs.gnu.org; Mon, 05 Mar 2018 18:21:33 -0500 Received: from out5-smtp.messagingengine.com ([66.111.4.29]:44553) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1eszQ4-0004Lp-Jf for 22533@debbugs.gnu.org; Mon, 05 Mar 2018 18:21:25 -0500 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id 0C40D21957; Mon, 5 Mar 2018 18:21:24 -0500 (EST) Received: from frontend2 ([10.202.2.161]) by compute5.internal (MEProxy); Mon, 05 Mar 2018 18:21:24 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.com; h= cc:content-type:date:from:in-reply-to:message-id:mime-version :references:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=cW2DVlZ6SG5dq7uWsaqu+k/yeEHYKMBRRWmGKdj8DaE=; b=AdeOmMoU J9/jUhYrok8Sn9cK/w2jW3ta04affTGT83bazNXmE1TavSiDGWwJKBxnLWa4xqwQ BP6lgWy3RWaFfmwxD+i3yQt0p9KkICasIB9DtS4739VUdglEZPxygRN/WbHWgtDr 1hgOiDdZRBIyuZ1gd0uokSxRvYkmrRmVHGdtNGpFljp2Xp4xcbAMVLi/ELRY5SPP 3iOMV0gsONEu5yL4xjMpxOcD2zxqJogmXFewe4eJlOCwOQbf4cqX3qF1VaalPHQD zaMaxNdDu3RJ+OoAX7ymdjfhmWOfWvobFio0n1pqBCyM3RRcPkijL2ftFQG3malX dyVg4J8DXRS0VA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm2; bh=cW2DVlZ6SG5dq7uWsaqu+k/yeEHYK MBRRWmGKdj8DaE=; b=VFL6mofot1iC2kxyOs38HlknAKFfWkAly0d9XJPNsJUHk 1hLeQFzfI8jxpb+/QhpEdnfLFvnpFamXi/eFECB491YqEek0y7Eh+aeDmHxEBc0Y mjApygMgr1ZDtWbED4/+lWq04PMufENkxgwkcB7W3vPbxoKq9sEQ6ovNoLGjEmbb UiATmJ8WFBpBAk9bEpbCD/96A4z5mymEYtSl5khc5rJTz0SdbdWfIjqCAtEwZ7lJ TBjJcuaq3QryxYPXQkVwjTc0S69tdk5tM620XpP3mtZC07BG7nHldAX1QYvGHgYG vZXAv8krzh37sC1c6rVj6gc01i03lhQfTVvb7ei2w== X-ME-Sender: Received: from localhost (unknown [185.3.4.227]) by mail.messagingengine.com (Postfix) with ESMTPA id 7D72024838; Mon, 5 Mar 2018 18:21:23 -0500 (EST) From: Marius Bakke To: Ricardo Wurmus , =?utf-8?Q?G=C3=A1bor?= Boskovits Subject: Re: bug#22533: Python bytecode reproducibility In-Reply-To: <871sgz1wg0.fsf@elephly.net> References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> User-Agent: Notmuch/0.26 (https://notmuchmail.org) Emacs/25.3.1 (x86_64-pc-linux-gnu) Date: Tue, 06 Mar 2018 00:21:21 +0100 Message-ID: <87h8pu153i.fsf@fastmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 22533 Cc: 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Ricardo Wurmus writes: > I have applied this patch locally: > > diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm > index 5f701701a..0d1ecc3c6 100644 > --- a/gnu/packages/python.scm > +++ b/gnu/packages/python.scm > @@ -359,8 +359,42 @@ data types.") > "Lib/ctypes/test/test_win32.py" ; fails on= aarch64 > "Lib/test/test_fcntl.py")) ; fails on aarc= h64 > #t)))) > - (arguments (substitute-keyword-arguments (package-arguments python-2) > - ((#:tests? _) #t))) > + (arguments > + (substitute-keyword-arguments (package-arguments python-2) > + ((#:tests? _) #t) > + ((#:phases phases) > + `(modify-phases ,phases > + (add-after 'unpack 'patch-timestamp-for-pyc-files > + (lambda _ > + ;; We set DETERMINISTIC_BUILD to only override the mtime = when > + ;; building with Guix, lest we break auto-compilation in > + ;; environments. > + (setenv "DETERMINISTIC_BUILD" "1") > + (substitute* "Lib/py_compile.py" > + (("source_stats\\['mtime'\\]") > + "(1 if 'DETERMINISTIC_BUILD' in os.environ else source= _stats['mtime'])")) > + > + ;; Use deterministic hashes for strings, bytes, and datet= ime > + ;; objects. > + (setenv "PYTHONHASHSEED" "0") > + > + ;; Reset mtime when validating bytecode header. > + (substitute* "Lib/importlib/_bootstrap_external.py" > + (("source_mtime =3D int\\(source_stats\\['mtime'\\]\\)") > + "source_mtime =3D 1")) > + #t)) > + (add-after 'unpack 'disable-timestamp-tests > + (lambda _ > + (substitute* "Lib/test/test_importlib/source/test_file_lo= ader.py" > + (("test_bad_marshal") > + "disable_test_bad_marshal") > + (("test_no_marshal") > + "disable_test_no_marshal") > + (("test_non_code_marshal") > + "disable_test_non_code_marshal")) > + #t)) > + (add-before 'check 'allow-non-deterministic-compilation > + (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t)))))) > (native-search-paths > (list (search-path-specification > (variable "PYTHONPATH") > > It allows me to build python-six and python-sip reproducibly. It does > not fix problems with Python 2, and I haven=E2=80=99t yet tested if it ca= uses > any new problems. > > It=E2=80=99s a little worrying that I had to disable three more tests tha= t I > think shouldn=E2=80=99t have failed. Woow, nice work! I can't tell what's going on with the tests, they do some bytecode manipulation stuff. Maybe it does not expect the low timestamp somehow? https://github.com/python/cpython/blob/374c6e178a7599aae46c857b17c6c8bc19df= e4c2/Lib/test/test_importlib/source/test_file_loader.py#L457-L484 I guess we'll do at least one 'core-updates' before 3.7 is released, so it makes sense to include this. It should also give us some experience that might be relevant for 2.7, since it probably won't get the upstream reproducibility patch that relies on 3.7 features. The only remark I have is: is introducing a new variable necessary? SOURCE_DATE_EPOCH implies that the user wants a deterministic build; the upstream patch doesn't actually honor it outside of making the hashing method deterministic. So, I think it might be enough to just test for SOURCE_DATE_EPOCH instead of DETERMINISTIC_BUILD. The former is also already set in the build environment. However, I just noticed that you unset DETERMINISTIC_BUILD before the 'check' phase. Did it break more things? I suppose we'll have to set PYTHONHASHSEED somewhere in python-build-system as well. Did you check if that makes a difference for numpy? Perhaps it's enough to set it if we add an auto-compilation step? --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCgAdFiEEu7At3yzq9qgNHeZDoqBt8qM6VPoFAlqd0PEACgkQoqBt8qM6 VPrDLwf7BrDvvnHtKJms7+uZjPg3VNiBNqXjdLm3+Ic8hXcD+T/rUnPbTDjQfKWS WlsmaUsDa8W5Xs6nTOaalEw2ifwrB0pqGpEZGOnHsGuldDRgt4AMZmKvojt9A4Bx kjuz06y9i1R1QQel04rQRpFoiC2D9MYgktJczpxeZJyp/wzbhw6BDBtEdCq7+Fap crIe2CdIepnHlDu44imBawPu59YHLfzXKA2GdwLS3zjfns6ZLaigQTqSNqqAYxML mpiwDsReZV/OmAa+xhStZMur2LxBIMSkh+PCm2xgW0H8ZaPdRV69bWmqopRynyLN uzzKniezMl+HM2SrVuljKa1RMMbfFw== =vpXg -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 06 08:29:14 2018 Received: (at 22533) by debbugs.gnu.org; 6 Mar 2018 13:29:14 +0000 Received: from localhost ([127.0.0.1]:46919 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1etCeY-0006jO-4W for submit@debbugs.gnu.org; Tue, 06 Mar 2018 08:29:14 -0500 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21111) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1etCeW-0006jF-6M for 22533@debbugs.gnu.org; Tue, 06 Mar 2018 08:29:12 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520342932; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; l=1070; bh=v66ZWnvWbKyG4FZccvwK5Il8gg4bgyP+Gir93W6aVHM=; b=byP5w1sn2X8M+Nj4j42xiuT8Jmi9itCRVqY9SG74jV+WdKn0//PS8mWHFLnIKCVJ rW1y2BhDniyEG+P9QIrm73rvQPM2RIhFw9Mwp/FkUurrx4vq1Epsx5CjuiuGeXq3pTU cZR0OF9vn1CVpy2epZ28s38y/l0J291IXT2wlDq4= Received: from localhost (141.80.247.222 [141.80.247.222]) by mx.zohomail.com with SMTPS id 152034293253063.76400782171072; Tue, 6 Mar 2018 05:28:52 -0800 (PST) References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87h8pu153i.fsf@fastmail.com> User-agent: mu4e 1.0; emacs 25.3.1 From: Ricardo Wurmus To: Marius Bakke Subject: Re: bug#22533: Python bytecode reproducibility In-reply-to: <87h8pu153i.fsf@fastmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Tue, 06 Mar 2018 14:28:49 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID: <87muzlib8u.fsf@elephly.net> X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: =?utf-8?Q?G=C3=A1bor?= Boskovits , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Marius Bakke writes: > The only remark I have is: is introducing a new variable necessary? > SOURCE_DATE_EPOCH implies that the user wants a deterministic build; > the upstream patch doesn't actually honor it outside of making the > hashing method deterministic. So, I think it might be enough to just > test for SOURCE_DATE_EPOCH instead of DETERMINISTIC_BUILD. The former > is also already set in the build environment. > However, I just noticed that you unset DETERMINISTIC_BUILD before the > 'check' phase. Did it break more things? Yes, it broke a bunch of tests that are all about recompiling files when they are considered stale. > I suppose we'll have to set PYTHONHASHSEED somewhere in > python-build-system as well. Did you check if that makes a difference > for numpy? Perhaps it's enough to set it if we add an auto-compilation > step? Right, I=E2=80=99m going to test this with numpy now. Thanks for the hint! --=20 Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 06 09:43:37 2018 Received: (at 22533) by debbugs.gnu.org; 6 Mar 2018 14:43:37 +0000 Received: from localhost ([127.0.0.1]:46953 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1etDoX-0001yw-7C for submit@debbugs.gnu.org; Tue, 06 Mar 2018 09:43:37 -0500 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21056) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1etDoV-0001yo-F3 for 22533@debbugs.gnu.org; Tue, 06 Mar 2018 09:43:36 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1520347394; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID; l=756; bh=gvrXlJH7v9grkQZFy6dwUpjOZZlyUM6p8LDM3YbmYbw=; b=aZfHRggoP5qCWvOZuEifJ2AEOtCja18gw4WpEY1qBud8S9vbDI+36ArSFJbggsIk vXZscaEP55wHrT9WTCFks7e3flMt5I+qBgCqIidWDKDurk6K6kmJcGf0330AzA36ora 62LMh57EeFeUBjlBqKI4LYHsja5/OXUFNBGWmmpk= Received: from localhost (141.80.247.222 [141.80.247.222]) by mx.zohomail.com with SMTPS id 1520347394133802.4637948601691; Tue, 6 Mar 2018 06:43:14 -0800 (PST) References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87h8pu153i.fsf@fastmail.com> <87muzlib8u.fsf@elephly.net> User-agent: mu4e 1.0; emacs 25.3.1 From: Ricardo Wurmus To: Marius Bakke Subject: Re: bug#22533: Python bytecode reproducibility In-reply-to: <87muzlib8u.fsf@elephly.net> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Tue, 06 Mar 2018 15:43:11 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Message-ID: <87ina9i7sw.fsf@elephly.net> X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: =?utf-8?Q?G=C3=A1bor?= Boskovits , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Ricardo Wurmus writes: > Marius Bakke writes: > >> I suppose we'll have to set PYTHONHASHSEED somewhere in >> python-build-system as well. Did you check if that makes a difference >> for numpy? Perhaps it's enough to set it if we add an auto-compilation >> step? > > Right, I=E2=80=99m going to test this with numpy now. Thanks for the hin= t! It did help with one file, which is now built reproducibly, namely lib/python3.6/site-packages/numpy/testing/nose_tools/__pycache__/utils.cp= ython-36.pyc This leaves five files in numpy that shouldn=E2=80=99t be but unfortunately= are different. -- Ricardo GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC https://elephly.net From debbugs-submit-bounces@debbugs.gnu.org Tue Mar 06 09:57:11 2018 Received: (at 22533) by debbugs.gnu.org; 6 Mar 2018 14:57:11 +0000 Received: from localhost ([127.0.0.1]:47852 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1etE1e-0002WL-Mz for submit@debbugs.gnu.org; Tue, 06 Mar 2018 09:57:10 -0500 Received: from mail-io0-f170.google.com ([209.85.223.170]:38873) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1etE1d-0002W7-2S for 22533@debbugs.gnu.org; Tue, 06 Mar 2018 09:57:09 -0500 Received: by mail-io0-f170.google.com with SMTP id g21so22321177ioj.5 for <22533@debbugs.gnu.org>; Tue, 06 Mar 2018 06:57:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=kZGw1z2xO281mLUXadEKwJZhPrIRLsn1F5fAawgUpPo=; b=ouYW6is7EaGhk/Mhn6I9GDd0GMGlw+n/O6ceyl1GynO2207Stvr0TK33EDkINBGgpY 6OUYvLHSQ9tjuJmSdICW2ZQHhs5cZ1jM30Ubq4+TipRXbrauM6t5rXh2XuRbDlXEShkc rFhyX9Hfa5//B/zquWjCaDa8aQmQMJ/Cgf8cy8Pyae2yHPiAkMYA3MZ8Vd62Yf02fZIV QMlKdXZefp6Pt2CheTSJLqFq/kstxEQP4kjmIS+IWCkD9ACLaO1wHyF9DvAW5k6I2qYH 0udEXLZOJGanl8OCcry9noNvo6t7NouUqAjg1gPzvRhgoBSGltStl7JZgPRqqnWRp6H9 9GMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=kZGw1z2xO281mLUXadEKwJZhPrIRLsn1F5fAawgUpPo=; b=Cr0sr0qqjiWEaIF+Quph7uS7lrvOo+iRQb9EAZ+AW8OwyaWuaHOEwfw3SrJIloWe9A gYctQiCqB/YMUv2lwzHML/00q1Fso6q3YR0zOSMbJZYVe85t3zkmGzEoJiUmCoT8SF23 o1r/nc4I4ckFweGds++mcbGVAz8PfmPLutApbA0n4Ir7HD0/hOJiI2kLBLy5ZW9p4YcK SytcwBBwdiuEor3uR8kwIonq+a2EUFhRpxL+ISQH9lWZfUV6alCKbe5OvuslVzmUiOeM JDpFi3R9WSYwegQjj4/HaVckBCf3WKPafW9j60OIPr0Veg5PdGBlOgdsRQbO7coR5JwI 0xtg== X-Gm-Message-State: APf1xPAHqxUOD7X6NR3eKChLfxzOn7iTrEvKDfaD/eOCI/MYxpb2DXWJ h7motXx5AMnZiREdCkvYr+CJTJ2PM2c63s8gKw== X-Google-Smtp-Source: AG47ELumAfpHsuSfWsadJWUzlQVk23JbOdurWrtifRNgrjZf8KnH80Og4+1Sqbe/+CBTIP1ao7sEmoA+/406A58Kmdc= X-Received: by 10.107.53.231 with SMTP id k100mr23601943ioo.292.1520348222937; Tue, 06 Mar 2018 06:57:02 -0800 (PST) MIME-Version: 1.0 Received: by 10.79.157.15 with HTTP; Tue, 6 Mar 2018 06:57:02 -0800 (PST) In-Reply-To: <87ina9i7sw.fsf@elephly.net> References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87h8pu153i.fsf@fastmail.com> <87muzlib8u.fsf@elephly.net> <87ina9i7sw.fsf@elephly.net> From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= Date: Tue, 6 Mar 2018 15:57:02 +0100 Message-ID: Subject: Re: bug#22533: Python bytecode reproducibility To: Ricardo Wurmus Content-Type: multipart/alternative; boundary="001a1144a158c35d230566bfa7af" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: Marius Bakke , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) --001a1144a158c35d230566bfa7af Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable 2018-03-06 15:43 GMT+01:00 Ricardo Wurmus : > > Ricardo Wurmus writes: > > > Marius Bakke writes: > > > >> I suppose we'll have to set PYTHONHASHSEED somewhere in > >> python-build-system as well. Did you check if that makes a difference > >> for numpy? Perhaps it's enough to set it if we add an auto-compilatio= n > >> step? > > > > Right, I=E2=80=99m going to test this with numpy now. Thanks for the h= int! > > It did help with one file, which is now built reproducibly, namely > > lib/python3.6/site-packages/numpy/testing/nose_tools/__ > pycache__/utils.cpython-36.pyc > > This leaves five files in numpy that shouldn=E2=80=99t be but unfortunate= ly are > different. > > Unfortunately backporting the upstream version is not straightforward at all. There are too many changes. I will have a look at those test failures instead. > -- > Ricardo > > GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC > https://elephly.net > > > --001a1144a158c35d230566bfa7af Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
2018= -03-06 15:43 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:=

Ricardo Wurmus <rekado@elephly.net= > writes:

> Marius Bakke <mbakke@fastmai= l.com> writes:
>
>> I suppose we'll have to set PYTHONHASHSEED somewhere in
>> python-build-system as well.=C2=A0 Did you check if that makes a d= ifference
>> for numpy?=C2=A0 Perhaps it's enough to set it if we add an au= to-compilation
>> step?
>
> Right, I=E2=80=99m going to test this with numpy now.=C2=A0 Thanks for= the hint!

It did help with one file, which is now built reproducibly, namely
=C2=A0 lib/python3.6/site-packages/numpy/testing/nose_tools/__pyc= ache__/utils.cpython-36.pyc

This leaves five files in numpy that shouldn=E2=80=99t be but unfortunately= are
different.

<= br>
Unfortunately backporting the upstream version is not straigh= tforward at all.
There are too many changes. I will have a look a= t those test failures instead.
=C2=A0
--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC
https:= //elephly.net



--001a1144a158c35d230566bfa7af-- From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 08 05:40:01 2018 Received: (at 22533) by debbugs.gnu.org; 8 Mar 2018 10:40:01 +0000 Received: from localhost ([127.0.0.1]:50224 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1etsxt-0005NM-0A for submit@debbugs.gnu.org; Thu, 08 Mar 2018 05:40:01 -0500 Received: from mail-io0-f179.google.com ([209.85.223.179]:34298) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1etsxq-0005N7-R6 for 22533@debbugs.gnu.org; Thu, 08 Mar 2018 05:39:59 -0500 Received: by mail-io0-f179.google.com with SMTP id e7so6571426ioj.1 for <22533@debbugs.gnu.org>; Thu, 08 Mar 2018 02:39:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=gH+oV29piaFeIxyPi0KtOVbkhHxRPe4yrBv4bE0U4Z4=; b=bHIjQMCUKo8q1LUQDHFRVm8v1/YzQqu/mhsSm9HWGPdD2A5ddk3DR2BtPSL0P5VaWS ErGcfDYart4gxpqPf+p7maHyld4Rrd1QxerW/mYjvW2e2GF+JyU5tyx86oWNnNQFTLFU 8h/Hp0EH9tGXy8YgH1WYaEdwNkoCWPbdC/HkdkqL3VrjUeSzrQ3Kz5O0WdDkHcFyKc7u kKZyD0sMTGg+oVsbgvsJrgApdZ9IbpVJrUEKQC2nBPd8qkUgcBKxLosTaxb6sLW732d4 277E4M1bFSIIg5fIvX28rHi4H7qqzKTR7cYCn0PxGQx68q47/S0Jyr/pExpldhSlgOMU oUTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=gH+oV29piaFeIxyPi0KtOVbkhHxRPe4yrBv4bE0U4Z4=; b=lAlbj9mllbVFmFPUCJCPFv14bNMpSoI+Q0LpvGG3nMKeJ61HxYluaAgv+HdxLuNv8y kYUtWDzqIkWehbRbU7PVQj7QJsTMbw75NmVmisVHj0PiZa3kCIuIJazNcaLaRptpP/O/ QgpXkqMcBjLleLFS+aAoC2a5cMvNDvgRv9131pLhbcK6kDdSskYKJLRJnIVEV/tMuP32 Gh55hSNAWxCRNOELvdTpCm0FvQ6n/9KkSHPi8JtALEOIQRwi30T3Lq1ZiHL5/CHW2Vmo bugjP/+h7yCKyJJvjfHpJ+IM9BYxJu9MqG25DV3jFPpquyB1+a7Pc3R4st+NrO7xK+wx x6EA== X-Gm-Message-State: AElRT7Hsb77rDPeL3e34GsvYEPWMKqf1yiU87bOxaJR7680Ohb45XKy2 mRegbKa66ePj59eEjG9OuVEMdEB4nxCfOdGd7Q== X-Google-Smtp-Source: AG47ELukVLtnj49nhofsDNYPLRr3u7KPT5QLkpCdPkKmZc+uTkrYGo90LiDwBYzYpoi9Wh0w45vovjQJ/PlXKxit5Bg= X-Received: by 10.107.178.206 with SMTP id b197mr32072660iof.9.1520505593104; Thu, 08 Mar 2018 02:39:53 -0800 (PST) MIME-Version: 1.0 Received: by 10.79.157.15 with HTTP; Thu, 8 Mar 2018 02:39:52 -0800 (PST) In-Reply-To: <871sgz1wg0.fsf@elephly.net> References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> From: =?UTF-8?Q?G=C3=A1bor_Boskovits?= Date: Thu, 8 Mar 2018 11:39:52 +0100 Message-ID: Subject: Re: bug#22533: Python bytecode reproducibility To: Ricardo Wurmus Content-Type: multipart/alternative; boundary="001a114c9d88c18df00566e44b12" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: Marius Bakke , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) --001a114c9d88c18df00566e44b12 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable 2018-03-04 20:18 GMT+01:00 Ricardo Wurmus : > I have applied this patch locally: > > > diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm > index 5f701701a..0d1ecc3c6 100644 > --- a/gnu/packages/python.scm > +++ b/gnu/packages/python.scm > @@ -359,8 +359,42 @@ data types.") > "Lib/ctypes/test/test_win32.py" ; fails on > aarch64 > "Lib/test/test_fcntl.py")) ; fails on > aarch64 > #t)))) > - (arguments (substitute-keyword-arguments (package-arguments python-2= ) > - ((#:tests? _) #t))) > + (arguments > + (substitute-keyword-arguments (package-arguments python-2) > + ((#:tests? _) #t) > + ((#:phases phases) > + `(modify-phases ,phases > + (add-after 'unpack 'patch-timestamp-for-pyc-files > + (lambda _ > + ;; We set DETERMINISTIC_BUILD to only override the mtime > when > + ;; building with Guix, lest we break auto-compilation in > + ;; environments. > + (setenv "DETERMINISTIC_BUILD" "1") > + (substitute* "Lib/py_compile.py" > + (("source_stats\\['mtime'\\]") > + "(1 if 'DETERMINISTIC_BUILD' in os.environ else > source_stats['mtime'])")) > + > + ;; Use deterministic hashes for strings, bytes, and > datetime > + ;; objects. > + (setenv "PYTHONHASHSEED" "0") > + > + ;; Reset mtime when validating bytecode header. > + (substitute* "Lib/importlib/_bootstrap_external.py" > + (("source_mtime =3D int\\(source_stats\\['mtime'\\]\\)"= ) > + "source_mtime =3D 1")) > + #t)) > + (add-after 'unpack 'disable-timestamp-tests > + (lambda _ > + (substitute* "Lib/test/test_importlib/ > source/test_file_loader.py" > + (("test_bad_marshal") > + "disable_test_bad_marshal") > + (("test_no_marshal") > + "disable_test_no_marshal") > + (("test_non_code_marshal") > + "disable_test_non_code_marshal")) > + #t)) > + (add-before 'check 'allow-non-deterministic-compilation > + (lambda _ (unsetenv "DETERMINISTIC_BUILD") #t)))))) > (native-search-paths > (list (search-path-specification > (variable "PYTHONPATH") > > > It allows me to build python-six and python-sip reproducibly. It does > not fix problems with Python 2, and I haven=E2=80=99t yet tested if it ca= uses > any new problems. > > It=E2=80=99s a little worrying that I had to disable three more tests tha= t I > think shouldn=E2=80=99t have failed. > > Ok, I've checked the test issue again. If we change the _bootstrap_external.py substitution to: "source_mtime =3D 1 if 'DETERMINISTIC_BUILD' in _os.environ else int(source_stats['mtime'])" the test do not fail any more. WDYT? > What do you think? > > -- > Ricardo > > GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC > https://elephly.net > > --001a114c9d88c18df00566e44b12 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
2018= -03-04 20:18 GMT+01:00 Ricardo Wurmus <rekado@elephly.net>:=
I have applied this patch locally:


diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 5f701701a..0d1ecc3c6 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -359,8 +359,42 @@ data types.")
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"Lib/ctypes/test/test_win32.py" ; fails on aarch64
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0"Lib/test/test_fcntl.py")) = ; fails on aarch64
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0#t))))=
-=C2=A0 =C2=A0 (arguments (substitute-keyword-arguments (package-arguments = python-2)
-=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0((#:tests? _= ) #t)))
+=C2=A0 =C2=A0 (arguments
+=C2=A0 =C2=A0 =C2=A0(substitute-keyword-arguments (package-arguments pytho= n-2)
+=C2=A0 =C2=A0 =C2=A0 =C2=A0((#:tests? _) #t)
+=C2=A0 =C2=A0 =C2=A0 =C2=A0((#:phases phases)
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 `(modify-phases ,phases
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(add-after 'unpack 'patch= -timestamp-for-pyc-files
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(lambda _
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; We set DETERMINI= STIC_BUILD to only override the mtime when
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; building with Gu= ix, lest we break auto-compilation in
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; environments. +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(setenv "DETER= MINISTIC_BUILD" "1")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(substitute* "= Lib/py_compile.py"
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(("sour= ce_stats\\['mtime'\\]")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "(1 if= 'DETERMINISTIC_BUILD' in os.environ else source_stats['mtime&#= 39;])"))
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; Use deterministi= c hashes for strings, bytes, and datetime
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; objects.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(setenv "PYTHO= NHASHSEED" "0")
+
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0;; Reset mtime when= validating bytecode header.
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(substitute* "= Lib/importlib/_bootstrap_external.py"
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(("sour= ce_mtime =3D int\\(source_stats\\['mtime'\\]\\)")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "sourc= e_mtime =3D 1"))
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0#t))
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(add-after 'unpack 'disab= le-timestamp-tests
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(lambda _
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(substitute* "= Lib/test/test_importlib/source/test_file_loader.py"
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(("test= _bad_marshal")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "disab= le_test_bad_marshal")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(("test= _no_marshal")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "disab= le_test_no_marshal")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(("test= _non_code_marshal")
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "disab= le_test_non_code_marshal"))
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0#t))
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(add-before 'check 'allow= -non-deterministic-compilation
+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(lambda _ (unsetenv "= DETERMINISTIC_BUILD") #t))))))
=C2=A0 =C2=A0 =C2=A0(native-search-paths
=C2=A0 =C2=A0 =C2=A0 (list (search-path-specification
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(variable "PYTHONPATH&= quot;)


It allows me to build python-six and python-sip reproducibly.=C2=A0 It does=
not fix problems with Python 2, and I haven=E2=80=99t yet tested if it caus= es
any new problems.

It=E2=80=99s a little worrying that I had to disable three more tests that = I
think shouldn=E2=80=99t have failed.


Ok, I've checked the test issue ag= ain. If we change the _bootstrap_external.py
substitution to:
"source_mtime =3D 1 if 'DETERMINISTIC_BUILD' in _os.env= iron else int(source_stats['mtime'])"
the test do no= t fail any more. WDYT?

=C2=A0
What do you think?

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6=C2=A0 2150 197A 5888 235F ACAC
https:= //elephly.net


--001a114c9d88c18df00566e44b12-- From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 14 08:40:42 2019 Received: (at 22533) by debbugs.gnu.org; 14 Jan 2019 13:40:42 +0000 Received: from localhost ([127.0.0.1]:58379 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gj2Tq-0004Ef-D9 for submit@debbugs.gnu.org; Mon, 14 Jan 2019 08:40:42 -0500 Received: from sender-of-o53.zoho.com ([135.84.80.218]:21812) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gj2To-0004EW-3L for 22533@debbugs.gnu.org; Mon, 14 Jan 2019 08:40:41 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1547473223; cv=none; d=zoho.com; s=zohoarc; b=eJUUpakaAFQWDaUGP/Hv9G39tPxReyPqKvbxdqwc8ZyyXSTbt530aJ+1N4bPqd6F8MUCE84ACApsnc+kbIO/v3adjMTqc9lXAlYFVCID9jTrjo4r870YZF3ZG/witZ46UN9KVdbHTlD03ddvj7MfKzz0D6FVFK9JnowUiXZLJDA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1547473223; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results; bh=/18kGDRKl2kk9R5xc4f1l7+/gcZOeNDiKiIEyIqfOhY=; b=TDcZDUc6gFABMNRAxLnF/WpPE6g/MqLDvo1lUTxTyGDqzrXP/AE52WEGSTEFkgE3pI3jBjLu8pQVUxl/GXaaoBUNuJIStPMuMXrwbXnnYiKArPA6FV5OZvq8O4pry95juJxWBtT+GmqRtfvi/kiTP4m9IXSMX3trJQ8XcsN3Cio= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1547473223; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; l=329; bh=/18kGDRKl2kk9R5xc4f1l7+/gcZOeNDiKiIEyIqfOhY=; b=FYIddyFw8N5H8E1gfCf9DxEHJVpqMk4wUbd4GumfGE60UkzmQTBAUtSVbHuuea62 CPPx+AVMm155fMwogz0tWGqBBr6khQP6yg6bnIIuUd6KNH5LcssoD6YIp9NXeGaC6hf Y9XWd9uhbCZDozxGLGuhcD+AOZ7mmvECFL3q+71M= Received: from localhost (141.80.247.192 [141.80.247.192]) by mx.zohomail.com with SMTPS id 1547473222247455.17296416035083; Mon, 14 Jan 2019 05:40:22 -0800 (PST) References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> User-agent: mu4e 1.0; emacs 26.1 From: Ricardo Wurmus To: =?utf-8?Q?G=C3=A1bor?= Boskovits Subject: Re: bug#22533: Python bytecode reproducibility In-reply-to: X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 14 Jan 2019 14:40:18 +0100 Message-ID: <87sgxvtl19.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 22533 Cc: Marius Bakke , 22533@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Now that we=E2=80=99re using Python 3.7 and this version supports hash-base= d pyc files, is this still an issue? Do we need to do anything to enable hash-based pyc compilation? See: https://docs.python.org/3/whatsnew/3.7.html#pep-552-hash-based-pyc-files https://www.python.org/dev/peps/pep-0552/ --=20 Ricardo From debbugs-submit-bounces@debbugs.gnu.org Sun Feb 03 16:22:54 2019 Received: (at 22533-done) by debbugs.gnu.org; 3 Feb 2019 21:22:54 +0000 Received: from localhost ([127.0.0.1]:57821 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gqPE5-00056c-Nr for submit@debbugs.gnu.org; Sun, 03 Feb 2019 16:22:53 -0500 Received: from sender-of-o51.zoho.com ([135.84.80.216]:21111) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gqPE3-00056S-4H for 22533-done@debbugs.gnu.org; Sun, 03 Feb 2019 16:22:52 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1549228948; cv=none; d=zoho.com; s=zohoarc; b=ICO3bVVlfJKS45PNXiI5e4N2bDj8to7VvzfEF9Isb2r2G+sEZZlRZVmEgKS+hSGk7L4wfHE5sPU0XvOAQ5o0B1NSVrFnT2Z6v6iQOFDpmwYstsMCLnc/K/e7vkm+/xjcs6Vvfkb1wnBizwQ8/bdfPh9mGnFot/tfH5sb0kqV0LU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1549228948; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To:ARC-Authentication-Results; bh=Qv5/AGQCxkAGV7x/YAbL0JwXg9Hggy5+bH4tvyKM7Ts=; b=l76/o0yUOZQjNhD6zZByF24yy0fwF88wo+ISLKjZKr4TNbsLRijUNWg90JTEtbgq4Kde42TNlp1mRJD9lGLjkGFNEKFuXpWZ9LGjczs6c6dDx17OT7IefK98eg8FzOQj1KZDEUHuMgGUTdvG5uuVV6XdyuThhb522FaK8VXXE+0= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1549228948; s=zoho; d=elephly.net; i=rekado@elephly.net; h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; l=581; bh=Qv5/AGQCxkAGV7x/YAbL0JwXg9Hggy5+bH4tvyKM7Ts=; b=hAzYtYosdon/CXH1P5CIGZpL3Va9h3GFAx3ZhG5z3L4bx/Nl4Un7GwaRU1Dsdop/ Cd18fsrsGLgPbyX8Ag+odsw+oIBzVJRs9UgK4ifYrHx59ySaCHCdu9FueothJHh7d4S CYQavK0xZwQ6wg6tkRQ3ICIWVupmmRGaUPUFJ8xc= Received: from localhost (252.139-78-194.adsl-static.isp.belgacom.be [194.78.139.252]) by mx.zohomail.com with SMTPS id 1549228946135147.2639457165526; Sun, 3 Feb 2019 13:22:26 -0800 (PST) References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87sgxvtl19.fsf@elephly.net> User-agent: mu4e 1.0; emacs 26.1 From: Ricardo Wurmus To: =?utf-8?Q?G=C3=A1bor?= Boskovits Subject: Re: bug#22533: Python bytecode reproducibility In-reply-to: <87sgxvtl19.fsf@elephly.net> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Sun, 03 Feb 2019 22:22:23 +0100 Message-ID: <87va20wos0.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 22533-done Cc: Marius Bakke , 22533-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ricardo Wurmus writes: > Now that we=E2=80=99re using Python 3.7 and this version supports hash-ba= sed pyc > files, is this still an issue? Do we need to do anything to enable > hash-based pyc compilation? > > See: > https://docs.python.org/3/whatsnew/3.7.html#pep-552-hash-based-pyc-files > https://www.python.org/dev/peps/pep-0552/ It looks like this is no longer a problem. I built borg just now and the pyc files are reproducible. (The man pages include a date stamp, though, which I=E2=80=99m trying to pa= tch now.) -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Mon Feb 04 17:39:25 2019 Received: (at 22533) by debbugs.gnu.org; 4 Feb 2019 22:39:25 +0000 Received: from localhost ([127.0.0.1]:59835 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gqmth-0002H0-Ez for submit@debbugs.gnu.org; Mon, 04 Feb 2019 17:39:25 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:41818) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gqmtf-0002Gs-Nh for 22533@debbugs.gnu.org; Mon, 04 Feb 2019 17:39:24 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 2A87DBAD2; Mon, 4 Feb 2019 23:39:23 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id V6xIuOkh4eYD; Mon, 4 Feb 2019 23:39:22 +0100 (CET) Received: from ribbon (unknown [IPv6:2a01:e0a:1d:7270:af76:b9b:ca24:c465]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 32F66B80B; Mon, 4 Feb 2019 23:39:22 +0100 (CET) From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 22533@debbugs.gnu.org Subject: Re: bug#22533: Python bytecode reproducibility References: <20160202051544.GA11744@jasmine> <87bmqfu44s.fsf@fastmail.com> <87606c23bq.fsf@elephly.net> <874llw101c.fsf@elephly.net> <871sgz1wg0.fsf@elephly.net> <87sgxvtl19.fsf@elephly.net> <87va20wos0.fsf@elephly.net> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 16 =?utf-8?Q?Pluvi=C3=B4se?= an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 04 Feb 2019 23:39:21 +0100 In-Reply-To: <87va20wos0.fsf@elephly.net> (Ricardo Wurmus's message of "Sun, 03 Feb 2019 22:22:23 +0100") Message-ID: <87h8djmb52.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 22533 Cc: rekado@elephly.net, leo@famulari.name X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Ricardo Wurmus skribis: > Ricardo Wurmus writes: > >> Now that we=E2=80=99re using Python 3.7 and this version supports hash-b= ased pyc >> files, is this still an issue? Do we need to do anything to enable >> hash-based pyc compilation? >> >> See: >> https://docs.python.org/3/whatsnew/3.7.html#pep-552-hash-based-pyc-fil= es >> https://www.python.org/dev/peps/pep-0552/ > > It looks like this is no longer a problem. I built borg just now and > the pyc files are reproducible. Yay! \o/ Ludo'. From unknown Tue Jun 17 20:12:39 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Tue, 05 Mar 2019 12:24:06 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator