GNU bug report logs - #38043
Incorrect Python byte-compiling for Python 3.5+ and PyPy3

Previous Next

Full log

View this message in rfc822 format

From: Michał Górny <mgorny <at> gentoo.org>
To: 38043 <at> debbugs.gnu.org
Subject: bug#38043: Incorrect Python byte-compiling for Python 3 and PyPy3
Date: Sun, 03 Nov 2019 11:45:39 +0100

[Message part 1 (text/plain, inline)]

Hello,

I've noticed that the logic in py-compile is built on assumptions from
Python 2 and does not fit Python 3 well.  Notably, there are two or
three bugs here:

1. .opt-2 (-OO) level is not compiled for py3.5+.

2. .opt-1 (-O) and .opt-2 (-OO) are wrongly skipped for PyPy3.


Firstly, the code in py-compile byte-compiles in two steps: without '-O' 
and with '-O'.  This is correct for Python <3.5 since '-O' and '-OO'
share the same file (.pyo).  However, since Python 3.5 they are written
into separate '.opt-1.pyc' and '.opt-2.pyc' files and therefore can
coexist.

It is therefore necessary to perform another byte-compilation step with
'-OO', so that all levels of optimization are installed and the module
can be correctly loaded from cache independently of Python options used.


Secondly, the code in '-O' block skips PyPy based on the existence of
sys.pypy_translation_info.  This is correct for PyPy2 that doesn't write
optimized caches.  However, PyPy3 (in particular versions based
on Python 3.5 and newer, I'm not sure about the older versions) do use
the same split naming scheme as CPython.

Therefore, the check here needs to be modified to apply only to Python 2
versions of PyPy.


To reproduce:

touch mytest.py
cat > configure.ac <<EOF
AC_INIT([test], [0])
AM_INIT_AUTOMAKE([foreign])
AM_PATH_PYTHON
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
EOF
cat > Makefile.am <<EOF
pyexec_PYTHON = mytest.py
EOF
autoreconf -i
./configure PYTHON=...
make DESTDIR=... install


With Python 3.5+, the following files are installed:
/tmp/z1/usr/local/lib64/python3.5/site-packages/mytest.py
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-35.opt-1.pyc
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-35.pyc

What should additionally be installed is:
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-35.opt-2.pyc


With PyPy3.5+, the following files are installed:
/tmp/z2/usr/local/lib/python3.6/site-packages/mytest.py
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.pyc

What should additionally be installed is:
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.opt-1.pyc
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.opt-2.pyc


Please let me know if you need any more details.

-- 
Best regards,
Michał Górny

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.