Subject: bug#38043: Incorrect Python byte-compiling for Python 3 and PyPy3

From: Michał Górny <[email-masked]>
Date: Sun, 03 Nov 2019 11:45:39 +0100

Hello,

I've noticed that the logic in py-compile is built on assumptions from
Python 2 and does not fit Python 3 well.  Notably, there are two or
three bugs here:

1. .opt-2 (-OO) level is not compiled for py3.5+.

2. .opt-1 (-O) and .opt-2 (-OO) are wrongly skipped for PyPy3.

Firstly, the code in py-compile byte-compiles in two steps: without '-O' 
and with '-O'.  This is correct for Python <3.5 since '-O' and '-OO'
share the same file (.pyo).  However, since Python 3.5 they are written
into separate '.opt-1.pyc' and '.opt-2.pyc' files and therefore can
coexist.

It is therefore necessary to perform another byte-compilation step with
'-OO', so that all levels of optimization are installed and the module
can be correctly loaded from cache independently of Python options used.

Secondly, the code in '-O' block skips PyPy based on the existence of
sys.pypy_translation_info.  This is correct for PyPy2 that doesn't write
optimized caches.  However, PyPy3 (in particular versions based
on Python 3.5 and newer, I'm not sure about the older versions) do use
the same split naming scheme as CPython.

Therefore, the check here needs to be modified to apply only to Python 2
versions of PyPy.

To reproduce:

touch mytest.py
cat > configure.ac <<EOF
AC_INIT([test], [0])
AM_INIT_AUTOMAKE([foreign])
AM_PATH_PYTHON
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
EOF
cat > Makefile.am <<EOF
pyexec_PYTHON = mytest.py
EOF
autoreconf -i
./configure PYTHON=...
make DESTDIR=... install

With Python 3.5+, the following files are installed:
/tmp/z1/usr/local/lib64/python3.5/site-packages/mytest.py
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-35.opt-1.pyc
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-35.pyc

What should additionally be installed is:
/tmp/z1/usr/local/lib64/python3.5/site-packages/__pycache__/mytest.cpython-35.opt-2.pyc

With PyPy3.5+, the following files are installed:
/tmp/z2/usr/local/lib/python3.6/site-packages/mytest.py
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.pyc

What should additionally be installed is:
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.opt-1.pyc
/tmp/z2/usr/local/lib/python3.6/site-packages/__pycache__/mytest.pypy3-72.opt-2.pyc

Please let me know if you need any more details.

-- 
Best regards,
Michał Górny