GNU bug report logs - #71672
[PATCH] python: Prefer python3 over python

Previous Next

To reply to this bug, email your comments to 71672 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Collin Funk <collin.funk1 <at> gmail.com>
To: automake-patches <at> gnu.org
Subject: [PATCH] python: Prefer python3 over python
Date: Thu, 20 Jun 2024 02:50:16 -0700

[Message part 1 (text/plain, inline)]

Hi,

Shouldn't AM_PATH_PYTHON check for the python3 command before python?
The full recommendation of how these should be setup is in PEP 394
[1].

On modern systems I don't think it should matter. On Fedora 40:

    $ command -v python
    /usr/bin/python
    $ command -v python3
    /usr/bin/python3
    $ python --version
    Python 3.12.3
    $ python3 --version
    Python 3.12.3

The difference mostly exists on older machines. On CentOS 7.9 for
example:

    $ command -v python
    /usr/bin/python
    $ command -v python3
    /usr/bin/python3
    $ python --version
    Python 2.7.5
    $ python3 --version
    Python 3.6.8

I've attached a proposed path. I don't see a situation where a
python3.* is installed without a python3 link. Therefore checking for
python3 and fallingback on python seems reasonable to me.

Collin

[1] https://peps.python.org/pep-0394/

[0001-python-Prefer-python3-over-python.patch (text/x-patch, attachment)]

Message #8 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: "Zack Weinberg" <zack <at> owlfolio.org>
To: "Collin Funk" <collin.funk1 <at> gmail.com>, 71672 <at> debbugs.gnu.org
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Thu, 20 Jun 2024 09:33:14 -0400

On Thu, Jun 20, 2024, at 5:50 AM, Collin Funk wrote:
> Shouldn't AM_PATH_PYTHON check for the python3 command before python?

Yes it should.

> The full recommendation of how these should be setup is in PEP 394

Note that PEP 394 is dangerously incorrect on one point: a system where
/usr/bin/python and/or the bare 'python' command invoke Python 3 is a
misconfigured system, because there's -- yes, even now -- a huge volume
of Python 2-only scripts out there in the wild that will malfunction if
run under a Python 3 interpreter.  In the worst case "malfunction" means
"cause silent data corruption."

This is mostly not Automake's problem, but -- assuming people who use
AM_PATH_PYTHON with current versions of Automake *want* the Python 3
interpreter, for the most part -- it might be an argument for *not*
probing unsuffixed 'python', only 'python3' and 'python3.*'.

zw

Message #11 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: zack <at> owlfolio.org
Cc: 71672 <at> debbugs.gnu.org, collin.funk1 <at> gmail.com
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Thu, 20 Jun 2024 15:28:33 -0600

Collin, Zack - 

    > Shouldn't AM_PATH_PYTHON check for the python3 command before python?

    Yes it should.

That seems like a huge change to make. I understand the reasoning, given
the world of Python, but the goal of "don't break existing projects"
argues against it.

    Note that PEP 394 is dangerously incorrect on one point: a system
    where /usr/bin/python and/or the bare 'python' command invoke
    Python3 is a misconfigured system

Nevertheless, that is the reality on Red Hat(-derived) 9 systems.
I don't know about any others.

    a huge volume of Python 2-only scripts out there in the wild 

Right.

    assuming people who use AM_PATH_PYTHON with current versions of
    Automake *want* the Python 3 interpreter, for the most part

That feels like an awfully big assumption to me. Ever since its inception,
Automake has looked for "python" first. We will surely be breaking many
existing projects if we change that default -- all the ones that are
still using Python 2, for starters.

Python created this insoluble problem. As far as I can see, we can't fix
it. People will just have to set PYTHON if they need to a force a
version different from the system, seems to me.

I can imagine preferring "python3" to "python" in the future, but this
seems like too big of a change to make at the very end (I hope) of the
pretest cycle. I think it would be better to make it well in advance of
the next release and try to garner feedback on the inevitable problems
that will result. --thanks, karl.

Message #14 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: Collin Funk <collin.funk1 <at> gmail.com>
To: Karl Berry <karl <at> freefriends.org>, zack <at> owlfolio.org
Cc: 71672 <at> debbugs.gnu.org
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Thu, 20 Jun 2024 17:50:54 -0700

Karl Berry wrote:
> I can imagine preferring "python3" to "python" in the future, but this
> seems like too big of a change to make at the very end (I hope) of the
> pretest cycle. I think it would be better to make it well in advance of
> the next release and try to garner feedback on the inevitable problems
> that will result. --thanks, karl.

Sounds reasonable to me. It was just something I stumbled upon last
night. I thought the intention was to prioritize python 3.X which
would cause problems using "python" first. We can just leave this bug
open or something until there is feedback and no release rush.

Collin

Message #17 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: collin.funk1 <at> gmail.com
Cc: zack <at> owlfolio.org, 71672 <at> debbugs.gnu.org
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Fri, 21 Jun 2024 16:12:34 -0600

    I thought the intention was to prioritize python 3.X which
    would cause problems using "python" first.

Yes ... but I feel strongly that we should try very hard not to break
existing configure setups.

It occurs to me that we could allow maintainers to specify p3-searching
options in their configure.ac, like

AM_PYTHON_OPTS(python3-check-first python-check-omit)

(Or additional args to AM_PATH_PYTHON, whatever.)

That wouldn't have any compatibility issues, and would allow maintainers
to control what gets looked for, for packages that need to force python3.
(Other options could force looking for p2 first.)

Would that make sense? --thanks, karl.

Message #20 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: Collin Funk <collin.funk1 <at> gmail.com>
To: Karl Berry <karl <at> freefriends.org>
Cc: zack <at> owlfolio.org, 71672 <at> debbugs.gnu.org
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Fri, 21 Jun 2024 18:57:28 -0700

Hi Karl,

Karl Berry wrote:
> Yes ... but I feel strongly that we should try very hard not to break
> existing configure setups.

Reasonable goal.

> It occurs to me that we could allow maintainers to specify p3-searching
> options in their configure.ac, like
> 
> AM_PYTHON_OPTS(python3-check-first python-check-omit)
> 
> (Or additional args to AM_PATH_PYTHON, whatever.)
> 
> That wouldn't have any compatibility issues, and would allow maintainers
> to control what gets looked for, for packages that need to force python3.
> (Other options could force looking for p2 first.)
> 
> Would that make sense?

Yes, I think that idea would work. Or maybe AM_PYTHON3_PATH which
requires Python3 so that the interface of the original macro isn't
changed.

Do you have any ideas/opinions Zach? I don't ever use Python 2 so
maybe my ideas for the interface are poor.

Collin

Message #23 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: collin.funk1 <at> gmail.com
Cc: zack <at> owlfolio.org, 71672 <at> debbugs.gnu.org
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Sat, 22 Jun 2024 15:10:23 -0600

    Or maybe AM_PYTHON3_PATH which requires Python3 so that the
    interface of the original macro isn't changed.

My idea for having a separate options command was for the future: who
knows what incompatibilities python will create in the future. Other
options besides p3 vs p2 might be needed. Indeed, could already be
needed; I know of a couple packages that require 3.8 or later. -k

Message #26 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: "Zack Weinberg" <zack <at> owlfolio.org>
To: "Karl Berry" <karl <at> freefriends.org>, collin.funk1 <at> gmail.com
Cc: 71672 <at> debbugs.gnu.org
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Sun, 23 Jun 2024 02:11:29 +0000

Here's what I think might be useful:

Add these three macros to Autoconf proper:

# AC_PYTHON_VERSION_RANGE_IFELSE([PROG], [MIN-VERSION], [MAX-VERSION = next-major],
#    [ACTION-IF-TRUE], [ACTION-IF-FALSE])
# ---------------------------------------------------------------------------------
# Run ACTION-IF-TRUE if the Python interpreter PROG has version >= MIN-VERSION
#  and < MAX_VERSION.  If MAX_VERSION is empty, it defaults to the next
#  major version after MIN-VERSION (e.g. 2.7 -> 3, 3.6 -> 4).
# Run ACTION-IF-FALSE otherwise.

# AC_PATH_PYTHON2([MINIMUM-VERSION], [VALUE-IF-NOT-FOUND])
# --------------------------------------------------------
# Find a Python 2 interpreter.  If MINIMUM-VERSION is not empty, it must be at
# least that version.  If one is found, set the substitution variable PYTHON2
# to the full pathname of the selected interpreter.  Otherwise set PYTHON2 to
# VALUE-IF-NOT-FOUND.  Does nothing else; in particular, does not do any of the
# additional checks and substitutions done by AM_PATH_PYTHON.

This one will try bare 'python'.

# AC_PATH_PYTHON3([MINIMUM-VERSION], [VALUE-IF-NOT-FOUND])
# --------------------------------------------------------
# Find a Python 3 interpreter.  If MINIMUM-VERSION is not empty, it must be at
# least that version.  If one is found, set the substitution variable PYTHON3
# to the full pathname of the selected interpreter.  Otherwise set PYTHON3 to
# VALUE-IF-NOT-FOUND.  Does nothing else; in particular, does not do any of the
# additional checks and substitutions done by AM_PATH_PYTHON.

This one will not.

Automake's python.m4 should also define these if they are not already defined.
Then, we split AM_PATH_PYTHON into the part that sets PYTHON, which is implemented
in terms of the above macros (but still looks for both 2 and 3) and the part that
does all the rest of it.  People who want a slightly more efficient v3-only configure
script can use AC_PATH_PYTHON3 + whatever the name of the new "all the rest of it"
macro is.

I think that should disentangle things reasonably well, but we should definitely
do this early rather than late in an automake release cycle, and seek out feedback
from existing users of AM_PATH_PYTHON.  What say you?

zw

Message #29 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: Collin Funk <collin.funk1 <at> gmail.com>
To: Zack Weinberg <zack <at> owlfolio.org>, Karl Berry <karl <at> freefriends.org>
Cc: 71672 <at> debbugs.gnu.org
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Sat, 22 Jun 2024 20:27:25 -0700

Zack Weinberg wrote:
> Automake's python.m4 should also define these if they are not already defined.
> Then, we split AM_PATH_PYTHON into the part that sets PYTHON, which is implemented
> in terms of the above macros (but still looks for both 2 and 3) and the part that
> does all the rest of it.  People who want a slightly more efficient v3-only configure
> script can use AC_PATH_PYTHON3 + whatever the name of the new "all the rest of it"
> macro is.
> 
> I think that should disentangle things reasonably well, but we should definitely
> do this early rather than late in an automake release cycle, and seek out feedback
> from existing users of AM_PATH_PYTHON.  What say you?

I like this idea and am happy to help after the upcoming release.

Maybe something like:

    AM_PYTHON_SETUP([PYTHON-EXECUTABLE])

Which does all the extra $(pyexecdir), $(pkgpyexecdir), etc. given the
python executable. It should be pretty simple to move all the code
from AM_PATH_PYTHON to that separate macro since it should run on all
supported versions of Python (2 and 3).

Collin

Message #32 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: zack <at> owlfolio.org
Cc: 71672 <at> debbugs.gnu.org, collin.funk1 <at> gmail.com
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Sun, 23 Jun 2024 15:17:58 -0600

    zw> Add these three macros to Autoconf proper:
    ...
    Automake's python.m4 should also define these if they are not
    already defined. Then, we split AM_PATH_PYTHON ...

Thanks Zack. It all sounds good to me.

    definitely do this early rather than late in an automake release
    cycle, and seek out feedback from existing users of AM_PATH_PYTHON.
    What say you?

I say +1 :).

    cf> I like this idea and am happy to help after the upcoming release.

Thanks Collin.

    Maybe something like:
        AM_PYTHON_SETUP([PYTHON-EXECUTABLE])

Sounds fine to me, at first blush.

My thought would be to add this stuff asap after the 1.17 release and
put out a 1.16.0.90 pretest purely for the purpose of "seeking
feedback", as you say, so users can give it a try without having other
unrelated changes. (I hope four-part version numbers don't cause problems.)

(Maybe along with Bruno's timestamp-fixing patch, again for feedback
purposes.)

So ... if either of you have time to work on implementing this, so it
can be pushed when ready, that would be great.

(Although in theory several branches are maintained in the automake repo
for purposes like this, in practice nothing but master has been used for
X years, and I myself have no time or interest in resurrecting others.)

Thanks! -k

Message #35 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: zack <at> owlfolio.org
Cc: 71672 <at> debbugs.gnu.org, collin.funk1 <at> gmail.com
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Mon, 24 Feb 2025 09:48:29 -0700

Returning to this issue of preferring python3, from last year:

    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=71672
    
    Add these three macros to Autoconf proper:
    ..._PYTHON_...

Have they been added?

    I think that should disentangle things reasonably well, but we should
    definitely do this early rather than late in an automake release cycle,

I hope to put out the next automake pretest->release very soon. So I
guess whatever happens

    and seek out feedback from existing users of AM_PATH_PYTHON.

To be clear, the above suggested factoring-out won't (intentionally)
affect the behavior of AM_PATH_PYTHON, right? It will still look for
"python" first, and still do all the stuff it does. I want Automake to
stay as backward-compatible as possible. --thanks, karl.

Message #40 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: praiskup <at> redhat.com
Cc: 17811 <at> debbugs.gnu.org, 71672 <at> debbugs.gnu.org
Subject: Re: bug#17811: RFE: build against multiple python stacks
Date: Mon, 24 Feb 2025 16:46:30 -0700

Hi Pavel - back in 2014 (!), you suggested supporting python2 and python3
simultaneously: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=17811
Essentially by defining a new "python3" primary to sit alongside
"python".

Agreed that this is desirable in general. It's now being discussed in
  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=71672
via a somewhat different approach. I'd worry about making python3 a new
primary, since 3 vs. 2 is not the only distinction needed, as has been
discussed.

I'm closing this one to reduce duplication in the bugdb, but feel free
to reply on 71762 or open a new issue if desired. --thanks, karl.

Message #43 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: Jacob Bachmeyer <jcb62281 <at> gmail.com>
To: Karl Berry <karl <at> freefriends.org>, zack <at> owlfolio.org
Cc: 71672 <at> debbugs.gnu.org, collin.funk1 <at> gmail.com
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Mon, 24 Feb 2025 22:03:58 -0600

On 2/24/25 10:48, Karl Berry wrote:
> [...]
>
> To be clear, the above suggested factoring-out won't (intentionally)
> affect the behavior of AM_PATH_PYTHON, right? It will still look for
> "python" first, and still do all the stuff it does. I want Automake to
> stay as backward-compatible as possible.

The problem is that Python did *not* stay backward-compatible and in the 
worst way:  running a program written for Python2 on a Python3 
interpreter can silently corrupt data if I understand correctly.

At minimum, a request for an unversioned python should raise a warning 
(if not an error:  "specify Python2 or Python3") and AM_PATH_PYTHON 
should probably be deprecated in favor of separate AM_PATH_PYTHON2 and 
AM_PATH_PYTHON3.

I am insufficiently familiar with Python to say if we should have finer 
granularity than "Python3" and I am uncertain how compatible the 
different releases of Python3 actually are.


-- Jacob

Message #46 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: "Zack Weinberg" <zack <at> owlfolio.org>
To: "Jacob Bachmeyer" <jcb62281 <at> gmail.com>, "Karl Berry" <karl <at> freefriends.org>
Cc: 71672 <at> debbugs.gnu.org, Collin Funk <collin.funk1 <at> gmail.com>
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Tue, 25 Feb 2025 10:14:55 -0500

On Mon, Feb 24, 2025, at 11:03 PM, Jacob Bachmeyer wrote:
> On 2/24/25 10:48, Karl Berry wrote:
>> [...]
>>
>> To be clear, the above suggested factoring-out won't (intentionally)
>> affect the behavior of AM_PATH_PYTHON, right? It will still look for
>> "python" first, and still do all the stuff it does. I want Automake
>> to stay as backward-compatible as possible.
>
> The problem is that Python did *not* stay backward-compatible and in
> the worst way:  running a program written for Python2 on a Python3
> interpreter can silently corrupt data if I understand correctly.

Yes. It takes some doing, but I *have* seen people write Python2
programs with this characteristic without knowing it.

> At minimum, a request for an unversioned python should raise a warning
> (if not an error:  "specify Python2 or Python3") and AM_PATH_PYTHON
> should probably be deprecated in favor of separate AM_PATH_PYTHON2 and
> AM_PATH_PYTHON3.

Agreed.  I would *like* to go a step farther and have AM_PATH_PYTHON*
bomb out if it detects that "python" on the PATH runs Python3, but
that would put us in direct conflict with a bunch of Linux distributions
so we probably shouldn't actually do that.

> I am insufficiently familiar with Python to say if we should have
> finer granularity than "Python3" and I am uncertain how compatible the
> different releases of Python3 actually are.

With all relatively recent Python 3 releases, you can have reasonably
high confidence that a program written against 3.x, that doesn't use third-
party Python libraries, will work correctly with 3.y where y > x.  The
only exception I know about is that the core Python devs recently decided
to start removing "old, unmaintained" modules from the standard library.
Also, I'm not sure how far back "relatively recent" extends.

The reverse is not true; a program written for 3.x is very likely to not
work at all with 3.z where z < x.  There have been a *lot* of shiny new
features added to the language in each 3.x release.

The compatibility story is much worse when third-party libraries are involved,
but that's not AM_PATH_PYTHON's problem.

zw

Message #49 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: "Zack Weinberg" <zack <at> owlfolio.org>
To: "Karl Berry" <karl <at> freefriends.org>
Cc: 71672 <at> debbugs.gnu.org, Collin Funk <collin.funk1 <at> gmail.com>
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Tue, 25 Feb 2025 10:18:00 -0500

On Mon, Feb 24, 2025, at 11:48 AM, Karl Berry wrote:
> Returning to this issue of preferring python3, from last year:
>
>     https://debbugs.gnu.org/cgi/bugreport.cgi?bug=71672
>
>     Add these three macros to Autoconf proper: ..._PYTHON_...
>
> Have they been added?

They have not.  Regrettably, I have not had any time at all for Autoconf
work recently.

What does your schedule look like for the next Automake release?  This
specific thing is probably only a weekend's worth of work, so I might be
able to scare up the time for it.  However, I don't think I can do all
the rest of the work involved in putting out an Autoconf release anytime
in the near future.

zw

Message #52 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: Karl Berry <karl <at> freefriends.org>
To: zack <at> owlfolio.org
Cc: 71672 <at> debbugs.gnu.org, collin.funk1 <at> gmail.com
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Tue, 25 Feb 2025 15:06:42 -0700

    What does your schedule look like for the next Automake release?

I'm hoping to put out the first pretest tomorrow and 1.18 not long
after. So clearly this Python stuff needs to wait for another release.

Thanks for the quick reply. -k

Message #55 received at 71672 <at> debbugs.gnu.org (full text, mbox):

From: Collin Funk <collin.funk1 <at> gmail.com>
To: Karl Berry <karl <at> freefriends.org>
Cc: zack <at> owlfolio.org, 71672 <at> debbugs.gnu.org
Subject: Re: [bug#71672] [PATCH] python: Prefer python3 over python
Date: Tue, 25 Feb 2025 14:15:20 -0800

Hi Karl,

Karl Berry <karl <at> freefriends.org> writes:

> I'm hoping to put out the first pretest tomorrow and 1.18 not long
> after. So clearly this Python stuff needs to wait for another release.

Thanks for mentioning this again. That plan sounds fine. Python packages
seem to have their own packaging stuff that works well so there is no
rush. However, I still think these would be nice for projects that might
use python scripts, etc.

I had a look at implementing them yesterday, but my m4 talent is
lacking. :)

Collin

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.