From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 01 Mar 2021 13:45:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 46848@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.161460624715468 (code B ref -1); Mon, 01 Mar 2021 13:45:01 +0000 Received: (at submit) by debbugs.gnu.org; 1 Mar 2021 13:44:07 +0000 Received: from localhost ([127.0.0.1]:48521 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lGiq5-00040s-W0 for submit@debbugs.gnu.org; Mon, 01 Mar 2021 08:44:07 -0500 Received: from lists.gnu.org ([209.51.188.17]:34676) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lGipz-00040g-09 for submit@debbugs.gnu.org; Mon, 01 Mar 2021 08:43:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:59510) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lGipy-0004en-Oi for guix-patches@gnu.org; Mon, 01 Mar 2021 08:43:50 -0500 Received: from mout-p-102.mailbox.org ([2001:67c:2050::465:102]:16092) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256) (Exim 4.90_1) (envelope-from ) id 1lGipt-0004lQ-1k for guix-patches@gnu.org; Mon, 01 Mar 2021 08:43:49 -0500 Received: from smtp2.mailbox.org (smtp2.mailbox.org [80.241.60.241]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4Dq1gR1VCBzQkJK for ; Mon, 1 Mar 2021 14:43:31 +0100 (CET) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1614606209; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=gszE9nJa3Nr0ww6hptcwp9mkhmP+fxtZ+I4I9OWz0K8=; b=JlGtx7QHCUxeel9+2zXBXXgRkdkROe6GH/85isarZMdvW8j6kF7zTtIi2VrEK/HlK2BvKG hNyioSoWFYXYwJFCpDdE8EMo3DuCRh3+JCd31EI+hlt+p71GpaMuK9w3vpNtAkmxwMs9yM PNEZFF3gcJjICXOJDkDd8CK1L36pxL/bcQWP7yM2O1sWQEIiixPnWiGomfjgOfO6u0s7t6 gR3hdvj65shvfV/zYx6IZaMVMIcGqzO8aU0ZkJpnIUkwIboNYrfs2uYFQ7KsfHrLnWuN9L rKCa/qZhCstPWsQLOJ1miUb/Cnn0mQE5tAd9pmsgZoqoWwrFPg347yZWQDtESw== Received: from smtp2.mailbox.org ([80.241.60.241]) by spamfilter03.heinlein-hosting.de (spamfilter03.heinlein-hosting.de [80.241.56.117]) (amavisd-new, port 10030) with ESMTP id jjDxxdmqfzvB for ; Mon, 1 Mar 2021 14:43:24 +0100 (CET) Date: Mon, 1 Mar 2021 14:43:22 +0100 From: Lars-Dominik Braun Message-ID: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="9EzuTbC0hpH3Zvh5" Content-Disposition: inline Content-Transfer-Encoding: 8bit X-MBO-SPAM-Probability: X-Rspamd-Score: -8.95 / 15.00 / 15.00 X-Rspamd-Queue-Id: C071C17B4 X-Rspamd-UID: 3d4170 Received-SPF: pass client-ip=2001:67c:2050::465:102; envelope-from=lars@6xq.net; helo=mout-p-102.mailbox.org X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_SBL_A=0.1 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.3 (/) --9EzuTbC0hpH3Zvh5 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit Hi everyone, the attached patches switch python-build-system to a PEP 517-based build system using python-pypa-build. As discussed previously Python is currently in the process of opening up for build systems other than setuptools using the API specified in PEP 517. python-pypa-build is a simple tool for building packages using that new API. It supports setup.py-based builds as a fallback, if no pyproject.toml is present. One downside is that this tool is not self-contained and has a few dependencies. Thus first I bootstrap setuptools using itself (possible because it bundles all of its own dependencies), then build python-pypa-build’s dependencies using setuptools (which is fortunately still possible) and then combine everything into a python-toolchain(-for-build), which is then used by the build-process. I can successfully build packages like python-pypa-build and python-pytest and python-pep517-bootstrap. The latter is using flit as its build backend. But other packages currently fail because I removed some arguments. Cheers, Lars --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-build-python-Handle-missing-setuptools-in-sanity-che.patch" >From 7ace01faef72f3a48b18fe241fe0524445a154fb Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Fri, 19 Feb 2021 17:22:35 +0100 Subject: [PATCH 01/12] build/python: Handle missing setuptools in sanity-check.py Just skip testing if required dependencies (setuptools) are not available. * gnu/packages/aux-files/python/sanity-check.py: Handle ImportError. --- gnu/packages/aux-files/python/sanity-check.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/gnu/packages/aux-files/python/sanity-check.py b/gnu/packages/aux-files/python/sanity-check.py index 83b6d583ca..240155cecc 100644 --- a/gnu/packages/aux-files/python/sanity-check.py +++ b/gnu/packages/aux-files/python/sanity-check.py @@ -19,9 +19,13 @@ from __future__ import print_function # Python 2 support. import importlib -import pkg_resources import sys import traceback +try: + import pkg_resources +except ImportError: + print('Warning: Skipping, because python-setuptools are not available.') + sys.exit(0) try: from importlib.machinery import PathFinder -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-gnu-python-pypa-build-Update-to-0.3.0.patch" >From 66033f14455456d465a3c9f5a2d1f4961cd3b989 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 12:50:42 +0100 Subject: [PATCH 02/12] gnu: python-pypa-build: Update to 0.3.0. * gnu/packages/python-build.scm (python-pypa-build): Update to 0.3.0. --- gnu/packages/python-build.scm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 140629ca43..92dbda314f 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -118,13 +118,13 @@ Language (TOML) configuration files.") (define-public python-pypa-build (package (name "python-pypa-build") - (version "0.1.0") + (version "0.3.0") (source (origin (method url-fetch) (uri (pypi-uri "build" version)) (sha256 (base32 - "1d6m21lijwm04g50nwgsgj7x3vhblzw7jv05ah8psqgzk20bbch8")))) + "1pazq66c35whrqd5b0zcydjvy2rmghi8riljkd67q3bpiln5pf8f")))) (build-system python-build-system) (arguments `(#:tests? #f ;to tests in the PyPI release -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0003-gnu-python-wheel-Install-entrypoint-scripts.patch" >From 00ef998413dc0fe6605653ae16f65d6377c2ec77 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:02:15 +0100 Subject: [PATCH 03/12] gnu: python-wheel: Install entrypoint scripts * gnu/packages/python-build.scm (pythont-wheel) [arguments]: Add phase 'patch-enable-entrypoints. --- gnu/packages/python-build.scm | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 92dbda314f..232e24f470 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -47,10 +47,17 @@ "0ii6f34rvpjg3nmw4bc2h7fhdsy38y1h93hghncfs5akfrldmj8h")))) (build-system python-build-system) (arguments - ;; FIXME: The test suite runs "python setup.py bdist_wheel", which in turn - ;; fails to find the newly-built bdist_wheel library, even though it is - ;; available on PYTHONPATH. What search path is consulted by setup.py? - '(#:tests? #f)) + `(#:phases + (modify-phases %standard-phases + (add-after 'unpack 'patch-enable-entrypoints + (lambda _ + ;; python-wheel tells setuptools to not install entry point scripts + ;; by default. Stop doing that, so wheels built contain all data + ;; required. + (substitute* "wheel/bdist_wheel.py" + (("(install_scripts\\.no_ep = )True" all assignment) + (string-append assignment "False"))) + #t))))) (home-page "https://bitbucket.org/pypa/wheel/") (synopsis "Format for built Python packages") (description -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0004-gnu-python-setuptools-Bootstrap-using-itself.patch" Content-Transfer-Encoding: 8bit >From 61313d8ddba30772e2587e3e16ca30d1565d3c7e Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:05:51 +0100 Subject: [PATCH 04/12] gnu: python-setuptools: Bootstrap using itself * gnu/packages/python-xyz.scm (python-setuptools) [arguments]: Add phase setting GUIX_PYTHONPATH to source directory. --- gnu/packages/python-xyz.scm | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index f8afa13f33..79d01f700a 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -1144,7 +1144,18 @@ other machines, such as over the network.") ;; FIXME: Tests require pytest, which itself relies on setuptools. ;; One could bootstrap with an internal untested setuptools. (arguments - `(#:tests? #f)) + `(#:tests? #f + #:python ,python-wrapper + #:phases (modify-phases %standard-phases + ;; Use this setuptools’ sources to bootstrap themselves. + (add-before 'build 'set-PYTHONPATH + (lambda _ + (format #t "current working dir ~s~%" (getcwd)) + (setenv "GUIX_PYTHONPATH" + (string-append ".:" (getenv "GUIX_PYTHONPATH"))) + #t))))) + ;; Not required when not building a wheel + ;(propagated-inputs `(("python-wheel" ,python-wheel))) (home-page "https://pypi.org/project/setuptools/") (synopsis "Library designed to facilitate packaging Python projects") -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0005-python-build-Switch-to-PEP-517-based-build.patch" Content-Transfer-Encoding: 8bit >From 7a99aaa40e65fde58ee2e78ad7d3e0ccd6d169ae Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:08:58 +0100 Subject: [PATCH 05/12] python-build: Switch to PEP 517-based build * gnu/packages/python-commencement.scm: New file, containing python-toolchain. * gnu/local.mk: Add it. * gnu/packages/python.scm (python): Disable installing bundled pip/setuptools. * guix/build/python-build-system.scm: Rewrite using python-pypa-build. * guix/build-system/python.scm (default-python): Switch to python-toolchain (lower): Remove unused parameter. XXX: rationale --- gnu/local.mk | 1 + gnu/packages/python-commencement.scm | 175 +++++++++++++++++++ gnu/packages/python.scm | 2 +- guix/build-system/python.scm | 8 +- guix/build/python-build-system.scm | 249 ++++++++++++++++++--------- 5 files changed, 342 insertions(+), 93 deletions(-) create mode 100644 gnu/packages/python-commencement.scm diff --git a/gnu/local.mk b/gnu/local.mk index 202677fed1..ef2532cb5d 100644 --- a/gnu/local.mk +++ b/gnu/local.mk @@ -463,6 +463,7 @@ GNU_SYSTEM_MODULES = \ %D%/packages/python.scm \ %D%/packages/python-build.scm \ %D%/packages/python-check.scm \ + %D%/packages/python-commencement.scm \ %D%/packages/python-compression.scm \ %D%/packages/python-crypto.scm \ %D%/packages/python-science.scm \ diff --git a/gnu/packages/python-commencement.scm b/gnu/packages/python-commencement.scm new file mode 100644 index 0000000000..2ced3079bc --- /dev/null +++ b/gnu/packages/python-commencement.scm @@ -0,0 +1,175 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès +;;; Copyright © 2014 Andreas Enge +;;; Copyright © 2012 Nikita Karetnikov +;;; Copyright © 2014, 2015, 2017 Mark H Weaver +;;; Copyright © 2017, 2018, 2019, 2021 Efraim Flashner +;;; Copyright © 2018 Tobias Geerinckx-Rice +;;; Copyright © 2018, 2019, 2020 Jan (janneke) Nieuwenhuizen +;;; Copyright © 2019, 2020 Marius Bakke +;;; Copyright © 2020 Timothy Sample +;;; Copyright © 2020 Guy Fleury Iteriteka +;;; Copyright © 2021 Maxim Cournoyer +;;; Copyright © 2021 Lars-Dominik Braun +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (gnu packages python-commencement) + #:use-module ((guix licenses) #:prefix license:) + #:use-module (guix download) + #:use-module (guix packages) + #:use-module (guix build-system trivial) + #:use-module (guix build-system python) + #:use-module (gnu packages) + #:use-module (gnu packages python) + #:use-module (gnu packages python-build) + #:use-module (gnu packages python-xyz) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26)) + +;; Python toolchain and all packages required to bootstrap it. + +(define-public python-toolchain + (package + (name "python-toolchain") + (version (package-version python)) + (source #f) + (build-system trivial-build-system) + (arguments + '(#:modules ((guix build union)) + #:builder (begin + (use-modules (ice-9 match) + (srfi srfi-1) + (srfi srfi-26) + (guix build union)) + + (let ((out (assoc-ref %outputs "out"))) + (union-build out (filter-map (match-lambda + ((_ . directory) directory)) + %build-inputs)) + #t)))) + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools) + ("python-pip" ,python-pip))) ; XXX Maybe virtualenv/venv too? It kind of + ; defeats the purpose of guix, but is used + ; alot in local development. + (native-search-paths + (package-native-search-paths python)) + (search-paths + (package-search-paths python)) + (license (package-license python)) ; XXX + (synopsis "Python toolchain") + (description + "Python toolchain including Python itself, setuptools and pip. Use this +package if you need a fully-fledged Python toolchain instead of just the +interpreter.") + (home-page (package-home-page python)))) + +;; Python 3 toolchain for python-build-system. We cannot use python-toolchain +;; here, since we’d need to bootstrap python-pip somehow. +(define-public python-toolchain-for-build + (package + (inherit python-toolchain) + (name "python-toolchain-for-build") + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools) + ("python-pypa-build" ,python-pypa-build-from-setuptools))))) + +;; Python 3 toolchain to bootstrap python-pypa-build +(define-public python-toolchain-only-setuptools + (package + (inherit python-toolchain) + (name "python-toolchain-only-setuptools") + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools))))) + +(define-public python-pypa-build-from-setuptools + (package + (inherit python-pypa-build) + (name "python-pypa-build-from-setuptools") + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools)) + (propagated-inputs + `(("python-pep517" ,python-pep517-from-setuptools) + ("python-packaging" ,python-packaging-from-setuptools))))) + +(define-public python-pep517-from-setuptools + (package + (inherit python-pep517-bootstrap) + (name "python-pep517-from-setuptools") + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools + #:phases (modify-phases %standard-phases + (add-after 'unpack 'patch + (lambda _ + (substitute* "setup.py" + (("distutils\\.core") "setuptools")) + #t))))) + (propagated-inputs + `(("python-toml" ,python-toml-from-setuptools) + ("python-wheel" ,python-wheel-from-setuptools))) + ;; Drop cyclic dependency. + (native-inputs '()))) + +(define-public python-toml-from-setuptools + (package + (inherit python-toml) + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools + ,@(package-arguments python-toml))))) + +(define-public python-packaging-from-setuptools + (package + (inherit python-packaging-bootstrap) + (name "python-packaging-from-setuptools") + (arguments + `(#:python ,python-toolchain-only-setuptools + ,@(package-arguments python-packaging-bootstrap))) + (propagated-inputs + `(("python-pyparsing" ,python-pyparsing-from-setuptools) + ("python-six" ,python-six-from-setuptools))))) + +(define-public python-pyparsing-from-setuptools + (package + (inherit python-pyparsing) + (name "python-pyparsing-from-setuptools") + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools + ,@(package-arguments python-pyparsing))))) + +(define-public python-six-from-setuptools + (package + (inherit python-six-bootstrap) + (name "python-six-from-setuptools") + (arguments + `(#:python ,python-toolchain-only-setuptools + ,@(package-arguments python-six-bootstrap))))) + +(define-public python-wheel-from-setuptools + (package + (inherit python-wheel) + (name "python-wheel-from-setuptools") + (arguments + `(#:python ,python-toolchain-only-setuptools + ,@(package-arguments python-wheel))))) + diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 8e8f46467b..ca5ce667ef 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -182,7 +182,7 @@ (list "--enable-shared" ;allow embedding "--with-system-expat" ;for XML support "--with-system-ffi" ;build ctypes - "--with-ensurepip=install" ;install pip and setuptools + "--with-ensurepip=no" ;do not install pip and setuptools "--enable-unicode=ucs4" ;; Prevent the installed _sysconfigdata.py from retaining a reference diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm index 2bb6fa87ca..998ea9323d 100644 --- a/guix/build-system/python.scm +++ b/guix/build-system/python.scm @@ -65,8 +65,8 @@ extension, such as '.tar.gz'." (define (default-python) "Return the default Python package." ;; Lazily resolve the binding to avoid a circular dependency. - (let ((python (resolve-interface '(gnu packages python)))) - (module-ref python 'python-wrapper))) + (let ((python (resolve-interface '(gnu packages python-commencement)))) + (module-ref python 'python-toolchain-for-build))) (define (default-python2) "Return the default Python 2 package." @@ -172,8 +172,6 @@ pre-defined variants." (define* (python-build store name inputs #:key (tests? #t) - (test-target "test") - (use-setuptools? #t) (configure-flags ''()) (phases '(@ (guix build python-build-system) %standard-phases)) @@ -199,9 +197,7 @@ provides a 'setup.py' file as its build system." source)) #:configure-flags ,configure-flags #:system ,system - #:test-target ,test-target #:tests? ,tests? - #:use-setuptools? ,use-setuptools? #:phases ,phases #:outputs %outputs #:search-paths ',(map search-path-specification->sexp diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm index 8ade1d5911..a5731511a9 100644 --- a/guix/build/python-build-system.scm +++ b/guix/build/python-build-system.scm @@ -34,6 +34,7 @@ #:use-module (ice-9 format) #:use-module (srfi srfi-1) #:use-module (srfi srfi-26) + #:use-module (srfi srfi-35) #:export (%standard-phases add-installed-pythonpath site-packages @@ -108,30 +109,17 @@ ;; "--single-version-externally-managed" is set, thus the .egg-info directory ;; and the scripts defined in entry-points will always be created. +;; Base error type. +(define-condition-type &python-build-error &error + python-build-error?) -(define setuptools-shim - ;; Run setup.py with "setuptools" being imported, which will patch - ;; "distutils". This is needed for packages using "distutils" instead of - ;; "setuptools" since the former does not understand the - ;; "--single-version-externally-managed" flag. - ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py - (string-append - "import setuptools, tokenize;__file__='setup.py';" - "f=getattr(tokenize, 'open', open)(__file__);" - "code=f.read().replace('\\r\\n', '\\n');" - "f.close();" - "exec(compile(code, __file__, 'exec'))")) - -(define (call-setuppy command params use-setuptools?) - (if (file-exists? "setup.py") - (begin - (format #t "running \"python setup.py\" with command ~s and parameters ~s~%" - command params) - (if use-setuptools? - (apply invoke "python" "-c" setuptools-shim - command params) - (apply invoke "python" "./setup.py" command params))) - (error "no setup.py found"))) +;; Raised when 'check cannot find a valid test system in the inputs. +(define-condition-type &test-system-not-found &python-build-error + test-system-not-found?) + +;; Raised when multiple wheels are created by 'build. +(define-condition-type &cannot-extract-multiple-wheels &python-build-error + cannot-extract-multiple-wheels?) (define* (sanity-check #:key tests? inputs outputs #:allow-other-keys) "Ensure packages depending on this package via setuptools work properly, @@ -142,23 +130,51 @@ without errors." (with-directory-excursion "/tmp" (invoke "python" sanity-check.py (site-packages inputs outputs))))) -(define* (build #:key use-setuptools? #:allow-other-keys) +(define* (build #:key outputs #:allow-other-keys) "Build a given Python package." - (call-setuppy "build" '() use-setuptools?) + + (define pyproject-build (which "pyproject-build")) + + (define (build-pep517) + ;; XXX: should probably use a different path, outside of source directory, + ;; maybe secondary output “wheel”? + (mkdir-p "dist") + (invoke pyproject-build "--outdir" "dist" "--no-isolation" "--wheel" ".")) + + ;; XXX Would be nice, if we could use bdist_wheel here to remove extra + ;; code path in 'install, but that depends on python-wheel. + (define (build-setuptools) + (invoke "python" "setup.py" "build")) + + (if pyproject-build + (build-pep517) + (build-setuptools)) #t) -(define* (check #:key tests? test-target use-setuptools? #:allow-other-keys) +(define* (check #:key inputs outputs tests? #:allow-other-keys) "Run the test suite of a given Python package." (if tests? - ;; Running `setup.py test` creates an additional .egg-info directory in - ;; build/lib in some cases, e.g. if the source is in a sub-directory - ;; (given with `package_dir`). This will by copied to the output, too, - ;; so we need to remove. - (let ((before (find-files "build" "\\.egg-info$" #:directories? #t))) - (call-setuppy test-target '() use-setuptools?) - (let* ((after (find-files "build" "\\.egg-info$" #:directories? #t)) - (inter (lset-difference string=? after before))) - (for-each delete-file-recursively inter))) + ;; Unfortunately with PEP 517 there is no common method to specify test + ;; systems. Guess test system based on inputs instead. + (let ((pytest (which "pytest")) + (have-setup-py (file-exists? "setup.py"))) + ;; Prefer pytest + ;; XXX: support nose + (cond + (pytest + (begin + (format #t "using pytest~%") + (invoke pytest "-vv"))) ; XXX: support skipping tests based on name/extra arguments? + ;; But fall back to setup.py, which should work for most + ;; packages. XXX: would be nice not to depend on setup.py here? fails + ;; more often than not to find any tests at all. Maybe we can run + ;; `python -m unittest`? + (have-setup-py + (begin + (format #t "using setup.py~%") + (invoke "python" "setup.py" "test" "-v"))) + ;; The developer should explicitly disable tests in this case. + (#t (raise (condition (&test-system-not-found)))))) (format #t "test suite not run~%")) #t) @@ -195,31 +211,109 @@ running checks after installing the package." "/bin:" (getenv "PATH")))) -(define* (install #:key inputs outputs (configure-flags '()) use-setuptools? - #:allow-other-keys) - "Install a given Python package." - (let* ((out (python-output outputs)) - (python (assoc-ref inputs "python")) - (major-minor (map string->number - (take (string-split (python-version python) #\.) 2))) - (<3.7? (match major-minor - ((major minor) - (or (< major 3) (and (= major 3) (< minor 7)))))) - (params (append (list (string-append "--prefix=" out) - "--no-compile") - (if use-setuptools? - ;; distutils does not accept these flags - (list "--single-version-externally-managed" - "--root=/") - '()) - configure-flags))) - (call-setuppy "install" params use-setuptools?) - ;; Rather than produce potentially non-reproducible .pyc files on Pythons - ;; older than 3.7, whose 'compileall' module lacks the - ;; '--invalidation-mode' option, do not generate any. - (unless <3.7? - (invoke "python" "-m" "compileall" "--invalidation-mode=unchecked-hash" - out)))) +(define* (install #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Install a wheel file according to PEP 427" + ;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl + (let* ((site-dir (site-packages inputs outputs)) + (out (assoc-ref outputs "out"))) + (define (extract file) + "Extract wheel (ZIP file) into site-packages directory" + ;; Use Python’s zipfile to avoid extra dependency + (invoke "python" "-m" "zipfile" "-e" file site-dir)) + + (define python-hashbang + (string-append "#!" (assoc-ref inputs "python") "/bin/python")) + + (define (move-data source destination) + (mkdir-p (dirname destination)) + (rename-file source destination)) + + (define (move-script source destination) + "Move executable script file from .data/scripts to out/bin and replace +temporary hashbang" + (move-data source destination) + ;; ZIP does not save/restore permissions, make executable + ;; XXX: might not be a file, but directory with subdirectories + (chmod destination #o755) + (substitute* destination (("#!python") python-hashbang))) + + ;; Python’s distutils.command.install defines this mapping from source to + ;; destination mapping. + (define install-schemes + `(("scripts" "bin" ,move-script) + ;; XXX: Why does Python not use share/ here? + ("data" "share" ,move-data))) + + (define (expand-data-directory directory) + "Move files from all .data subdirectories to their respective +destinations." + (for-each + (match-lambda ((source destination function) + (let ((source-path (string-append directory "/" source)) + (destination-path (string-append out "/" destination))) + (when (file-exists? source-path) + (begin + ;; This assumes only files exist in the scripts/ directory. + (for-each + (lambda (file) + (apply + function + (list + (string-append source-path "/" file) + (string-append destination-path "/" file)))) + (scandir source-path (negate (cut member <> '("." ".."))))) + (rmdir source-path)))))) + install-schemes)) + + (define pyproject-build (which "pyproject-build")) + + (define (list-directories base predicate) + ;; Cannot use find-files here, because it’s recursive. + (scandir + base + (lambda (name) + (let ((stat (lstat (string-append base "/" name)))) + (and + (not (member name '("." ".."))) + (eq? (stat:type stat) 'directory) + (predicate name stat)))))) + + (define (install-pep517) + "Install a wheel generated by a PEP 517-compatible builder." + (let ((wheels (find-files "dist" "\\.whl$"))) ; XXX: do not recurse + (when (> (length wheels) 1) ; This code does not support multiple wheels + ; yet, because their outputs would have to be + ; merged properly. + (raise (condition (&cannot-extract-multiple-wheels)))) + (for-each extract wheels)) + (let ((datadirs (map + (cut string-append site-dir "/" <>) + (list-directories site-dir (file-name-predicate "\\.data$"))))) + (for-each (lambda (directory) + (expand-data-directory directory) + (rmdir directory)) + datadirs))) + + (define (install-setuptools) + "Install using setuptools." + (let ((out (assoc-ref outputs "out"))) + (invoke "python" "setup.py" + "install" + "--prefix" out + "--single-version-externally-managed" + "--root=/"))) + + (if pyproject-build + (install-pep517) + (install-setuptools)) + #t)) + +(define* (compile-bytecode #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Compile installed byte-code in site-packages." + (let ((site-dir (site-packages inputs outputs))) + (invoke "python" "-m" "compileall" site-dir) + ;; XXX: We could compile with -O and -OO too here, at the cost of more space. + #t)) (define* (wrap #:key inputs outputs #:allow-other-keys) (define (list-of-files dir) @@ -243,29 +337,12 @@ running checks after installing the package." files))) bindirs))) -(define* (rename-pth-file #:key name inputs outputs #:allow-other-keys) - "Rename easy-install.pth to NAME.pth to avoid conflicts between packages -installed with setuptools." - ;; Even if the "easy-install.pth" is not longer created, we kept this phase. - ;; There still may be packages creating an "easy-install.pth" manually for - ;; some good reason. - (let* ((site-packages (site-packages inputs outputs)) - (easy-install-pth (string-append site-packages "/easy-install.pth")) - (new-pth (string-append site-packages "/" name ".pth"))) - (when (file-exists? easy-install-pth) - (rename-file easy-install-pth new-pth)))) - -(define* (ensure-no-mtimes-pre-1980 #:rest _) - "Ensure that there are no mtimes before 1980-01-02 in the source tree." - ;; Rationale: patch-and-repack creates tarballs with timestamps at the POSIX - ;; epoch, 1970-01-01 UTC. This causes problems with Python packages, - ;; because Python eggs are ZIP files, and the ZIP format does not support - ;; timestamps before 1980. - (let ((early-1980 315619200)) ; 1980-01-02 UTC - (ftw "." (lambda (file stat flag) - (unless (<= early-1980 (stat:mtime stat)) - (utime file early-1980 early-1980)) - #t)))) +(define* (set-SOURCE-DATE-EPOCH #:rest _) + "Set the 'SOURCE_DATE_EPOCH' environment variable. This is used by tools +that incorporate timestamps as a way to tell them to use a fixed timestamp. +See https://reproducible-builds.org/specs/source-date-epoch/." + (setenv "SOURCE_DATE_EPOCH" "315619200") ;; python-wheel respects this variable and sets pre-1980 times on files in zip files, which is unsupported + #t) (define* (enable-bytecode-determinism #:rest _) "Improve determinism of pyc files." @@ -292,11 +369,11 @@ by Cython." ;; prefix directory. The check phase is moved after the installation phase ;; to ease testing the built package. (modify-phases gnu:%standard-phases - (add-after 'unpack 'ensure-no-mtimes-pre-1980 ensure-no-mtimes-pre-1980) - (add-after 'ensure-no-mtimes-pre-1980 'enable-bytecode-determinism + (add-after 'unpack 'enable-bytecode-determinism enable-bytecode-determinism) (add-after 'enable-bytecode-determinism 'ensure-no-cythonized-files ensure-no-cythonized-files) + (replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH) (delete 'bootstrap) (delete 'configure) ;not needed (replace 'build build) @@ -308,7 +385,7 @@ by Cython." (add-after 'add-install-to-path 'wrap wrap) (add-after 'wrap 'check check) (add-after 'check 'sanity-check sanity-check) - (add-before 'strip 'rename-pth-file rename-pth-file))) + (add-before 'check 'compile-bytecode compile-bytecode))) (define* (python-build #:key inputs (phases %standard-phases) #:allow-other-keys #:rest args) -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0006-gnu-Add-python-pytoml.patch" >From 14b70dea3d21684e7fdb66dd072031cef3214b07 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:17:10 +0100 Subject: [PATCH 06/12] gnu: Add python-pytoml. * gnu/packages/python-build.scm (python-pytoml): Add new variable. --- gnu/packages/python-build.scm | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 232e24f470..a6a310177c 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -173,3 +173,20 @@ implementation developed for Poetry. This project is intended to be a light weight, fully compliant, self-contained package allowing PEP 517 compatible build front-ends to build Poetry managed projects.") (license license:expat))) + +(define-public python-pytoml + (package + (name "python-pytoml") + (version "0.1.21") + (source + (origin + (method url-fetch) + (uri (pypi-uri "pytoml" version)) + (sha256 + (base32 + "1rv1byiw82k7mj6aprcrqi2vdabs801y97xhfnrz7kxds34ggv4f")))) + (build-system python-build-system) + (home-page "https://github.com/avakar/pytoml") + (synopsis "A parser for TOML-0.4.0") + (description "A parser for TOML-0.4.0") + (license license:expat))) -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0007-gnu-Add-python-flit-core.patch" >From e8b1f40157b3d884bb2fc3e3d10bbfbd469c67b9 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:18:17 +0100 Subject: [PATCH 07/12] gnu: Add python-flit-core. * gnu/packages/python-build.scm (python-flit-core): New variable. --- gnu/packages/python-build.scm | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index a6a310177c..f767704a78 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -190,3 +190,30 @@ compatible build front-ends to build Poetry managed projects.") (synopsis "A parser for TOML-0.4.0") (description "A parser for TOML-0.4.0") (license license:expat))) + +(define-public python-flit-core + (package + (name "python-flit-core") + (version "3.0.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "flit_core" version)) + (sha256 + (base32 + "0bbw84r33gwi0xyp7m8dzp2dzpjs4harj3l5wrbxkmp2awh0ard4")))) + (build-system python-build-system) + (arguments + `(;; No tests. + #:tests? #f)) + (propagated-inputs + `(("python-pytoml" ,python-pytoml))) + (home-page "https://github.com/takluyver/flit") + (synopsis + "Simplified packaging of Python modules, distribution-building parts") + (description + "Flit is a simple way to put Python packages and modules on PyPI. It +tries to require less thought about packaging and help you avoid common +mistakes. Distribution-building parts of Flit.") + (license license:bsd-3))) + -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0008-gnu-python-pep517-bootstrap-Build-using-flit-core.patch" >From 316c25f310cad4dd8f0cb73a914a4776b1b1375c Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:20:48 +0100 Subject: [PATCH 08/12] gnu: python-pep517-bootstrap: Build using flit-core. * gnu/packages/python-build.scm (python-pep517-bootstrap) [arguments]: Relax dependency on flit-core version. [native-inputs]: Add flit-core. --- gnu/packages/python-build.scm | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index f767704a78..f74a3ee49e 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -110,10 +110,21 @@ Language (TOML) configuration files.") "0zqidxah03qpnp6zkg3zd1kmd5f79hhdsfmlc0cldaniy80qddxf")))) (build-system python-build-system) (arguments - `(#:tests? #f)) ;to avoid circular dependencies + `(#:tests? #f ; To avoid circular dependencies. + #:phases + (modify-phases %standard-phases + (add-after 'unpack 'relax-dependency + (lambda _ + (substitute* "pyproject.toml" + (("flit_core >=2,<3") + "flit_core >=2,<4")) + #t))))) (propagated-inputs `(("python-toml" ,python-toml) ("python-wheel" ,python-wheel))) + (native-inputs + `(;; Build system. + ("python-flit-core" ,python-flit-core))) (home-page "https://github.com/pypa/pep517") (synopsis "Wrappers to build Python packages using PEP 517 hooks") (description -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0009-gnu-python-iniconfig-Add-missing-build-input.patch" >From c8898a6a282daaa2cceabfac96e417f7b09e8d5f Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:23:53 +0100 Subject: [PATCH 09/12] gnu: python-iniconfig: Add missing build input. * gnu/packages/python-xyz.scm (python-iniconfig) [native-inputs] Add setuptools-scm. --- gnu/packages/python-xyz.scm | 1 + 1 file changed, 1 insertion(+) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index 79d01f700a..d598a380a9 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -15704,6 +15704,7 @@ in other versions.") (base32 "0ckzngs3scaa1mcfmsi1w40a1l8cxxnncscrxzjjwjyisx8z0fmw")))) (build-system python-build-system) + (native-inputs `(("python-setuptools-scm" ,python-setuptools-scm))) (home-page "https://github.com/RonnyPfannschmidt/iniconfig") (synopsis "Simple INI-file parser") (description "The @code{iniconfig} package provides a small and simple -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0010-gnu-Add-python-u-msgpack.patch" >From 8f5c9398ac1a6ac1871d5cf8a1efbe6a70ed4a1b Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 14:16:07 +0100 Subject: [PATCH 10/12] gnu: Add python-u-msgpack. * gnu/packages/python-xyz.scm (python-u-msgpack): New variable. The redundant -python postfix from the original package name has been removed. --- gnu/packages/python-xyz.scm | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index d598a380a9..ab3c6d301e 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -23659,3 +23659,27 @@ Application Programming Interface based on the Open Inventor 2.1 API.") Crayons automatically wraps a given string in the foreground color and restores the original state after the string is printed.") (license license:expat))) + +(define-public python-u-msgpack + (package + (name "python-u-msgpack") + (version "2.7.1") + (source + (origin + (method url-fetch) + (uri (pypi-uri "u-msgpack-python" version)) + (sha256 + (base32 + "0lcmlr7gc4dydpxn6l5bdcq40c3ghf8mv1sjqyj72wdpr8rx9rxp")))) + (build-system python-build-system) + (home-page + "https://github.com/vsergeev/u-msgpack-python") + (synopsis + "Portable, lightweight MessagePack serializer and deserializer") + (description + "A portable, lightweight MessagePack serializer and deserializer written +in pure Python. u-msgpack-python is fully compliant with the latest MessagePack +specification. In particular, it supports the new binary, UTF-8 string, +application-defined ext, and timestamp types.") + (license license:expat))) + -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0011-gnu-Add-python-pytest-expect.patch" >From 39eef77658f400ccfceb65b6fcd3f4996ae90807 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 14:20:03 +0100 Subject: [PATCH 11/12] gnu: Add python-pytest-expect. * gnu/packages/python-check.scm (python-pytest-expect): New variable. --- gnu/packages/python-check.scm | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/gnu/packages/python-check.scm b/gnu/packages/python-check.scm index 9849b16685..4139f3cdde 100644 --- a/gnu/packages/python-check.scm +++ b/gnu/packages/python-check.scm @@ -1251,3 +1251,31 @@ help in debugging failures and optimizing the scheduler to improve speed.") (description "A pytest plugin for Sanic. It helps you to test your code asynchronously.") (license license:expat))) + +(define-public python-pytest-expect + (package + (name "python-pytest-expect") + (version "1.1.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "pytest-expect" version)) + (sha256 + (base32 + "0iyq3zd1g5ffaz2wv6mskjfn84sfbyh0j209glcrh1s50hkldd1n")))) + (build-system python-build-system) + (arguments `(#:tests? #f)) ; no tests via pypi + (propagated-inputs + `(("python-pytest" ,python-pytest) ; package declares this dependency + ("python-u-msgpack" ,python-u-msgpack))) + (home-page + "https://github.com/gsnedders/pytest-expect") + (synopsis + "Py.test plugin to store test expectations and mark tests based on them") + (description + "A py.test plugin that stores test expectations by saving the set of +failing tests, allowing them to be marked as xfail when running them in future. +The tests expectations are stored such that they can be distributed alongside +the tests.") + (license license:expat))) + -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0012-gnu-python-html5lib-Fix-tests-with-pytest-6.patch" >From 4ed7eba12c80dd33f20626ddb81abc34dd667ab3 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 14:23:07 +0100 Subject: [PATCH 12/12] gnu: python-html5lib: Fix tests with pytest 6. * gnu/packages/python-web.scm (python-html5lib) [source]: Add upstream patch. [native-inputs]: Add test dependencies. [arguments]: Remove unsupported #:test-target. --- gnu/packages/python-web.scm | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm index 947c200253..c484d7ba36 100644 --- a/gnu/packages/python-web.scm +++ b/gnu/packages/python-web.scm @@ -1020,13 +1020,27 @@ storage.") (uri (pypi-uri "html5lib" version)) (sha256 (base32 - "0vqlhk0hgbsfkh7ybmby93xhlx8dq6pr5blf356ka3z2c41b9rdj")))) + "0vqlhk0hgbsfkh7ybmby93xhlx8dq6pr5blf356ka3z2c41b9rdj")) + (patches + (list + ;; Adds Pytest 6 support. + (origin + (method url-fetch) + (uri (string-append + "https://github.com/html5lib/" + "html5lib-python/commit/" + "2c19b9899ab3a3e8bd0ca35e5d78544334204169.patch")) + (file-name "python-html5lib-support-pytest6.patch") + (sha256 + (base32 + "0jg2ry0439q8n7j1mf4p2hdq54i704pq9scv4wwa2pp3cwvb6dvg"))))))) (build-system python-build-system) (propagated-inputs `(("python-six" ,python-six) ("python-webencodings" ,python-webencodings))) - (arguments - `(#:test-target "check")) + (native-inputs + `(("python-pytest" ,python-pytest) + ("python-pytest-expect" ,python-pytest-expect))) (home-page "https://github.com/html5lib/html5lib-python") (synopsis -- 2.26.2 --9EzuTbC0hpH3Zvh5-- From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 15 May 2021 09:33:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 46848@debbugs.gnu.org Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.162107114925211 (code B ref 46848); Sat, 15 May 2021 09:33:01 +0000 Received: (at 46848) by debbugs.gnu.org; 15 May 2021 09:32:29 +0000 Received: from localhost ([127.0.0.1]:47569 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhqef-0006YF-AZ for submit@debbugs.gnu.org; Sat, 15 May 2021 05:32:29 -0400 Received: from mout-p-102.mailbox.org ([80.241.56.152]:35742) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lhqea-0006Xv-HE for 46848@debbugs.gnu.org; Sat, 15 May 2021 05:32:15 -0400 Received: from smtp2.mailbox.org (smtp2.mailbox.org [IPv6:2001:67c:2050:105:465:1:2:0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4Fj0Xk1mRzzQk2j for <46848@debbugs.gnu.org>; Sat, 15 May 2021 11:32:06 +0200 (CEST) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1621071124; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K/HzaYGlgZT5yJWEONi/5i+7eeLt2bnx4i/cIDRhVnI=; b=J4lrD/BVbhMyxyXi3cVPxe79lPn/PNTubKEglsPIbNb3t5h0NKynHw9kuTbCC+UJ4jaE3g GkFRHE2liMyI9YOY9+XVTTz7n+hEs8rmfDgVI5kjKMtnoJ9YbtPMXWHMyS9uWmBBB+srAI p+R67fDIfApDhZLMUitQeVtebRZrFzwFygVKuwoDOUVXS04qrnUUWblcrY+q9ie0njvDI1 3Y6wZHFNlIwFDm4K0zpSCaRm6M1coL/SA8IKBPszjsW+5dn5wJ/Ui0DaD6TzQ/5ViitKYN WJFgLRAHCCDwhaNlyCUwnStCuO3AA/rSUkN6tiZD+4ARe7lvlC1vj/g2WL7k0A== Received: from smtp2.mailbox.org ([80.241.60.241]) by spamfilter02.heinlein-hosting.de (spamfilter02.heinlein-hosting.de [80.241.56.116]) (amavisd-new, port 10030) with ESMTP id 88ji7F7xrrrh for <46848@debbugs.gnu.org>; Sat, 15 May 2021 11:32:01 +0200 (CEST) Date: Sat, 15 May 2021 11:31:59 +0200 From: Lars-Dominik Braun Message-ID: References: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="lI7hEO8aOAzcKIP4" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-MBO-SPAM-Probability: X-Rspamd-Score: -8.85 / 15.00 / 15.00 X-Rspamd-Queue-Id: D9356180D X-Rspamd-UID: e4faad X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --lI7hEO8aOAzcKIP4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, I rebased my changes on top of the current core-updates HEAD. Cheers, Lars --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-build-python-Handle-missing-setuptools-in-sanity-che.patch" >From 9d9c417fba869913366a12c89b7309ecc402777d Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Fri, 19 Feb 2021 17:22:35 +0100 Subject: [PATCH 01/14] build/python: Handle missing setuptools in sanity-check.py Just skip testing if required dependencies (setuptools) are not available. * gnu/packages/aux-files/python/sanity-check.py: Handle ImportError. --- gnu/packages/aux-files/python/sanity-check.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/gnu/packages/aux-files/python/sanity-check.py b/gnu/packages/aux-files/python/sanity-check.py index 83b6d583ca..240155cecc 100644 --- a/gnu/packages/aux-files/python/sanity-check.py +++ b/gnu/packages/aux-files/python/sanity-check.py @@ -19,9 +19,13 @@ from __future__ import print_function # Python 2 support. import importlib -import pkg_resources import sys import traceback +try: + import pkg_resources +except ImportError: + print('Warning: Skipping, because python-setuptools are not available.') + sys.exit(0) try: from importlib.machinery import PathFinder -- 2.26.3 --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-gnu-python-pypa-build-Update-to-0.3.0.patch" >From 5cdfe3f6de27be54eeaa5c507a62319c3783ff1a Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 12:50:42 +0100 Subject: [PATCH 02/14] gnu: python-pypa-build: Update to 0.3.0. * gnu/packages/python-build.scm (python-pypa-build): Update to 0.3.0. --- gnu/packages/python-build.scm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 59ee91aa9c..25e2f1e60f 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -136,13 +136,13 @@ Language (TOML) configuration files.") (define-public python-pypa-build (package (name "python-pypa-build") - (version "0.1.0") + (version "0.3.0") (source (origin (method url-fetch) (uri (pypi-uri "build" version)) (sha256 (base32 - "1d6m21lijwm04g50nwgsgj7x3vhblzw7jv05ah8psqgzk20bbch8")))) + "1pazq66c35whrqd5b0zcydjvy2rmghi8riljkd67q3bpiln5pf8f")))) (build-system python-build-system) (arguments `(#:tests? #f ;to tests in the PyPI release -- 2.26.3 --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0003-gnu-python-wheel-Install-entrypoint-scripts.patch" >From 9516a9e734009f7e38acb37a51f704de9e6198ef Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:02:15 +0100 Subject: [PATCH 03/14] gnu: python-wheel: Install entrypoint scripts * gnu/packages/python-build.scm (pythont-wheel) [arguments]: Add phase 'patch-enable-entrypoints. --- gnu/packages/python-build.scm | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 25e2f1e60f..bbd273e5de 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -49,10 +49,17 @@ "0ii6f34rvpjg3nmw4bc2h7fhdsy38y1h93hghncfs5akfrldmj8h")))) (build-system python-build-system) (arguments - ;; FIXME: The test suite runs "python setup.py bdist_wheel", which in turn - ;; fails to find the newly-built bdist_wheel library, even though it is - ;; available on PYTHONPATH. What search path is consulted by setup.py? - '(#:tests? #f)) + `(#:phases + (modify-phases %standard-phases + (add-after 'unpack 'patch-enable-entrypoints + (lambda _ + ;; python-wheel tells setuptools to not install entry point scripts + ;; by default. Stop doing that, so wheels built contain all data + ;; required. + (substitute* "wheel/bdist_wheel.py" + (("(install_scripts\\.no_ep = )True" all assignment) + (string-append assignment "False"))) + #t))))) (home-page "https://bitbucket.org/pypa/wheel/") (synopsis "Format for built Python packages") (description -- 2.26.3 --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0004-gnu-python-setuptools-Bootstrap-using-itself.patch" Content-Transfer-Encoding: 8bit >From 19349bf56a88feb5ebc0d24c4f3324e776b2f5ff Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:05:51 +0100 Subject: [PATCH 04/14] gnu: python-setuptools: Bootstrap using itself * gnu/packages/python-xyz.scm (python-setuptools) [arguments]: Add phase setting GUIX_PYTHONPATH to source directory. --- gnu/packages/python-xyz.scm | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index eebb44b9dc..8f472dea42 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -1390,7 +1390,18 @@ other machines, such as over the network.") ;; FIXME: Tests require pytest, which itself relies on setuptools. ;; One could bootstrap with an internal untested setuptools. (arguments - `(#:tests? #f)) + `(#:tests? #f + #:python ,python-wrapper + #:phases (modify-phases %standard-phases + ;; Use this setuptools’ sources to bootstrap themselves. + (add-before 'build 'set-PYTHONPATH + (lambda _ + (format #t "current working dir ~s~%" (getcwd)) + (setenv "GUIX_PYTHONPATH" + (string-append ".:" (getenv "GUIX_PYTHONPATH"))) + #t))))) + ;; Not required when not building a wheel + ;(propagated-inputs `(("python-wheel" ,python-wheel))) (home-page "https://pypi.org/project/setuptools/") (synopsis "Library designed to facilitate packaging Python projects") -- 2.26.3 --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0005-python-build-Switch-to-PEP-517-based-build.patch" Content-Transfer-Encoding: 8bit >From 0e023d986fe6a093d5f13923b84c54a674fd2278 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:08:58 +0100 Subject: [PATCH 05/14] python-build: Switch to PEP 517-based build * gnu/packages/python-commencement.scm: New file, containing python-toolchain. * gnu/local.mk: Add it. * gnu/packages/python.scm (python): Disable installing bundled pip/setuptools. * guix/build/python-build-system.scm: Rewrite using python-pypa-build. * guix/build-system/python.scm (default-python): Switch to python-toolchain (lower): Remove unused parameter. XXX: rationale --- gnu/local.mk | 1 + gnu/packages/python-commencement.scm | 175 +++++++++++++++++++ gnu/packages/python.scm | 2 +- guix/build-system/python.scm | 6 +- guix/build/python-build-system.scm | 249 ++++++++++++++++++--------- 5 files changed, 343 insertions(+), 90 deletions(-) create mode 100644 gnu/packages/python-commencement.scm diff --git a/gnu/local.mk b/gnu/local.mk index 0e1e83b3d5..e1e41adff7 100644 --- a/gnu/local.mk +++ b/gnu/local.mk @@ -465,6 +465,7 @@ GNU_SYSTEM_MODULES = \ %D%/packages/python.scm \ %D%/packages/python-build.scm \ %D%/packages/python-check.scm \ + %D%/packages/python-commencement.scm \ %D%/packages/python-compression.scm \ %D%/packages/python-crypto.scm \ %D%/packages/python-science.scm \ diff --git a/gnu/packages/python-commencement.scm b/gnu/packages/python-commencement.scm new file mode 100644 index 0000000000..2ced3079bc --- /dev/null +++ b/gnu/packages/python-commencement.scm @@ -0,0 +1,175 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès +;;; Copyright © 2014 Andreas Enge +;;; Copyright © 2012 Nikita Karetnikov +;;; Copyright © 2014, 2015, 2017 Mark H Weaver +;;; Copyright © 2017, 2018, 2019, 2021 Efraim Flashner +;;; Copyright © 2018 Tobias Geerinckx-Rice +;;; Copyright © 2018, 2019, 2020 Jan (janneke) Nieuwenhuizen +;;; Copyright © 2019, 2020 Marius Bakke +;;; Copyright © 2020 Timothy Sample +;;; Copyright © 2020 Guy Fleury Iteriteka +;;; Copyright © 2021 Maxim Cournoyer +;;; Copyright © 2021 Lars-Dominik Braun +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (gnu packages python-commencement) + #:use-module ((guix licenses) #:prefix license:) + #:use-module (guix download) + #:use-module (guix packages) + #:use-module (guix build-system trivial) + #:use-module (guix build-system python) + #:use-module (gnu packages) + #:use-module (gnu packages python) + #:use-module (gnu packages python-build) + #:use-module (gnu packages python-xyz) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26)) + +;; Python toolchain and all packages required to bootstrap it. + +(define-public python-toolchain + (package + (name "python-toolchain") + (version (package-version python)) + (source #f) + (build-system trivial-build-system) + (arguments + '(#:modules ((guix build union)) + #:builder (begin + (use-modules (ice-9 match) + (srfi srfi-1) + (srfi srfi-26) + (guix build union)) + + (let ((out (assoc-ref %outputs "out"))) + (union-build out (filter-map (match-lambda + ((_ . directory) directory)) + %build-inputs)) + #t)))) + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools) + ("python-pip" ,python-pip))) ; XXX Maybe virtualenv/venv too? It kind of + ; defeats the purpose of guix, but is used + ; alot in local development. + (native-search-paths + (package-native-search-paths python)) + (search-paths + (package-search-paths python)) + (license (package-license python)) ; XXX + (synopsis "Python toolchain") + (description + "Python toolchain including Python itself, setuptools and pip. Use this +package if you need a fully-fledged Python toolchain instead of just the +interpreter.") + (home-page (package-home-page python)))) + +;; Python 3 toolchain for python-build-system. We cannot use python-toolchain +;; here, since we’d need to bootstrap python-pip somehow. +(define-public python-toolchain-for-build + (package + (inherit python-toolchain) + (name "python-toolchain-for-build") + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools) + ("python-pypa-build" ,python-pypa-build-from-setuptools))))) + +;; Python 3 toolchain to bootstrap python-pypa-build +(define-public python-toolchain-only-setuptools + (package + (inherit python-toolchain) + (name "python-toolchain-only-setuptools") + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools))))) + +(define-public python-pypa-build-from-setuptools + (package + (inherit python-pypa-build) + (name "python-pypa-build-from-setuptools") + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools)) + (propagated-inputs + `(("python-pep517" ,python-pep517-from-setuptools) + ("python-packaging" ,python-packaging-from-setuptools))))) + +(define-public python-pep517-from-setuptools + (package + (inherit python-pep517-bootstrap) + (name "python-pep517-from-setuptools") + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools + #:phases (modify-phases %standard-phases + (add-after 'unpack 'patch + (lambda _ + (substitute* "setup.py" + (("distutils\\.core") "setuptools")) + #t))))) + (propagated-inputs + `(("python-toml" ,python-toml-from-setuptools) + ("python-wheel" ,python-wheel-from-setuptools))) + ;; Drop cyclic dependency. + (native-inputs '()))) + +(define-public python-toml-from-setuptools + (package + (inherit python-toml) + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools + ,@(package-arguments python-toml))))) + +(define-public python-packaging-from-setuptools + (package + (inherit python-packaging-bootstrap) + (name "python-packaging-from-setuptools") + (arguments + `(#:python ,python-toolchain-only-setuptools + ,@(package-arguments python-packaging-bootstrap))) + (propagated-inputs + `(("python-pyparsing" ,python-pyparsing-from-setuptools) + ("python-six" ,python-six-from-setuptools))))) + +(define-public python-pyparsing-from-setuptools + (package + (inherit python-pyparsing) + (name "python-pyparsing-from-setuptools") + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools + ,@(package-arguments python-pyparsing))))) + +(define-public python-six-from-setuptools + (package + (inherit python-six-bootstrap) + (name "python-six-from-setuptools") + (arguments + `(#:python ,python-toolchain-only-setuptools + ,@(package-arguments python-six-bootstrap))))) + +(define-public python-wheel-from-setuptools + (package + (inherit python-wheel) + (name "python-wheel-from-setuptools") + (arguments + `(#:python ,python-toolchain-only-setuptools + ,@(package-arguments python-wheel))))) + diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index b2ec486d7a..38a1ebe49f 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -183,7 +183,7 @@ (list "--enable-shared" ;allow embedding "--with-system-expat" ;for XML support "--with-system-ffi" ;build ctypes - "--with-ensurepip=install" ;install pip and setuptools + "--with-ensurepip=no" ;do not install pip and setuptools "--enable-unicode=ucs4" ;; Prevent the installed _sysconfigdata.py from retaining a reference diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm index efade6f74b..4a23b42628 100644 --- a/guix/build-system/python.scm +++ b/guix/build-system/python.scm @@ -67,8 +67,8 @@ extension, such as '.tar.gz'." (define (default-python) "Return the default Python package." ;; Lazily resolve the binding to avoid a circular dependency. - (let ((python (resolve-interface '(gnu packages python)))) - (module-ref python 'python-wrapper))) + (let ((python (resolve-interface '(gnu packages python-commencement)))) + (module-ref python 'python-toolchain-for-build))) (define (default-python2) "Return the default Python 2 package." @@ -172,9 +172,9 @@ pre-defined variants." (define* (python-build name inputs #:key source (tests? #t) + (configure-flags ''()) (test-target "test") (use-setuptools? #t) - (configure-flags ''()) (phases '%standard-phases) (outputs '("out")) (search-paths '()) diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm index 5b1339d14c..fbc90d3655 100644 --- a/guix/build/python-build-system.scm +++ b/guix/build/python-build-system.scm @@ -34,6 +34,7 @@ #:use-module (ice-9 format) #:use-module (srfi srfi-1) #:use-module (srfi srfi-26) + #:use-module (srfi srfi-35) #:export (%standard-phases add-installed-pythonpath site-packages @@ -108,30 +109,17 @@ ;; "--single-version-externally-managed" is set, thus the .egg-info directory ;; and the scripts defined in entry-points will always be created. +;; Base error type. +(define-condition-type &python-build-error &error + python-build-error?) -(define setuptools-shim - ;; Run setup.py with "setuptools" being imported, which will patch - ;; "distutils". This is needed for packages using "distutils" instead of - ;; "setuptools" since the former does not understand the - ;; "--single-version-externally-managed" flag. - ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py - (string-append - "import setuptools, tokenize;__file__='setup.py';" - "f=getattr(tokenize, 'open', open)(__file__);" - "code=f.read().replace('\\r\\n', '\\n');" - "f.close();" - "exec(compile(code, __file__, 'exec'))")) - -(define (call-setuppy command params use-setuptools?) - (if (file-exists? "setup.py") - (begin - (format #t "running \"python setup.py\" with command ~s and parameters ~s~%" - command params) - (if use-setuptools? - (apply invoke "python" "-c" setuptools-shim - command params) - (apply invoke "python" "./setup.py" command params))) - (error "no setup.py found"))) +;; Raised when 'check cannot find a valid test system in the inputs. +(define-condition-type &test-system-not-found &python-build-error + test-system-not-found?) + +;; Raised when multiple wheels are created by 'build. +(define-condition-type &cannot-extract-multiple-wheels &python-build-error + cannot-extract-multiple-wheels?) (define* (sanity-check #:key tests? inputs outputs #:allow-other-keys) "Ensure packages depending on this package via setuptools work properly, @@ -142,23 +130,51 @@ without errors." (with-directory-excursion "/tmp" (invoke "python" sanity-check.py (site-packages inputs outputs))))) -(define* (build #:key use-setuptools? #:allow-other-keys) +(define* (build #:key outputs #:allow-other-keys) "Build a given Python package." - (call-setuppy "build" '() use-setuptools?) + + (define pyproject-build (which "pyproject-build")) + + (define (build-pep517) + ;; XXX: should probably use a different path, outside of source directory, + ;; maybe secondary output “wheel”? + (mkdir-p "dist") + (invoke pyproject-build "--outdir" "dist" "--no-isolation" "--wheel" ".")) + + ;; XXX Would be nice, if we could use bdist_wheel here to remove extra + ;; code path in 'install, but that depends on python-wheel. + (define (build-setuptools) + (invoke "python" "setup.py" "build")) + + (if pyproject-build + (build-pep517) + (build-setuptools)) #t) -(define* (check #:key tests? test-target use-setuptools? #:allow-other-keys) +(define* (check #:key inputs outputs tests? #:allow-other-keys) "Run the test suite of a given Python package." (if tests? - ;; Running `setup.py test` creates an additional .egg-info directory in - ;; build/lib in some cases, e.g. if the source is in a sub-directory - ;; (given with `package_dir`). This will by copied to the output, too, - ;; so we need to remove. - (let ((before (find-files "build" "\\.egg-info$" #:directories? #t))) - (call-setuppy test-target '() use-setuptools?) - (let* ((after (find-files "build" "\\.egg-info$" #:directories? #t)) - (inter (lset-difference string=? after before))) - (for-each delete-file-recursively inter))) + ;; Unfortunately with PEP 517 there is no common method to specify test + ;; systems. Guess test system based on inputs instead. + (let ((pytest (which "pytest")) + (have-setup-py (file-exists? "setup.py"))) + ;; Prefer pytest + ;; XXX: support nose + (cond + (pytest + (begin + (format #t "using pytest~%") + (invoke pytest "-vv"))) ; XXX: support skipping tests based on name/extra arguments? + ;; But fall back to setup.py, which should work for most + ;; packages. XXX: would be nice not to depend on setup.py here? fails + ;; more often than not to find any tests at all. Maybe we can run + ;; `python -m unittest`? + (have-setup-py + (begin + (format #t "using setup.py~%") + (invoke "python" "setup.py" "test" "-v"))) + ;; The developer should explicitly disable tests in this case. + (#t (raise (condition (&test-system-not-found)))))) (format #t "test suite not run~%")) #t) @@ -195,31 +211,109 @@ running checks after installing the package." "/bin:" (getenv "PATH")))) -(define* (install #:key inputs outputs (configure-flags '()) use-setuptools? - #:allow-other-keys) - "Install a given Python package." - (let* ((out (python-output outputs)) - (python (assoc-ref inputs "python")) - (major-minor (map string->number - (take (string-split (python-version python) #\.) 2))) - (<3.7? (match major-minor - ((major minor) - (or (< major 3) (and (= major 3) (< minor 7)))))) - (params (append (list (string-append "--prefix=" out) - "--no-compile") - (if use-setuptools? - ;; distutils does not accept these flags - (list "--single-version-externally-managed" - "--root=/") - '()) - configure-flags))) - (call-setuppy "install" params use-setuptools?) - ;; Rather than produce potentially non-reproducible .pyc files on Pythons - ;; older than 3.7, whose 'compileall' module lacks the - ;; '--invalidation-mode' option, do not generate any. - (unless <3.7? - (invoke "python" "-m" "compileall" "--invalidation-mode=unchecked-hash" - out)))) +(define* (install #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Install a wheel file according to PEP 427" + ;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl + (let* ((site-dir (site-packages inputs outputs)) + (out (assoc-ref outputs "out"))) + (define (extract file) + "Extract wheel (ZIP file) into site-packages directory" + ;; Use Python’s zipfile to avoid extra dependency + (invoke "python" "-m" "zipfile" "-e" file site-dir)) + + (define python-hashbang + (string-append "#!" (assoc-ref inputs "python") "/bin/python")) + + (define (move-data source destination) + (mkdir-p (dirname destination)) + (rename-file source destination)) + + (define (move-script source destination) + "Move executable script file from .data/scripts to out/bin and replace +temporary hashbang" + (move-data source destination) + ;; ZIP does not save/restore permissions, make executable + ;; XXX: might not be a file, but directory with subdirectories + (chmod destination #o755) + (substitute* destination (("#!python") python-hashbang))) + + ;; Python’s distutils.command.install defines this mapping from source to + ;; destination mapping. + (define install-schemes + `(("scripts" "bin" ,move-script) + ;; XXX: Why does Python not use share/ here? + ("data" "share" ,move-data))) + + (define (expand-data-directory directory) + "Move files from all .data subdirectories to their respective +destinations." + (for-each + (match-lambda ((source destination function) + (let ((source-path (string-append directory "/" source)) + (destination-path (string-append out "/" destination))) + (when (file-exists? source-path) + (begin + ;; This assumes only files exist in the scripts/ directory. + (for-each + (lambda (file) + (apply + function + (list + (string-append source-path "/" file) + (string-append destination-path "/" file)))) + (scandir source-path (negate (cut member <> '("." ".."))))) + (rmdir source-path)))))) + install-schemes)) + + (define pyproject-build (which "pyproject-build")) + + (define (list-directories base predicate) + ;; Cannot use find-files here, because it’s recursive. + (scandir + base + (lambda (name) + (let ((stat (lstat (string-append base "/" name)))) + (and + (not (member name '("." ".."))) + (eq? (stat:type stat) 'directory) + (predicate name stat)))))) + + (define (install-pep517) + "Install a wheel generated by a PEP 517-compatible builder." + (let ((wheels (find-files "dist" "\\.whl$"))) ; XXX: do not recurse + (when (> (length wheels) 1) ; This code does not support multiple wheels + ; yet, because their outputs would have to be + ; merged properly. + (raise (condition (&cannot-extract-multiple-wheels)))) + (for-each extract wheels)) + (let ((datadirs (map + (cut string-append site-dir "/" <>) + (list-directories site-dir (file-name-predicate "\\.data$"))))) + (for-each (lambda (directory) + (expand-data-directory directory) + (rmdir directory)) + datadirs))) + + (define (install-setuptools) + "Install using setuptools." + (let ((out (assoc-ref outputs "out"))) + (invoke "python" "setup.py" + "install" + "--prefix" out + "--single-version-externally-managed" + "--root=/"))) + + (if pyproject-build + (install-pep517) + (install-setuptools)) + #t)) + +(define* (compile-bytecode #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Compile installed byte-code in site-packages." + (let ((site-dir (site-packages inputs outputs))) + (invoke "python" "-m" "compileall" site-dir) + ;; XXX: We could compile with -O and -OO too here, at the cost of more space. + #t)) (define* (wrap #:key inputs outputs #:allow-other-keys) (define (list-of-files dir) @@ -243,29 +337,12 @@ running checks after installing the package." files))) bindirs))) -(define* (rename-pth-file #:key name inputs outputs #:allow-other-keys) - "Rename easy-install.pth to NAME.pth to avoid conflicts between packages -installed with setuptools." - ;; Even if the "easy-install.pth" is not longer created, we kept this phase. - ;; There still may be packages creating an "easy-install.pth" manually for - ;; some good reason. - (let* ((site-packages (site-packages inputs outputs)) - (easy-install-pth (string-append site-packages "/easy-install.pth")) - (new-pth (string-append site-packages "/" name ".pth"))) - (when (file-exists? easy-install-pth) - (rename-file easy-install-pth new-pth)))) - -(define* (ensure-no-mtimes-pre-1980 #:rest _) - "Ensure that there are no mtimes before 1980-01-02 in the source tree." - ;; Rationale: patch-and-repack creates tarballs with timestamps at the POSIX - ;; epoch, 1970-01-01 UTC. This causes problems with Python packages, - ;; because Python eggs are ZIP files, and the ZIP format does not support - ;; timestamps before 1980. - (let ((early-1980 315619200)) ; 1980-01-02 UTC - (ftw "." (lambda (file stat flag) - (unless (<= early-1980 (stat:mtime stat)) - (utime file early-1980 early-1980)) - #t)))) +(define* (set-SOURCE-DATE-EPOCH #:rest _) + "Set the 'SOURCE_DATE_EPOCH' environment variable. This is used by tools +that incorporate timestamps as a way to tell them to use a fixed timestamp. +See https://reproducible-builds.org/specs/source-date-epoch/." + (setenv "SOURCE_DATE_EPOCH" "315619200") ;; python-wheel respects this variable and sets pre-1980 times on files in zip files, which is unsupported + #t) (define* (enable-bytecode-determinism #:rest _) "Improve determinism of pyc files." @@ -292,11 +369,11 @@ by Cython." ;; prefix directory. The check phase is moved after the installation phase ;; to ease testing the built package. (modify-phases gnu:%standard-phases - (add-after 'unpack 'ensure-no-mtimes-pre-1980 ensure-no-mtimes-pre-1980) - (add-after 'ensure-no-mtimes-pre-1980 'enable-bytecode-determinism + (add-after 'unpack 'enable-bytecode-determinism enable-bytecode-determinism) (add-after 'enable-bytecode-determinism 'ensure-no-cythonized-files ensure-no-cythonized-files) + (replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH) (delete 'bootstrap) (delete 'configure) ;not needed (replace 'build build) @@ -308,7 +385,7 @@ by Cython." (add-after 'add-install-to-path 'wrap wrap) (add-after 'wrap 'check check) (add-after 'check 'sanity-check sanity-check) - (add-before 'strip 'rename-pth-file rename-pth-file))) + (add-before 'check 'compile-bytecode compile-bytecode))) (define* (python-build #:key inputs (phases %standard-phases) #:allow-other-keys #:rest args) -- 2.26.3 --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0006-gnu-Add-python-pytoml.patch" >From 881a8e817a1fe21a39bcfdc3e5be2d2ac7760cb9 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:17:10 +0100 Subject: [PATCH 06/14] gnu: Add python-pytoml. * gnu/packages/python-build.scm (python-pytoml): Add new variable. --- gnu/packages/python-build.scm | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index bbd273e5de..238fc8b3a5 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -191,3 +191,20 @@ implementation developed for Poetry. This project is intended to be a light weight, fully compliant, self-contained package allowing PEP 517 compatible build front-ends to build Poetry managed projects.") (license license:expat))) + +(define-public python-pytoml + (package + (name "python-pytoml") + (version "0.1.21") + (source + (origin + (method url-fetch) + (uri (pypi-uri "pytoml" version)) + (sha256 + (base32 + "1rv1byiw82k7mj6aprcrqi2vdabs801y97xhfnrz7kxds34ggv4f")))) + (build-system python-build-system) + (home-page "https://github.com/avakar/pytoml") + (synopsis "A parser for TOML-0.4.0") + (description "A parser for TOML-0.4.0") + (license license:expat))) -- 2.26.3 --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0007-gnu-Add-python-flit-core.patch" >From 545fa78c3a42f5cc22ccc9ea70e0ff1f9bcebe5c Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:18:17 +0100 Subject: [PATCH 07/14] gnu: Add python-flit-core. * gnu/packages/python-build.scm (python-flit-core): New variable. --- gnu/packages/python-build.scm | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 238fc8b3a5..8940e6490b 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -208,3 +208,30 @@ compatible build front-ends to build Poetry managed projects.") (synopsis "A parser for TOML-0.4.0") (description "A parser for TOML-0.4.0") (license license:expat))) + +(define-public python-flit-core + (package + (name "python-flit-core") + (version "3.0.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "flit_core" version)) + (sha256 + (base32 + "0bbw84r33gwi0xyp7m8dzp2dzpjs4harj3l5wrbxkmp2awh0ard4")))) + (build-system python-build-system) + (arguments + `(;; No tests. + #:tests? #f)) + (propagated-inputs + `(("python-pytoml" ,python-pytoml))) + (home-page "https://github.com/takluyver/flit") + (synopsis + "Simplified packaging of Python modules, distribution-building parts") + (description + "Flit is a simple way to put Python packages and modules on PyPI. It +tries to require less thought about packaging and help you avoid common +mistakes. Distribution-building parts of Flit.") + (license license:bsd-3))) + -- 2.26.3 --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0008-gnu-python-pep517-bootstrap-Build-using-flit-core.patch" >From 4ffe19ccc9b6d077c811cfbdd715d10a79730e8f Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:20:48 +0100 Subject: [PATCH 08/14] gnu: python-pep517-bootstrap: Build using flit-core. * gnu/packages/python-build.scm (python-pep517-bootstrap) [arguments]: Relax dependency on flit-core version. [native-inputs]: Add flit-core. --- gnu/packages/python-build.scm | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 8940e6490b..c53d49c287 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -128,10 +128,21 @@ Language (TOML) configuration files.") "0zqidxah03qpnp6zkg3zd1kmd5f79hhdsfmlc0cldaniy80qddxf")))) (build-system python-build-system) (arguments - `(#:tests? #f)) ;to avoid circular dependencies + `(#:tests? #f ; To avoid circular dependencies. + #:phases + (modify-phases %standard-phases + (add-after 'unpack 'relax-dependency + (lambda _ + (substitute* "pyproject.toml" + (("flit_core >=2,<3") + "flit_core >=2,<4")) + #t))))) (propagated-inputs `(("python-toml" ,python-toml) ("python-wheel" ,python-wheel))) + (native-inputs + `(;; Build system. + ("python-flit-core" ,python-flit-core))) (home-page "https://github.com/pypa/pep517") (synopsis "Wrappers to build Python packages using PEP 517 hooks") (description -- 2.26.3 --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0009-gnu-python-iniconfig-Add-missing-build-input.patch" >From f1eea04f204dc0fef64675ac2605f42535aa0815 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:23:53 +0100 Subject: [PATCH 09/14] gnu: python-iniconfig: Add missing build input. * gnu/packages/python-xyz.scm (python-iniconfig) [native-inputs] Add setuptools-scm. --- gnu/packages/python-xyz.scm | 1 + 1 file changed, 1 insertion(+) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index 8f472dea42..47466d31f0 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -17110,6 +17110,7 @@ in other versions.") (base32 "0ckzngs3scaa1mcfmsi1w40a1l8cxxnncscrxzjjwjyisx8z0fmw")))) (build-system python-build-system) + (native-inputs `(("python-setuptools-scm" ,python-setuptools-scm))) (home-page "https://github.com/RonnyPfannschmidt/iniconfig") (synopsis "Simple INI-file parser") (description "The @code{iniconfig} package provides a small and simple -- 2.26.3 --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0010-gnu-Add-python-u-msgpack.patch" >From 271997517b2b03aec44e8b158c911412caccf7bf Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 14:16:07 +0100 Subject: [PATCH 10/14] gnu: Add python-u-msgpack. * gnu/packages/python-xyz.scm (python-u-msgpack): New variable. The redundant -python postfix from the original package name has been removed. --- gnu/packages/python-xyz.scm | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index 47466d31f0..fdc9bd85b7 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -25575,3 +25575,27 @@ is the cythonized version of @code{fractions.Fraction}.") "@code{pathvalidate} is a Python library to sanitize/validate strings representing paths or filenames.") (license license:expat))) + +(define-public python-u-msgpack + (package + (name "python-u-msgpack") + (version "2.7.1") + (source + (origin + (method url-fetch) + (uri (pypi-uri "u-msgpack-python" version)) + (sha256 + (base32 + "0lcmlr7gc4dydpxn6l5bdcq40c3ghf8mv1sjqyj72wdpr8rx9rxp")))) + (build-system python-build-system) + (home-page + "https://github.com/vsergeev/u-msgpack-python") + (synopsis + "Portable, lightweight MessagePack serializer and deserializer") + (description + "A portable, lightweight MessagePack serializer and deserializer written +in pure Python. u-msgpack-python is fully compliant with the latest MessagePack +specification. In particular, it supports the new binary, UTF-8 string, +application-defined ext, and timestamp types.") + (license license:expat))) + -- 2.26.3 --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0011-gnu-Add-python-pytest-expect.patch" >From 77f5e2e445e01c3ce7b1c9484ca27ba86b9d5def Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 14:20:03 +0100 Subject: [PATCH 11/14] gnu: Add python-pytest-expect. * gnu/packages/python-check.scm (python-pytest-expect): New variable. --- gnu/packages/python-check.scm | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/gnu/packages/python-check.scm b/gnu/packages/python-check.scm index ff29f5d7ce..80fb5f704a 100644 --- a/gnu/packages/python-check.scm +++ b/gnu/packages/python-check.scm @@ -1515,3 +1515,31 @@ allows one to create a set of tests using @emph{pairwise combinations} method, reducing a number of combinations of variables into a lesser set that covers most situations.") (license license:expat))) + +(define-public python-pytest-expect + (package + (name "python-pytest-expect") + (version "1.1.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "pytest-expect" version)) + (sha256 + (base32 + "0iyq3zd1g5ffaz2wv6mskjfn84sfbyh0j209glcrh1s50hkldd1n")))) + (build-system python-build-system) + (arguments `(#:tests? #f)) ; no tests via pypi + (propagated-inputs + `(("python-pytest" ,python-pytest) ; package declares this dependency + ("python-u-msgpack" ,python-u-msgpack))) + (home-page + "https://github.com/gsnedders/pytest-expect") + (synopsis + "Py.test plugin to store test expectations and mark tests based on them") + (description + "A py.test plugin that stores test expectations by saving the set of +failing tests, allowing them to be marked as xfail when running them in future. +The tests expectations are stored such that they can be distributed alongside +the tests.") + (license license:expat))) + -- 2.26.3 --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0012-gnu-python-html5lib-Fix-tests-with-pytest-6.patch" >From 2e2eb3c05f285f3bfc4c990309584eb98503cd91 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 14:23:07 +0100 Subject: [PATCH 12/14] gnu: python-html5lib: Fix tests with pytest 6. * gnu/packages/python-web.scm (python-html5lib) [source]: Add upstream patch. [native-inputs]: Add test dependencies. [arguments]: Remove unsupported #:test-target. --- gnu/packages/python-web.scm | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm index 8b31368424..8ab783bbb3 100644 --- a/gnu/packages/python-web.scm +++ b/gnu/packages/python-web.scm @@ -1079,7 +1079,20 @@ storage.") (uri (pypi-uri "html5lib" version)) (sha256 (base32 - "0vqlhk0hgbsfkh7ybmby93xhlx8dq6pr5blf356ka3z2c41b9rdj")))) + "0vqlhk0hgbsfkh7ybmby93xhlx8dq6pr5blf356ka3z2c41b9rdj")) + (patches + (list + ;; Adds Pytest 6 support. + (origin + (method url-fetch) + (uri (string-append + "https://github.com/html5lib/" + "html5lib-python/commit/" + "2c19b9899ab3a3e8bd0ca35e5d78544334204169.patch")) + (file-name "python-html5lib-support-pytest6.patch") + (sha256 + (base32 + "0jg2ry0439q8n7j1mf4p2hdq54i704pq9scv4wwa2pp3cwvb6dvg"))))))) (build-system python-build-system) (propagated-inputs `(("python-six" ,python-six) @@ -1088,6 +1101,9 @@ storage.") ("python-chardet" ,python-chardet))) (arguments `(#:test-target "check")) + (native-inputs + `(("python-pytest" ,python-pytest) + ("python-pytest-expect" ,python-pytest-expect))) (home-page "https://github.com/html5lib/html5lib-python") (synopsis -- 2.26.3 --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0013-gnu-python-libxml2-Fix-build.patch" >From 312ee407e5d878da99098753ba24641c2c468451 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 17:18:50 +0100 Subject: [PATCH 13/14] gnu: python-libxml2: Fix build. * gnu/packages/xml.scm (python-libxml2) [#:phases]: Replace setuptools and do not exit at the end of setup.py. --- gnu/packages/xml.scm | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/gnu/packages/xml.scm b/gnu/packages/xml.scm index 9accd08a7e..9d93d21e27 100644 --- a/gnu/packages/xml.scm +++ b/gnu/packages/xml.scm @@ -370,7 +370,10 @@ It uses libxml2 to access the XML files.") (format #f "ROOT = r'~a'" libxml2)) ;; For 'iconv.h'. (("/opt/include") - (string-append glibc "/include")))) + (string-append glibc "/include")) + (("distutils\\.core") "setuptools") + ;; python-pypa-build does not like it if setup.py exits. + (("sys\\.exit\\(0\\)") ""))) #t))))) (inputs `(("libxml2" ,libxml2))) (synopsis "Python bindings for the libxml2 library"))) -- 2.26.3 --lI7hEO8aOAzcKIP4 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0014-gnu-scons-Remove-obsolete-argument.patch" >From ec72099a90b0fc7f6538f578ab3e28411391dacb Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 17:21:35 +0100 Subject: [PATCH 14/14] gnu: scons: Remove obsolete argument. * gnu/packages/python-xyz.scm (scons) [arguments]: Remove #:use-setuptools? argument. --- gnu/packages/python-xyz.scm | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index fdc9bd85b7..a61451f8ea 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -2641,8 +2641,7 @@ and is not compatible with JSON.") "1xy8jrwz87y589ihcld4hv7wn122sjbz914xn8h50ww77wbhk8hn")))) (build-system python-build-system) (arguments - `(#:use-setuptools? #f ; still relies on distutils - #:tests? #f ; no 'python setup.py test' command + `(#:tests? #f ; no 'python setup.py test' command #:phases (modify-phases %standard-phases (add-before 'build 'bootstrap -- 2.26.3 --lI7hEO8aOAzcKIP4-- From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] Questions regarding Python packaging Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 06 Jun 2021 19:45:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Tanguy LE CARROUR Cc: guix-devel@gnu.org, 46848@debbugs.gnu.org Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.162300868915833 (code B ref 46848); Sun, 06 Jun 2021 19:45:02 +0000 Received: (at 46848) by debbugs.gnu.org; 6 Jun 2021 19:44:49 +0000 Received: from localhost ([127.0.0.1]:53957 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lpyhV-00047J-05 for submit@debbugs.gnu.org; Sun, 06 Jun 2021 15:44:49 -0400 Received: from mout-p-201.mailbox.org ([80.241.56.171]:13890) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lpyhR-000474-Q6 for 46848@debbugs.gnu.org; Sun, 06 Jun 2021 15:44:47 -0400 Received: from smtp1.mailbox.org (smtp1.mailbox.org [80.241.60.240]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-201.mailbox.org (Postfix) with ESMTPS id 4Fyn5M1VpmzQk3L; Sun, 6 Jun 2021 21:44:39 +0200 (CEST) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1623008677; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J5bpL8cW7tMxPOVbMmAz4hyHYIg9U31uuqEvsh3eqSU=; b=k9YbVyrFIx04K9qzG2cKgBYIsxLjqBjcFb5dKWo+xdHHoaLrKBt10EmX8ioSmwo9F+eud5 tJoTUkrsiHtmWBVh0JP5nAt/2cqJcb2slDLaQa9qDknKp4FsjxI59Hpo+T+3C/V32kGX0M Bw66DC/zzvYHhwOv/mmZSY+4XZJ1CqVP9kvZYI62FI2wz0qV0pNwPA6NuFMo5JHQiu5Nl0 doz5NC2AbQ2VwyHi2ukA91C/fmRxRS6/pTpFeNQ5uPfV4qTt3KyzEDjoATA9a7av6lrwv+ s+pLHd+NYI2ZXfPpCUorD0XV98dbI1k7TbYDUsVT7Uz5Hgt3nyHHUlEes2KOLg== Received: from smtp1.mailbox.org ([80.241.60.240]) by spamfilter06.heinlein-hosting.de (spamfilter06.heinlein-hosting.de [80.241.56.125]) (amavisd-new, port 10030) with ESMTP id hxWPYYFpoO12; Sun, 6 Jun 2021 21:44:36 +0200 (CEST) Date: Sun, 6 Jun 2021 21:44:33 +0200 From: Lars-Dominik Braun Message-ID: References: <20201108142717.lmud5h4gh44vtjc6@melmoth> <1609946775.8blxygrg9p.astroid@rafflesia.none> <1611303651.35tpgtn1z1.astroid@melmoth.none> <1622997703.qcpe1ehxem.astroid@melmoth.none> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1622997703.qcpe1ehxem.astroid@melmoth.none> X-MBO-SPAM-Probability: X-Rspamd-Score: -3.74 / 15.00 / 15.00 X-Rspamd-Queue-Id: 053AB1887 X-Rspamd-UID: 963543 X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi Tanguy, (cross-posting this to the issue itself too) > Sorry if I'm (very) late, but apprently this hasn't made it to master > yet, so… What the status? Do you still need a willing-but-maybe-not-qualified > person to review or discuss your patch? the patch set works, I can build many Python packages, although some require changes. Still, multiple things need to be done before merging is possible imo: 1) Validate the general idea of using pypa-build is viable and sustainable in the long run – ideally through review by someone else than me. We can’t touch python-build-system every week to solve structural issues, so it needs to be bullet-proof. 2) Figure out how to run testing code. Currently python-build-system just picks pytest, if available – not sure this is the best option we have. How do we deal with other test systems? How do we pass options? 3) Determine the fate of Python 2, which is probably broken through this patch set. Shall we remove it entirely? Is it worth to keep support? 4) Iron out minor details like including pip in the python package or create a new python-toolchain package? What do we include in that meta-package? pip? virtualenv? …? 5) Fix my awkward Scheme code, especially regarding unpacking of the built wheels. Should we be using Python’s unzip module or can be assumed unzip is available in the build environment? (Should we add it?) I’m by no means a Python packaging expert, so any help would be appreciated, even if it’s just a question or thumbs-up/thumbs-down on my ideas. Cheers, Lars From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] Questions regarding Python packaging Resent-From: Hartmut Goebel Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 22 Jun 2021 07:02:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Lars-Dominik Braun Cc: guix-devel@gnu.org, Tanguy LE CARROUR , 46848@debbugs.gnu.org Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.162434526412957 (code B ref 46848); Tue, 22 Jun 2021 07:02:02 +0000 Received: (at 46848) by debbugs.gnu.org; 22 Jun 2021 07:01:04 +0000 Received: from localhost ([127.0.0.1]:37641 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvaPA-0003MK-0v for submit@debbugs.gnu.org; Tue, 22 Jun 2021 03:01:04 -0400 Received: from mail-out.m-online.net ([212.18.0.10]:44158) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lvaP6-0003Hx-G0 for 46848@debbugs.gnu.org; Tue, 22 Jun 2021 03:01:02 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 4G8HNq1NSDz1s6gn; Tue, 22 Jun 2021 09:00:59 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 4G8HNq0nWmz1qqkQ; Tue, 22 Jun 2021 09:00:59 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id zZJNOq6Yg3wW; Tue, 22 Jun 2021 09:00:57 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-52-81.dynamic.mnet-online.de [188.174.52.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS; Tue, 22 Jun 2021 09:00:57 +0200 (CEST) Received: from thisbe.goebel-consult.de (hermia.goebel-consult.de [192.168.110.7]) by hermia.goebel-consult.de (Postfix) with ESMTP id 7DFAE600F9; Tue, 22 Jun 2021 09:00:57 +0200 (CEST) References: <20201108142717.lmud5h4gh44vtjc6@melmoth> <1609946775.8blxygrg9p.astroid@rafflesia.none> <1611303651.35tpgtn1z1.astroid@melmoth.none> <1622997703.qcpe1ehxem.astroid@melmoth.none> From: Hartmut Goebel Organization: crazy-compilers.com Message-ID: <520a5492-6467-bbfc-3252-f17a5cc5d16f@crazy-compilers.com> Date: Tue, 22 Jun 2021 09:00:54 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------78C137176FE39D2B54BEEB94" Content-Language: en-US X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) This is a multi-part message in MIME format. --------------78C137176FE39D2B54BEEB94 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Hi Lars, sorry for being late for commenting on this (the time I can spend on guix is rather limited atm). Here are some general remarks on this patch-set (in order of appearance): * Not installing pip by default might break some user's environments. Anyhow, since using pip in guix is not such a good idea anyway, this should be okay. * "use-setuptools" is gone. There are still about 10 packages with "#:use-setuptools #f" - which means they are (expected to be) incompatible with setuptools for some reason. You might want to check whether these packages actually still can't be packages with setuptools. * setuptools-shim has been removed. I don't think this is a good idea, since this peace of code enforces packages to be actually build with setuptools instead of old distutils. This code is still in current pip, so I assume it is still required. (This shim ensures setuptools is used, even if setup.py only imports distutils. And setuptools is required for some options like ""--single-version-externally-managed" - as the comment for the shim says.) * set-SOURCE-DATE-EPOCH: Please keep the verbose rational. It's much more helpful than the new one-line comment. * set-SOURCE-DATE-EPOCH: This implementation makes the code depend on wheel and wheel being used for installation. * Why has rename-pth-file been removed? Are you sure .pth-files are never created anymore nowerdays? * python-hashbang: Isn't this done already by the normal "patch-shebangs" phase after install in  gnu-build-system? (BTW: these are called *she*bangs). * I suggest to have phase compile-bytecode still honor older versions of python > 1) Validate the general idea of using pypa-build is viable and > sustainable in the long run – ideally through review by someone else > than me. We can’t touch python-build-system every week to solve > structural issues, so it needs to be bullet-proof. pypa bulld is where the PyPA is pushing towards. Anyhow, as of today, as far as I can see, adoption is low. > 2) Figure out how to run testing code. Currently python-build-system > just picks pytest, if available – not sure this is the best option we > have. How do we deal with other test systems? How do we pass options? AFAIK fhere is no standard way for running tests in python. pytest seems to be the most modern test-system. Anyhow packages still use nose or tox (which again might run pytest or nose, with parameters fetched from tox.ini). So I'm afraid, there is no general rule. Did the PyPA publish some recommendations or PEP on this? > 4) Iron out minor details like including pip in the python package or > create a new python-toolchain package? What do we include in that > meta-package? pip? virtualenv? …? As I Python developer I nowerdays would expect pip and venv (which is part of the std-lib - but not the virualenv, which is a separate module) to be availalbe when installing "python". Anyhow I could live with pip being a separate package. "python-toolchain" sounds oversized for me. Would this include the C-compiler, too (which one? maybe I want to build cross). I'd rather not have such a package. > 5) Fix my awkward Scheme code, especially regarding unpacking of the > built wheels. Should we be using Python’s unzip module or can be > assumed unzip is available in the build environment? (Should we add > it?) The gnu-build-system already provides the "unzip" binary (used in phase "unpack"). So we could simply use this. Otherwise I recommend using the Python zip module, as this is what is used for creating the zip-archives :-) -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | --------------78C137176FE39D2B54BEEB94 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit
Hi Lars,

sorry for being late for commenting on this (the time I can spend on guix is rather limited atm).

Here are some general remarks on this patch-set (in order of appearance):

  • Not installing pip by default might break some user's environments. Anyhow, since using pip in guix is not such a good idea anyway, this should be okay.

  • "use-setuptools" is gone. There are still about 10 packages with "#:use-setuptools #f" - which means they are (expected to be) incompatible with setuptools for some reason. You might want to check whether these packages actually still can't be packages with setuptools.

  • setuptools-shim has been removed. I don't think this is a good idea, since this peace of code enforces packages to be actually build with setuptools instead of old distutils. This code is still in current pip, so I assume it is still required.

(This shim ensures setuptools is used, even if setup.py only imports distutils. And setuptools is required for some options like ""--single-version-externally-managed" - as the comment for the shim says.)

  • set-SOURCE-DATE-EPOCH: Please keep the verbose rational. It's much more helpful than the new one-line comment.

  • set-SOURCE-DATE-EPOCH: This implementation makes the code depend on wheel and wheel being used for installation.

  • Why has rename-pth-file been removed? Are you sure .pth-files are never created anymore nowerdays?

  • python-hashbang: Isn't this done already by the normal "patch-shebangs" phase after install in  gnu-build-system? (BTW: these are called *she*bangs).

  • I suggest to have phase compile-bytecode still honor older versions of python 


1) Validate the general idea of using pypa-build is viable and
   sustainable in the long run – ideally through review by someone else
   than me. We can’t touch python-build-system every week to solve
   structural issues, so it needs to be bullet-proof.

pypa bulld is where the PyPA is pushing towards. Anyhow, as of today, as far as I can see, adoption is low.

2) Figure out how to run testing code. Currently python-build-system
   just picks pytest, if available – not sure this is the best option we
   have. How do we deal with other test systems? How do we pass options?

AFAIK fhere is no standard way for running tests in python. pytest seems to be the most modern test-system. Anyhow packages still use nose or tox (which again might run pytest or nose, with parameters fetched from tox.ini). So I'm afraid, there is no general rule.

Did the PyPA publish some recommendations or PEP on this?

4) Iron out minor details like including pip in the python package or
   create a new python-toolchain package? What do we include in that
   meta-package? pip? virtualenv? …?

As I Python developer I nowerdays would expect pip and venv (which is part of the std-lib - but not the virualenv, which is a separate module) to be availalbe when installing "python". Anyhow I could live with pip being a separate package.

"python-toolchain" sounds oversized for me. Would this include the C-compiler, too (which one? maybe I want to build cross). I'd rather not have such a package. 

5) Fix my awkward Scheme code, especially regarding unpacking of the
   built wheels. Should we be using Python’s unzip module or can be
   assumed unzip is available in the build environment? (Should we add
   it?)
The gnu-build-system already provides the "unzip" binary (used in phase "unpack"). So we could simply use this. Otherwise I recommend using the Python zip module, as this is what is used for creating the zip-archives :-) 
-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |
--------------78C137176FE39D2B54BEEB94-- From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] Questions regarding Python packaging Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 28 Jun 2021 12:00:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Hartmut Goebel Cc: guix-devel@gnu.org, Tanguy LE CARROUR , 46848@debbugs.gnu.org Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.162488158718761 (code B ref 46848); Mon, 28 Jun 2021 12:00:02 +0000 Received: (at 46848) by debbugs.gnu.org; 28 Jun 2021 11:59:47 +0000 Received: from localhost ([127.0.0.1]:51192 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lxpvW-0004sW-Gl for submit@debbugs.gnu.org; Mon, 28 Jun 2021 07:59:46 -0400 Received: from mout-p-202.mailbox.org ([80.241.56.172]:16006) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lxpvT-0004sE-PS for 46848@debbugs.gnu.org; Mon, 28 Jun 2021 07:59:45 -0400 Received: from smtp1.mailbox.org (smtp1.mailbox.org [IPv6:2001:67c:2050:105:465:1:1:0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-202.mailbox.org (Postfix) with ESMTPS id 4GD5kd4L0KzQjbF; Mon, 28 Jun 2021 13:59:37 +0200 (CEST) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1624881575; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O43vvCscLBCAWmXk4mcJskRqp3wXeaZ6QLKnm8sOtgg=; b=FIQ8ptnPH+f2QqwgKa6o4FP/FeJ80VfGVBpTRbDbio6s4v6m5Mpfhw8tXTZOQ4n+IW2ZSz PqWutFRfRthxcPCQ1klb9JgJp2SQmQE0WVOt2cR/dsVzS3PfSFqkJt9WXaecPxDfqZdFgW g9I5RI9D4C/1cEosl6o3S9B4HRxBWNmDTqVZWQuAKPD72qbh9GBzk4CyNn9JLd4NVcc4NJ t8dc4v2mn7xGS+nONGGQfJ1UteGEjBZX+0OEPY9iivVof20fvOVJpHr/mrVDU8rxqaCVfT 1FkWFQbkRzbXj6ds80BlhR72Ud9V0ybtVobRDFsdZWNuRqxT59MRJe7ukImQwA== Received: from smtp1.mailbox.org ([80.241.60.240]) by gerste.heinlein-support.de (gerste.heinlein-support.de [91.198.250.173]) (amavisd-new, port 10030) with ESMTP id 4Z3uC8ySDuZF; Mon, 28 Jun 2021 13:59:33 +0200 (CEST) Date: Mon, 28 Jun 2021 13:59:31 +0200 From: Lars-Dominik Braun Message-ID: References: <20201108142717.lmud5h4gh44vtjc6@melmoth> <1609946775.8blxygrg9p.astroid@rafflesia.none> <1611303651.35tpgtn1z1.astroid@melmoth.none> <1622997703.qcpe1ehxem.astroid@melmoth.none> <520a5492-6467-bbfc-3252-f17a5cc5d16f@crazy-compilers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <520a5492-6467-bbfc-3252-f17a5cc5d16f@crazy-compilers.com> X-MBO-SPAM-Probability: X-Rspamd-Score: -6.99 / 15.00 / 15.00 X-Rspamd-Queue-Id: 351451847 X-Rspamd-UID: 9e6b9b X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Hartmut, > sorry for being late for commenting on this (the time I can spend on > guix is rather limited atm). no problem, same thing on my side. > * > > Not installing pip by default might break some user's environments. > Anyhow, since using pip in guix is not such a good idea anyway, this > should be okay. True. We could rename python→python-minimal-interpreteronly (or similar; given that python-minimal already exists) and python-toolchain→python to work around that. > * > > "use-setuptools" is gone. There are still about 10 packages with > "#:use-setuptools #f" - which means they are (expected to be) > incompatible with setuptools for some reason. You might want to > check whether these packages actually still can't be packages with > setuptools. Yeah, I’ve seen those, but the number was too small to bother for now. I’ll have a look later. > * > > setuptools-shim has been removed. I don't think this is a good idea, > since this peace of code enforces packages to be actually build with > setuptools instead of old distutils. This code is still in current > pip, so I assume it is still required. > > (This shim ensures setuptools is used, even if setup.py only imports > distutils. And setuptools is required for some options like > ""--single-version-externally-managed" - as the comment for the shim > says.) Is this relevant though? I doubt many packages are still importing distutils and the few that do can be patched. > * > > set-SOURCE-DATE-EPOCH: Please keep the verbose rational. It's much > more helpful than the new one-line comment. You mean the one from the now-removed ensure-no-mtimes-pre-1980? Sure. > * > > set-SOURCE-DATE-EPOCH: This implementation makes the code depend on > wheel and wheel being used for installation. Technically it depends on the wheel builder understanding SOURCE_DATE_EPOCH (not necessarily python-wheel). I’d say that’s acceptable and it’d be preferable to fix build systems not respecting this variable imo. > * > > Why has rename-pth-file been removed? Are you sure .pth-files are > never created anymore nowerdays? Given that easy-install has been deprecated I think it’s safe to remove this phase and flag any packages creating this easy-install.pth as broken. (There are, however, legitimate packages creating files like ruamel.yaml-0.15.83-py3.8-nspkg.pth.) > * > > python-hashbang: Isn't this done already by the normal > "patch-shebangs" phase after install in  gnu-build-system? (BTW: > these are called *she*bangs). Afaik the function patch-shebang expects a leading slash and thus it does not replace this “special” shebang (see https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl; Spread, point 3). > * > > I suggest to have phase compile-bytecode still honor older versions > of python I’m not sure what you mean. compileall is also part of Python 2. > pypa bulld is where the PyPA is pushing towards. Anyhow, as of today, as > far as I can see, adoption is low. Of pypa build? That is true. > AFAIK fhere is no standard way for running tests in python. pytest seems > to be the most modern test-system. Anyhow packages still use nose or tox > (which again might run pytest or nose, with parameters fetched from > tox.ini). So I'm afraid, there is no general rule. > > Did the PyPA publish some recommendations or PEP on this? I’m not aware of any accepted PEP’s. There is a discussion about the removal of `python setup.py test`: https://github.com/pypa/setuptools/issues/931 And a proposal for pyproject.toml going nowhere: https://discuss.python.org/t/proposal-for-tests-entry-point-in-pyproject-toml/2077/2 > As I Python developer I nowerdays would expect pip and venv (which is > part of the std-lib - but not the virualenv, which is a separate module) > to be availalbe when installing "python". Anyhow I could live with pip > being a separate package. If we keep setuptools/pip bundled, we don’t have to do any of this pypa-build dance. We could also modernize python-build-system around `pip install` and just be done with it. (I don’t have a proof-of-concept for that yet.) > "python-toolchain" sounds oversized for me. Would this include the > C-compiler, too (which one? maybe I want to build cross). I'd rather not > have such a package. See suggestion above wrt renaming. > The gnu-build-system already provides the "unzip" binary (used in phase > "unpack"). So we could simply use this. Otherwise I recommend using the > Python zip module, as this is what is used for creating the zip-archives > :-) I’m using Python’s zipfile module already. Cheers, Lars From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] Questions regarding Python packaging Resent-From: Hartmut Goebel Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 28 Jun 2021 20:38:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Lars-Dominik Braun Cc: guix-devel@gnu.org, Tanguy LE CARROUR , 46848@debbugs.gnu.org Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.162491263323345 (code B ref 46848); Mon, 28 Jun 2021 20:38:02 +0000 Received: (at 46848) by debbugs.gnu.org; 28 Jun 2021 20:37:13 +0000 Received: from localhost ([127.0.0.1]:53052 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lxy0G-00064T-Gb for submit@debbugs.gnu.org; Mon, 28 Jun 2021 16:37:12 -0400 Received: from mail-out.m-online.net ([212.18.0.10]:49087) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lxy0E-00064K-6V for 46848@debbugs.gnu.org; Mon, 28 Jun 2021 16:37:11 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 4GDKCm2pzrz1s5l5; Mon, 28 Jun 2021 22:37:08 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 4GDKCm1l74z1r0wp; Mon, 28 Jun 2021 22:37:08 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id y9o-7BVejmon; Mon, 28 Jun 2021 22:37:06 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-63-88.dynamic.mnet-online.de [188.174.63.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS; Mon, 28 Jun 2021 22:37:06 +0200 (CEST) Received: from thisbe.goebel-consult.de (hermia.goebel-consult.de [192.168.110.7]) by hermia.goebel-consult.de (Postfix) with ESMTP id 6C59060082; Mon, 28 Jun 2021 22:37:21 +0200 (CEST) References: <20201108142717.lmud5h4gh44vtjc6@melmoth> <1609946775.8blxygrg9p.astroid@rafflesia.none> <1611303651.35tpgtn1z1.astroid@melmoth.none> <1622997703.qcpe1ehxem.astroid@melmoth.none> <520a5492-6467-bbfc-3252-f17a5cc5d16f@crazy-compilers.com> From: Hartmut Goebel Organization: crazy-compilers.com Message-ID: <681eb450-0185-a465-3ed1-8446e8ad0974@crazy-compilers.com> Date: Mon, 28 Jun 2021 22:37:04 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi Lars-Dominik, Am 28.06.21 um 13:59 schrieb Lars-Dominik Braun:* >> Not installing pip by default might break some user's environments. >> Anyhow, since using pip in guix is not such a good idea anyway, this >> should be okay. > True. We could rename python→python-minimal-interpreteronly (or similar; > given that python-minimal already exists) and python-toolchain→python to > work around that. What should be the use of having a package without pip? Anything else than saving a few KB? >> [setuptools-shim has been removed] > Is this relevant though? I doubt many packages are still importing > distutils and the few that do can be patched. Was I wrote: This code is still in pip, so I assume it is still relevant. I don't think patching is a good idea. It requires effort (implementing, reviewing), which can be saved by keeping exisiting and tested code. >> set-SOURCE-DATE-EPOCH: This implementation makes the code depend on >> wheel and wheel being used for installation. > Technically it depends on the wheel builder understanding > SOURCE_DATE_EPOCH (not necessarily python-wheel). I’d say that’s > acceptable and it’d be preferable to fix build systems not respecting > this variable imo. For this case please change the comment not not referring to wheel in this way. More something like "we expect the builder to support SOURCE_DATE_EPOCH, like wheel does" Anyhow, *m not actually convinced that we should throw away the old code. I can imagine in the next cuple of years quite some new build-systems to arrive, most of which will most probably not support SOURCE_DATE_EPOCH in the beginning, and thus making package's life harder. > >> Why has rename-pth-file been removed? Are you sure .pth-files are >> never created anymore nowerdays? > Given that easy-install has been deprecated I think it’s safe to remove > this phase and flag any packages creating this easy-install.pth as > broken. (There are, however, legitimate packages creating files like > ruamel.yaml-0.15.83-py3.8-nspkg.pth.) What exaclty do you mean with "flag as broken"? Will anybody (you? ;-) verify *all* current packages to not be "broken" prior to merging this change? Anyhow, again, I'm not convinced we should remove this phase now. .pth-file are deprecated only, but still supported. By removing this phase we might create conflict cased we can not foresee. And I would keep it even if one analyzes none of the current packages is "broken" - just do be on the save side fpr avoiding user trouble. (These issues will show up at the user, and are hard to track down, since noone will think about .pth files) > >> python-hashbang: Isn't this done already by the normal >> "patch-shebangs" phase after install in  gnu-build-system? (BTW: >> these are called *she*bangs). > Afaik the function patch-shebang expects a leading slash and thus it > does not replace this “special” shebang (see > https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl; > Spread, point 3). IC. Please add a comment to make this clear (e.g. "handle shebang of scripts generated by wheel missing leading slash") >> * >> >> I suggest to have phase compile-bytecode still honor older versions >> of python > I’m not sure what you mean. compileall is also part of Python 2. The old code did not compile the source for Python <3.7. Please see the comment of the old code for rational. >> As I Python developer I nowerdays would expect pip and venv (which is >> part of the std-lib - but not the virualenv, which is a separate module) >> to be availalbe when installing "python". Anyhow I could live with pip >> being a separate package. > If we keep setuptools/pip bundled, we don’t have to do any of this > pypa-build dance. We could also modernize python-build-system around > `pip install` and just be done with it. (I don’t have a proof-of-concept > for that yet.) AFAIK this might not be true if other build systems not using setuptools at all might show up. And isn't this the main reason for all your work? > >> The gnu-build-system already provides the "unzip" binary (used in phase >> "unpack"). So we could simply use this. Otherwise I recommend using the >> Python zip module, as this is what is used for creating the zip-archives >> :-) > I’m using Python’s zipfile module already. Fine, so you can safely remove the respective comment ;-) -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] Questions regarding Python packaging Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 29 Jun 2021 07:21:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Hartmut Goebel Cc: guix-devel@gnu.org, Tanguy LE CARROUR , 46848@debbugs.gnu.org Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.162495124231361 (code B ref 46848); Tue, 29 Jun 2021 07:21:02 +0000 Received: (at 46848) by debbugs.gnu.org; 29 Jun 2021 07:20:42 +0000 Received: from localhost ([127.0.0.1]:53484 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ly830-00089l-60 for submit@debbugs.gnu.org; Tue, 29 Jun 2021 03:20:42 -0400 Received: from mout-p-202.mailbox.org ([80.241.56.172]:49452) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ly82w-00089P-JR for 46848@debbugs.gnu.org; Tue, 29 Jun 2021 03:20:41 -0400 Received: from smtp2.mailbox.org (smtp2.mailbox.org [IPv6:2001:67c:2050:105:465:1:2:0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-202.mailbox.org (Postfix) with ESMTPS id 4GDbV84v4RzQk4Q; Tue, 29 Jun 2021 09:20:32 +0200 (CEST) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1624951230; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RD4G0u69JO8HtDEdMdaooKxX6ObkQ4i5xuqxYIHdDj8=; b=tRZASMCWGAFhq+BF0rKMWwAKxaXHsXAsJXABKSX3yWc/QUUwW7nfDcJJh6nseBIG8vW8R7 13hu5B+oLwdFxbVAFbpWPQNB5aVEUYt1KdHhRGjLYQu3y5EtmHh3FyjlJX4wXTIPXNpuu7 KJSj6cfHK3aI7Hp9HidOdavAM467rUmxmicKd2Zh2v2rEAOgdHASojtvebZ2jOLZifeZDx 8NIV4f58OOy4O23ms1f7Hr6Icb3wu2Wilv6liZ5t8BkT0Yod0wOSwHtR4C7Hyhxz43zcGR RN2MWWkF19d3tQapT9DsYv55V54XBHc5huVo+GQOM9uD1wzqJqcTMleIieSHAQ== Received: from smtp2.mailbox.org ([80.241.60.241]) by spamfilter01.heinlein-hosting.de (spamfilter01.heinlein-hosting.de [80.241.56.115]) (amavisd-new, port 10030) with ESMTP id o7ml6uhnTq9z; Tue, 29 Jun 2021 09:20:25 +0200 (CEST) Date: Tue, 29 Jun 2021 09:20:23 +0200 From: Lars-Dominik Braun Message-ID: References: <1609946775.8blxygrg9p.astroid@rafflesia.none> <1611303651.35tpgtn1z1.astroid@melmoth.none> <1622997703.qcpe1ehxem.astroid@melmoth.none> <520a5492-6467-bbfc-3252-f17a5cc5d16f@crazy-compilers.com> <681eb450-0185-a465-3ed1-8446e8ad0974@crazy-compilers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <681eb450-0185-a465-3ed1-8446e8ad0974@crazy-compilers.com> X-MBO-SPAM-Probability: X-Rspamd-Score: -7.24 / 15.00 / 15.00 X-Rspamd-Queue-Id: 4C45A181C X-Rspamd-UID: 3f03f3 X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Hartmut, > What should be the use of having a package without pip? Anything else > than saving a few KB? saving some space and unvendoring components that we also have separate packages for. (As I understand it, ensurepip, which installs both pip and setuptools, is merely a convenience tool if you install Python from source yourself – not for distributions.) > AFAIK this might not be true if other build systems not using setuptools > at all might show up. And isn't this the main reason for all your work? No, try git clone https://github.com/pypa/pep517.git cd pep517 pip wheel --use-pep517 -v . which has no setup.py and uses flit instead. pip can build it, because it supports PEP 517-based builds. As I said, if we decide to keep it bundled with our python package, there’s no good reason to choose pypa-build. Cheers, Lars From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] Questions regarding Python packaging Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 06 Jul 2021 12:17:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Hartmut Goebel Cc: guix-devel@gnu.org, 46848@debbugs.gnu.org Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.1625573790736 (code B ref 46848); Tue, 06 Jul 2021 12:17:02 +0000 Received: (at 46848) by debbugs.gnu.org; 6 Jul 2021 12:16:30 +0000 Received: from localhost ([127.0.0.1]:47558 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m0jzw-0000Be-I2 for submit@debbugs.gnu.org; Tue, 06 Jul 2021 08:16:30 -0400 Received: from mout-p-201.mailbox.org ([80.241.56.171]:24830) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m0jzs-0000BO-Ok for 46848@debbugs.gnu.org; Tue, 06 Jul 2021 08:16:19 -0400 Received: from smtp1.mailbox.org (smtp1.mailbox.org [IPv6:2001:67c:2050:105:465:1:1:0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-201.mailbox.org (Postfix) with ESMTPS id 4GK1k268HnzQkBv; Tue, 6 Jul 2021 14:16:10 +0200 (CEST) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1625573768; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kaG6VNJJty9xD8VZJalEGuS/bvkEBLt3J6gywW9g/FE=; b=AS+BvMta1I60rCfI01/ndRjOLeRwAARp3HNWyYP55B51p2FZtH+1ujMJV08uesabLosyca UHPt8FA8btqcaQzlTHUENdKkEivGrLyzMHnaVsQXI+dUBFno/chXDz22eIDkDJMRqgrsDO Tr4Yq9OUQGf3sAxWpsPnf0PK37oSjBZAaC6905stBE9cOA8KHhgwsT4AMcRVUHJnpUhTSW n8LWgLfm73SX0f7etcu+HEK9y4tZGkiguIX/dAtIt1I8227TgV6BuHwtO1PX9EzG4uSRQ8 fp9vz22xUaehrwixtxGMcmrZl7hbaAQYYEj8Yg3s6edW+DaHlf6YIqDbAmk7aw== Received: from smtp1.mailbox.org ([80.241.60.240]) by gerste.heinlein-support.de (gerste.heinlein-support.de [91.198.250.173]) (amavisd-new, port 10030) with ESMTP id HlXG7a2YG9nd; Tue, 6 Jul 2021 14:16:05 +0200 (CEST) Date: Tue, 6 Jul 2021 14:16:03 +0200 From: Lars-Dominik Braun Message-ID: References: <1609946775.8blxygrg9p.astroid@rafflesia.none> <1611303651.35tpgtn1z1.astroid@melmoth.none> <1622997703.qcpe1ehxem.astroid@melmoth.none> <520a5492-6467-bbfc-3252-f17a5cc5d16f@crazy-compilers.com> <681eb450-0185-a465-3ed1-8446e8ad0974@crazy-compilers.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="ZimuCUS9BpRiMKpV" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-MBO-SPAM-Probability: X-Rspamd-Score: -3.11 / 15.00 / 15.00 X-Rspamd-Queue-Id: 97B8718B4 X-Rspamd-UID: 6ba2c3 X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --ZimuCUS9BpRiMKpV Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit Hi again, > No, try > > git clone https://github.com/pypa/pep517.git > cd pep517 > pip wheel --use-pep517 -v . > > which has no setup.py and uses flit instead. pip can build it, because > it supports PEP 517-based builds. As I said, if we decide to keep it > bundled with our python package, there’s no good reason to choose > pypa-build. I now have a proof of concept for a pip-based python-build-system, see 1st patch. The changeset is quite small, but it does not handle testing at all right now. Note that alot of setuptools-based packages lack a dependency on python-wheel (2nd patch; pardon the whitespace errors), which is required when using pyproject.toml (and also declared there, but our Python importer cannot read that file yet). But – as expected – that’s really the only issue I’ve encountered while rebuilding alot of Python packages so far. Cheers, Lars --ZimuCUS9BpRiMKpV Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0001-dirty-build-Build-Python-packages-using-pip.patch" Content-Transfer-Encoding: 8bit >From a7c6750917f5dc2e1eaf34520f7e6b0e3d5e0d3c Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Tue, 6 Jul 2021 14:13:51 +0200 Subject: [PATCH 1/2] dirty: build: Build Python packages using pip. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * guix/build/python-build-system.scm … --- guix/build/python-build-system.scm | 109 +++++++++++------------------ 1 file changed, 42 insertions(+), 67 deletions(-) diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm index 8ade1d5911..e4b63e131e 100644 --- a/guix/build/python-build-system.scm +++ b/guix/build/python-build-system.scm @@ -109,30 +109,6 @@ ;; and the scripts defined in entry-points will always be created. -(define setuptools-shim - ;; Run setup.py with "setuptools" being imported, which will patch - ;; "distutils". This is needed for packages using "distutils" instead of - ;; "setuptools" since the former does not understand the - ;; "--single-version-externally-managed" flag. - ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py - (string-append - "import setuptools, tokenize;__file__='setup.py';" - "f=getattr(tokenize, 'open', open)(__file__);" - "code=f.read().replace('\\r\\n', '\\n');" - "f.close();" - "exec(compile(code, __file__, 'exec'))")) - -(define (call-setuppy command params use-setuptools?) - (if (file-exists? "setup.py") - (begin - (format #t "running \"python setup.py\" with command ~s and parameters ~s~%" - command params) - (if use-setuptools? - (apply invoke "python" "-c" setuptools-shim - command params) - (apply invoke "python" "./setup.py" command params))) - (error "no setup.py found"))) - (define* (sanity-check #:key tests? inputs outputs #:allow-other-keys) "Ensure packages depending on this package via setuptools work properly, their advertised endpoints work and their top level modules are importable @@ -142,26 +118,6 @@ without errors." (with-directory-excursion "/tmp" (invoke "python" sanity-check.py (site-packages inputs outputs))))) -(define* (build #:key use-setuptools? #:allow-other-keys) - "Build a given Python package." - (call-setuppy "build" '() use-setuptools?) - #t) - -(define* (check #:key tests? test-target use-setuptools? #:allow-other-keys) - "Run the test suite of a given Python package." - (if tests? - ;; Running `setup.py test` creates an additional .egg-info directory in - ;; build/lib in some cases, e.g. if the source is in a sub-directory - ;; (given with `package_dir`). This will by copied to the output, too, - ;; so we need to remove. - (let ((before (find-files "build" "\\.egg-info$" #:directories? #t))) - (call-setuppy test-target '() use-setuptools?) - (let* ((after (find-files "build" "\\.egg-info$" #:directories? #t)) - (inter (lset-difference string=? after before))) - (for-each delete-file-recursively inter))) - (format #t "test suite not run~%")) - #t) - (define (python-version python) (let* ((version (last (string-split python #\-))) (components (string-split version #\.)) @@ -195,31 +151,40 @@ running checks after installing the package." "/bin:" (getenv "PATH")))) -(define* (install #:key inputs outputs (configure-flags '()) use-setuptools? +;; Some packages expect 'build and 'check exist. If they don’t replacing them +;; or adding phases before/after will fail. Preserve them as dummy-phases. +(define* (build #:key outputs (configure-flags '()) use-setuptools? + #:allow-other-keys) + #t) + +(define* (check #:key outputs (configure-flags '()) use-setuptools? + #:allow-other-keys) + #t) + +(define* (install #:key outputs (configure-flags '()) use-setuptools? #:allow-other-keys) "Install a given Python package." - (let* ((out (python-output outputs)) - (python (assoc-ref inputs "python")) - (major-minor (map string->number - (take (string-split (python-version python) #\.) 2))) - (<3.7? (match major-minor - ((major minor) - (or (< major 3) (and (= major 3) (< minor 7)))))) - (params (append (list (string-append "--prefix=" out) - "--no-compile") - (if use-setuptools? - ;; distutils does not accept these flags - (list "--single-version-externally-managed" - "--root=/") - '()) - configure-flags))) - (call-setuppy "install" params use-setuptools?) - ;; Rather than produce potentially non-reproducible .pyc files on Pythons - ;; older than 3.7, whose 'compileall' module lacks the - ;; '--invalidation-mode' option, do not generate any. - (unless <3.7? - (invoke "python" "-m" "compileall" "--invalidation-mode=unchecked-hash" - out)))) + (let* ((out (assoc-ref outputs "out"))) + (setenv "HOME" "/tmp") ; silence warning + (invoke "pip" "install" + (string-append "--prefix=" out) + "--root=/" + ;; Hide pip’s own version check. Nothing we can do anyway. + "--disable-pip-version-check" + ;; Do not search and install dependencies. + "--no-deps" + ;; Do not search PyPi. + "--no-index" + ;; Don’t reinstall build dependencies into virtal environent, + ;; instead use local ones. Otherwise build process will fail + ;; finding dependencies. + "--no-build-isolation" + ;; Ignore installed packages, important for packages bundled by + ;; Python, like setuptools. + "--ignore-installed" + ;; Library is in the current directory. + ".") + #t)) (define* (wrap #:key inputs outputs #:allow-other-keys) (define (list-of-files dir) @@ -267,6 +232,15 @@ installed with setuptools." (utime file early-1980 early-1980)) #t)))) +;; python-wheel respects this virable and passes an invalid early timestamp to +;; Python’s zip module if not redefined to a later date. +(define* (set-SOURCE-DATE-EPOCH #:rest _) + "Set the 'SOURCE_DATE_EPOCH' environment variable. This is used by tools +that incorporate timestamps as a way to tell them to use a fixed timestamp. +See https://reproducible-builds.org/specs/source-date-epoch/." + (setenv "SOURCE_DATE_EPOCH" "315619200") + #t) + (define* (enable-bytecode-determinism #:rest _) "Improve determinism of pyc files." ;; Use deterministic hashes for strings, bytes, and datetime objects. @@ -297,6 +271,7 @@ by Cython." enable-bytecode-determinism) (add-after 'enable-bytecode-determinism 'ensure-no-cythonized-files ensure-no-cythonized-files) + (replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH) (delete 'bootstrap) (delete 'configure) ;not needed (replace 'build build) -- 2.31.1 --ZimuCUS9BpRiMKpV Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-dirty-Fix-build-errors.patch" >From 7e6d12aac228f369a1f09870529dc003a8ad356e Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Tue, 6 Jul 2021 14:14:21 +0200 Subject: [PATCH 2/2] dirty: Fix build errors. --- gnu/packages/build-tools.scm | 2 ++ gnu/packages/check.scm | 16 ++++++++---- gnu/packages/python-build.scm | 47 ++++++++++++++++++++++++++++++++++ gnu/packages/python-crypto.scm | 9 ++++--- gnu/packages/python-xyz.scm | 44 ++++++++++++++++--------------- gnu/packages/time.scm | 4 ++- 6 files changed, 92 insertions(+), 30 deletions(-) diff --git a/gnu/packages/build-tools.scm b/gnu/packages/build-tools.scm index dc2411801c..6c29fb9bdc 100644 --- a/gnu/packages/build-tools.scm +++ b/gnu/packages/build-tools.scm @@ -50,6 +50,7 @@ #:use-module (gnu packages pretty-print) #:use-module (gnu packages protobuf) #:use-module (gnu packages python) + #:use-module (gnu packages python-build) #:use-module (gnu packages python-crypto) #:use-module (gnu packages python-web) #:use-module (gnu packages python-xyz) @@ -237,6 +238,7 @@ files and generates build instructions for the Ninja build system.") (delete 'wrap)))) (inputs `(("ninja" ,ninja))) (propagated-inputs `(("python" ,python))) + (native-inputs `(("python-wheel" ,python-wheel))) (home-page "https://mesonbuild.com/") (synopsis "Build system designed to be fast and user-friendly") (description diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm index 11117b88e4..12f17c41c6 100644 --- a/gnu/packages/check.scm +++ b/gnu/packages/check.scm @@ -975,6 +975,7 @@ standard library.") ("python-mock" ,python-mock) ("python-pytest" ,python-pytest-bootstrap) ("python-setuptools-scm" ,python-setuptools-scm) + ("python-wheel" ,python-wheel) ("python-toml" ,python-toml) ("python-xmlschema" ,python-xmlschema))) (home-page "https://docs.pytest.org/en/latest/") @@ -1028,7 +1029,8 @@ and many external plugins.") (name "python-pytest-bootstrap") (native-inputs `(("python-iniconfig" ,python-iniconfig) ("python-setuptools-scm" ,python-setuptools-scm) - ("python-toml" ,python-toml))) + ("python-toml" ,python-toml) + ("python-wheel" ,python-wheel))) (arguments `(#:tests? #f)) (properties `((python2-variant . ,(delay python2-pytest-bootstrap)))))) @@ -1113,7 +1115,8 @@ supports coverage of subprocesses.") (format #t "test suite not run~%")) #t))))) (native-inputs - `(("python-setuptools-scm" ,python-setuptools-scm))) + `(("python-setuptools-scm" ,python-setuptools-scm) + ("python-wheel" ,python-wheel))) (home-page "https://github.com/pytest-dev/pytest-runner") (synopsis "Invoke py.test as a distutils command") (description @@ -1236,7 +1239,8 @@ same arguments.") (when tests? (invoke "py.test" "-v"))))))) (native-inputs - `(("python-setuptools-scm" ,python-setuptools-scm))) + `(("python-setuptools-scm" ,python-setuptools-scm) + ("python-wheel" ,python-wheel))) (propagated-inputs `(("python-execnet" ,python-execnet) ("python-pytest" ,python-pytest) @@ -1302,7 +1306,8 @@ timeout has been exceeded.") (build-system python-build-system) (native-inputs `(("python-pytest" ,python-pytest) - ("python-setuptools-scm" ,python-setuptools-scm))) + ("python-setuptools-scm" ,python-setuptools-scm) + ("python-wheel" ,python-wheel))) (home-page "https://github.com/pytest-dev/pytest-forked") (synopsis @@ -2486,7 +2491,8 @@ create data based on random numbers and yet remain repeatable.") (build-system python-build-system) (native-inputs `(("python-mock" ,python-mock) - ("python-pytest" ,python-pytest))) + ("python-pytest" ,python-pytest) + ("python-wheel" ,python-wheel))) (propagated-inputs `(("python-six" ,python-six) ("python-dateutil" ,python-dateutil))) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 140629ca43..913d1b04ce 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -89,6 +89,52 @@ installed with a newer @code{pip} or with wheel's own command line utility.") Language (TOML) configuration files.") (license license:expat))) +(define-public python-pytoml + (package + (name "python-pytoml") + (version "0.1.21") + (source + (origin + (method url-fetch) + (uri (pypi-uri "pytoml" version)) + (sha256 + (base32 + "1rv1byiw82k7mj6aprcrqi2vdabs801y97xhfnrz7kxds34ggv4f")))) + (build-system python-build-system) + (home-page "https://github.com/avakar/pytoml") + (synopsis "Parser for TOML") + (description "This package provides a Python parser for TOML-0.4.0.") + (license license:expat))) + +(define-public python-flit-core + (package + (name "python-flit-core") + (version "3.2.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "flit-core" version)) + (sha256 + (base32 + "0cclv7v4cmzi457bzqsx9ig8ir1ha3ip8h1kx8qfy95wbmfg51zz")))) + (build-system python-build-system) + (arguments + `(#:phases + (modify-phases %standard-phases + ;; Requirements refer to a specific version of dependencies, + ;; which are too old. So we patch to refer to any later version. + (add-after 'unpack 'use-local-sourcetree + (lambda _ + (setenv "GUIX_PYTHONPATH" (string-append (getenv "GUIX_PYTHONPATH") ":" (getcwd)))))))) + (propagated-inputs + `(("python-toml" ,python-toml))) + (home-page "https://github.com/takluyver/flit") + (synopsis + "Distribution-building parts of Flit. See flit package for more information") + (description + "Distribution-building parts of Flit. See flit package for more information") + (license #f))) + (define-public python-pep517-bootstrap (hidden-package (package @@ -104,6 +150,7 @@ Language (TOML) configuration files.") (build-system python-build-system) (arguments `(#:tests? #f)) ;to avoid circular dependencies + (native-inputs `(("python-flit-core" ,python-flit-core))) (propagated-inputs `(("python-toml" ,python-toml) ("python-wheel" ,python-wheel))) diff --git a/gnu/packages/python-crypto.scm b/gnu/packages/python-crypto.scm index 39b194b25e..e40a60a9ca 100644 --- a/gnu/packages/python-crypto.scm +++ b/gnu/packages/python-crypto.scm @@ -124,7 +124,8 @@ with what is used by the Bitcoin network.") (build-system python-build-system) (native-inputs `(("python-pycparser" ,python-pycparser) - ("python-pytest" ,python-pytest))) + ("python-pytest" ,python-pytest) + ("python-wheel" ,python-wheel))) (propagated-inputs `(("python-cffi" ,python-cffi) ("python-six" ,python-six))) @@ -544,7 +545,8 @@ is used by the Requests library to verify HTTPS requests.") ("python-hypothesis" ,python-hypothesis) ("python-pretend" ,python-pretend) ("python-pytz" ,python-pytz) - ("python-pytest" ,python-pytest))) + ("python-pytest" ,python-pytest) + ("python-wheel" ,python-wheel))) (home-page "https://github.com/pyca/cryptography") (synopsis "Cryptographic recipes and primitives for Python") (description @@ -611,7 +613,8 @@ message digests and key derivation functions.") `(("libfaketime" ,libfaketime) ("python-flaky" ,python-flaky) ("python-pretend" ,python-pretend) - ("python-pytest" ,python-pytest))) + ("python-pytest" ,python-pytest) + ("python-wheel" ,python-wheel))) (home-page "https://github.com/pyca/pyopenssl") (synopsis "Python wrapper module around the OpenSSL library") (description diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index 7a2b9eecc6..59dbe45a4f 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -2307,6 +2307,7 @@ and is not compatible with JSON.") (invoke "python" "bootstrap.py" "build/scons" "DEVELOPER=guix") (chdir "build/scons") #t))))) + (native-inputs `(("python-wheel" ,python-wheel))) (home-page "https://scons.org/") (synopsis "Software construction tool written in Python") (description @@ -3161,7 +3162,8 @@ e.g. filters, callbacks and errbacks can all be promises.") (native-inputs `(("python-mock" ,python-mock) ("python-pytest" ,python-pytest) - ("python-setuptools-scm" ,python-setuptools-scm))) + ("python-setuptools-scm" ,python-setuptools-scm) + ("python-wheel" ,python-wheel))) (propagated-inputs `(("python-appdirs" ,python-appdirs) ("python-distlib" ,python-distlib/next) @@ -6498,6 +6500,7 @@ child application and control it as if a human were typing commands.") (base32 "0ahlrxxkx2xhmxskx57gc96w3bdndflxx30304ihvm7ds136nny8")))) (build-system python-build-system) + (native-inputs `(("python-wheel" ,python-wheel))) (home-page "https://github.com/pypa/setuptools_scm/") (synopsis "Manage Python package versions in SCM metadata") (description @@ -8284,7 +8287,8 @@ PEP 8.") (native-inputs `(("python-mock" ,python-mock) ("python-pytest" ,python-pytest) - ("python-testpath" ,python-testpath))) + ("python-testpath" ,python-testpath) + ,@(package-native-inputs python-pep517-bootstrap))) (properties `((python2-variant . ,(delay python2-pep517)))))) ;; Skip the tests so we don't create a cyclical dependency with pytest. @@ -8752,6 +8756,7 @@ output.") (sha256 (base32 "1p9p7mn8x2j9psc4jxab98897v4i9s4fliyfw8rp8v4bx1n7pjj2")))) (build-system python-build-system) + (native-inputs `(("python-poetry-core" ,python-poetry-core))) (home-page "https://github.com/sdispater/crashtest") (synopsis "Manage Python errors with ease") (description @@ -9891,20 +9896,7 @@ Jupyter Notebook format and Python APIs for working with notebooks.") (base32 "0lc4si3xb7hza424414rdqdc3vng3kcrph8jbvjqb32spqddf3f7")))) (build-system python-build-system) - ;; The package does not come with a setup.py file, so we have to generate - ;; one ourselves. - (arguments - `(#:tests? #f - #:phases - (modify-phases %standard-phases - (add-after 'unpack 'create-setup.py - (lambda _ - (call-with-output-file "setup.py" - (lambda (port) - (format port "\ -from setuptools import setup -setup(name='entrypoints', version='~a', py_modules=['entrypoints']) -" ,version)))))))) + (native-inputs `(("python-flit" ,python-flit))) (home-page "https://github.com/takluyver/entrypoints") (synopsis "Discover and load entry points from installed Python packages") (description "Entry points are a way for Python packages to advertise @@ -11176,7 +11168,8 @@ library as well as on the command line.") "1c35qyhvy27q9ih9n899f3h4sdnpgq027dbiilly2qb5cvgarchm")))) (build-system python-build-system) (native-inputs - `(("python-setuptools-scm" ,python-setuptools-scm))) + `(("python-setuptools-scm" ,python-setuptools-scm) + ("python-wheel" ,python-wheel))) (synopsis "Plugin and hook calling mechanism for Python") (description "Pluggy is an extraction of the plugin manager as used by Pytest but stripped of Pytest specific details.") @@ -14657,7 +14650,8 @@ than during a preprocessing step).") "17xbrgi23l87yg6h0qcknssp2q812miiy33qw6v45v5gx0jwv5xh")))) (build-system python-build-system) (propagated-inputs - `(("python-setuptools-scm" ,python-setuptools-scm))) + `(("python-setuptools-scm" ,python-setuptools-scm) + ("python-wheel" ,python-wheel))) (home-page "https://github.com/marcelm/xopen/") (synopsis "Open compressed files transparently") (description "This module provides an @code{xopen} function that works like @@ -15194,7 +15188,8 @@ and bit flag values.") (lambda _ (invoke "pytest")))))) (native-inputs - `(("python-coverage" ,python-coverage) + `(("python-wheel" ,python-wheel) + ("python-coverage" ,python-coverage) ("python-hypothesis" ,python-hypothesis) ("python-pympler" ,python-pympler) ("python-pytest" ,python-pytest) @@ -15215,7 +15210,7 @@ protocols.") (package (inherit python-attrs) (name "python-attrs-bootstrap") - (native-inputs `()) + (native-inputs `(("python-wheel" ,python-wheel))) (arguments `(#:tests? #f)))) (define-public python2-attrs-bootstrap @@ -15606,6 +15601,7 @@ in other versions.") (base32 "0ckzngs3scaa1mcfmsi1w40a1l8cxxnncscrxzjjwjyisx8z0fmw")))) (build-system python-build-system) + (native-inputs `(("python-wheel" ,python-wheel))) (home-page "https://github.com/RonnyPfannschmidt/iniconfig") (synopsis "Simple INI-file parser") (description "The @code{iniconfig} package provides a small and simple @@ -20784,7 +20780,8 @@ register custom encoders and decoders.") (native-inputs `(("double-conversion" ,double-conversion) ("python-setuptools-scm" ,python-setuptools-scm) - ("python-pytest" ,python-pytest))) + ("python-pytest" ,python-pytest) + ("python-wheel" ,python-wheel))) (home-page "https://github.com/ultrajson/ultrajson") (synopsis "Ultra fast JSON encoder and decoder for Python") (description @@ -20980,6 +20977,11 @@ the syntactic logic to configure and launch jobs in an execution environment.") (build-system python-build-system) (arguments `(#:tests? #f)) ; XXX: Check requires network access. + (propagated-inputs + `(("python-docutils" ,python-docutils) + ("python-pytoml" ,python-pytoml) + ("python-requests" ,python-requests) + ("python-flit-core" ,python-flit-core))) (home-page "https://flit.readthedocs.io/") (synopsis "Simple packaging tool for simple packages") diff --git a/gnu/packages/time.scm b/gnu/packages/time.scm index 9b006a5438..60a0584202 100644 --- a/gnu/packages/time.scm +++ b/gnu/packages/time.scm @@ -46,6 +46,7 @@ #:use-module (gnu packages compression) #:use-module (gnu packages perl) #:use-module (gnu packages python) + #:use-module (gnu packages python-build) #:use-module (gnu packages python-xyz)) (define-public time @@ -194,7 +195,8 @@ Pendulum instances.") (native-inputs `(("python-pytest" ,python-pytest) ("python-pytest-cov" ,python-pytest-cov) - ("python-setuptools-scm" ,python-setuptools-scm))) + ("python-setuptools-scm" ,python-setuptools-scm) + ("python-wheel" ,python-wheel))) (propagated-inputs `(("python-six" ,python-six))) (home-page "https://dateutil.readthedocs.io/en/stable/") -- 2.31.1 --ZimuCUS9BpRiMKpV-- From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] Questions regarding Python packaging Resent-From: Hartmut Goebel Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 07 Jul 2021 15:02:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Lars-Dominik Braun Cc: guix-devel@gnu.org, Tanguy LE CARROUR , 46848@debbugs.gnu.org Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.16256700754965 (code B ref 46848); Wed, 07 Jul 2021 15:02:01 +0000 Received: (at 46848) by debbugs.gnu.org; 7 Jul 2021 15:01:15 +0000 Received: from localhost ([127.0.0.1]:53235 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m1935-0001I1-Je for submit@debbugs.gnu.org; Wed, 07 Jul 2021 11:01:15 -0400 Received: from mail-out.m-online.net ([212.18.0.10]:50838) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1m1933-0001Hs-Cf for 46848@debbugs.gnu.org; Wed, 07 Jul 2021 11:01:14 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 4GKjKz3zmqz1s60D; Wed, 7 Jul 2021 17:01:11 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 4GKjKz1tHnz1qwYb; Wed, 7 Jul 2021 17:01:11 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id x_-d5RsbzEj9; Wed, 7 Jul 2021 17:01:09 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-62-244.dynamic.mnet-online.de [188.174.62.244]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS; Wed, 7 Jul 2021 17:01:09 +0200 (CEST) Received: from thisbe.goebel-consult.de (hermia.goebel-consult.de [192.168.110.7]) by hermia.goebel-consult.de (Postfix) with ESMTP id 62DD260106; Wed, 7 Jul 2021 17:01:17 +0200 (CEST) References: <1609946775.8blxygrg9p.astroid@rafflesia.none> <1611303651.35tpgtn1z1.astroid@melmoth.none> <1622997703.qcpe1ehxem.astroid@melmoth.none> <520a5492-6467-bbfc-3252-f17a5cc5d16f@crazy-compilers.com> <681eb450-0185-a465-3ed1-8446e8ad0974@crazy-compilers.com> From: Hartmut Goebel Organization: crazy-compilers.com Message-ID: <716d2bce-bc89-2442-a02a-3bcbcbe19219@crazy-compilers.com> Date: Wed, 7 Jul 2021 17:01:05 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------7146F635C4E75FB9B7D3E096" Content-Language: en-US X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) This is a multi-part message in MIME format. --------------7146F635C4E75FB9B7D3E096 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Am 29.06.21 um 09:20 schrieb Lars-Dominik Braun: >> AFAIK this might not be true if other build systems not using setuptools >> at all might show up. And isn't this the main reason for all your work? > No, try Sorry, I've been inprecise on this: There might still be quite some packages out there importing plain, old distutils (and not setuptools) in their setup.py. These are what I meant with "other build systems not using setuptools". For these setup.py to understand the options we (and pip) need for installation, "import distutils" has to be hacked to actually become "import setuptools" - which is what setuptools-shim does -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | --------------7146F635C4E75FB9B7D3E096 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit
Am 29.06.21 um 09:20 schrieb Lars-Dominik Braun:
AFAIK this might not be true if other build systems not using setuptools 
at all might show up. And isn't this the main reason for all your work?

No, try

Sorry, I've been inprecise on this:

There might still be quite some packages out there importing plain, old distutils (and not setuptools) in their setup.py. These are what I meant with "other build systems not using setuptools". For these setup.py to understand the options we (and pip) need for installation, "import distutils" has to be hacked to actually become "import setuptools" - which is what setuptools-shim does

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | h.goebel@crazy-compilers.com               |
| www.crazy-compilers.com | compilers which you thought are impossible |
--------------7146F635C4E75FB9B7D3E096-- From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] Questions regarding Python packaging Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 25 Aug 2021 07:57:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 46848@debbugs.gnu.org Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.162987821115297 (code B ref 46848); Wed, 25 Aug 2021 07:57:01 +0000 Received: (at 46848) by debbugs.gnu.org; 25 Aug 2021 07:56:51 +0000 Received: from localhost ([127.0.0.1]:45626 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mInmE-0003yf-Vu for submit@debbugs.gnu.org; Wed, 25 Aug 2021 03:56:51 -0400 Received: from mout-p-102.mailbox.org ([80.241.56.152]:20414) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mInmC-0003yO-LD for 46848@debbugs.gnu.org; Wed, 25 Aug 2021 03:56:49 -0400 Received: from smtp1.mailbox.org (smtp1.mailbox.org [80.241.60.240]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4GvdbZ2YmQzQkHL for <46848@debbugs.gnu.org>; Wed, 25 Aug 2021 09:56:42 +0200 (CEST) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1629878200; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I0d7GRT8qfJIQ4zG7ap4VEHjRALQSmt8kJn6Xm2uem8=; b=tXe5Eqzf80d9fgj4iQCzQaW5g/MQKIIe61Y77PPeKKKod73JyJ8IcUssGu/l1cdILAnFjr 5VlIlYJmxrF7N6TMIW9wvRthFiRZpwLemDvzMTV6OKZ4oK67uiZtgQ1VzJ7OhoC7AG3NgD 1R5WtRLlquqakl+aee9lboL0dP6Yf2YNKx2nJz1DHzvOaDxH0TmFSoWbVqlJREYi5XV2XH LwR/qjppnTPZjpNM3HnL+UwGgDhitP8Muv1gl9wdC89IfyCuiAoNi9xSEB2VyNyElhPfsB 1kwFF/Lsc5KM7XokjzcMkZ627h56IO+WUK6/lbWZTqORYvziaEjM2BZu6erizw== Received: from smtp1.mailbox.org ([80.241.60.240]) by spamfilter04.heinlein-hosting.de (spamfilter04.heinlein-hosting.de [80.241.56.122]) (amavisd-new, port 10030) with ESMTP id i88MEjegd6iU for <46848@debbugs.gnu.org>; Wed, 25 Aug 2021 09:56:39 +0200 (CEST) Date: Wed, 25 Aug 2021 09:56:37 +0200 From: Lars-Dominik Braun Message-ID: References: <1609946775.8blxygrg9p.astroid@rafflesia.none> <1611303651.35tpgtn1z1.astroid@melmoth.none> <1622997703.qcpe1ehxem.astroid@melmoth.none> <520a5492-6467-bbfc-3252-f17a5cc5d16f@crazy-compilers.com> <681eb450-0185-a465-3ed1-8446e8ad0974@crazy-compilers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Rspamd-Queue-Id: 9613F18B8 X-Rspamd-UID: e110ca X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi, > > What should be the use of having a package without pip? Anything else > > than saving a few KB? > saving some space and unvendoring components that we also have separate > packages for. (As I understand it, ensurepip, which installs both pip and > setuptools, is merely a convenience tool if you install Python from > source yourself – not for distributions.) there is another crucial one: Collisions. Right now our python-pip and python-setuptools packages are rather useless, because they collide with the Python-bundled one. I’m trying to update Jupyter currently and python-jupyter-packaging depends on a newer setuptools (v46ish) than our python package provides (v41). Adding python-setuptools (v52) to the propagated inputs (yes, it’s a runtime dependency), would be the natural solution, but the resulting setuptools package in the build environment for packages depending on python-jupyter-packaging is a mix between v42 and v52 and does not work. Cheers, Lars From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 13 Dec 2021 20:11:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 46848@debbugs.gnu.org Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.163942621614488 (code B ref 46848); Mon, 13 Dec 2021 20:11:02 +0000 Received: (at 46848) by debbugs.gnu.org; 13 Dec 2021 20:10:16 +0000 Received: from localhost ([127.0.0.1]:56902 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mwreK-0003lc-2j for submit@debbugs.gnu.org; Mon, 13 Dec 2021 15:10:16 -0500 Received: from mout-p-102.mailbox.org ([80.241.56.152]:59386) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mwreI-0003lI-GV for 46848@debbugs.gnu.org; Mon, 13 Dec 2021 15:10:15 -0500 Received: from smtp102.mailbox.org (smtp102.mailbox.org [IPv6:2001:67c:2050:105:465:1:3:0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4JCXg21FjCzQjDq for <46848@debbugs.gnu.org>; Mon, 13 Dec 2021 21:10:06 +0100 (CET) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1639426203; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SbGpKGP1U6AmTrSN34ixDJNOPVsowjUCgNJqHumvoRk=; b=OvwIiNHayYoHNcCA/3aadJIfdvqZOY5o7Isby8sRJz6wkWtRYVMDfKQUkdo+2SUH5mVCvs yNgrWkp0QuqiPOnih7zia+YEqsWNaJ4NoHt1FuFShc1oNROL0pEro1j56t6MKxv/thD7Cc 7QlqC42JffRnkNTTKipmuYc+xyQoCEagu8urOCjjbyWMqDzE/6NOd8OL/05SnMYQlAVUwX p/58Epyiwsnnt8UKAO8+lXxQ/lwe0y6jnrXNNfxXK5L/OpdJKkq8MY+pEcYnWVkq96/kEn Q0tczVBJpbYmNLme7/B+UzPdF4pjwFI919PY4Iq4LlGO0g1nEVAVzv9oXq80vg== Date: Mon, 13 Dec 2021 21:10:00 +0100 From: Lars-Dominik Braun Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi, I’m working on version 3 of this patchset, removing the dependency on python-pypa-build, which simplifies bootstrapping. A rough version is available at https://github.com/PromyLOPh/guix/commits/work-python-build-pep517 Cheers, Lars From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 05 Jan 2022 14:52:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 46848@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.164139430221857 (code B ref 46848); Wed, 05 Jan 2022 14:52:02 +0000 Received: (at 46848) by debbugs.gnu.org; 5 Jan 2022 14:51:42 +0000 Received: from localhost ([127.0.0.1]:39831 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n57de-0005gT-0B for submit@debbugs.gnu.org; Wed, 05 Jan 2022 09:51:42 -0500 Received: from mout-p-201.mailbox.org ([80.241.56.171]:45864) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1n57dc-0005gE-Gg for 46848@debbugs.gnu.org; Wed, 05 Jan 2022 09:51:41 -0500 Received: from smtp102.mailbox.org (smtp102.mailbox.org [IPv6:2001:67c:2050:105:465:1:3:0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-201.mailbox.org (Postfix) with ESMTPS id 4JTXVs75X6zQjmx; Wed, 5 Jan 2022 15:51:33 +0100 (CET) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1641394292; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/nZ7dRjk3aCVlxOwhwqUgOUQwhYqH/ZGRy25s8XUACk=; b=NCgfTLMqrpVLgh2RjNH24progoqOMNBH2fX6zc7/ZgV+C76ppqdULEISydvZj48u9s7Vb5 yMkd6DFARwOD9xcuMUsApCCJ6KIOOzWqP0KjJCwDT/OgydQ9DTIbb0jzorhyT/0ubwhEgR uoReQqAnrupu1ieTGRFbTuI9kKrTF8S6Uj+7oMXvIAwcxWpjYM8vksTCcxUzhEyY80ice1 8XxGBdNfOWKxsiYmw03G4lNSWU43iytCbE3lF5bdHZiAVuy0kduyJ36zcbvu9tOldqHDRx pvpx1wPENFDblk85n1rHP6SFLsWOoTVFQwjowiZRR+Y3YGmsgWeVtwmQT4/LGw== Date: Wed, 5 Jan 2022 15:51:29 +0100 From: Lars-Dominik Braun Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi everyone, I pushed the 3rd version of my new PEP 517-based python-build-system into wip-python-pep517. I removed the dependency on python-pypa-build for building packages and instead implemented building with a single python invokation. This simplifies bootstrapping Python. The 'install phase also learned how to create scripts in bin/ from an entry points specification. Currently 600 of 2212 packages using python-build-system are failing to build. That’s alot, but most failures so far were related to testsuites, which were not enabled correctly previously because 'check was not replaced manually. And some fail because upstream does not separate sources and tests and a name clash occurs. This also happens when C extensions are build and there does not seem to be a workaround. This number does not include Python 2 packages, which will probably break with this new build system. Since Python 2 is EOL anyway I suggest to entirely remove Python 2 support before merging this changeset. After merging we should upgrade the entire Python ecosystem, because alot of packages are already years old. Cheers, Lars PS: Maxim: This is your ping for a review (via #52269). Thanks :) From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Marius Bakke Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 20 Jan 2022 15:42:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Lars-Dominik Braun , 46848@debbugs.gnu.org Cc: Maxim Cournoyer Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.164269328115151 (code B ref 46848); Thu, 20 Jan 2022 15:42:02 +0000 Received: (at 46848) by debbugs.gnu.org; 20 Jan 2022 15:41:21 +0000 Received: from localhost ([127.0.0.1]:58988 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nAZYv-0003wI-A3 for submit@debbugs.gnu.org; Thu, 20 Jan 2022 10:41:21 -0500 Received: from eggs.gnu.org ([209.51.188.92]:57694) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nAZYt-0003w2-Hk for 46848@debbugs.gnu.org; Thu, 20 Jan 2022 10:41:19 -0500 Received: from [2001:470:142:3::e] (port=44102 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nAZYn-0007ij-46; Thu, 20 Jan 2022 10:41:13 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=LrMVRFT79ZQXzhLae5b0vtznC1CRQ1FY/7CqxPMFd2U=; b=dLMEWUYAGNMd9SpgCb1I vR5V3tqCNkz20IkoNjuhq0Exucfof4ipGmbU0wUdew3iWmTsynICVbRy79SeJD2AoGynbfJ4LP2e9 +zPbXJtyiyzGHpczp8DvkqYeKPyaPlpCWZ2zl/ySLEDSKkhKMmpQa5TETsKHGMZ2nvznkKp1UjCQW yRMx2PNMMXIKd8rurWZh1T+2qT8UnFZkpLcoegYVA0/zxYcaZlkEi73rxZAiNwOhKBip79RtEBZMc 2K9wVQAaS/GGfuty6weja/nHvzgC5hRg7qLiyapb0dQevs7G+JCE9aYtEDOf6oC45dYWnEKHzSQ36 dhPpepxJHpCTBg==; Received: from host-37-191-236-102.lynet.no ([37.191.236.102]:59886 helo=localhost) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nAZYn-0001ed-0o; Thu, 20 Jan 2022 10:41:13 -0500 From: Marius Bakke In-Reply-To: References: Date: Thu, 20 Jan 2022 16:41:09 +0100 Message-ID: <877daubeju.fsf@gnu.org> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Lars-Dominik Braun skriver: > I pushed the 3rd version of my new PEP 517-based python-build-system > into wip-python-pep517. I removed the dependency on python-pypa-build > for building packages and instead implemented building with a single > python invokation. This simplifies bootstrapping Python. The 'install > phase also learned how to create scripts in bin/ from an entry points > specification. This is great, thank you. > Currently 600 of 2212 packages using python-build-system are failing to > build. That=E2=80=99s alot, but most failures so far were related to test= suites, > which were not enabled correctly previously because 'check was not > replaced manually. And some fail because upstream does not separate > sources and tests and a name clash occurs. This also happens when C > extensions are build and there does not seem to be a workaround. Can you elaborate on the name clash issue? > This number does not include Python 2 packages, which will probably > break with this new build system. Since Python 2 is EOL anyway I suggest > to entirely remove Python 2 support before merging this changeset. After > merging we should upgrade the entire Python ecosystem, because alot of > packages are already years old. Unfortunately we need Python 2 for some time still. It is used to bootstrap old versions of GHC, Node, and probably other things. Do you think it is feasible to provide this as a new build system, say pep517-build-system, during a transitional period? Then we can a) start using it right away for the increasing amount of packages that lack a setup.py; and b) flesh out most bugs before eventually merging it back (possibly piecemeal) into python-build-system. I've had a cursory look and it looks good overall. A few comments: * Zipping a wheel just to unpack it afterwards is weird, but there seems to be no way around it. * I also think trying "python setup.py test" is unnecessary. * It would be nice to support the "no tests" case without having to add explicit #:tests? everywhere. Perhaps along the lines of... (match use-test-backend ('pytest (apply invoke ...)) [...] (_ (match (find-files "." "^test_.*\\.py$") (() (format #t "no tests found~%")) (_ (apply invoke `("python" "-m" "unittest" ,@test-flags)))))) WDYT? Great work, and apologies for not chiming in earlier! --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iIUEARYKAC0WIQRNTknu3zbaMQ2ddzTocYulkRQQdwUCYemClQ8cbWFyaXVzQGdu dS5vcmcACgkQ6HGLpZEUEHekigD/fOVOE7lCfRW0OuNWk9gGTLOZOQpqO8+MmiJI CaWULd4A/3mf5Srz1hublc4NlZMb5NanVzduIrpl7YlBn1XN2NMJ =JNtc -----END PGP SIGNATURE----- --=-=-=-- From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 20 Jan 2022 18:44:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Marius Bakke Cc: 46848@debbugs.gnu.org, Maxim Cournoyer Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.164270421810828 (code B ref 46848); Thu, 20 Jan 2022 18:44:01 +0000 Received: (at 46848) by debbugs.gnu.org; 20 Jan 2022 18:43:38 +0000 Received: from localhost ([127.0.0.1]:59184 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nAcPK-0002oa-2J for submit@debbugs.gnu.org; Thu, 20 Jan 2022 13:43:38 -0500 Received: from mout-p-102.mailbox.org ([80.241.56.152]:40992) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nAcPI-0002oK-JK for 46848@debbugs.gnu.org; Thu, 20 Jan 2022 13:43:37 -0500 Received: from smtp2.mailbox.org (smtp2.mailbox.org [IPv6:2001:67c:2050:105:465:1:2:0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4JfrxY5cQ6z9sjs; Thu, 20 Jan 2022 19:43:29 +0100 (CET) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1642704208; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=byf06nzMITKrSDZsPhyRivdana1vgV5lHyxqplO28C4=; b=LOU+JQ0R6CcrKj4P4MXTwlYo4YXELWAOixq38dbq/LWt76hccfjrymP80f2FMunB/Gb/cL d81zKsepjfobRoqo+yUD6lpB06wkyNrYi0Vr1t37xvZ3+ivS46SuP+LnnQxG9eWT+vnLfB tSh5vJVaO5dqSfMQn40GtK5ZOKshqX3R2F47f2hgjS9OJE5G3oQ8clMm4rd6bWBJxIu9+n gxaVwnf3Ar2s7AHTI4SuMz0Ww1wK8FstTpfy7foF0YDE21+UITmJiTasM3s0vv9PfIEsh7 iJzq5jVXA7awnG32Y7YGRcULMBrmnIOqAG0SNPxFyvzb2m3ICTieW2nryvz5pQ== Date: Thu, 20 Jan 2022 19:43:24 +0100 From: Lars-Dominik Braun Message-ID: References: <877daubeju.fsf@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <877daubeju.fsf@gnu.org> X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi Marius, > Can you elaborate on the name clash issue? the problem seems to be that most packages do not put the source code into a subdirectory in the sdist. I.e. package foobar will have a foobar/ directory, instead of src/foobar. Thus Python tries to load the module foobar.barbaz from that directory, instead of the store. Thus it’ll also look there for C extensions and fail. I don’t know why it does that, even when *not* using `python -m` – according to the documentation it should not. If anyone has any pointers I’d like to have a look again. > Unfortunately we need Python 2 for some time still. It is used to > bootstrap old versions of GHC, Node, and probably other things. But as far as I see these specific examples do not use python-build-system. Only if some package using `#:python python-2` is required during bootstrapping, then we’d need a way to support Python 2, right? > Do you think it is feasible to provide this as a new build system, say > pep517-build-system, during a transitional period? Then we can a) start > using it right away for the increasing amount of packages that lack a > setup.py; and b) flesh out most bugs before eventually merging it back > (possibly piecemeal) into python-build-system. Actually, I think that’s a good idea. I tried, but I cannot fix all packages broken by this change on my own, so smoothing the transition could help and – as you said – we could catch bugs early on. I could prepare a patch, if there is interest in this path. > * Zipping a wheel just to unpack it afterwards is weird, but there seems > to be no way around it. Indeed, that’s how PEP 517 works, which always builds a wheel (i.e. ZIP file). > * I also think trying "python setup.py test" is unnecessary. It still works quite often, although its usefulness will decrease in the future I guess. Another problem I see is that this command will not fail if there are no tests. > * It would be nice to support the "no tests" case without having to add > explicit #:tests? everywhere. Perhaps along the lines of... My idea was to force packagers to make and explain this decision explicitly. If you don’t run tests, you have to add `#:tests? #f` and leave a comment why they are disabled. I do see this could become a hassle with packages that simply don’t have any tests. But my hope is that it increases package quality. > (_ (apply invoke `("python" "-m" "unittest" ,@test-flags)))))) If I remember correctly I tried this and it did not work for some reason. I’ll have a look again. Cheers, Lars From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Marius Bakke Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 20 Jan 2022 20:45:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Lars-Dominik Braun Cc: 46848@debbugs.gnu.org, Maxim Cournoyer Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.164271144831444 (code B ref 46848); Thu, 20 Jan 2022 20:45:02 +0000 Received: (at 46848) by debbugs.gnu.org; 20 Jan 2022 20:44:08 +0000 Received: from localhost ([127.0.0.1]:59312 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nAeHv-0008B6-W0 for submit@debbugs.gnu.org; Thu, 20 Jan 2022 15:44:08 -0500 Received: from eggs.gnu.org ([209.51.188.92]:52148) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nAeHu-0008Ac-GJ for 46848@debbugs.gnu.org; Thu, 20 Jan 2022 15:44:07 -0500 Received: from [2001:470:142:3::e] (port=51594 helo=fencepost.gnu.org) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nAeHh-0002jD-04; Thu, 20 Jan 2022 15:43:59 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To: From; bh=pXkj+Jzo7eEicfiKAYkrEIYSP5Gb+CFNo93zR/uwn6c=; b=rw2k8zw9PAubwiGt6ppt cJZHoQvhLGyjBxA2OTOT2XJEEGkROWtImAq2rvvgoy7Ejr75GZQhlKh5r4RVPYEYjexTvVyKlx1Ag fxglJKeCKkQQol59kSyEIzyOHQU3Wu2TCgBFXdBV2pD5cDvFaJPeJiBhVz0wRDJgt7BL3pd0TPiU+ awfFJo+ke8+Qwfnb+zVenuVl+TM+8a8vxqwnxPfM3iVYohdb5m8O6q3uPmbJMOGEp7mAvSTGbgKJ5 hsCB0PR5s5r9+gRJkKvnszrXe/Z18nj1R6QFnqvNjXSQJ62XY7e91UtA3v/wDKGr13hPVOES6CV+q TvJhT6Swuf0E4Q==; Received: from host-37-191-236-102.lynet.no ([37.191.236.102]:59890 helo=localhost) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nAeHh-0001p3-4P; Thu, 20 Jan 2022 15:43:53 -0500 From: Marius Bakke In-Reply-To: References: <877daubeju.fsf@gnu.org> Date: Thu, 20 Jan 2022 21:43:50 +0100 Message-ID: <874k5yb0jd.fsf@gnu.org> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha512; protocol="application/pgp-signature" X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Lars-Dominik Braun skriver: > Hi Marius, > >> Can you elaborate on the name clash issue? > the problem seems to be that most packages do not put the source code > into a subdirectory in the sdist. I.e. package foobar will have a foobar/ > directory, instead of src/foobar. Thus Python tries to load the module > foobar.barbaz from that directory, instead of the store. Thus it=E2=80=99= ll > also look there for C extensions and fail. I don=E2=80=99t know why it do= es > that, even when *not* using `python -m` =E2=80=93 according to the docume= ntation > it should not. If anyone has any pointers I=E2=80=99d like to have a look= again. Perhaps it could be worked around by building in a separate directory, i.e. ./build/lib, like setuptools does? And ensure that the original source directory does not end up on {,GUIX_}PYTHONPATH. (I don't really understand the problem, just throwing ideas around.) >> Unfortunately we need Python 2 for some time still. It is used to >> bootstrap old versions of GHC, Node, and probably other things. > But as far as I see these specific examples do not use > python-build-system. Only if some package using `#:python python-2` is > required during bootstrapping, then we=E2=80=99d need a way to support Py= thon > 2, right? Right. If we don't need any Python 2 modules it should be fine to remove support from the build system. >> Do you think it is feasible to provide this as a new build system, say >> pep517-build-system, during a transitional period? Then we can a) start >> using it right away for the increasing amount of packages that lack a >> setup.py; and b) flesh out most bugs before eventually merging it back >> (possibly piecemeal) into python-build-system. > Actually, I think that=E2=80=99s a good idea. I tried, but I cannot fix a= ll > packages broken by this change on my own, so smoothing the transition > could help and =E2=80=93 as you said =E2=80=93 we could catch bugs early = on. > > I could prepare a patch, if there is interest in this path. It would lower the bar significantly for testing and contributing, so I would appreciate it. :-) >> * I also think trying "python setup.py test" is unnecessary. > It still works quite often, although its usefulness will decrease in > the future I guess. Another problem I see is that this command will not > fail if there are no tests. > >> * It would be nice to support the "no tests" case without having to add >> explicit #:tests? everywhere. Perhaps along the lines of... > My idea was to force packagers to make and explain this decision > explicitly. If you don=E2=80=99t run tests, you have to add `#:tests? #f` > and leave a comment why they are disabled. I do see this could become > a hassle with packages that simply don=E2=80=99t have any tests. But my h= ope > is that it increases package quality. My main concern is end-user experience when they just want to get some random library working on their local channel. But it's arguably something that can be solved on the importer level and not a strong opinion. Thinking about importers, perhaps they could start emitting git origins when possible, as there is a trend to strip tests before uploading to PyPI. >> (_ (apply invoke `("python" "-m" "unittest" ,@test-flags)))))) > If I remember correctly I tried this and it did not work for some > reason. I=E2=80=99ll have a look again. Eh, it should be (apply invoke "python" "-m" "unittest" test-flags), but you probably got that. :-P Thanks! --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iIUEARYKAC0WIQRNTknu3zbaMQ2ddzTocYulkRQQdwUCYenJhg8cbWFyaXVzQGdu dS5vcmcACgkQ6HGLpZEUEHeGuwD9H2BY3O9ueLy3SB08wMJF/fov3rf5Yv8pnOR5 U1ljuasA/340yEq0yCLyQD4/iqk1v7G4JOwTP5DtgHDLiC+GTi0J =3u8K -----END PGP SIGNATURE----- --=-=-=-- From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 23 Jan 2022 05:30:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Lars-Dominik Braun Cc: 46848@debbugs.gnu.org Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.164291579118024 (code B ref 46848); Sun, 23 Jan 2022 05:30:02 +0000 Received: (at 46848) by debbugs.gnu.org; 23 Jan 2022 05:29:51 +0000 Received: from localhost ([127.0.0.1]:38406 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nBVRg-0004gU-Fm for submit@debbugs.gnu.org; Sun, 23 Jan 2022 00:29:51 -0500 Received: from mail-qv1-f47.google.com ([209.85.219.47]:38411) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nBVRe-0004gD-43 for 46848@debbugs.gnu.org; Sun, 23 Jan 2022 00:29:43 -0500 Received: by mail-qv1-f47.google.com with SMTP id kl12so16103543qvb.5 for <46848@debbugs.gnu.org>; Sat, 22 Jan 2022 21:29:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=G3LXihQLy5xEL4n12cXs3iSYOkLRnIvip6p/JlZvfSk=; b=TRNT/lbJ0qDYYbAVTLadeub8TZxt4keeLt9BkcTJY3WasWFyWue3xkS+nxlN34s7NF HGk7SI1xP0US+rztVFZusNAvqsw7R441TwI8xikdxNNYl213Sth3/g9yZoK8becZOyYp Qp2oMQS7I1/BF0GSfd+sMvtBMbJXn4tQbE6Vdx6ROZvSNnoOox5r4nomg60icfgP09Jq fBg8jufhRWezW0H3NNZVGi1CZRfH7Q23FWgwiY5YFJN3jGKvKjGLTke87mjN7SGNRDd+ TUjA3f3vINzsxtL1qIS585PgrVfYMwcR9zVF6PsO6gKD8ANWRdwePK4oER2UIYb/Ok00 c09w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=G3LXihQLy5xEL4n12cXs3iSYOkLRnIvip6p/JlZvfSk=; b=kS8xFM+pER9TiT4A+haz3qsH3XkHBoA5RUHmbM37JsBoRgXCh1zGjOYVo5Iz/hrc9E cKegJ2c4S75R+otQ24px5RrVsbeFNSgSGyO8jtNCcG1CG9l/2BAAj7Z0tZ6gAE8cvis2 q6vL4XY4fhA7NVNCxIPIMb5gEoVmzhwM4yFou/tMNyJO3rRskxlEIWG6WL9epez6IgXd /C7BdRdnRgf/K3c6kWONxZUkdEfnc7pEkXl/Klt4hMp+qIuW+YwiYoPSn3JiHtKHX7Um hqW0y8QbNyaDluIqDYRU3k8XgtIp3J9D4rgxRxsblrWMb3md3AZnaCNBfuWHJBoMehku PTnw== X-Gm-Message-State: AOAM530JWV6rbQa/K3jchD3rmv8sQxQxlZcU9vBU7s72ZpCYxXBHEj0n RKgimFkR89/gusS4fAMwFADQSGspjYQ= X-Google-Smtp-Source: ABdhPJxSRvONPB86BEnfFDfiTAWIvBbG9SrsI5IXSvvwnZM+Qa6d84W17niOcM16HV4BmXNZg3wezw== X-Received: by 2002:a05:6214:1ccb:: with SMTP id g11mr10112553qvd.97.1642915776274; Sat, 22 Jan 2022 21:29:36 -0800 (PST) Received: from hurd (dsl-205-236-230-254.b2b2c.ca. [205.236.230.254]) by smtp.gmail.com with ESMTPSA id t20sm5570067qta.41.2022.01.22.21.29.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 22 Jan 2022 21:29:35 -0800 (PST) From: Maxim Cournoyer References: Date: Sun, 23 Jan 2022 00:29:34 -0500 In-Reply-To: (Lars-Dominik Braun's message of "Mon, 1 Mar 2021 14:43:22 +0100") Message-ID: <87ee4z6mv5.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Lars, Here's a review of patches 1 to 6. Lars-Dominik Braun writes: > the attached patches switch python-build-system to a PEP 517-based build > system using python-pypa-build. Neat! Thank you for working on this! > One downside is that this tool is not self-contained and has a few > dependencies. Thus first I bootstrap setuptools using itself (possible > because it bundles all of its own dependencies), then build > python-pypa-build=E2=80=99s dependencies using setuptools (which is fortu= nately > still possible) and then combine everything into a > python-toolchain(-for-build), which is then used by the build-process. > > I can successfully build packages like python-pypa-build and > python-pytest and python-pep517-bootstrap. The latter is using flit as > its build backend. But other packages currently fail because I removed > some arguments. > Lars > > > > >>>From 61313d8ddba30772e2587e3e16ca30d1565d3c7e Mon Sep 17 00:00:00 2001 > From: Lars-Dominik Braun > Date: Sun, 28 Feb 2021 13:05:51 +0100 > Subject: [PATCH 04/12] gnu: python-setuptools: Bootstrap using itself > > * gnu/packages/python-xyz.scm (python-setuptools) [arguments]: Add phase > setting GUIX_PYTHONPATH to source directory. > --- > gnu/packages/python-xyz.scm | 13 ++++++++++++- > 1 file changed, 12 insertions(+), 1 deletion(-) > > diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm > index f8afa13f33..79d01f700a 100644 > --- a/gnu/packages/python-xyz.scm > +++ b/gnu/packages/python-xyz.scm > @@ -1144,7 +1144,18 @@ other machines, such as over the network.") > ;; FIXME: Tests require pytest, which itself relies on setuptools. > ;; One could bootstrap with an internal untested setuptools. > (arguments > - `(#:tests? #f)) > + `(#:tests? #f > + #:python ,python-wrapper > + #:phases (modify-phases %standard-phases > + ;; Use this setuptools=E2=80=99 sources to bootstrap t= hemselves. > + (add-before 'build 'set-PYTHONPATH ^ nitpick: GUIX_PYTHONPATH > + (lambda _ > + (format #t "current working dir ~s~%" (getcwd)) > + (setenv "GUIX_PYTHONPATH" > + (string-append ".:" (getenv "GUIX_PYTHONPA= TH"))) > + #t))))) > + ;; Not required when not building a wheel > + ;(propagated-inputs `(("python-wheel" ,python-wheel))) > (home-page "https://pypi.org/project/setuptools/") > (synopsis > "Library designed to facilitate packaging Python projects") > --=20 > 2.26.2 > > >>>From 7a99aaa40e65fde58ee2e78ad7d3e0ccd6d169ae Mon Sep 17 00:00:00 2001 > From: Lars-Dominik Braun > Date: Sun, 28 Feb 2021 13:08:58 +0100 > Subject: [PATCH 05/12] python-build: Switch to PEP 517-based build > > * gnu/packages/python-commencement.scm: New file, containing > python-toolchain. > * gnu/local.mk: Add it. > * gnu/packages/python.scm (python): Disable installing bundled > pip/setuptools. > * guix/build/python-build-system.scm: Rewrite using python-pypa-build. > * guix/build-system/python.scm (default-python): Switch to > python-toolchain > (lower): Remove unused parameter. > > XXX: rationale Forgotten XXX comment (perhaps just drop it). > --- > gnu/local.mk | 1 + > gnu/packages/python-commencement.scm | 175 +++++++++++++++++++ > gnu/packages/python.scm | 2 +- > guix/build-system/python.scm | 8 +- > guix/build/python-build-system.scm | 249 ++++++++++++++++++--------- > 5 files changed, 342 insertions(+), 93 deletions(-) > create mode 100644 gnu/packages/python-commencement.scm [...] > + (use-modules (ice-9 match) > + (srfi srfi-1) > + (srfi srfi-26) > + (guix build union)) > + > + (let ((out (assoc-ref %outputs "out"))) > + (union-build out (filter-map (match-lambda > + ((_ . directory) directo= ry)) > + %build-inputs)) > + #t)))) Note that returning #t after phases and snippets is obsolete; you can safely take them all out now. > + (inputs > + `(("python" ,python-wrapper) > + ("python-setuptools" ,python-setuptools) > + ("python-pip" ,python-pip))) ; XXX Maybe virtualenv/venv too? It = kind of > + ; defeats the purpose of guix, but i= s used > + ; alot in local development. I think it's OK to have people explicitly add virtualenv if they want it, rather than grow the set of minimal packages here. I'd simply remove the above XXX comment. > + (native-search-paths > + (package-native-search-paths python)) > + (search-paths > + (package-search-paths python)) > + (license (package-license python)) ; XXX > + (synopsis "Python toolchain") > + (description > + "Python toolchain including Python itself, setuptools and pip. Use= this > +package if you need a fully-fledged Python toolchain instead of just the > +interpreter.") > + (home-page (package-home-page python)))) Following my above comment, perhaps s/fully-fledged/minimal/. [...] > diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm > index 8e8f46467b..ca5ce667ef 100644 > --- a/gnu/packages/python.scm > +++ b/gnu/packages/python.scm > @@ -182,7 +182,7 @@ > (list "--enable-shared" ;allow embedding > "--with-system-expat" ;for XML support > "--with-system-ffi" ;build ctypes > - "--with-ensurepip=3Dinstall" ;install pip and setuptools > + "--with-ensurepip=3Dno" ;do not install pip and setupt= ools > "--enable-unicode=3Ducs4" >=20=20 > ;; Prevent the installed _sysconfigdata.py from retaining a= reference > diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm > index 2bb6fa87ca..998ea9323d 100644 > --- a/guix/build-system/python.scm > +++ b/guix/build-system/python.scm > @@ -65,8 +65,8 @@ extension, such as '.tar.gz'." > (define (default-python) > "Return the default Python package." > ;; Lazily resolve the binding to avoid a circular dependency. > - (let ((python (resolve-interface '(gnu packages python)))) > - (module-ref python 'python-wrapper))) > + (let ((python (resolve-interface '(gnu packages python-commencement)))) > + (module-ref python 'python-toolchain-for-build))) >=20=20 > (define (default-python2) > "Return the default Python 2 package." > @@ -172,8 +172,6 @@ pre-defined variants." > (define* (python-build store name inputs > #:key > (tests? #t) > - (test-target "test") > - (use-setuptools? #t) > (configure-flags ''()) > (phases '(@ (guix build python-build-system) > %standard-phases)) > @@ -199,9 +197,7 @@ provides a 'setup.py' file as its build system." > source)) > #:configure-flags ,configure-flags > #:system ,system > - #:test-target ,test-target > #:tests? ,tests? > - #:use-setuptools? ,use-setuptools? > #:phases ,phases > #:outputs %outputs > #:search-paths ',(map search-path-specification->se= xp > diff --git a/guix/build/python-build-system.scm b/guix/build/python-build= -system.scm > index 8ade1d5911..a5731511a9 100644 > --- a/guix/build/python-build-system.scm > +++ b/guix/build/python-build-system.scm > @@ -34,6 +34,7 @@ > #:use-module (ice-9 format) > #:use-module (srfi srfi-1) > #:use-module (srfi srfi-26) > + #:use-module (srfi srfi-35) > #:export (%standard-phases > add-installed-pythonpath > site-packages > @@ -108,30 +109,17 @@ > ;; "--single-version-externally-managed" is set, thus the .egg-info dire= ctory > ;; and the scripts defined in entry-points will always be created. >=20=20 > +;; Base error type. > +(define-condition-type &python-build-error &error > + python-build-error?) >=20=20 > -(define setuptools-shim > - ;; Run setup.py with "setuptools" being imported, which will patch > - ;; "distutils". This is needed for packages using "distutils" instead = of > - ;; "setuptools" since the former does not understand the > - ;; "--single-version-externally-managed" flag. > - ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py > - (string-append > - "import setuptools, tokenize;__file__=3D'setup.py';" > - "f=3Dgetattr(tokenize, 'open', open)(__file__);" > - "code=3Df.read().replace('\\r\\n', '\\n');" > - "f.close();" > - "exec(compile(code, __file__, 'exec'))")) > - > -(define (call-setuppy command params use-setuptools?) > - (if (file-exists? "setup.py") > - (begin > - (format #t "running \"python setup.py\" with command ~s and par= ameters ~s~%" > - command params) > - (if use-setuptools? > - (apply invoke "python" "-c" setuptools-shim > - command params) > - (apply invoke "python" "./setup.py" command params))) > - (error "no setup.py found"))) > +;; Raised when 'check cannot find a valid test system in the inputs. > +(define-condition-type &test-system-not-found &python-build-error > + test-system-not-found?) > + > +;; Raised when multiple wheels are created by 'build. > +(define-condition-type &cannot-extract-multiple-wheels &python-build-err= or > + cannot-extract-multiple-wheels?) >=20=20 > (define* (sanity-check #:key tests? inputs outputs #:allow-other-keys) > "Ensure packages depending on this package via setuptools work properl= y, > @@ -142,23 +130,51 @@ without errors." > (with-directory-excursion "/tmp" > (invoke "python" sanity-check.py (site-packages inputs outputs))))) >=20=20 > -(define* (build #:key use-setuptools? #:allow-other-keys) > +(define* (build #:key outputs #:allow-other-keys) > "Build a given Python package." > - (call-setuppy "build" '() use-setuptools?) > + > + (define pyproject-build (which "pyproject-build")) > + > + (define (build-pep517) > + ;; XXX: should probably use a different path, outside of source dire= ctory, > + ;; maybe secondary output =E2=80=9Cwheel=E2=80=9D? > + (mkdir-p "dist") > + (invoke pyproject-build "--outdir" "dist" "--no-isolation" "--wheel"= ".")) > + > + ;; XXX Would be nice, if we could use bdist_wheel here to remove e= xtra > + ;; code path in 'install, but that depends on python-wheel. > + (define (build-setuptools) > + (invoke "python" "setup.py" "build")) > + > + (if pyproject-build > + (build-pep517) > + (build-setuptools)) > #t) >=20=20 > -(define* (check #:key tests? test-target use-setuptools? #:allow-other-k= eys) > +(define* (check #:key inputs outputs tests? #:allow-other-keys) > "Run the test suite of a given Python package." > (if tests? > - ;; Running `setup.py test` creates an additional .egg-info directo= ry in > - ;; build/lib in some cases, e.g. if the source is in a sub-directo= ry > - ;; (given with `package_dir`). This will by copied to the output, = too, > - ;; so we need to remove. > - (let ((before (find-files "build" "\\.egg-info$" #:directories? #t= ))) > - (call-setuppy test-target '() use-setuptools?) > - (let* ((after (find-files "build" "\\.egg-info$" #:directories? = #t)) > - (inter (lset-difference string=3D? after before))) > - (for-each delete-file-recursively inter))) > + ;; Unfortunately with PEP 517 there is no common method to specify t= est > + ;; systems. Guess test system based on inputs instead. > + (let ((pytest (which "pytest")) > + (have-setup-py (file-exists? "setup.py"))) ^ indentation =20=20=20=20=20=20=20=20=20=20=20=20=20 > + ;; Prefer pytest > + ;; XXX: support nose You can remove this; nose is stale/deprecated. > + (cond > + (pytest > + (begin > + (format #t "using pytest~%") > + (invoke pytest "-vv"))) ; XXX: support skipping tests base= d on name/extra arguments? We could have a #:test-command argument to specify an arbitrary command as a list of strings, such as used by the emacs-build-system; that'd allow us to avoid overriding the phase just to add a '-k "not this-test"' or similar. > + ;; But fall back to setup.py, which should work for most > + ;; packages. XXX: would be nice not to depend on setup.py here= ? fails > + ;; more often than not to find any tests at all. Maybe we can = run > + ;; `python -m unittest`? > + (have-setup-py > + (begin > + (format #t "using setup.py~%") > + (invoke "python" "setup.py" "test" "-v"))) As Marius noted, falling back to 'python setup.py test' is not desirable; it's scheduled to be removed already. > + ;; The developer should explicitly disable tests in this case. > + (#t (raise (condition (&test-system-not-found)))))) > (format #t "test suite not run~%")) > #t) >=20=20 > @@ -195,31 +211,109 @@ running checks after installing the package." > "/bin:" > (getenv "PATH")))) >=20=20 > -(define* (install #:key inputs outputs (configure-flags '()) use-setupto= ols? > - #:allow-other-keys) > - "Install a given Python package." > - (let* ((out (python-output outputs)) > - (python (assoc-ref inputs "python")) > - (major-minor (map string->number > - (take (string-split (python-version python) #= \.) 2))) > - (<3.7? (match major-minor > - ((major minor) > - (or (< major 3) (and (=3D major 3) (< minor 7)))))) > - (params (append (list (string-append "--prefix=3D" out) > - "--no-compile") > - (if use-setuptools? > - ;; distutils does not accept these flags > - (list "--single-version-externally-managed" > - "--root=3D/") > - '()) > - configure-flags))) > - (call-setuppy "install" params use-setuptools?) > - ;; Rather than produce potentially non-reproducible .pyc files on Py= thons > - ;; older than 3.7, whose 'compileall' module lacks the > - ;; '--invalidation-mode' option, do not generate any. > - (unless <3.7? > - (invoke "python" "-m" "compileall" "--invalidation-mode=3Dunchecke= d-hash" > - out)))) > +(define* (install #:key inputs outputs (configure-flags '()) #:allow-oth= er-keys) > + "Install a wheel file according to PEP 427" > + ;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-di= stribution-1-0-py32-none-any-whl > + (let* ((site-dir (site-packages inputs outputs)) > + (out (assoc-ref outputs "out"))) > + (define (extract file) > + "Extract wheel (ZIP file) into site-packages directory" > + ;; Use Python=E2=80=99s zipfile to avoid extra dependency =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20= =20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20=20 > + (invoke "python" "-m" "zipfile" "-e" file site-dir)) > + > + (define python-hashbang > + (string-append "#!" (assoc-ref inputs "python") "/bin/python")) > + > + (define (move-data source destination) > + (mkdir-p (dirname destination)) > + (rename-file source destination)) > + > + (define (move-script source destination) > + "Move executable script file from .data/scripts to out/bin and rep= lace > +temporary hashbang" > + (move-data source destination) > + ;; ZIP does not save/restore permissions, make executable > + ;; XXX: might not be a file, but directory with subdirectories > + (chmod destination #o755) > + (substitute* destination (("#!python") python-hashbang))) It seems the directory case should be handled? Otherwise the substitute* call would error out upon encountering it. > + ;; Python=E2=80=99s distutils.command.install defines this mapping f= rom source to > + ;; destination mapping. > + (define install-schemes > + `(("scripts" "bin" ,move-script) > + ;; XXX: Why does Python not use share/ here? > + ("data" "share" ,move-data))) > + > + (define (expand-data-directory directory) > + "Move files from all .data subdirectories to their respective > +destinations." > + (for-each > + (match-lambda ((source destination function) > + (let ((source-path (string-append directory "/" source)) > + (destination-path (string-append out "/" destination))) > + (when (file-exists? source-path) > + (begin > + ;; This assumes only files exist in the scripts/ directo= ry. > + (for-each > + (lambda (file) > + (apply > + function > + (list > + (string-append source-path "/" file) > + (string-append destination-path "/" file)))) > + (scandir source-path (negate (cut member <> '("." ".."= ))))) > + (rmdir source-path)))))) > + install-schemes)) > +=20=20=20=20 > + (define pyproject-build (which "pyproject-build")) > + > + (define (list-directories base predicate) > + ;; Cannot use find-files here, because it=E2=80=99s recursive. > + (scandir > + base > + (lambda (name) > + (let ((stat (lstat (string-append base "/" name)))) > + (and > + (not (member name '("." ".."))) > + (eq? (stat:type stat) 'directory) > + (predicate name stat)))))) > + > + (define (install-pep517) > + "Install a wheel generated by a PEP 517-compatible builder." > + (let ((wheels (find-files "dist" "\\.whl$"))) ; XXX: do not recurse If we do not want to recurse, we should use scandir? > + (when (> (length wheels) 1) ; This code does not support multiple = wheels > + ; yet, because their outputs would hav= e to be > + ; merged properly. > + (raise (condition (&cannot-extract-multiple-wheels)))) > + (for-each extract wheels)) > + (let ((datadirs (map > + (cut string-append site-dir "/" <>) > + (list-directories site-dir (file-name-predicate "\\.data$"))))) > + (for-each (lambda (directory) > + (expand-data-directory directory) > + (rmdir directory)) > + datadirs))) > + > + (define (install-setuptools) > + "Install using setuptools." > + (let ((out (assoc-ref outputs "out"))) > + (invoke "python" "setup.py" > + "install" > + "--prefix" out > + "--single-version-externally-managed" > + "--root=3D/"))) > + > + (if pyproject-build > + (install-pep517) > + (install-setuptools)) > + #t)) So, IIUC, this complicated install phase is because we no longer take 'pip' for granted and is only later available, built from this very build system, right? Otherwise installing a wheel with pip would be trivial (c.f. python-isort). > + > +(define* (compile-bytecode #:key inputs outputs (configure-flags '()) #:= allow-other-keys) > + "Compile installed byte-code in site-packages." > + (let ((site-dir (site-packages inputs outputs))) > + (invoke "python" "-m" "compileall" site-dir) > + ;; XXX: We could compile with -O and -OO too here, at the cost of mo= re space. > + #t)) I think you can drop that comment; the default sounds reasonable: -o OPT_LEVELS Optimization levels to run compilation with. Defaul= t is -1 which uses the optimization level of the Python interpreter itself= (see -O). If we ever want to change we could globally change it for our Python. I think we keep using "--invalidation-mode=3Dunchecked-hash" though, for performance [0]: Unchecked hash-based .pyc files are a useful performance optimization for environments where a system external to Python (e.g., the build system) is responsible for keeping .pyc files up-to-date. [0] https://docs.python.org/3.7/whatsnew/3.7.html#pep-552-hash-based-pyc-fi= les [...] It looks rather good to me, with the above comments! I'll be looking forward reviewing the rest of this series shortly. Thank you! Maxim From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 23 Jan 2022 10:22:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxim Cournoyer Cc: 46848@debbugs.gnu.org Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.164293329415605 (code B ref 46848); Sun, 23 Jan 2022 10:22:02 +0000 Received: (at 46848) by debbugs.gnu.org; 23 Jan 2022 10:21:34 +0000 Received: from localhost ([127.0.0.1]:38586 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nBa05-00043d-MQ for submit@debbugs.gnu.org; Sun, 23 Jan 2022 05:21:33 -0500 Received: from mout-p-101.mailbox.org ([80.241.56.151]:34616) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nBa04-00043O-02 for 46848@debbugs.gnu.org; Sun, 23 Jan 2022 05:21:33 -0500 Received: from smtp102.mailbox.org (smtp102.mailbox.org [IPv6:2001:67c:2050:105:465:1:3:0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-101.mailbox.org (Postfix) with ESMTPS id 4JhTfs3W1yz9sl0; Sun, 23 Jan 2022 11:21:25 +0100 (CET) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1642933281; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=076ctM8sG9BBWoVa8w+krO0Gge6eatZ/+gvKm6RxsWA=; b=lYnLQoLONOcqxf2rML2zUXdobOgy/E+FjGe65If68sHbRMqBUQ/Cii5kyOWB4FMM7869bM FQRT09r+yO88Vy0WikZjj/Ow1DlvA0N3DEaaMWyINR6PCgPvL7+p9nkDb5O6vj1hGJVecW iM7wOu0IH4O9t5W911VWpAcQPEOxgFBSmZ50+86xLMNokRvEnRnHezy4nl0uCGZhk4MH/T H7zIFhlv8jZj8duSKrBZU8NzrrRE13B2HpBZcGk8dJhXpI9PG6gBwFIHmQX52lK67nv0x/ PY7o/e3tXcvLAiEb3GhWU26nJVG+MVGaJvyqgQmGBUp4vsYc1aY66J1aLtO1jw== Date: Sun, 23 Jan 2022 11:21:17 +0100 From: Lars-Dominik Braun Message-ID: References: <87ee4z6mv5.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <87ee4z6mv5.fsf@gmail.com> X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi Maxim, > Here's a review of patches 1 to 6. thanks for the review. Unfortunately this is not the most recent proposal and I have no way to retract the previous patches. I pushed v3 to the wip-python-pep517 branch, because of the sheer number of patches and so the CI could build it (since it requires a rebuild of the entire rust bootstrap chain). > > + ;; Prefer pytest > > + ;; XXX: support nose > > You can remove this; nose is stale/deprecated. So it’s preferred to replace 'check in cases where python-nose is still in use? > > > + (cond > > + (pytest > > + (begin > > + (format #t "using pytest~%") > > + (invoke pytest "-vv"))) ; XXX: support skipping tests based on name/extra arguments? > > We could have a #:test-command argument to specify an arbitrary command > as a list of strings, such as used by the emacs-build-system; that'd > allow us to avoid overriding the phase just to add a '-k "not > this-test"' or similar. I added #:test-flags in my v3 proposal. > > + ;; But fall back to setup.py, which should work for most > > + ;; packages. XXX: would be nice not to depend on setup.py here? fails > > + ;; more often than not to find any tests at all. Maybe we can run > > + ;; `python -m unittest`? > > + (have-setup-py > > + (begin > > + (format #t "using setup.py~%") > > + (invoke "python" "setup.py" "test" "-v"))) > > As Marius noted, falling back to 'python setup.py test' is not > desirable; it's scheduled to be removed already. Sure, but using `python -m unittest` instead requires some investigation. > > + (define (move-script source destination) > > + "Move executable script file from .data/scripts to out/bin and replace > > +temporary hashbang" > > + (move-data source destination) > > + ;; ZIP does not save/restore permissions, make executable > > + ;; XXX: might not be a file, but directory with subdirectories > > + (chmod destination #o755) > > + (substitute* destination (("#!python") python-hashbang))) > > It seems the directory case should be handled? Otherwise the > substitute* call would error out upon encountering it. I have not seen anyone using subdirectories in bin/ yet. Is that supported anywhere? > So, IIUC, this complicated install phase is because we no longer take > 'pip' for granted and is only later available, built from this very > build system, right? Otherwise installing a wheel with pip would be > trivial (c.f. python-isort). If we want to bootstrap these two packages easily (and possibly start unvendoring their vendored dependencies later), they cannot be part of this build system and thus we need to implement building/installing ourselves. I tried using pypa-build in an earlier version, but the bootstrap chain is unmaintainable. There also is a project called installer[1], but it does not have a CLI yet. Cheers, Lars [1] https://github.com/pradyunsg/installer From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sat, 26 Feb 2022 14:11:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Lars-Dominik Braun Cc: 46848@debbugs.gnu.org, Marius Bakke Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.164588464028488 (code B ref 46848); Sat, 26 Feb 2022 14:11:01 +0000 Received: (at 46848) by debbugs.gnu.org; 26 Feb 2022 14:10:40 +0000 Received: from localhost ([127.0.0.1]:54560 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nNxmR-0007PQ-RF for submit@debbugs.gnu.org; Sat, 26 Feb 2022 09:10:40 -0500 Received: from mail-qt1-f178.google.com ([209.85.160.178]:33719) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nNxmQ-0007PE-Py for 46848@debbugs.gnu.org; Sat, 26 Feb 2022 09:10:39 -0500 Received: by mail-qt1-f178.google.com with SMTP id bt3so5090946qtb.0 for <46848@debbugs.gnu.org>; Sat, 26 Feb 2022 06:10:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=+IafuyIVFUbPbdWcvsX5K2XzamJ29hI10nu437w7zk4=; b=ZeJ/61GM4WnY5C7bCL4VmZZNXfiX/yIhxrbaY/D1zLjurXNPJPlphqlYnAUvU10v6T nxXv1GbXflh49J6q0+TzKcqEsEPi8UMBohFkeoFuSEe51Vq/HHRCLgKznEp+RRJ/d4AZ BulsxwxD5bRo5oQll8/jksMIyg+pA3Q2cm6F4gaFssbNs8gmBkkCNjrDnly5aEc5154V 1LLTFYyDAsqe6I+ut+kJexCX8ajLWUwbe0A3o//mmBXDYQX1PAixUl0M/mMt1sNEthf2 sivXiiQpdxIxxEQS10oHRyuOHX/HXcHFqVaz9ty9J1l8sIG1/2ESF6VOj8MfidJqDaLD zVXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=+IafuyIVFUbPbdWcvsX5K2XzamJ29hI10nu437w7zk4=; b=bLzcwRFrAlvXvq9G9YlX7gJ2pXqRJx0GTeAZhqrqCwQ0wxAzzaPeFAtDYeWosxAvHW rm5ezKSgskIR9vihEwZmJNDO/UIYwVqsasz8BrbCwjum1H91VolpXPaE12SE9av/S5QD ltf313xYrA5PiKNcplsbbsXalzfj1siqbXhYaY2Nko3par17ayxubNHsdGVdNKmGBtks q9v7fc+T5eUX3nc87ItHoJnV3KQCEq2YN/sgAVeEpcVQNanAPV/nqEy98R+k9RIUEXpW aPdO40UHhaO8tS4NdDJ3PMtxM8sIA3aWGICX9vWA32/5WjtQ09jUJYx0G/hhBPkqeMcY OYfQ== X-Gm-Message-State: AOAM531xt7oOGHE6kmR094wg05NheddkLIAzgJ/7+68OF4QqEjryvvNl tbMNIpqFr8D3XBrTBJKa08lMDE3Oqm7EWQ== X-Google-Smtp-Source: ABdhPJz7TVIWX9pLmXS0jlRlA/qMN0skHZE95BrYf4JWHExiAecSvS7yMgjPpPQQdWSR1IYAQUOuzA== X-Received: by 2002:a05:622a:49:b0:2de:4d52:5e76 with SMTP id y9-20020a05622a004900b002de4d525e76mr10280396qtw.633.1645884633077; Sat, 26 Feb 2022 06:10:33 -0800 (PST) Received: from hurd (dsl-149-67.b2b2c.ca. [66.158.149.67]) by smtp.gmail.com with ESMTPSA id u14-20020ac858ce000000b002de89087e7dsm3477326qta.78.2022.02.26.06.10.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 26 Feb 2022 06:10:32 -0800 (PST) From: Maxim Cournoyer References: <877daubeju.fsf@gnu.org> Date: Sat, 26 Feb 2022 09:10:31 -0500 In-Reply-To: (Lars-Dominik Braun's message of "Thu, 20 Jan 2022 19:43:24 +0100") Message-ID: <87h78lwwfc.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Lars-Dominik, Lars-Dominik Braun writes: When you judge the branch ready to merge, could you please send a subset of the patches (at least the ones touching the python-build-system directly) to this issue (marked as v2 -- git send-email -v2) so that they can be more easily commented? Thanks, Maxim From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 28 Feb 2022 19:26:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxim Cournoyer Cc: 46848@debbugs.gnu.org, Marius Bakke Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.164607634911766 (code B ref 46848); Mon, 28 Feb 2022 19:26:01 +0000 Received: (at 46848) by debbugs.gnu.org; 28 Feb 2022 19:25:49 +0000 Received: from localhost ([127.0.0.1]:34814 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nOleX-00033i-1O for submit@debbugs.gnu.org; Mon, 28 Feb 2022 14:25:49 -0500 Received: from mout-p-202.mailbox.org ([80.241.56.172]:44560) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nOleV-00033V-L2 for 46848@debbugs.gnu.org; Mon, 28 Feb 2022 14:25:48 -0500 Received: from smtp1.mailbox.org (smtp1.mailbox.org [IPv6:2001:67c:2050:105:465:1:1:0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-202.mailbox.org (Postfix) with ESMTPS id 4K6r2F2D9pz9sSP; Mon, 28 Feb 2022 20:25:41 +0100 (CET) Date: Mon, 28 Feb 2022 20:25:35 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1646076339; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=xKT+HiigVesq6XJmnILTGYdVR1W7wl1SiFr2Hslmd4o=; b=Ly9G01JHMsDvIQATPv0lz/qwIlC526/xLSAoCd6xkPVr0kOqF7JQiKsDpDtHJ+l5X4fs/N W1+qOKq81Rbawb8DHK0ingtPxnBHEYBUrrl/OjI2jBnTW1wTvKJW/yUXwgufYLKp0C9XXl Z+QgmUy7TpSGXbxTVSDxqJYS97Fnjh7lpmHJKLqXnhC284cr8shZijLJpbos7JzTI/7L2J KLghG1ghQeUSAwMlGpkJE5Q/eAbLNyCvmvNeGMr1P1jEopyYpCbNEQo0jAtP3tH3oCJwEz vYi2OE2rrW7ngWVzfkBXs+bLSB7Aa3IquaUIqUeykVy45h+xAbk6SzdEvL1MaQ== From: Lars-Dominik Braun Message-ID: References: <877daubeju.fsf@gnu.org> <87h78lwwfc.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87h78lwwfc.fsf@gmail.com> X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi Maxim, > When you judge the branch ready to merge, could you please send a subset > of the patches (at least the ones touching the python-build-system > directly) to this issue (marked as v2 -- git send-email -v2) so that > they can be more easily commented? as soon as time permits I can do that. Which route do we want to take? Replace python-build-system entirely in one big merge or add a new one and slowly migrate? Cheers, Lars From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 28 Feb 2022 22:34:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Lars-Dominik Braun Cc: 46848@debbugs.gnu.org, Marius Bakke Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.164608758329713 (code B ref 46848); Mon, 28 Feb 2022 22:34:01 +0000 Received: (at 46848) by debbugs.gnu.org; 28 Feb 2022 22:33:03 +0000 Received: from localhost ([127.0.0.1]:34937 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nOoZj-0007jB-Ek for submit@debbugs.gnu.org; Mon, 28 Feb 2022 17:33:03 -0500 Received: from mail-qk1-f181.google.com ([209.85.222.181]:44932) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nOoZi-0007ih-En for 46848@debbugs.gnu.org; Mon, 28 Feb 2022 17:33:02 -0500 Received: by mail-qk1-f181.google.com with SMTP id q4so11559702qki.11 for <46848@debbugs.gnu.org>; Mon, 28 Feb 2022 14:33:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=GG/Borwkd5kRIIfjeYsGuF9q5EuBgjTAtIize6ztPkM=; b=qoVQ+CUbH/xbVr+P1eMswc5H4/PPvLlBG7YPayzK6xIpKfXb4JxC4jroKxI5sHtthg oBBXqlfom4DpW4Nj/C0bjHhF1J+BFKrMaIs1JimrOSrdZaT+xpkZw/1WW2PbnrCXVUbP rRDQPSGdVbyEQWRCoQm9WTM8qQypBG52K/YOZS7j2MB5fF/xmVvIaZMvUcrDNfKaShcd FH9fSrdvkodI1IUIA5pjO3pkWsFROCRgowFwDEQeQ3MPyR0fzQDpTOzL6fULothnzy5e 7CN/0F1zV1HVtWpfGtmG4oyb5Wk6CQR+Gcko4dbAGLuT7RNEFO2iR2oxtJoXCprjO3qz 5Ajg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=GG/Borwkd5kRIIfjeYsGuF9q5EuBgjTAtIize6ztPkM=; b=Bwm2Gb4wDA03V2C+DP1LM05fhERYlXTaqXge2y06rjn/pLDNuA8+Q8BJf19WgiaAyS BcR9RKJjREeAmImnRKD5f7dLngNwkz2O0QJ3xWHRQcvgNKjOYp9SHljKFprDG98oIMUg /UtLABXotnMOBaI+nEO66OXhk7NiJ8EjRUhRwHHD5frcloC3+dvtu1WfYncxedCKHpv2 BNNIVtyKgaIB8pme6+K2SeQQl5gB4ecJ0y6Aivae91QZoudztiJvaE5AHZAeWNt3eG8U UFYWJg1QxE76DupsZlMqJ8f6ltlPPA4beY9AbGiLhCzrS5ubmhBuPZLQi3jLYCDo6koi fyMg== X-Gm-Message-State: AOAM530hGM9dew8+SeQEmGGl9hNvkmC8hq4Pwa7aNkHf2fFfLpzzVfht kioA9fBvSFH4DCcyy7IAKlHDJ1790eQR5A== X-Google-Smtp-Source: ABdhPJwMpwgLPI1CAFhQIR0M2WlVhFoKwCJrMhhDLZ+1ULvf1aYPVC85lA7EcE4FTixrAHJfFyxtcg== X-Received: by 2002:a37:b885:0:b0:606:f607:d820 with SMTP id i127-20020a37b885000000b00606f607d820mr12157614qkf.124.1646087575707; Mon, 28 Feb 2022 14:32:55 -0800 (PST) Received: from hurd (dsl-141-227.b2b2c.ca. [66.158.141.227]) by smtp.gmail.com with ESMTPSA id w3-20020ac857c3000000b002de99dba1c3sm7356083qta.90.2022.02.28.14.32.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Feb 2022 14:32:55 -0800 (PST) From: Maxim Cournoyer References: <877daubeju.fsf@gnu.org> <87h78lwwfc.fsf@gmail.com> Date: Mon, 28 Feb 2022 17:32:54 -0500 In-Reply-To: (Lars-Dominik Braun's message of "Mon, 28 Feb 2022 20:25:35 +0100") Message-ID: <878rtusju1.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Lars, Lars-Dominik Braun writes: > Hi Maxim, > >> When you judge the branch ready to merge, could you please send a subset >> of the patches (at least the ones touching the python-build-system >> directly) to this issue (marked as v2 -- git send-email -v2) so that >> they can be more easily commented? > > as soon as time permits I can do that. Which route do we want to > take? Replace python-build-system entirely in one big merge or add a > new one and slowly migrate? After the Berlin can be rebooted onto its new file system, I expect it should allow us to keep the build farm much busier than in the past give us the headroom to experiment with a PEP 517-focused world rebuilding branch. I'd aim to have the build system completely replaced; unless your experience suggests it's going to take multiple weeks to fix iron out the kinks? Thanks, Maxim From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 24 Apr 2022 09:14:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxim Cournoyer Cc: 46848@debbugs.gnu.org, Marius Bakke Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.16507916371704 (code B ref 46848); Sun, 24 Apr 2022 09:14:02 +0000 Received: (at 46848) by debbugs.gnu.org; 24 Apr 2022 09:13:57 +0000 Received: from localhost ([127.0.0.1]:57609 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1niYJZ-0000RQ-Cb for submit@debbugs.gnu.org; Sun, 24 Apr 2022 05:13:57 -0400 Received: from mout-p-102.mailbox.org ([80.241.56.152]:53594) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1niYJX-0000R2-Dt for 46848@debbugs.gnu.org; Sun, 24 Apr 2022 05:13:55 -0400 Received: from smtp102.mailbox.org (smtp102.mailbox.org [IPv6:2001:67c:2050:105:465:1:3:0]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4KmMrr69x5z9sT4; Sun, 24 Apr 2022 11:13:48 +0200 (CEST) Date: Sun, 24 Apr 2022 11:13:39 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1650791626; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kzs3otwLpZAn6ibU6bOVsP/FFnSrJlJWEXDgPiCOIa0=; b=juSKHVoh88CUQRfCl3Z1AceGoxk0w53HQ+yZB4dh9KsIXU4dgyeT4jRbIVvShd8DCM/AEI CLCZNC6aygUkyYFfog4yqw5d+nvXyXd1aOYnUf9CzaJ6DtRacDgRmYYhEfUFeZZfSZ2ZSu xejc3eJ03FfuMwMOuMMYUEhsspbKhjmeIklV5IPIt2TGoiqnueAJuiWARhVZ2mjl3AKvum QVc8kLJe6lBMcptF6wycPHwKNbeTDJuMbLzE81SnuITuOM4HcrsTb240R3F/BHg9W/M3QL OYjiik/TGleV2gAYwseRbUy0Zpydiz+MvCAa8/wwTBbv55lyLmgnAawrkSrBYA== From: Lars-Dominik Braun Message-ID: References: <877daubeju.fsf@gnu.org> <87h78lwwfc.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="JgXvKQ6Yd69jxylY" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <87h78lwwfc.fsf@gmail.com> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" --JgXvKQ6Yd69jxylY Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit Hi Maxim, > When you judge the branch ready to merge, could you please send a subset > of the patches (at least the ones touching the python-build-system > directly) to this issue (marked as v2 -- git send-email -v2) so that > they can be more easily commented? I had some time to finish my work, so I pushed all of my changes to wip-python-pep517 and attached changes that do not fix individual packages to this email. Since my last version I added support for building Python 2 packages, although we should really phase out Python 2 asap. I kept support for nose and setup.py’s test target, because they are still in use/valuable, but we can also remove them if you prefer. Please have a look when time permits. If all is good we can move on to fix failing packages. Thank you very much, Lars --JgXvKQ6Yd69jxylY Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v3-0001-build-python-Handle-missing-setuptools-in-sanity-.patch" >From 720dbe22d431262938be29dd9a9ddb78c44a99b3 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Fri, 19 Feb 2021 17:22:35 +0100 Subject: [PATCH v3 001/150] build/python: Handle missing setuptools in sanity-check.py Just skip testing if required dependencies (setuptools) are not available. * gnu/packages/aux-files/python/sanity-check.py: Handle ImportError. --- gnu/packages/aux-files/python/sanity-check.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/gnu/packages/aux-files/python/sanity-check.py b/gnu/packages/aux-files/python/sanity-check.py index 182133bb3d..1366b68e3d 100644 --- a/gnu/packages/aux-files/python/sanity-check.py +++ b/gnu/packages/aux-files/python/sanity-check.py @@ -19,9 +19,13 @@ from __future__ import print_function # Python 2 support. import importlib -import pkg_resources import sys import traceback +try: + import pkg_resources +except ImportError: + print('Warning: Skipping, because python-setuptools are not available.') + sys.exit(0) try: from importlib.machinery import PathFinder -- 2.35.1 --JgXvKQ6Yd69jxylY Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="v3-0002-gnu-python-2-setuptools-Move-to-python-build.patch" Content-Transfer-Encoding: 8bit >From 4539d2994f454ca3528985bb140ec4aba8e919da Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sat, 23 Apr 2022 11:07:58 +0200 Subject: [PATCH v3 002/150] gnu: python{,2}-setuptools: Move to python-build. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * gnu/packages/python-xyz.scm (python{,2}-setuptools): Move… * gnu/packages/python-build.scm: …here. --- gnu/packages/python-build.scm | 90 +++++++++++++++++++++++++++++++++++ gnu/packages/python-xyz.scm | 90 ----------------------------------- 2 files changed, 90 insertions(+), 90 deletions(-) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 2abb781f76..f167c3953f 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -39,6 +39,96 @@ (define-module (gnu packages python-build) ;;; ;;; Code: +(define-public python-setuptools + (package + (name "python-setuptools") + (version "52.0.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "setuptools" version)) + (sha256 + (base32 + "15ibjdjhkwgj6qbmpsxikkqdfsb1550z46fly7dm15ah4bk1wfpv")) + (modules '((guix build utils))) + (snippet + '(begin + ;; Remove included binaries which are used to build self-extracting + ;; installers for Windows. + ;; TODO: Find some way to build them ourself so we can include them. + (for-each delete-file (find-files "setuptools" "^(cli|gui).*\\.exe$")) + #t)))) + (build-system python-build-system) + ;; FIXME: Tests require pytest, which itself relies on setuptools. + ;; One could bootstrap with an internal untested setuptools. + (arguments + `(#:tests? #f)) + (home-page "https://pypi.org/project/setuptools/") + (synopsis + "Library designed to facilitate packaging Python projects") + (description + "Setuptools is a fully-featured, stable library designed to facilitate +packaging Python projects, where packaging includes: +Python package and module definitions, +distribution package metadata, +test hooks, +project installation, +platform-specific details, +Python 3 support.") + ;; TODO: setuptools now bundles the following libraries: + ;; packaging, pyparsing, six and appdirs. How to unbundle? + (license (list license:psfl ; setuptools itself + license:expat ; six, appdirs, pyparsing + license:asl2.0 ; packaging is dual ASL2/BSD-2 + license:bsd-2)) + (properties `((python2-variant . ,(delay python2-setuptools)))))) + +;; Newer versions of setuptools no longer support Python 2. +(define-public python2-setuptools + (package + (name "python2-setuptools") + (version "41.0.1") + (source + (origin + (method url-fetch) + (uri (pypi-uri "setuptools" version ".zip")) + (sha256 + (base32 + "04sns22y2hhsrwfy1mha2lgslvpjsjsz8xws7h2rh5a7ylkd28m2")) + (modules '((guix build utils))) + (snippet + '(begin + ;; Remove included binaries which are used to build self-extracting + ;; installers for Windows. + ;; TODO: Find some way to build them ourself so we can include them. + (for-each delete-file (find-files "setuptools" "^(cli|gui).*\\.exe$")) + #t)))) + (build-system python-build-system) + ;; FIXME: Tests require pytest, which itself relies on setuptools. + ;; One could bootstrap with an internal untested setuptools. + (arguments + `(#:tests? #f)) + (native-inputs + (list unzip)) + (home-page "https://pypi.org/project/setuptools/") + (synopsis + "Library designed to facilitate packaging Python projects") + (description + "Setuptools is a fully-featured, stable library designed to facilitate +packaging Python projects, where packaging includes: +Python package and module definitions, +distribution package metadata, +test hooks, +project installation, +platform-specific details, +Python 3 support.") + ;; TODO: setuptools now bundles the following libraries: + ;; packaging, pyparsing, six and appdirs. How to unbundle? + (license (list license:psfl ; setuptools itself + license:expat ; six, appdirs, pyparsing + license:asl2.0 ; packaging is dual ASL2/BSD-2 + license:bsd-2)))) + (define-public python-wheel (package (name "python-wheel") diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index 219a0d9213..f2532abc52 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -1792,96 +1792,6 @@ (define-public python2-serpent `(("python-enum34" ,python2-enum34) ,@(package-propagated-inputs base)))))) -(define-public python-setuptools - (package - (name "python-setuptools") - (version "52.0.0") - (source - (origin - (method url-fetch) - (uri (pypi-uri "setuptools" version)) - (sha256 - (base32 - "15ibjdjhkwgj6qbmpsxikkqdfsb1550z46fly7dm15ah4bk1wfpv")) - (modules '((guix build utils))) - (snippet - '(begin - ;; Remove included binaries which are used to build self-extracting - ;; installers for Windows. - ;; TODO: Find some way to build them ourself so we can include them. - (for-each delete-file (find-files "setuptools" "^(cli|gui).*\\.exe$")) - #t)))) - (build-system python-build-system) - ;; FIXME: Tests require pytest, which itself relies on setuptools. - ;; One could bootstrap with an internal untested setuptools. - (arguments - `(#:tests? #f)) - (home-page "https://pypi.org/project/setuptools/") - (synopsis - "Library designed to facilitate packaging Python projects") - (description - "Setuptools is a fully-featured, stable library designed to facilitate -packaging Python projects, where packaging includes: -Python package and module definitions, -distribution package metadata, -test hooks, -project installation, -platform-specific details, -Python 3 support.") - ;; TODO: setuptools now bundles the following libraries: - ;; packaging, pyparsing, six and appdirs. How to unbundle? - (license (list license:psfl ; setuptools itself - license:expat ; six, appdirs, pyparsing - license:asl2.0 ; packaging is dual ASL2/BSD-2 - license:bsd-2)) - (properties `((python2-variant . ,(delay python2-setuptools)))))) - -;; Newer versions of setuptools no longer support Python 2. -(define-public python2-setuptools - (package - (name "python2-setuptools") - (version "41.0.1") - (source - (origin - (method url-fetch) - (uri (pypi-uri "setuptools" version ".zip")) - (sha256 - (base32 - "04sns22y2hhsrwfy1mha2lgslvpjsjsz8xws7h2rh5a7ylkd28m2")) - (modules '((guix build utils))) - (snippet - '(begin - ;; Remove included binaries which are used to build self-extracting - ;; installers for Windows. - ;; TODO: Find some way to build them ourself so we can include them. - (for-each delete-file (find-files "setuptools" "^(cli|gui).*\\.exe$")) - #t)))) - (build-system python-build-system) - ;; FIXME: Tests require pytest, which itself relies on setuptools. - ;; One could bootstrap with an internal untested setuptools. - (arguments - `(#:tests? #f)) - (native-inputs - (list unzip)) - (home-page "https://pypi.org/project/setuptools/") - (synopsis - "Library designed to facilitate packaging Python projects") - (description - "Setuptools is a fully-featured, stable library designed to facilitate -packaging Python projects, where packaging includes: -Python package and module definitions, -distribution package metadata, -test hooks, -project installation, -platform-specific details, -Python 3 support.") - ;; TODO: setuptools now bundles the following libraries: - ;; packaging, pyparsing, six and appdirs. How to unbundle? - (license (list license:psfl ; setuptools itself - license:expat ; six, appdirs, pyparsing - license:asl2.0 ; packaging is dual ASL2/BSD-2 - license:bsd-2)))) - (define-public python-setuptools-declarative-requirements (package (name "python-setuptools-declarative-requirements") -- 2.35.1 --JgXvKQ6Yd69jxylY Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="v3-0003-python-build-system-Use-PEP-517-compatible-builds.patch" Content-Transfer-Encoding: 8bit >From 9a120ae0f1791410e41951982ea3f44ad602dfec Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sat, 23 Apr 2022 11:36:55 +0200 Subject: [PATCH v3 003/150] python-build-system: Use PEP 517-compatible builds. This is effectively an entire rewrite of python-build-system. It supports all PEP 517-compatible build backends. * gnu/packages/python-commencement.scm: New file containing new Python toolchain package(s). * gnu/local.mk (GNU_SYSTEM_MODULES): Add it. * gnu/packages/python-build.scm (python-setuptools-bootstrap): New variable. (python2-setuptools-bootstrap): New variable. (python-wheel): Break bootstrap cycle. (python-wheel-bootstrap): New variable. (python2-wheel-bootstrap): New variable. * gnu/packages/python.scm (python-2.7): Do not install setuptools and pip. * guix/build-system/python.scm (%python-build-system-modules): Use (guix build json). (default-python): Default to python-toolchain-for-build. (lower): Add default wheel output, remove test-target and use-setuptools? flags, add build-backend, test-backend and test-flags. * guix/build/python-build-system.scm: Rewrite build system. --- gnu/local.mk | 1 + gnu/packages/python-build.scm | 104 ++++++- gnu/packages/python.scm | 2 +- guix/build-system/python.scm | 26 +- guix/build/python-build-system.scm | 436 +++++++++++++++++++---------- 5 files changed, 397 insertions(+), 172 deletions(-) diff --git a/gnu/local.mk b/gnu/local.mk index 0e721236d9..4298a2621b 100644 --- a/gnu/local.mk +++ b/gnu/local.mk @@ -495,6 +495,7 @@ GNU_SYSTEM_MODULES = \ %D%/packages/python.scm \ %D%/packages/python-build.scm \ %D%/packages/python-check.scm \ + %D%/packages/python-commencement.scm \ %D%/packages/python-compression.scm \ %D%/packages/python-crypto.scm \ %D%/packages/python-science.scm \ diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index f167c3953f..e23382972c 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -29,7 +29,9 @@ (define-module (gnu packages python-build) #:use-module (guix gexp) #:use-module (guix download) #:use-module (guix git-download) - #:use-module (guix packages)) + #:use-module (guix packages) + #:use-module (gnu packages) + #:use-module (gnu packages python)) ;;; Commentary: ;;; @@ -58,11 +60,23 @@ (define-public python-setuptools ;; TODO: Find some way to build them ourself so we can include them. (for-each delete-file (find-files "setuptools" "^(cli|gui).*\\.exe$")) #t)))) + (outputs '("out" "wheel")) (build-system python-build-system) ;; FIXME: Tests require pytest, which itself relies on setuptools. ;; One could bootstrap with an internal untested setuptools. (arguments - `(#:tests? #f)) + `(#:tests? #f + #:python ,python-wrapper ; Break cycle with default build system’s setuptools dependency. + #:phases (modify-phases %standard-phases + ;; Use this setuptools’ sources to bootstrap themselves. + (add-before 'build 'set-PYTHONPATH + (lambda _ + (format #t "current working dir ~s~%" (getcwd)) + (setenv "GUIX_PYTHONPATH" + (string-append ".:" (getenv "GUIX_PYTHONPATH"))) + #t))))) + ;; Required to build wheels. + (propagated-inputs `(("python-wheel" ,python-wheel))) (home-page "https://pypi.org/project/setuptools/") (synopsis "Library designed to facilitate packaging Python projects") @@ -83,6 +97,14 @@ (define-public python-setuptools license:bsd-2)) (properties `((python2-variant . ,(delay python2-setuptools)))))) +;; Break loop between python-setuptools and python-wheel. +(define-public python-setuptools-bootstrap + (package + (inherit python-setuptools) + (name "python-setuptools-bootstrap") + (propagated-inputs `(("python-wheel" ,python-wheel-bootstrap))) + (properties `((python2-variant . ,(delay python2-setuptools-bootstrap)))))) + ;; Newer versions of setuptools no longer support Python 2. (define-public python2-setuptools (package @@ -107,9 +129,10 @@ (define-public python2-setuptools ;; FIXME: Tests require pytest, which itself relies on setuptools. ;; One could bootstrap with an internal untested setuptools. (arguments - `(#:tests? #f)) - (native-inputs - (list unzip)) + `(#:tests? #f + #:python ,python-2 ; Break loop to python2-toolchain-for-build + )) + (propagated-inputs `(("python2-wheel" ,python2-wheel))) (home-page "https://pypi.org/project/setuptools/") (synopsis "Library designed to facilitate packaging Python projects") @@ -129,6 +152,12 @@ (define-public python2-setuptools license:asl2.0 ; packaging is dual ASL2/BSD-2 license:bsd-2)))) +(define-public python2-setuptools-bootstrap + (package + (inherit python2-setuptools) + (name "python2-setuptools-bootstrap") + (propagated-inputs `(("python2-wheel" ,python2-wheel-bootstrap))))) + (define-public python-wheel (package (name "python-wheel") @@ -142,10 +171,8 @@ (define-public python-wheel "1bbga5i49rj1cwi4sjpkvfhl1f8vl9lfky2lblsy768nk4wp5vz2")))) (build-system python-build-system) (arguments - ;; FIXME: The test suite runs "python setup.py bdist_wheel", which in turn - ;; fails to find the newly-built bdist_wheel library, even though it is - ;; available on PYTHONPATH. What search path is consulted by setup.py? - '(#:tests? #f)) + `(#:python ,python-wrapper)) ; Break cycle with python-toolchain-for-build. + (native-inputs `(("python-setuptools" ,python-setuptools-bootstrap))) (home-page "https://bitbucket.org/pypa/wheel/") (synopsis "Format for built Python packages") (description @@ -158,8 +185,65 @@ (define-public python-wheel installed with a newer @code{pip} or with wheel's own command line utility.") (license license:expat))) +(define-public python-wheel-bootstrap + (package + (inherit python-wheel) + (name "python-wheel-bootstrap") + (build-system copy-build-system) + (native-inputs '()) ; Break cycle to setuptools. + (arguments + `(#:install-plan + ;; XXX: Do not hard-code Python version. + '(("src/wheel" "lib/python3.9/site-packages/wheel")) + #:phases + (modify-phases %standard-phases + ;; Add metadata for setuptools, so it will find the wheel-building code. + (add-after 'install 'install-metadata + (lambda* (#:key outputs #:allow-other-keys) + (let* ((out (assoc-ref outputs "out")) + (site-dir (string-append out "/lib/python3.9/site-packages")) + (metadata-dir (string-append site-dir "/wheel.egg-info"))) + (mkdir-p metadata-dir) + (call-with-output-file (string-append metadata-dir "/entry_points.txt") + (lambda (port) + (format port "~ + [distutils.commands]~@ + bdist_wheel = wheel.bdist_wheel:bdist_wheel~%"))))))))) + (properties `((python2-variant . ,(delay python2-wheel-bootstrap)))))) + (define-public python2-wheel - (package-with-python2 python-wheel)) + (package + (inherit (package-with-python2 python-wheel)) + (arguments `(#:python ,python-2)))) + +(define-public python2-wheel-bootstrap + (package + (inherit python2-wheel) + (name "python2-wheel-bootstrap") + (build-system copy-build-system) + (native-inputs '()) ; Break cycle to setuptools. + (arguments + `(#:install-plan + ;; XXX: Do not hard-code Python version. + '(("src/wheel" "lib/python2.7/site-packages/wheel")) + #:phases + (modify-phases %standard-phases + ;; Add metadata for setuptools, so it will find the wheel-building code. + (add-after 'install 'install-metadata + (lambda* (#:key outputs #:allow-other-keys) + (let* ((out (assoc-ref outputs "out")) + (site-dir (string-append out "/lib/python2.7/site-packages")) + (metadata-dir (string-append site-dir "/wheel.egg-info"))) + (mkdir-p metadata-dir) + (call-with-output-file (string-append metadata-dir "/entry_points.txt") + (lambda (port) + (format port "~ + [distutils.commands]~@ + bdist_wheel = wheel.bdist_wheel:bdist_wheel~%"))) + (call-with-output-file (string-append metadata-dir "/PKG-INFO") + (lambda (port) + (format port "~ + Version: ~a" (version)))))))))))) ;;; XXX: Not really at home, but this seems the best place to prevent circular ;;; module dependencies. diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 3bc3346c21..4399d30aad 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -183,7 +183,7 @@ (define-public python-2.7 (list "--enable-shared" ;allow embedding "--with-system-expat" ;for XML support "--with-system-ffi" ;build ctypes - "--with-ensurepip=install" ;install pip and setuptools + "--with-ensurepip=no" ;do not install pip and setuptools "--with-computed-gotos" ;main interpreter loop optimization "--enable-unicode=ucs4" diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm index efade6f74b..aad861d278 100644 --- a/guix/build-system/python.scm +++ b/guix/build-system/python.scm @@ -62,18 +62,19 @@ (define* (pypi-uri name version #:optional (extension ".tar.gz")) (define %python-build-system-modules ;; Build-side modules imported by default. `((guix build python-build-system) + (guix build json) ,@%gnu-build-system-modules)) (define (default-python) "Return the default Python package." ;; Lazily resolve the binding to avoid a circular dependency. - (let ((python (resolve-interface '(gnu packages python)))) - (module-ref python 'python-wrapper))) + (let ((python (resolve-interface '(gnu packages python-commencement)))) + (module-ref python 'python-toolchain-for-build))) (define (default-python2) "Return the default Python 2 package." - (let ((python (resolve-interface '(gnu packages python)))) - (module-ref python 'python-2))) + (let ((python (resolve-interface '(gnu packages python-commencement)))) + (module-ref python 'python2-toolchain-for-build))) (define sanity-check.py ;; The script used to validate the installation of a Python package. @@ -165,26 +166,26 @@ (define private-keywords (build-inputs `(("python" ,python) ("sanity-check.py" ,(local-file sanity-check.py)) ,@native-inputs)) - (outputs outputs) + (outputs (append outputs '(wheel))) (build python-build) (arguments (strip-keyword-arguments private-keywords arguments))))) (define* (python-build name inputs #:key source (tests? #t) - (test-target "test") - (use-setuptools? #t) (configure-flags ''()) + (build-backend #f) + (test-backend #f) + (test-flags #f) (phases '%standard-phases) - (outputs '("out")) + (outputs '("out" "wheel")) (search-paths '()) (system (%current-system)) (guile #f) (imported-modules %python-build-system-modules) (modules '((guix build python-build-system) (guix build utils)))) - "Build SOURCE using PYTHON, and with INPUTS. This assumes that SOURCE -provides a 'setup.py' file as its build system." + "Build SOURCE using PYTHON, and with INPUTS." (define build (with-imported-modules imported-modules #~(begin @@ -194,9 +195,10 @@ (define build #~(python-build #:name #$name #:source #+source #:configure-flags #$configure-flags - #:use-setuptools? #$use-setuptools? #:system #$system - #:test-target #$test-target + #:build-backend #$build-backend + #:test-backend #$test-backend + #:test-flags #$test-flags #:tests? #$tests? #:phases #$(if (pair? phases) (sexp->gexp phases) diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm index 08871f60cd..15cbdd4e7c 100644 --- a/guix/build/python-build-system.scm +++ b/guix/build/python-build-system.scm @@ -30,11 +30,16 @@ (define-module (guix build python-build-system) #:use-module ((guix build gnu-build-system) #:prefix gnu:) #:use-module (guix build utils) + #:use-module (guix build json) #:use-module (ice-9 match) #:use-module (ice-9 ftw) #:use-module (ice-9 format) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 regex) #:use-module (srfi srfi-1) #:use-module (srfi srfi-26) + #:use-module (srfi srfi-34) + #:use-module (srfi srfi-35) #:export (%standard-phases add-installed-pythonpath site-packages @@ -43,96 +48,45 @@ (define-module (guix build python-build-system) ;; Commentary: ;; -;; Builder-side code of the standard Python package build procedure. +;; PEP 517-compatible build system for Python packages. ;; +;; PEP 517 mandates the use of a TOML file called pyproject.toml at the +;; project root, describing build and runtime dependencies, as well as the +;; build system, which can be different from setuptools. This module uses +;; that file to extract the build system used and call its wheel-building +;; entry point build_wheel (see 'build). setuptools’ wheel builder is +;; used as a fallback if either no pyproject.toml exists or it does not +;; declare a build-system. It supports config_settings through the +;; standard #:configure-flags argument. ;; -;; Backgound about the Python installation methods +;; This wheel, which is just a ZIP file with a file structure defined +;; by PEP 427 (https://www.python.org/dev/peps/pep-0427/), is then unpacked +;; and its contents are moved to the appropriate locations in 'install. ;; -;; In Python there are different ways to install packages: distutils, -;; setuptools, easy_install and pip. All of these are sharing the file -;; setup.py, introduced with distutils in Python 2.0. The setup.py file can be -;; considered as a kind of Makefile accepting targets (or commands) like -;; "build" and "install". As of autumn 2016 the recommended way to install -;; Python packages is using pip. +;; Then entry points, as defined by the PyPa Entry Point Specification +;; (https://packaging.python.org/specifications/entry-points/) are read +;; from a file called entry_points.txt in the package’s site-packages +;; subdirectory and scripts are written to bin/. These are not part of a +;; wheel and expected to be created by the installing utility. ;; -;; For both distutils and setuptools, running "python setup.py install" is the -;; way to install Python packages. With distutils the "install" command -;; basically copies all packages into /lib/pythonX.Y/site-packages. -;; -;; Some time later "setuptools" was established to enhance distutils. To use -;; setuptools, the developer imports setuptools in setup.py. When importing -;; setuptools, the original "install" command gets overwritten by setuptools' -;; "install" command. -;; -;; The command-line tools easy_install and pip are both capable of finding and -;; downloading the package source from PyPI (the Python Package Index). Both -;; of them import setuptools and execute the "setup.py" file under their -;; control. Thus the "setup.py" behaves as if the developer had imported -;; setuptools within setup.py - even is still using only distutils. -;; -;; Setuptools' "install" command (to be more precise: the "easy_install" -;; command which is called by "install") will put the path of the currently -;; installed version of each package and it's dependencies (as declared in -;; setup.py) into an "easy-install.pth" file. In Guix each packages gets its -;; own "site-packages" directory and thus an "easy-install.pth" of its own. -;; To avoid conflicts, the python build system renames the file to -;; .pth in the phase rename-pth-file. To ensure that Python will -;; process the .pth file, easy_install also creates a basic "site.py" in each -;; "site-packages" directory. The file is the same for all packages, thus -;; there is no need to rename it. For more information about .pth files and -;; the site module, please refere to -;; https://docs.python.org/3/library/site.html. -;; -;; The .pth files contain the file-system paths (pointing to the store) of all -;; dependencies. So the dependency is hidden in the .pth file but is not -;; visible in the file-system. Now if packages A and B both required packages -;; P, but in different versions, Guix will not detect this when installing -;; both A and B to a profile. (For details and example see -;; https://lists.gnu.org/archive/html/guix-devel/2016-10/msg01233.html.) -;; -;; Pip behaves a bit different then easy_install: it always executes -;; "setup.py" with the option "--single-version-externally-managed" set. This -;; makes setuptools' "install" command run the original "install" command -;; instead of the "easy_install" command, so no .pth file (and no site.py) -;; will be created. The "site-packages" directory only contains the package -;; and the related .egg-info directory. -;; -;; This is exactly what we need for Guix and this is what we mimic in the -;; install phase below. -;; -;; As a draw back, the magic of the .pth file of linking to the other required -;; packages is gone and these packages have now to be declared as -;; "propagated-inputs". -;; -;; Note: Importing setuptools also adds two sub-commands: "install_egg_info" -;; and "install_scripts". These sub-commands are executed even if -;; "--single-version-externally-managed" is set, thus the .egg-info directory -;; and the scripts defined in entry-points will always be created. - - -(define setuptools-shim - ;; Run setup.py with "setuptools" being imported, which will patch - ;; "distutils". This is needed for packages using "distutils" instead of - ;; "setuptools" since the former does not understand the - ;; "--single-version-externally-managed" flag. - ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py - (string-append - "import setuptools, tokenize;__file__='setup.py';" - "f=getattr(tokenize, 'open', open)(__file__);" - "code=f.read().replace('\\r\\n', '\\n');" - "f.close();" - "exec(compile(code, __file__, 'exec'))")) - -(define (call-setuppy command params use-setuptools?) - (if (file-exists? "setup.py") - (begin - (format #t "running \"python setup.py\" with command ~s and parameters ~s~%" - command params) - (if use-setuptools? - (apply invoke "python" "-c" setuptools-shim - command params) - (apply invoke "python" "./setup.py" command params))) - (error "no setup.py found"))) +;; Caveats: +;; - There is no support for in-tree build backends. + +;; Base error type. +(define-condition-type &python-build-error &error + python-build-error?) + +;; Raised when 'check cannot find a valid test system in the inputs. +(define-condition-type &test-system-not-found &python-build-error + test-system-not-found?) + +;; Raised when multiple wheels are created by 'build. +(define-condition-type &cannot-extract-multiple-wheels &python-build-error + cannot-extract-multiple-wheels?) + +;; Raised, when no wheel has been built by the build system. +(define-condition-type &no-wheels-built &python-build-error + no-wheels-built?) (define* (sanity-check #:key tests? inputs outputs #:allow-other-keys) "Ensure packages depending on this package via setuptools work properly, @@ -143,25 +97,83 @@ (define* (sanity-check #:key tests? inputs outputs #:allow-other-keys) (with-directory-excursion "/tmp" (invoke "python" sanity-check.py (site-packages inputs outputs))))) -(define* (build #:key use-setuptools? #:allow-other-keys) +(define* (build #:key outputs build-backend configure-flags #:allow-other-keys) "Build a given Python package." - (call-setuppy "build" '() use-setuptools?) - #t) -(define* (check #:key tests? test-target use-setuptools? #:allow-other-keys) + (define (pyproject.toml->build-backend file) + "Look up the build backend in a pyproject.toml file." + (call-with-input-file file + (lambda (in) + (let loop ((line (read-line in 'concat))) + (if (eof-object? line) + #f + (let ((m (string-match "build-backend = [\"'](.+)[\"']" line))) + (if m (match:substring m 1) + (loop (read-line in 'concat))))))))) + + (let* ((wheel-output (assoc-ref outputs "wheel")) + (wheel-dir (if wheel-output wheel-output "dist")) + ;; There is no easy way to get data from Guile into Python via + ;; s-expressions, but we have JSON serialization already, which Python + ;; also supports out-of-the-box. + (config-settings (call-with-output-string (cut write-json configure-flags <>))) + ;; python-setuptools’ default backend supports setup.py *and* + ;; pyproject.toml. Allow overriding this automatic detection via + ;; build-backend. + (auto-build-backend (if (file-exists? "pyproject.toml") + (pyproject.toml->build-backend "pyproject.toml") + #f)) + ;; Use build system detection here and not in importer, because a) we + ;; have alot of legacy packages and b) the importer cannot update arbitrary + ;; fields in case a package switches its build system. + (use-build-backend (or + build-backend + auto-build-backend + "setuptools.build_meta"))) + (format #t "Using '~a' to build wheels, auto-detected '~a', override '~a'.~%" + use-build-backend auto-build-backend build-backend) + (mkdir-p wheel-dir) + ;; Call the PEP 517 build function, which drops a .whl into wheel-dir. + (invoke "python" "-c" "import sys, importlib, json +config_settings = json.loads (sys.argv[3]) +builder = importlib.import_module(sys.argv[1]) +builder.build_wheel(sys.argv[2], config_settings=config_settings)" + use-build-backend wheel-dir config-settings))) + +(define* (check #:key inputs outputs tests? test-backend test-flags #:allow-other-keys) "Run the test suite of a given Python package." (if tests? - ;; Running `setup.py test` creates an additional .egg-info directory in - ;; build/lib in some cases, e.g. if the source is in a sub-directory - ;; (given with `package_dir`). This will by copied to the output, too, - ;; so we need to remove. - (let ((before (find-files "build" "\\.egg-info$" #:directories? #t))) - (call-setuppy test-target '() use-setuptools?) - (let* ((after (find-files "build" "\\.egg-info$" #:directories? #t)) - (inter (lset-difference string=? after before))) - (for-each delete-file-recursively inter))) - (format #t "test suite not run~%")) - #t) + ;; Unfortunately with PEP 517 there is no common method to specify test + ;; systems. Guess test system based on inputs instead. + (let* ((pytest (which "pytest")) + (nosetests (which "nosetests")) + (nose2 (which "nose2")) + (have-setup-py (file-exists? "setup.py")) + (use-test-backend + (or + test-backend + ;; Prefer pytest + (if pytest 'pytest #f) + (if nosetests 'nose #f) + (if nose2 'nose2 #f) + ;; But fall back to setup.py, which should work for most + ;; packages. XXX: would be nice not to depend on setup.py here? fails + ;; more often than not to find any tests at all. Maybe we can run + ;; `python -m unittest`? + (if have-setup-py 'setup.py #f)))) + (format #t "Using ~a~%" use-test-backend) + (match use-test-backend + ('pytest + (apply invoke (cons pytest (or test-flags '("-vv"))))) + ('nose + (apply invoke (cons nosetests (or test-flags '("-v"))))) + ('nose2 + (apply invoke (cons nose2 (or test-flags '("-v" "--pretty-assert"))))) + ('setup.py + (apply invoke (append '("python" "setup.py") (or test-flags '("test" "-v"))))) + ;; The developer should explicitly disable tests in this case. + (else (raise (condition (&test-system-not-found)))))) + (format #t "test suite not run~%"))) (define (python-version python) (let* ((version (last (string-split python #\-))) @@ -196,33 +208,175 @@ (define* (add-install-to-path #:key outputs #:allow-other-keys) "/bin:" (getenv "PATH")))) -(define* (install #:key inputs outputs (configure-flags '()) use-setuptools? - #:allow-other-keys) - "Install a given Python package." - (let* ((out (python-output outputs)) +(define* (install #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Install a wheel file according to PEP 427" + ;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl + (let* ((site-dir (site-packages inputs outputs)) + (python (assoc-ref inputs "python")) + (out (assoc-ref outputs "out"))) + (define (extract file) + "Extract wheel (ZIP file) into site-packages directory" + ;; Use Python’s zipfile to avoid extra dependency + (invoke "python" "-m" "zipfile" "-e" file site-dir)) + + (define python-hashbang + (string-append "#!" python "/bin/python")) + + (define* (merge-directories source destination #:optional (post-move #f)) + "Move all files in SOURCE into DESTINATION, merging the two directories." + (format #t "Merging directory ~a into ~a~%" source destination) + (for-each + (lambda (file) + (format #t "~a/~a -> ~a/~a~%" source file destination file) + (mkdir-p destination) + (rename-file + (string-append source "/" file) + (string-append destination "/" file)) + (when post-move + (post-move file))) + (scandir source (negate (cut member <> '("." ".."))))) + (rmdir source)) + + (define (expand-data-directory directory) + "Move files from all .data subdirectories to their respective +destinations." + ;; Python’s distutils.command.install defines this mapping from source to + ;; destination mapping. + (let ((source (string-append directory "/scripts")) + (destination (string-append out "/bin"))) + (when (file-exists? source) + (merge-directories + source + destination + (lambda (f) + (let ((dest-path (string-append destination "/" f))) + (chmod dest-path #o755) + (substitute* dest-path (("#!python") python-hashbang))))))) + ;; data can create arbitrary directory structures. Most commonly + ;; it is used for share/. + (let ((source (string-append directory "/data")) + (destination out)) + (when (file-exists? source) + (merge-directories source destination))) + (let* ((distribution (car (string-split (basename directory) #\-))) + (source (string-append directory "/headers")) + (destination (string-append out "/include/python" (python-version python) "/" distribution))) + (when (file-exists? source) + (merge-directories source destination)))) + + (define (list-directories base predicate) + ;; Cannot use find-files here, because it’s recursive. + (scandir + base + (lambda (name) + (let ((stat (lstat (string-append base "/" name)))) + (and + (not (member name '("." ".."))) + (eq? (stat:type stat) 'directory) + (predicate name stat)))))) + + (let* ((wheel-output (assoc-ref outputs "wheel")) + (wheel-dir (if wheel-output wheel-output "dist")) + (wheels (map (cut string-append wheel-dir "/" <>) + (scandir wheel-dir (cut string-suffix? ".whl" <>))))) + (cond + ((> (length wheels) 1) ; This code does not support multiple wheels + ; yet, because their outputs would have to be + ; merged properly. + (raise (condition (&cannot-extract-multiple-wheels)))) + ((= (length wheels) 0) + (raise (condition (&no-wheels-built))))) + (for-each extract wheels)) + (let ((datadirs (map + (cut string-append site-dir "/" <>) + (list-directories site-dir (file-name-predicate "\\.data$"))))) + (for-each (lambda (directory) + (expand-data-directory directory) + (rmdir directory)) + datadirs)))) + +(define* (compile-bytecode #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Compile installed byte-code in site-packages." + (let* ((site-dir (site-packages inputs outputs)) (python (assoc-ref inputs "python")) (major-minor (map string->number (take (string-split (python-version python) #\.) 2))) (<3.7? (match major-minor ((major minor) - (or (< major 3) (and (= major 3) (< minor 7)))))) - (params (append (list (string-append "--prefix=" out) - "--no-compile") - (if use-setuptools? - ;; distutils does not accept these flags - (list "--single-version-externally-managed" - "--root=/") - '()) - configure-flags))) - (call-setuppy "install" params use-setuptools?) - ;; Rather than produce potentially non-reproducible .pyc files on Pythons - ;; older than 3.7, whose 'compileall' module lacks the - ;; '--invalidation-mode' option, do not generate any. - (unless <3.7? - (invoke "python" "-m" "compileall" "--invalidation-mode=unchecked-hash" - out)))) - -(define* (wrap #:key inputs outputs #:allow-other-keys) + (or (< major 3) (and (= major 3) (< minor 7))))))) + (if <3.7? + ;; These versions don’t have the hash invalidation modes and do + ;; not produce reproducible bytecode files. + (format #t "Skipping bytecode compilation for Python version ~a < 3.7~%" (python-version python)) + (invoke "python" "-m" "compileall" "--invalidation-mode=unchecked-hash" site-dir)))) + +(define* (create-entrypoints #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Implement Entry Points Specification +(https://packaging.python.org/specifications/entry-points/) by PyPa, +which creates runnable scripts in bin/ from entry point specification +file entry_points.txt. This is necessary, because wheels do not contain +these binaries and installers are expected to create them." + + (define (entry-points.txt->entry-points file) + "Specialized parser for Python configfile-like files, in particular +entry_points.txt. Returns a list of console_script and gui_scripts +entry points." + (call-with-input-file file + (lambda (in) + (let loop ((line (read-line in)) + (inside #f) + (result '())) + (if (eof-object? line) + result + (let* ((group-match (string-match "^\\[(.+)\\]$" line)) + (group-name (if group-match (match:substring group-match 1) #f)) + (next-inside + (if (not group-name) + inside + (or + (string=? group-name "console_scripts") + (string=? group-name "gui_scripts")))) + (item-match (string-match "^([^ =]+)\\s*=\\s*([^:]+):(.+)$" line))) + (if (and inside item-match) + (loop (read-line in) next-inside (cons (list + (match:substring item-match 1) + (match:substring item-match 2) + (match:substring item-match 3)) + result)) + (loop (read-line in) next-inside result)))))))) + + (define (create-script path name module function) + "Create a Python script from an entry point’s NAME, MODULE and + FUNCTION and return write it to PATH/NAME." + (let ((interpreter (which "python")) + (file-path (string-append path "/" name))) + (format #t "Creating entry point for '~a.~a' at '~a'.~%" module function + file-path) + (call-with-output-file file-path + (lambda (port) + ;; Technically the script could also include search-paths, + ;; but having a generic 'wrap phases also handles manually + ;; written entry point scripts. + (format port "#!~a +# Auto-generated entry point script. +import sys +import ~a as mod +sys.exit (mod.~a ())~%" interpreter module function))) + (chmod file-path #o755))) + + (let* ((site-dir (site-packages inputs outputs)) + (out (assoc-ref outputs "out")) + (bin-dir (string-append out "/bin")) + (entry-point-files (find-files site-dir "^entry_points.txt$"))) + (mkdir-p bin-dir) + (for-each + (lambda (f) + (for-each + (lambda (ep) (apply create-script (cons bin-dir ep))) + (entry-points.txt->entry-points f))) + entry-point-files))) + +(define* (wrap #:key inputs outputs search-paths #:allow-other-keys) (define (list-of-files dir) (find-files dir (lambda (file stat) (and (eq? 'regular (stat:type stat)) @@ -250,29 +404,11 @@ (define (sh) (force %sh)) files))) bindirs))) -(define* (rename-pth-file #:key name inputs outputs #:allow-other-keys) - "Rename easy-install.pth to NAME.pth to avoid conflicts between packages -installed with setuptools." - ;; Even if the "easy-install.pth" is not longer created, we kept this phase. - ;; There still may be packages creating an "easy-install.pth" manually for - ;; some good reason. - (let* ((site-packages (site-packages inputs outputs)) - (easy-install-pth (string-append site-packages "/easy-install.pth")) - (new-pth (string-append site-packages "/" name ".pth"))) - (when (file-exists? easy-install-pth) - (rename-file easy-install-pth new-pth)))) - -(define* (ensure-no-mtimes-pre-1980 #:rest _) - "Ensure that there are no mtimes before 1980-01-02 in the source tree." - ;; Rationale: patch-and-repack creates tarballs with timestamps at the POSIX - ;; epoch, 1970-01-01 UTC. This causes problems with Python packages, - ;; because Python eggs are ZIP files, and the ZIP format does not support - ;; timestamps before 1980. - (let ((early-1980 315619200)) ; 1980-01-02 UTC - (ftw "." (lambda (file stat flag) - (unless (<= early-1980 (stat:mtime stat)) - (utime file early-1980 early-1980)) - #t)))) +(define* (set-SOURCE-DATE-EPOCH #:rest _) + "Set the 'SOURCE_DATE_EPOCH' environment variable. This is used by tools +that incorporate timestamps as a way to tell them to use a fixed timestamp. +See https://reproducible-builds.org/specs/source-date-epoch/." + (setenv "SOURCE_DATE_EPOCH" "315619200")) ;; python-wheel respects this variable and sets pre-1980 times on files in zip files, which is unsupported (define* (enable-bytecode-determinism #:rest _) "Improve determinism of pyc files." @@ -299,11 +435,11 @@ (define %standard-phases ;; prefix directory. The check phase is moved after the installation phase ;; to ease testing the built package. (modify-phases gnu:%standard-phases - (add-after 'unpack 'ensure-no-mtimes-pre-1980 ensure-no-mtimes-pre-1980) - (add-after 'ensure-no-mtimes-pre-1980 'enable-bytecode-determinism + (add-after 'unpack 'enable-bytecode-determinism enable-bytecode-determinism) (add-after 'enable-bytecode-determinism 'ensure-no-cythonized-files ensure-no-cythonized-files) + (replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH) (delete 'bootstrap) (delete 'configure) ;not needed (replace 'build build) @@ -313,9 +449,11 @@ (define %standard-phases (add-after 'add-install-to-pythonpath 'add-install-to-path add-install-to-path) (add-after 'add-install-to-path 'wrap wrap) + ;; must be before tests, so they can use installed packages’ entry points. + (add-before 'wrap 'create-entrypoints create-entrypoints) (add-after 'wrap 'check check) (add-after 'check 'sanity-check sanity-check) - (add-before 'strip 'rename-pth-file rename-pth-file))) + (add-before 'check 'compile-bytecode compile-bytecode))) (define* (python-build #:key inputs (phases %standard-phases) #:allow-other-keys #:rest args) -- 2.35.1 --JgXvKQ6Yd69jxylY Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v3-0004-gnu-Use-python-build-system-module-inside-importe.patch" >From 7e9bea152a81cfe69c4641a26469219a147f1d87 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sat, 8 Jan 2022 14:56:04 +0100 Subject: [PATCH v3 004/150] gnu: Use %python-build-system-module inside #:imported-modules. It includes (guix build json), which is now required for (guix build python-build-system). * gnu/packages/boost.scm (boost): Use %python-build-system-module. * gnu/packages/cups.scm (hplip): Likewise. * gnu/packages/djvu.scm (didjvu): Likewise. * gnu/packages/finance.scm (ledger): Likewise. * gnu/packages/games.scm (fifengine): Likewise. * gnu/packages/geo.scm (grass): Likewise. (qgis): Likewise. * gnu/packages/gnome.scm (system-config-printer): Likewise. (gnome-tweaks): Likewise. (lollypop): Likewise. (soundconverter): Likewise. (piper): Likewise. * gnu/packages/graph.scm (python-graph-tool): Likewise. * gnu/packages/graphics.scm (lib2geom): Likewise. * gnu/packages/gstreamer.scm (python-gst): Likewise. * gnu/packages/julia-xyz.scm (julia-pycall): Likewise. * gnu/packages/lisp-xyz.scm (sbcl-burgled-batteries3): Likewise. * gnu/packages/machine-learning.scm (ghmm): Likewise. (tensorflow): Likewise. * gnu/packages/maths.scm (nomad-optimizer): Likewise. (gmsh): Likewise. (z3): Likewise. (ocaml-z3): Likewise. (fp16): Likewise. (optizelle): Likewise. * gnu/packages/mpd.scm (sonata): Likewise. * gnu/packages/music.scm (jack-select): Likewise. * gnu/packages/networking.scm (opendht): Likewise. * gnu/packages/openldap.scm (389-ds-base): Likewise. * gnu/packages/qt.scm (python-sip-4): Likewise. (python-pyqt): Likewise. (python-pyqtwebengine): Likewise. * gnu/packages/radio.scm (gnuradio): Likewise. (gr-osmosdr): Likewise. (gr-satellites): Likewise. * gnu/packages/rpm.scm (libmodulemd): Likewise. (createrepo-c): Likewise. * gnu/packages/textutils.scm (opencc): Likewise. * gnu/packages/version-control.scm (hg-commitsigs): Likewise. * gnu/packages/virtualization.scm (ganeti): Likewise. * gnu/packages/vpn.scm (bitmask): Likewise. * gnu/packages/xdisorg.scm (redshift): Likewise. --- gnu/packages/boost.scm | 3 ++- gnu/packages/cups.scm | 2 +- gnu/packages/djvu.scm | 2 +- gnu/packages/finance.scm | 2 +- gnu/packages/geo.scm | 4 ++-- gnu/packages/gnome.scm | 10 +++++----- gnu/packages/graph.scm | 2 +- gnu/packages/graphics.scm | 2 +- gnu/packages/gstreamer.scm | 3 ++- gnu/packages/julia-xyz.scm | 3 ++- gnu/packages/lisp-xyz.scm | 3 ++- gnu/packages/machine-learning.scm | 4 ++-- gnu/packages/maths.scm | 12 ++++++------ gnu/packages/mpd.scm | 2 +- gnu/packages/music.scm | 2 +- gnu/packages/networking.scm | 2 +- gnu/packages/openldap.scm | 2 +- gnu/packages/qt.scm | 6 +++--- gnu/packages/radio.scm | 6 +++--- gnu/packages/rpm.scm | 5 +++-- gnu/packages/textutils.scm | 2 +- gnu/packages/version-control.scm | 2 +- gnu/packages/virtualization.scm | 2 +- gnu/packages/vpn.scm | 2 +- gnu/packages/xdisorg.scm | 2 +- 25 files changed, 46 insertions(+), 41 deletions(-) diff --git a/gnu/packages/boost.scm b/gnu/packages/boost.scm index b53b1f4257..ff9b6a3167 100644 --- a/gnu/packages/boost.scm +++ b/gnu/packages/boost.scm @@ -42,6 +42,7 @@ (define-module (gnu packages boost) #:use-module (guix download) #:use-module (guix git-download) #:use-module (guix build-system gnu) + #:use-module (guix build-system python) #:use-module (guix build-system trivial) #:use-module (gnu packages) #:use-module (gnu packages compression) @@ -86,7 +87,7 @@ (define-public boost `(("python" ,python-minimal-wrapper))) ("tcsh" ,tcsh))) (arguments - `(#:imported-modules ((guix build python-build-system) + `(#:imported-modules (,@%python-build-system-modules ,@%gnu-build-system-modules) #:modules (((guix build python-build-system) #:select (python-version)) ,@%gnu-build-system-modules) diff --git a/gnu/packages/cups.scm b/gnu/packages/cups.scm index da113ac168..3a006c3993 100644 --- a/gnu/packages/cups.scm +++ b/gnu/packages/cups.scm @@ -562,7 +562,7 @@ (define-public hplip (assoc-ref %outputs "out") "/etc/xdg") "--enable-qt5" "--disable-qt4") - #:imported-modules ((guix build python-build-system) + #:imported-modules (,@%python-build-system-modules ,@%gnu-build-system-modules) #:modules ((guix build gnu-build-system) (guix build utils) diff --git a/gnu/packages/djvu.scm b/gnu/packages/djvu.scm index 07c77f318e..b5ee1ebe19 100644 --- a/gnu/packages/djvu.scm +++ b/gnu/packages/djvu.scm @@ -356,7 +356,7 @@ (define-public didjvu ((guix build python-build-system) #:prefix python:) (guix build utils)) #:imported-modules (,@%gnu-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:test-target "test" #:phases (modify-phases %standard-phases diff --git a/gnu/packages/finance.scm b/gnu/packages/finance.scm index 84547639e4..1160fc341b 100644 --- a/gnu/packages/finance.scm +++ b/gnu/packages/finance.scm @@ -319,7 +319,7 @@ (define-public ledger `(#:modules (,@%cmake-build-system-modules ((guix build python-build-system) #:select (python-version))) #:imported-modules (,@%cmake-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:configure-flags `("-DBUILD_DOCS:BOOL=ON" "-DBUILD_WEB_DOCS:BOOL=ON" diff --git a/gnu/packages/geo.scm b/gnu/packages/geo.scm index c2569c27d4..ec6740ba9a 100644 --- a/gnu/packages/geo.scm +++ b/gnu/packages/geo.scm @@ -2182,7 +2182,7 @@ (define-public grass ((guix build python-build-system) #:prefix python:) (guix build utils)) #:imported-modules (,@%gnu-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:phases (modify-phases %standard-phases (replace 'configure @@ -2309,7 +2309,7 @@ (define-public qgis (guix build qt-utils) (guix build utils)) #:imported-modules (,@%cmake-build-system-modules - (guix build python-build-system) + ,@%python-build-system-modules (guix build qt-utils)) #:configure-flags '("-DWITH_QTWEBKIT=NO") diff --git a/gnu/packages/gnome.scm b/gnu/packages/gnome.scm index 19adb67423..a55c54e8cb 100644 --- a/gnu/packages/gnome.scm +++ b/gnu/packages/gnome.scm @@ -2941,7 +2941,7 @@ (define-public system-config-printer (build-system glib-or-gtk-build-system) (arguments (list - #:imported-modules `((guix build python-build-system) + #:imported-modules `(,@%python-build-system-modules ,@%glib-or-gtk-build-system-modules) #:phases #~(modify-phases %standard-phases @@ -9935,7 +9935,7 @@ (define-public gnome-tweaks `(#:glib-or-gtk? #t #:configure-flags '("-Dlocalstatedir=/tmp" "-Dsysconfdir=/tmp") - #:imported-modules ((guix build python-build-system) + #:imported-modules (,@%python-build-system-modules ,@%meson-build-system-modules) #:phases (modify-phases %standard-phases @@ -10629,7 +10629,7 @@ (define-public lollypop (arguments `(#:imported-modules (,@%meson-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:modules ((guix build meson-build-system) ((guix build python-build-system) #:prefix python:) @@ -10889,7 +10889,7 @@ (define-public soundconverter (base32 "1jv8m82hi23ilrgdznlc1jhp2jm8bw1yrw0chh3qw2l0sixvkl11")))) (build-system glib-or-gtk-build-system) (arguments - `(#:imported-modules ((guix build python-build-system) + `(#:imported-modules (,@%python-build-system-modules (guix build glib-or-gtk-build-system) ,@%gnu-build-system-modules) @@ -12133,7 +12133,7 @@ (define-public piper python-pycairo python-pygobject)) (arguments - `(#:imported-modules ((guix build python-build-system) + `(#:imported-modules (,@%python-build-system-modules ,@%meson-build-system-modules) #:modules (((guix build python-build-system) #:prefix python:) (guix build meson-build-system) diff --git a/gnu/packages/graph.scm b/gnu/packages/graph.scm index 981c4d4793..1e5e64de62 100644 --- a/gnu/packages/graph.scm +++ b/gnu/packages/graph.scm @@ -621,7 +621,7 @@ (define-public python-graph-tool (build-system gnu-build-system) (arguments `(#:imported-modules (,@%gnu-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:modules (,@%gnu-build-system-modules ((guix build python-build-system) #:select (site-packages))) #:configure-flags diff --git a/gnu/packages/graphics.scm b/gnu/packages/graphics.scm index 2cc75d57cf..b48b105d68 100644 --- a/gnu/packages/graphics.scm +++ b/gnu/packages/graphics.scm @@ -752,7 +752,7 @@ (define-public lib2geom "03bx9k1m4bfhmx0ldsg0bks6i8h7fmvl5vbg6gmpq0bk0nkmpnmv")))) (build-system cmake-build-system) (arguments - `(#:imported-modules ((guix build python-build-system) + `(#:imported-modules (,@%python-build-system-modules ,@%cmake-build-system-modules) #:configure-flags '("-D2GEOM_BUILD_SHARED=ON" "-D2GEOM_BOOST_PYTHON=ON" diff --git a/gnu/packages/gstreamer.scm b/gnu/packages/gstreamer.scm index 8b01c9b04b..1df699b6fe 100644 --- a/gnu/packages/gstreamer.scm +++ b/gnu/packages/gstreamer.scm @@ -34,6 +34,7 @@ (define-module (gnu packages gstreamer) #:use-module (guix build-system cmake) #:use-module (guix build-system gnu) #:use-module (guix build-system meson) + #:use-module (guix build-system python) #:use-module (guix build-system trivial) #:use-module (guix utils) #:use-module (gnu packages) @@ -1072,7 +1073,7 @@ (define-public python-gst (guix build utils) ((guix build python-build-system) #:prefix python:)) #:imported-modules (,@%meson-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:configure-flags (list (string-append "-Dpygi-overrides-dir=" diff --git a/gnu/packages/julia-xyz.scm b/gnu/packages/julia-xyz.scm index 5d45c07115..ab0b8a37b4 100644 --- a/gnu/packages/julia-xyz.scm +++ b/gnu/packages/julia-xyz.scm @@ -28,6 +28,7 @@ (define-module (gnu packages julia-xyz) #:use-module (guix packages) #:use-module (guix git-download) #:use-module (guix build-system julia) + #:use-module (guix build-system python) #:use-module (gnu packages gcc) #:use-module (gnu packages julia-jll) #:use-module (gnu packages python) @@ -4160,7 +4161,7 @@ (define-public julia-pycall (build-system julia-build-system) (arguments (list - #:imported-modules `((guix build python-build-system) + #:imported-modules `(,@%python-build-system-modules ,@%julia-build-system-modules) #:modules '((guix build julia-build-system) (guix build utils) diff --git a/gnu/packages/lisp-xyz.scm b/gnu/packages/lisp-xyz.scm index 9317108f4c..263dd29234 100644 --- a/gnu/packages/lisp-xyz.scm +++ b/gnu/packages/lisp-xyz.scm @@ -64,6 +64,7 @@ (define-module (gnu packages lisp-xyz) #:use-module (guix utils) #:use-module (guix build-system asdf) #:use-module (guix build-system gnu) + #:use-module (guix build-system python) #:use-module (guix build-system trivial) #:use-module (gnu packages base) #:use-module (gnu packages c) @@ -5344,7 +5345,7 @@ (define-public sbcl-burgled-batteries3 `(#:tests? #f #:modules (((guix build python-build-system) #:select (python-version)) ,@%asdf-build-system-modules) - #:imported-modules ((guix build python-build-system) + #:imported-modules (,@%python-build-system-modules ,@%asdf-build-system-modules) #:phases (modify-phases (@ (guix build asdf-build-system) %standard-phases) diff --git a/gnu/packages/machine-learning.scm b/gnu/packages/machine-learning.scm index 7f082ee677..48e6c1489d 100644 --- a/gnu/packages/machine-learning.scm +++ b/gnu/packages/machine-learning.scm @@ -234,7 +234,7 @@ (define-public ghmm (build-system gnu-build-system) (arguments `(#:imported-modules (,@%gnu-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:modules ((guix build python-build-system) ,@%gnu-build-system-modules) #:phases @@ -1884,7 +1884,7 @@ (define-public tensorflow ((guix build python-build-system) #:select (python-version))) #:imported-modules (,@%cmake-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:phases (modify-phases %standard-phases (add-after 'unpack 'set-source-file-times-to-1980 diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm index f9767d2430..d02d1c2303 100644 --- a/gnu/packages/maths.scm +++ b/gnu/packages/maths.scm @@ -1953,7 +1953,7 @@ (define-public nomad-optimizer `(("python" ,python-wrapper) ("python-cython" ,python-cython))) (arguments - `(#:imported-modules ((guix build python-build-system) + `(#:imported-modules (,@%python-build-system-modules ,@%cmake-build-system-modules) #:modules (((guix build python-build-system) #:select (python-version site-packages)) @@ -2853,7 +2853,7 @@ (define-public gmsh "-DENABLE_BUILD_SHARED:BOOL=ON" "-DENABLE_BUILD_DYNAMIC:BOOL=ON") #:imported-modules (,@%cmake-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:modules (((guix build python-build-system) #:select (site-packages)) (guix build cmake-build-system) (guix build utils)) @@ -5710,7 +5710,7 @@ (define-public z3 "1hnbzq10d23drd7ksm3c1n2611c3kd0q0yxgz8y78zaafwczvwxx")))) (build-system gnu-build-system) (arguments - `(#:imported-modules ((guix build python-build-system) + `(#:imported-modules (,@%python-build-system-modules ,@%gnu-build-system-modules) #:modules (((guix build python-build-system) #:select (site-packages)) (guix build gnu-build-system) @@ -5765,7 +5765,7 @@ (define-public ocaml-z3 (inherit z3) (name "ocaml-z3") (arguments - `(#:imported-modules ((guix build python-build-system) + `(#:imported-modules (,@%python-build-system-modules ,@%gnu-build-system-modules) #:modules (((guix build python-build-system) #:select (site-packages)) (guix build gnu-build-system) @@ -7333,7 +7333,7 @@ (define-public fp16 (patches (search-patches "fp16-system-libraries.patch")))) (build-system cmake-build-system) (arguments - `(#:imported-modules ((guix build python-build-system) + `(#:imported-modules (,@%python-build-system-modules ,@%cmake-build-system-modules) #:modules (((guix build python-build-system) #:select (site-packages)) @@ -7405,7 +7405,7 @@ (define-public optizelle #t)))) (build-system cmake-build-system) (arguments - `(#:imported-modules ((guix build python-build-system) + `(#:imported-modules (,@%python-build-system-modules ,@%cmake-build-system-modules) #:modules (((guix build python-build-system) #:select (python-version)) diff --git a/gnu/packages/mpd.scm b/gnu/packages/mpd.scm index 1ee6806735..e5795a724c 100644 --- a/gnu/packages/mpd.scm +++ b/gnu/packages/mpd.scm @@ -328,7 +328,7 @@ (define-public sonata ((guix build glib-or-gtk-build-system) #:prefix glib-or-gtk:) (guix build utils)) #:imported-modules (,@%gnu-build-system-modules - (guix build python-build-system) + ,@%python-build-system-modules (guix build glib-or-gtk-build-system)) #:phases (modify-phases %standard-phases diff --git a/gnu/packages/music.scm b/gnu/packages/music.scm index 89d64957c8..2e4a08c283 100644 --- a/gnu/packages/music.scm +++ b/gnu/packages/music.scm @@ -3226,7 +3226,7 @@ (define-public jack-select ((guix build python-build-system) #:prefix python:) (guix build utils)) #:imported-modules (,@%gnu-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:make-flags (list (string-append "PREFIX=" (assoc-ref %outputs "out"))) #:tests? #f ; there are none diff --git a/gnu/packages/networking.scm b/gnu/packages/networking.scm index b45f2f79f2..35cee1877a 100644 --- a/gnu/packages/networking.scm +++ b/gnu/packages/networking.scm @@ -3412,7 +3412,7 @@ (define-public opendht (build-system gnu-build-system) (arguments (list - #:imported-modules `((guix build python-build-system) ;for site-packages + #:imported-modules `(,@%python-build-system-modules ;for site-packages ,@%gnu-build-system-modules) #:modules '(((guix build python-build-system) #:prefix python:) (guix build gnu-build-system) diff --git a/gnu/packages/openldap.scm b/gnu/packages/openldap.scm index c8a47e45d5..ee3c2de367 100644 --- a/gnu/packages/openldap.scm +++ b/gnu/packages/openldap.scm @@ -259,7 +259,7 @@ (define-public 389-ds-base ((guix build python-build-system) #:select (add-installed-pythonpath python-version)) (guix build utils)) - #:imported-modules ((guix build python-build-system) + #:imported-modules (,@%python-build-system-modules ,@%gnu-build-system-modules) #:configure-flags (list (string-append "--with-db=" diff --git a/gnu/packages/qt.scm b/gnu/packages/qt.scm index 0163d02bc0..f98920373d 100644 --- a/gnu/packages/qt.scm +++ b/gnu/packages/qt.scm @@ -2017,7 +2017,7 @@ (define-public python-sip-4 (propagated-inputs `()) (arguments `(#:tests? #f ; no check target - #:imported-modules ((guix build python-build-system) + #:imported-modules (,@%python-build-system-modules ,@%gnu-build-system-modules) #:modules ((srfi srfi-1) ((guix build python-build-system) #:select (python-version)) @@ -2082,7 +2082,7 @@ (define-public python-pyqt `(#:modules ((srfi srfi-1) ((guix build python-build-system) #:select (python-version)) ,@%gnu-build-system-modules) - #:imported-modules ((guix build python-build-system) + #:imported-modules (,@%python-build-system-modules ,@%gnu-build-system-modules) #:phases (modify-phases %standard-phases @@ -2183,7 +2183,7 @@ (define-public python-pyqtwebengine `(#:modules ((srfi srfi-1) ((guix build python-build-system) #:select (python-version)) ,@%gnu-build-system-modules) - #:imported-modules ((guix build python-build-system) + #:imported-modules (,@%python-build-system-modules ,@%gnu-build-system-modules) #:phases (modify-phases %standard-phases diff --git a/gnu/packages/radio.scm b/gnu/packages/radio.scm index dfc274a1de..d49ae0f591 100644 --- a/gnu/packages/radio.scm +++ b/gnu/packages/radio.scm @@ -545,7 +545,7 @@ (define-public gnuradio (ice-9 match)) #:imported-modules (,@%cmake-build-system-modules (guix build glib-or-gtk-build-system) - (guix build python-build-system)) + ,@%python-build-system-modules) #:configure-flags (list (string-append "-DMATHJAX2_ROOT=" (assoc-ref %build-inputs "js-mathjax") @@ -675,7 +675,7 @@ (define-public gr-osmosdr ((guix build python-build-system) #:prefix python:) (guix build utils)) #:imported-modules (,@%cmake-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:phases (modify-phases %standard-phases (add-after 'install 'wrap-python @@ -807,7 +807,7 @@ (define-public gr-satellites ((guix build python-build-system) #:prefix python:) (guix build utils)) #:imported-modules (,@%cmake-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:phases (modify-phases %standard-phases (add-before 'check 'set-test-environment diff --git a/gnu/packages/rpm.scm b/gnu/packages/rpm.scm index 96c6776d1a..4583f1afc1 100644 --- a/gnu/packages/rpm.scm +++ b/gnu/packages/rpm.scm @@ -23,6 +23,7 @@ (define-module (gnu packages rpm) #:use-module (guix packages) #:use-module (guix build-system cmake) #:use-module (guix build-system meson) + #:use-module (guix build-system python) #:use-module (gnu packages bash) #:use-module (gnu packages check) #:use-module (gnu packages compression) @@ -90,7 +91,7 @@ (define-public libmodulemd (list (string-append "-Dgobject_overrides_dir_py3=" (python:site-packages %build-inputs %outputs))) #:imported-modules (,@%meson-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:modules ((guix build meson-build-system) ((guix build python-build-system) #:prefix python:) (guix build utils)) @@ -160,7 +161,7 @@ (define-public createrepo-c (build-system cmake-build-system) (arguments `(#:imported-modules (,@%cmake-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:modules ((guix build cmake-build-system) ((guix build python-build-system) #:prefix python:) (guix build utils)) diff --git a/gnu/packages/textutils.scm b/gnu/packages/textutils.scm index cbc657172d..faba62673f 100644 --- a/gnu/packages/textutils.scm +++ b/gnu/packages/textutils.scm @@ -1198,7 +1198,7 @@ (define-public opencc #:parallel-build? #f ;occasionally failed. #:imported-modules (,@%cmake-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:modules ((guix build cmake-build-system) ((guix build python-build-system) #:prefix python:) (guix build utils)) diff --git a/gnu/packages/version-control.scm b/gnu/packages/version-control.scm index d77c2e51f6..5fb7de5f15 100644 --- a/gnu/packages/version-control.scm +++ b/gnu/packages/version-control.scm @@ -1835,7 +1835,7 @@ (define-public hg-commitsigs "059gm66q06m6ayl4brsc517zkw3ahmz249b6xm1m32ac5y24wb9x")))) (build-system copy-build-system) (arguments - `(#:imported-modules ((guix build python-build-system) + `(#:imported-modules (,@%python-build-system-modules ,@%copy-build-system-modules) #:modules ((srfi srfi-1) (guix build python-build-system) diff --git a/gnu/packages/virtualization.scm b/gnu/packages/virtualization.scm index f3396e7c94..127306eb28 100644 --- a/gnu/packages/virtualization.scm +++ b/gnu/packages/virtualization.scm @@ -565,7 +565,7 @@ (define-public ganeti (arguments `(#:imported-modules (,@%gnu-build-system-modules (guix build haskell-build-system) - (guix build python-build-system)) + ,@%python-build-system-modules) #:modules (,@%gnu-build-system-modules ((guix build haskell-build-system) #:prefix haskell:) ((guix build python-build-system) #:select (site-packages)) diff --git a/gnu/packages/vpn.scm b/gnu/packages/vpn.scm index 5bf5a62481..9e83f06551 100644 --- a/gnu/packages/vpn.scm +++ b/gnu/packages/vpn.scm @@ -114,7 +114,7 @@ (define-public bitmask `(#:imported-modules ((guix build cmake-build-system) (guix build copy-build-system) - (guix build python-build-system) + ,@%python-build-system-modules (guix build qt-build-system) (guix build qt-utils) ,@%go-build-system-modules) diff --git a/gnu/packages/xdisorg.scm b/gnu/packages/xdisorg.scm index d2caee85ec..ae610b0bea 100644 --- a/gnu/packages/xdisorg.scm +++ b/gnu/packages/xdisorg.scm @@ -1356,7 +1356,7 @@ (define-public redshift (build-system gnu-build-system) (arguments `(#:imported-modules (,@%gnu-build-system-modules - (guix build python-build-system)) + ,@%python-build-system-modules) #:phases (modify-phases %standard-phases (add-after 'install 'split-outputs -- 2.35.1 --JgXvKQ6Yd69jxylY Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v3-0005-gnu-Use-python2-toolchain-for-build-instead-of-py.patch" >From dc9e54629f9f68060db6597686d30d4c44948062 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sat, 23 Apr 2022 10:56:01 +0200 Subject: [PATCH v3 005/150] gnu: Use python2-toolchain-for-build instead of python-2. * gnu/packages/check.scm (module-variable-resolver): New variable. (python-commencement-packager): Likewise. (python2-pytest): Use python2-toolchain-for-build. (python2-pytest-mock): Likewise. (python2-pyfakefs-bootstrap): Likewise. * gnu/packages/python-xyz.scm (module-variable-resolver): Likewise. (python-commencement-package): Likewise. (python2-twodict): Likewise. (python2-backport-ssl-match-hostname): Likewise. (python2-openpyxl): Likewise. (python2-element-tree): Likewise. (python2-pybugz): Likewise. (python2-enum): Likewise. (python2-funcsigs): Likewise. (python2-pystache): Likewise. (python2-cython): Likewise. (python2-matplotlib): Likewise. (python2-elib.intl): Likewise. (python2-importlib-resources): Likewise. (python-dbus): Likewise. (python2-beautifulsoup4): Likewise. (python2-networkx): Likewise. (python2-backports-shutil-get-terminal-size): Likewise. (python2-pyroute2): Likewise. (python2-ipaddr): Likewise. (python-tlsh): Likewise. (python2-functools32): Likewise. (python2-subprocess32): Likewise. (python2-futures): Likewise. (python2-pathlib2): Likewise. (python2-unicodecsv): Likewise. (python2-s3cmd): Likewise. (python2-shedskin): Likewise. (python2-jsonrpclib): Likewise. (python2-ruamel.ordereddict): Likewise. (python2-tracing): Likewise. (python2-backports-functools-lru-cache): Likewise. (python2-argparse): Likewise. (python2-stemming): Likewise. (python2-couleur): Likewise. (python2-steadymark): Likewise. (python2-booleanoperations): Likewise. (python2-pyro): Likewise. (python2-scientific): Likewise. (python2-mmtk): Likewise. (python2-typing): Likewise. (python2-quex-0.67.3): Likewise. (python2-more-itertools): Likewise. (python2-zeroconf): Likewise. (python2-gamera): Likewise. --- gnu/packages/check.scm | 13 +++-- gnu/packages/python-xyz.scm | 97 ++++++++++++++++++++----------------- 2 files changed, 63 insertions(+), 47 deletions(-) diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm index 335ab7ab11..c0eadbad4b 100644 --- a/gnu/packages/check.scm +++ b/gnu/packages/check.scm @@ -91,6 +91,13 @@ (define-module (gnu packages check) #:use-module (guix build-system trivial) #:use-module (srfi srfi-1)) +(define (module-variable-resolver module) + (lambda (variable) + (module-ref (resolve-interface module) variable))) + +(define python-commencement-package + (module-variable-resolver '(gnu packages python-commencement))) + (define-public pict (package (name "pict") @@ -1062,7 +1069,7 @@ (define-public python2-pytest "0ls3pqr86xgif6bphsb6wrww9r2vc7p7a2naq8zcq8115wwq5yjh")))) (build-system python-build-system) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) ,@(package-arguments python-pytest))) (propagated-inputs `(("python-atomicwrites" ,python2-atomicwrites) @@ -1380,7 +1387,7 @@ (define-public python2-pytest-mock (base32 "1i5mg3ff1qk0wqfcxfz60hwy3q5dskdp36i10ckigkzffg8hc3ad")))) (arguments - `(#:python ,python-2)) + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))) (native-inputs `(("python2-setuptools-scm" ,python2-setuptools-scm))) (propagated-inputs @@ -3024,7 +3031,7 @@ (define-public python2-pyfakefs-bootstrap (name "python2-pyfakefs-bootstrap") (native-inputs '()) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:tests? #f))))) (define-public python-aiounittest diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index f2532abc52..c300e973c5 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -196,6 +196,8 @@ (define-module (gnu packages python-xyz) #:use-module (gnu packages python) #:use-module (gnu packages python-build) #:use-module (gnu packages python-check) + ;; Importing this module results in a cycle, see below. + ;#:use-module (gnu packages python-commencement) #:use-module (gnu packages python-compression) #:use-module (gnu packages python-crypto) #:use-module (gnu packages python-science) @@ -243,6 +245,13 @@ (define-module (gnu packages python-xyz) #:use-module (srfi srfi-1) #:use-module (srfi srfi-26)) +(define (module-variable-resolver module) + (lambda (variable) + (module-ref (resolve-interface module) variable))) + +(define python-commencement-package + (module-variable-resolver '(gnu packages python-commencement))) + (define-public python-xmldiff (package (name "python-xmldiff") @@ -463,7 +472,7 @@ (define-public python2-twodict (base32 "0ifv7dv18jn2lg0a3l6zdlvmmlda2ivixfjbsda58a2ay6kxznr0")))) (build-system python-build-system) (arguments - `(#:python ,python-2)))) + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))))) (define-public python-argopt (package @@ -1210,7 +1219,7 @@ (define-public python2-backport-ssl-match-hostname "1wndipik52cyqy0677zdgp90i435pmvwd89cz98lm7ri0y3xjajh")))) (build-system python-build-system) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:tests? #f)) ; no test target (home-page "https://bitbucket.org/brandon/backports.ssl_match_hostname") (synopsis "Backport of ssl.match_hostname() function from Python 3.5") @@ -2400,7 +2409,7 @@ (define-public python2-openpyxl (base32 "1qzjj8nwj4dn0mhq1j64f136afiqqb81lvqiikipz3g1g0b80lqx")))) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:tests? #f))))) ; No test suite. (define-public python-eventlet @@ -2734,7 +2743,7 @@ (define-public python2-element-tree "016bphqnlg0l4vslahhw4r0aanw95bpypy65r1i1acyb2wj5z7dj")))) (build-system python-build-system) (arguments - `(#:python ,python-2 ; seems to be part of Python 3 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) ; seems to be part of Python 3 #:tests? #f)) ; no 'test' sub-command (synopsis "Toolkit for XML processing in Python") (description @@ -2760,7 +2769,7 @@ (define-public python2-pybugz "pybugz-encode-error.patch")))) (build-system python-build-system) (arguments - `(#:python ,python-2 ; SyntaxError with Python 3 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) ; SyntaxError with Python 3 #:tests? #f)) ; no 'test' sub-command (propagated-inputs `(("element-tree" ,python2-element-tree))) @@ -2784,7 +2793,7 @@ (define-public python2-enum "13lk3yrwj42vl30kw3c194f739nrfrdg64s6i0v2p636n4k8brsl")))) (build-system python-build-system) (arguments - `(#:python ,python-2)) + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))) (home-page "https://pypi.org/project/enum/") (synopsis "Robust enumerated type support in Python") (description @@ -3153,7 +3162,7 @@ (define-public python2-funcsigs "0l4g5818ffyfmfs1a924811azhjj8ax9xd1cffr1mzd3ycn0zfx7")))) (build-system python-build-system) (arguments - `(#:python ,python-2)) + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))) (native-inputs (list python2-unittest2)) (home-page "http://funcsigs.readthedocs.org") @@ -4051,7 +4060,7 @@ (define-public python2-pystache (strip-python2-variant python-pystache)))) (package/inherit base (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:phases (modify-phases %standard-phases (replace 'check @@ -5541,7 +5550,7 @@ (define-public python2-cython (package/inherit base (name "python2-cython") (inputs - `(("python-2" ,python-2))) ;this is not automatically changed + `(("python-2" ,(python-commencement-package 'python2-toolchain-for-build)))) ;this is not automatically changed (arguments (substitute-keyword-arguments (package-arguments base) ((#:phases phases) @@ -6606,7 +6615,7 @@ (define-public python2-matplotlib ("python2-pytz" ,python2-pytz) ("python2-six" ,python2-six) ("python2-subprocess32" ,python2-subprocess32) - ("python2-tkinter" ,python-2 "tk")))))) + ("python2-tkinter" ,(python-commencement-package 'python2-toolchain-for-build) "tk")))))) (define-public python-matplotlib-documentation (package @@ -7084,7 +7093,7 @@ (define-public python2-elib.intl (build-system python-build-system) (arguments ;; incompatible with Python 3 (exception syntax) - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:tests? #f)) (home-page "https://github.com/dieterv/elib.intl") (synopsis "Enhanced internationalization for Python") @@ -8236,7 +8245,7 @@ (define-public python2-importlib-resources "0y3hg12iby1qyaspnbisz4s4vxax7syikk3skznwqizqyv89y9yk")))) (build-system python-build-system) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:phases (modify-phases %standard-phases ;; The build system tests for python-wheel, but it is ;; not required for Guix nor the test suite. Just drop @@ -9303,7 +9312,7 @@ (define-public python-dbus (define-public python2-dbus (package/inherit python-dbus (name "python2-dbus") - (inputs `(("python" ,python-2) + (inputs `(("python" ,(python-commencement-package 'python2-toolchain-for-build)) ,@(alist-delete "python" (package-inputs python-dbus)))) (arguments @@ -9402,7 +9411,7 @@ (define-public python2-beautifulsoup4 (sha256 (base32 "09gbd49mwz86k572r1231x2rdp82p42zlnw0bz9b9mfi58r9wwl4")))) - (arguments `(#:python ,python-2))))) + (arguments `(#:python ,(python-commencement-package 'python2-toolchain-for-build)))))) (define-public python-soupsieve (package @@ -9521,7 +9530,7 @@ (define-public python2-networkx (base32 "12swxb15299v9vqjsq4z8rgh5sdhvpx497xwnhpnb0gynrx6zra5")))) (arguments - `(#:python ,python-2)) + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))) (native-inputs (list python2-nose unzip))))) @@ -10089,7 +10098,7 @@ (define-public python2-backports-shutil-get-terminal-size "107cmn7g3jnbkp826zlj8rrj19fam301qvaqf0f3905f5217lgki")))) (build-system python-build-system) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:phases (modify-phases %standard-phases (replace 'check @@ -11442,7 +11451,7 @@ (define-public python2-pyroute2 "1gmz4r1w0yzj6fjjypnalmfyy0lnfznydyn62gi3wk50j5hhxbny")))) (build-system python-build-system) (arguments - `(#:python ,python-2)) ;Python 3.x is not supported + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))) ;Python 3.x is not supported (home-page "https://github.com/svinota/pyroute2") (synopsis "Python netlink library") (description @@ -12225,7 +12234,7 @@ (define-public python2-ipaddr (base32 "1dwq3ngsapjc93fw61rp17fvzggmab5x1drjzvd4y4q0i255nm8v")))) (build-system python-build-system) (arguments - `(#:python ,python-2 ;version 2 only + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) ;version 2 only #:phases (modify-phases %standard-phases (replace 'check @@ -12430,7 +12439,7 @@ (define-public python-tlsh (define-public python2-tlsh (package/inherit python-tlsh (name "python2-tlsh") - (inputs `(("python" ,python-2))))) + (inputs `(("python" ,(python-commencement-package 'python2-toolchain-for-build)))))) (define-public python-termcolor (package @@ -13985,7 +13994,7 @@ (define-public python2-functools32 "0v8ya0b58x47wp216n1zamimv4iw57cxz3xxhzix52jkw3xks9gn")))) (build-system python-build-system) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:tests? #f)) ; no test target (home-page "https://github.com/MiCHiLU/python-functools32") (synopsis @@ -14009,7 +14018,7 @@ (define-public python2-subprocess32 (search-patches "python2-subprocess32-disable-input-test.patch")))) (build-system python-build-system) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) ;; The test suite fails with Python > 2.7.13: ;; import test.support ;; ImportError: No module named support @@ -14045,7 +14054,7 @@ (define-public python2-futures "0rdjmmsab550kxsssdq49jcniz77zlkpw4pvi9hvib3lsskjmh4y")))) (build-system python-build-system) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:phases (modify-phases %standard-phases (replace 'check @@ -15406,7 +15415,7 @@ (define-public python2-pathlib2 ;; version is 3.4 which already includes this package as part of the ;; standard library. (arguments - `(#:python ,python-2)) + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))) (propagated-inputs (list python2-scandir python2-six)) (home-page "https://pypi.org/project/pathlib2/") @@ -15484,7 +15493,7 @@ (define-public python2-unicodecsv (build-system python-build-system) (arguments `(;; It supports Python 3, but Python 3 can already do Unicode CSV. - #:python ,python-2)) + #:python ,(python-commencement-package 'python2-toolchain-for-build))) (native-inputs (list python2-unittest2)) (home-page "https://github.com/jdunck/python-unicodecsv") @@ -15687,7 +15696,7 @@ (define-public python2-s3cmd (build-system python-build-system) (arguments ;; s3cmd is written for python2 only and contains no tests. - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:tests? #f)) (propagated-inputs (list python2-dateutil @@ -15876,7 +15885,7 @@ (define-public python2-shedskin "0nzwrzgw1ga8rw6f0ryq7zr9kkiavd1cqz5hzxkcbicl1dk7kz41")))) (build-system python-build-system) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:phases (modify-phases %standard-phases (add-after 'unpack 'fix-resulting-include-libs (lambda* (#:key inputs #:allow-other-keys) @@ -16406,7 +16415,7 @@ (define-public python2-jsonrpclib (build-system python-build-system) (arguments `(#:tests? #f - #:python ,python-2)) + #:python ,(python-commencement-package 'python2-toolchain-for-build))) (home-page "https://github.com/joshmarshall/jsonrpclib/") (synopsis "Implementation of JSON-RPC specification for Python") (description @@ -18635,7 +18644,7 @@ (define-public python2-ruamel.ordereddict "1xmkl8v9l9inm2pyxgc1fm5005yxm7fkd5gv74q7lj1iy5qc8n3h")))) (build-system python-build-system) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:phases (modify-phases %standard-phases (delete 'check) @@ -18957,7 +18966,7 @@ (define-public python2-tracing "06cw4zg42fsvqy372vi2whj26w56vzg5axhzwdjc2bgwf03garbw")))) (build-system python-build-system) (arguments - `(#:python ,python-2)) + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))) (home-page "https://liw.fi/tracing/") (synopsis "Python debug logging helper") (description "@code{python2-tracing} is a python library for @@ -19136,7 +19145,7 @@ (define-public python2-backports-functools-lru-cache (native-inputs (list python2-setuptools-scm)) (arguments - `(#:python ,python-2)) + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))) (home-page "https://github.com/jaraco/backports.functools_lru_cache") (synopsis "Backport of functools.lru_cache from Python 3.3") (description "@code{python2-backports-functools-lru-cache} is a backport @@ -19252,7 +19261,7 @@ (define-public python2-argparse "1r6nznp64j68ih1k537wms7h57nvppq0szmwsaf99n71bfjqkc32")))) (build-system python-build-system) (arguments - `(#:python ,python-2)) + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))) (home-page "https://github.com/ThomasWaldmann/argparse/") (synopsis "Python command-line parsing library") (description @@ -19429,7 +19438,7 @@ (define-public python2-stemming (base32 "0ldwa24gnnxhniv0fhygkpc2mwgd93q10ag8rvzayv6hw418frsr")))) (build-system python-build-system) (arguments - `(#:python ,python-2)) + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))) (home-page "https://bitbucket.org/mchaput/stemming/overview") (synopsis "Python implementations of various stemming algorithms") (description @@ -19679,7 +19688,7 @@ (define-public python2-couleur "1qqaxyqz74wvid0cr119dhcwz0h0if5b5by44zl49pd5z65v58k1")))) (build-system python-build-system) (arguments - `(#:python ,python-2)) + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))) (home-page "https://github.com/gabrielfalcao/couleur") (synopsis "ANSI terminal tool for python, colored shell and other handy fancy features") @@ -19740,7 +19749,7 @@ (define-public python2-steadymark (native-inputs (list python2-couleur python2-sure python2-misaka)) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:phases (modify-phases %standard-phases (add-before 'build 'patch-setup-py @@ -20909,7 +20918,7 @@ (define-public python2-booleanoperations "1hw42fazdpvsn77glx96hwsj9l17mvx37sc5707s08y5w6fx16mn")))) (build-system python-build-system) (arguments - `(#:python ,python-2)) + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))) (native-inputs (list unzip python2-pytest python2-pytest-runner)) (propagated-inputs @@ -21085,7 +21094,7 @@ (define-public python2-pyro (build-system python-build-system) (arguments ;; Pyro is not compatible with Python 3 - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) ;; Pyro has no test cases for automatic execution #:tests? #f)) (home-page "https://pythonhosted.org/Pyro/") @@ -21120,7 +21129,7 @@ (define-public python2-scientific (list python2-numpy-1.8 python2-pyro)) (arguments ;; ScientificPython is not compatible with Python 3 - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:tests? #f ; No test suite #:phases (modify-phases %standard-phases @@ -21157,9 +21166,9 @@ (define-public python2-mmtk (list netcdf)) (propagated-inputs `(("python-scientific" ,python2-scientific) - ("python-tkinter" ,python-2 "tk"))) + ("python-tkinter" ,(python-commencement-package 'python2-toolchain-for-build) "tk"))) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:tests? #f #:phases (modify-phases %standard-phases @@ -21754,7 +21763,7 @@ (define-public python2-typing (sha256 (base32 "0c5il4d68fd4qrm5k3dps70j0xz0n5krj6lhwn9vzpal3whsvd0k")))) (build-system python-build-system) - (arguments (list #:python python-2)) + (arguments (list #:python (python-commencement-package 'python2-toolchain-for-build))) (home-page "https://docs.python.org/3/library/typing.html") (synopsis "Type hints for Python") (description "This is a backport of the standard library @code{typing} @@ -21926,7 +21935,7 @@ (define-public python2-quex-0.67.3 (native-inputs (list unzip)) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:tests? #f #:phases (modify-phases %standard-phases @@ -21998,7 +22007,7 @@ (define-public python2-more-itertools (base32 "1r12cm6mcdwdzz7d47a6g4l437xsvapdlgyhqay3i2nrlv03da9q")))) (arguments - `(#:python ,python-2)) + `(#:python ,(python-commencement-package 'python2-toolchain-for-build))) (propagated-inputs `(("python2-six" ,python2-six-bootstrap))))) @@ -24203,7 +24212,7 @@ (define-public python2-zeroconf "0ykzg730n915qbrq9bn5pn06bv6rb5zawal4sqjyfnjjm66snkj3")))) (build-system python-build-system) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:phases (modify-phases %standard-phases (add-after 'unpack 'patch-requires @@ -27189,7 +27198,7 @@ (define-public python2-gamera (inputs (list libpng libtiff zlib)) (arguments - `(#:python ,python-2 + `(#:python ,(python-commencement-package 'python2-toolchain-for-build) #:phases (modify-phases %standard-phases (add-after 'unpack 'disable-wx-support -- 2.35.1 --JgXvKQ6Yd69jxylY Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="v3-0146-gnu-meson-Match-shebang-instead-of-setuptools-spe.patch" >From 84c3af5cf41d402847adb33b11897d34f5a14179 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Wed, 19 Jan 2022 09:48:44 +0100 Subject: [PATCH v3 146/150] gnu: meson: Match shebang instead of setuptools-specific line. * gnu/packages/build-tools.scm (meson)[arguments]: Replace substitute* pattern. --- gnu/packages/build-tools.scm | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/gnu/packages/build-tools.scm b/gnu/packages/build-tools.scm index 437b0d3550..b343467699 100644 --- a/gnu/packages/build-tools.scm +++ b/gnu/packages/build-tools.scm @@ -289,12 +289,12 @@ (define-public meson (python-version (assoc-ref inputs "python"))) (output (assoc-ref outputs "out"))) (substitute* (string-append output "/bin/meson") - (("# EASY-INSTALL-ENTRY-SCRIPT") - (format #f "\ + (("#!/(.+)" all) + (format #f "~a\ import sys sys.path.insert(0, '~a/lib/python~a/site-packages') -# EASY-INSTALL-ENTRY-SCRIPT" - output python-version))))))))) +" + all output python-version))))))))) (inputs (list python-wrapper ninja)) (home-page "https://mesonbuild.com/") (synopsis "Build system designed to be fast and user-friendly") -- 2.35.1 --JgXvKQ6Yd69jxylY-- From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Lars-Dominik Braun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Sun, 24 Apr 2022 09:23:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxim Cournoyer Cc: 46848@debbugs.gnu.org, Marius Bakke Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.16507921712635 (code B ref 46848); Sun, 24 Apr 2022 09:23:02 +0000 Received: (at 46848) by debbugs.gnu.org; 24 Apr 2022 09:22:51 +0000 Received: from localhost ([127.0.0.1]:57613 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1niYS3-0000gH-3e for submit@debbugs.gnu.org; Sun, 24 Apr 2022 05:22:51 -0400 Received: from mout-p-202.mailbox.org ([80.241.56.172]:37401) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1niYS0-0000fy-1S for 46848@debbugs.gnu.org; Sun, 24 Apr 2022 05:22:41 -0400 Received: from smtp2.mailbox.org (smtp2.mailbox.org [80.241.60.241]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-202.mailbox.org (Postfix) with ESMTPS id 4KmN2x61Mjz9sTC; Sun, 24 Apr 2022 11:22:33 +0200 (CEST) Date: Sun, 24 Apr 2022 11:22:12 +0200 From: Lars-Dominik Braun Message-ID: References: <877daubeju.fsf@gnu.org> <87h78lwwfc.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="hrBvXOG8QWMXBmMC" Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --hrBvXOG8QWMXBmMC Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hey again, sorry for the noise. During one of my many rebases I missed to add a file that should have been included in patch 3. See attached patch. Cheers, Lars --hrBvXOG8QWMXBmMC Content-Type: text/plain; charset=utf-8 Content-Disposition: attachment; filename="v3-0151-dirty-squash-Add-missing-file.patch" Content-Transfer-Encoding: 8bit >From 6bb5578a47f795f54456d2f7e3c6949f5e0b3e13 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 24 Apr 2022 11:18:37 +0200 Subject: [PATCH v3 151/151] dirty: squash: Add missing file. --- gnu/packages/python-commencement.scm | 129 +++++++++++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 gnu/packages/python-commencement.scm diff --git a/gnu/packages/python-commencement.scm b/gnu/packages/python-commencement.scm new file mode 100644 index 0000000000..e0959c6e79 --- /dev/null +++ b/gnu/packages/python-commencement.scm @@ -0,0 +1,129 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès +;;; Copyright © 2014 Andreas Enge +;;; Copyright © 2012 Nikita Karetnikov +;;; Copyright © 2014, 2015, 2017 Mark H Weaver +;;; Copyright © 2017, 2018, 2019, 2021 Efraim Flashner +;;; Copyright © 2018 Tobias Geerinckx-Rice +;;; Copyright © 2018, 2019, 2020 Jan (janneke) Nieuwenhuizen +;;; Copyright © 2019, 2020 Marius Bakke +;;; Copyright © 2020 Timothy Sample +;;; Copyright © 2020 Guy Fleury Iteriteka +;;; Copyright © 2021 Maxim Cournoyer +;;; Copyright © 2021 Lars-Dominik Braun +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (gnu packages python-commencement) + #:use-module ((guix licenses) #:prefix license:) + #:use-module (guix download) + #:use-module (guix packages) + #:use-module (guix build-system trivial) + #:use-module (gnu packages) + #:use-module (gnu packages python) + #:use-module (gnu packages python-build) + #:use-module (gnu packages python-xyz) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26)) + +(define-public python-toolchain + (package + (name "python-toolchain") + (version (package-version python)) + (source #f) + (build-system trivial-build-system) + (arguments + '(#:modules ((guix build union)) + #:builder (begin + (use-modules (ice-9 match) + (srfi srfi-1) + (srfi srfi-26) + (guix build union)) + + (let ((out (assoc-ref %outputs "out"))) + (union-build out (filter-map (match-lambda + ((_ . directory) directory)) + %build-inputs)) + #t)))) + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools) + ("python-pip" ,python-pip))) + (native-search-paths + (package-native-search-paths python)) + (search-paths + (package-search-paths python)) + (license (package-license python)) + (synopsis "Python toolchain") + (description + "Python toolchain including Python itself, setuptools and pip. Use this +package if you need a minimal Python toolchain instead of just the +interpreter.") + (home-page (package-home-page python)))) + +(define-public python2-toolchain + (package + (name "python2-toolchain") + (version (package-version python-2)) + (source #f) + (build-system trivial-build-system) + (arguments + '(#:modules ((guix build union)) + #:builder (begin + (use-modules (ice-9 match) + (srfi srfi-1) + (srfi srfi-26) + (guix build union)) + + (let ((out (assoc-ref %outputs "out"))) + (union-build out (filter-map (match-lambda + ((_ . directory) directory)) + %build-inputs)) + #t)))) + (inputs + `(("python" ,python-2) + ("python2-setuptools" ,python2-setuptools) + ("python2-pip" ,python2-pip))) + (native-search-paths + (package-native-search-paths python-2)) + (search-paths + (package-search-paths python-2)) + (license (package-license python-2)) + (synopsis "Python toolchain") + (description + "Python toolchain including Python itself, setuptools and pip. Use this +package if you need a minimal Python toolchain instead of just the +interpreter.") + (home-page (package-home-page python-2)))) + +;; Python 3 toolchain for python-build-system. We cannot use python-toolchain +;; here, since we’d need to bootstrap python-pip somehow. +(define-public python-toolchain-for-build + (package + (inherit python-toolchain) + (name "python-toolchain-for-build") + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools))))) + +(define-public python2-toolchain-for-build + (package + (inherit python2-toolchain) + (name "python2-toolchain-for-build") + (inputs + `(("python" ,python-2) + ("python2-setuptools" ,python2-setuptools))))) + -- 2.35.1 --hrBvXOG8QWMXBmMC-- From unknown Wed Sep 10 13:56:00 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#46848] [PATCHES] [core-updates] PEP 517 python-build-system Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Wed, 11 Jan 2023 15:42:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 46848 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Marius Bakke Cc: Lars-Dominik Braun , 46848@debbugs.gnu.org Received: via spool by 46848-submit@debbugs.gnu.org id=B46848.16734516703252 (code B ref 46848); Wed, 11 Jan 2023 15:42:02 +0000 Received: (at 46848) by debbugs.gnu.org; 11 Jan 2023 15:41:10 +0000 Received: from localhost ([127.0.0.1]:43582 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pFdDx-0000qO-U2 for submit@debbugs.gnu.org; Wed, 11 Jan 2023 10:41:10 -0500 Received: from mail-yw1-f174.google.com ([209.85.128.174]:41829) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pFdDv-0000q9-Pq for 46848@debbugs.gnu.org; Wed, 11 Jan 2023 10:41:08 -0500 Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-4a2f8ad29d5so199751117b3.8 for <46848@debbugs.gnu.org>; Wed, 11 Jan 2023 07:41:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=uSDWroPFEOeRaf5tdpW/H8Zg+PpFiGWBRLUGkiMMxqM=; b=Mz/lCUx19sgAUjfBrRZRDWCgqOc2UhV4VAzWavwDXCxzp3yqbYyunkMXsrB4i1TBP+ Nzv+egO6tOKiPSX4bWan185TpW6uKMFNLEOoXEB4XkgU3Bk5NNI6+vvn4iDe9PQOwSgO rnG3r3xkxWPU0bt5UTOOWePfRW1dE25JnQ+nsmyB/GwcK9d/FOwFPlnfP5eTQQBPpkP/ 8SRGmOheHPgJpJwwsYunKWqzAyN+NXT7P2n1CdGWxzVNPQLhe4nnqIf881wwsDy/BXE6 lJB/Md3YDlZyiLoEaarteRqOs1W87AoUfS/KpupjuLCBgCxDXn0PmAJ5JysigfNHITUf fMtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:user-agent:message-id:in-reply-to:date:references :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=uSDWroPFEOeRaf5tdpW/H8Zg+PpFiGWBRLUGkiMMxqM=; b=zQMcL306X/5AFfJH6f2FOQILFQmSCok10VSkA5its+u23uUwsjDRkE9DTF6G28GwtS erGVkIqhOBoXh1YkczY/wA2BXMnws+SpYsveGWcxFfH05aZnizjO35lGMkUTbhCVnJwf wcRkIpwh6m/zTuB+twD5KiZWZwDPiuO4ES0thrkUM/F6LPIdbINicnB1jr1p47RvyaBb b6+xotLLjdrHKb/aYPmLqIobLgEEOjfUyCyYnvCakjcPIJOgVQQwxbCv4RUZlv0solqg 8bshQdytsJ8Xcn9AfdxJ1f98tC5XFX3vo7C1KIJLE3WJfYBc1+RJXU90cR0RMx3R6Hnt MHHw== X-Gm-Message-State: AFqh2kqk2OBXVbuU/PkwpueeE0g8uNCPuImFcTbeN9CcwjAVxVs8OPk+ UjckM0eORBlPVzb7AhaCix7PuydqQBpqKxHp X-Google-Smtp-Source: AMrXdXuHuqmWkazxvFC+shs/0+dExkoNTpFp5pV1vx5QVLSujHpID4VjVFanqAcCh6SU8Olcn4stVg== X-Received: by 2002:a81:4895:0:b0:370:3fb8:184c with SMTP id v143-20020a814895000000b003703fb8184cmr19585944ywa.33.1673451662079; Wed, 11 Jan 2023 07:41:02 -0800 (PST) Received: from hurd (dsl-159-79.b2b2c.ca. [66.158.159.79]) by smtp.gmail.com with ESMTPSA id n15-20020a05620a294f00b0070383f1b6f1sm9245508qkp.31.2023.01.11.07.41.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Jan 2023 07:41:01 -0800 (PST) From: Maxim Cournoyer References: <877daubeju.fsf@gnu.org> <874k5yb0jd.fsf@gnu.org> Date: Wed, 11 Jan 2023 10:41:00 -0500 In-Reply-To: <874k5yb0jd.fsf@gnu.org> (Marius Bakke's message of "Thu, 20 Jan 2022 21:43:50 +0100") Message-ID: <87v8ldhyer.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello! With the pyproject build system now in master, can we close this issue? -- Thanks, Maxim From unknown Wed Sep 10 13:56:00 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Lars-Dominik Braun Subject: bug#46848: closed (Re: bug#46848: [PATCHES] [core-updates] PEP 517 python-build-system) Message-ID: References: X-Gnu-PR-Message: they-closed 46848 X-Gnu-PR-Package: guix-patches X-Gnu-PR-Keywords: patch Reply-To: 46848@debbugs.gnu.org Date: Fri, 10 Feb 2023 10:14:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1676024042-16198-1" This is a multi-part message in MIME format... ------------=_1676024042-16198-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #46848: [PATCHES] [core-updates] PEP 517 python-build-system which was filed against the guix-patches package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 46848@debbugs.gnu.org. --=20 46848: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D46848 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1676024042-16198-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 46848-done) by debbugs.gnu.org; 10 Feb 2023 10:13:48 +0000 Received: from localhost ([127.0.0.1]:34392 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pQQPb-0004Cf-Ss for submit@debbugs.gnu.org; Fri, 10 Feb 2023 05:13:48 -0500 Received: from mout-p-103.mailbox.org ([80.241.56.161]:33556) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pQQPZ-0004CP-K0 for 46848-done@debbugs.gnu.org; Fri, 10 Feb 2023 05:13:46 -0500 Received: from smtp102.mailbox.org (smtp102.mailbox.org [IPv6:2001:67c:2050:b231:465::102]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-103.mailbox.org (Postfix) with ESMTPS id 4PCqM56kDmz9sS1; Fri, 10 Feb 2023 11:13:37 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1676024017; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=yLXBP6ZIm3RevwmmZA60y6IgQgBtydTAzUVcUwMzSkg=; b=jc2uTlyAl//mdLWY64WsEKSyfKEYacpxuBTvkA3Gdshnj8V4dkxoq/Dl9Pb1LjQ9oStYnq Xpdn78j7/KKC7yeXiQ6C6yk4zmnvzjZP4CzMk4as4YsLCiBUScjzENPSE67lXmMx2n9MVT RCsdCyKglBOSCvv0uv8cQSKJVWMy0XSoO5nbHhoVb9KO15RbcU4LNL0zMsZpnhu/fxvDaa 9/kLtI8RcZnwVDJF8F4lx/Cg4viQ5bbq28ly90JUPa3kbeazL2mEmycUAWcJQ8htx7qYA2 Y5ezA6pVQ7kha9zw4XqBdMnkfP1eV47ol27uTy1ROjILYu8qF6kzb1Sytilu2g== Date: Fri, 10 Feb 2023 11:13:35 +0100 From: Lars-Dominik Braun To: Maxim Cournoyer Subject: Re: bug#46848: [PATCHES] [core-updates] PEP 517 python-build-system Message-ID: References: <877daubeju.fsf@gnu.org> <874k5yb0jd.fsf@gnu.org> <87v8ldhyer.fsf_-_@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87v8ldhyer.fsf_-_@gmail.com> X-Rspamd-Queue-Id: 4PCqM56kDmz9sS1 X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 46848-done Cc: 46848-done@debbugs.gnu.org, Marius Bakke X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi, > With the pyproject build system now in master, can we close this issue? yes, I think so. Cheers, Lars ------------=_1676024042-16198-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 1 Mar 2021 13:44:07 +0000 Received: from localhost ([127.0.0.1]:48521 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lGiq5-00040s-W0 for submit@debbugs.gnu.org; Mon, 01 Mar 2021 08:44:07 -0500 Received: from lists.gnu.org ([209.51.188.17]:34676) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lGipz-00040g-09 for submit@debbugs.gnu.org; Mon, 01 Mar 2021 08:43:56 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:59510) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lGipy-0004en-Oi for guix-patches@gnu.org; Mon, 01 Mar 2021 08:43:50 -0500 Received: from mout-p-102.mailbox.org ([2001:67c:2050::465:102]:16092) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256) (Exim 4.90_1) (envelope-from ) id 1lGipt-0004lQ-1k for guix-patches@gnu.org; Mon, 01 Mar 2021 08:43:49 -0500 Received: from smtp2.mailbox.org (smtp2.mailbox.org [80.241.60.241]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-102.mailbox.org (Postfix) with ESMTPS id 4Dq1gR1VCBzQkJK for ; Mon, 1 Mar 2021 14:43:31 +0100 (CET) X-Virus-Scanned: amavisd-new at heinlein-support.de DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=6xq.net; s=MBO0001; t=1614606209; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=gszE9nJa3Nr0ww6hptcwp9mkhmP+fxtZ+I4I9OWz0K8=; b=JlGtx7QHCUxeel9+2zXBXXgRkdkROe6GH/85isarZMdvW8j6kF7zTtIi2VrEK/HlK2BvKG hNyioSoWFYXYwJFCpDdE8EMo3DuCRh3+JCd31EI+hlt+p71GpaMuK9w3vpNtAkmxwMs9yM PNEZFF3gcJjICXOJDkDd8CK1L36pxL/bcQWP7yM2O1sWQEIiixPnWiGomfjgOfO6u0s7t6 gR3hdvj65shvfV/zYx6IZaMVMIcGqzO8aU0ZkJpnIUkwIboNYrfs2uYFQ7KsfHrLnWuN9L rKCa/qZhCstPWsQLOJ1miUb/Cnn0mQE5tAd9pmsgZoqoWwrFPg347yZWQDtESw== Received: from smtp2.mailbox.org ([80.241.60.241]) by spamfilter03.heinlein-hosting.de (spamfilter03.heinlein-hosting.de [80.241.56.117]) (amavisd-new, port 10030) with ESMTP id jjDxxdmqfzvB for ; Mon, 1 Mar 2021 14:43:24 +0100 (CET) Date: Mon, 1 Mar 2021 14:43:22 +0100 From: Lars-Dominik Braun To: guix-patches@gnu.org Subject: [PATCHES] [core-updates] PEP 517 python-build-system Message-ID: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="9EzuTbC0hpH3Zvh5" Content-Disposition: inline Content-Transfer-Encoding: 8bit X-MBO-SPAM-Probability: X-Rspamd-Score: -8.95 / 15.00 / 15.00 X-Rspamd-Queue-Id: C071C17B4 X-Rspamd-UID: 3d4170 Received-SPF: pass client-ip=2001:67c:2050::465:102; envelope-from=lars@6xq.net; helo=mout-p-102.mailbox.org X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_SBL_A=0.1 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.3 (/) --9EzuTbC0hpH3Zvh5 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit Hi everyone, the attached patches switch python-build-system to a PEP 517-based build system using python-pypa-build. As discussed previously Python is currently in the process of opening up for build systems other than setuptools using the API specified in PEP 517. python-pypa-build is a simple tool for building packages using that new API. It supports setup.py-based builds as a fallback, if no pyproject.toml is present. One downside is that this tool is not self-contained and has a few dependencies. Thus first I bootstrap setuptools using itself (possible because it bundles all of its own dependencies), then build python-pypa-build’s dependencies using setuptools (which is fortunately still possible) and then combine everything into a python-toolchain(-for-build), which is then used by the build-process. I can successfully build packages like python-pypa-build and python-pytest and python-pep517-bootstrap. The latter is using flit as its build backend. But other packages currently fail because I removed some arguments. Cheers, Lars --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-build-python-Handle-missing-setuptools-in-sanity-che.patch" >From 7ace01faef72f3a48b18fe241fe0524445a154fb Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Fri, 19 Feb 2021 17:22:35 +0100 Subject: [PATCH 01/12] build/python: Handle missing setuptools in sanity-check.py Just skip testing if required dependencies (setuptools) are not available. * gnu/packages/aux-files/python/sanity-check.py: Handle ImportError. --- gnu/packages/aux-files/python/sanity-check.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/gnu/packages/aux-files/python/sanity-check.py b/gnu/packages/aux-files/python/sanity-check.py index 83b6d583ca..240155cecc 100644 --- a/gnu/packages/aux-files/python/sanity-check.py +++ b/gnu/packages/aux-files/python/sanity-check.py @@ -19,9 +19,13 @@ from __future__ import print_function # Python 2 support. import importlib -import pkg_resources import sys import traceback +try: + import pkg_resources +except ImportError: + print('Warning: Skipping, because python-setuptools are not available.') + sys.exit(0) try: from importlib.machinery import PathFinder -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-gnu-python-pypa-build-Update-to-0.3.0.patch" >From 66033f14455456d465a3c9f5a2d1f4961cd3b989 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 12:50:42 +0100 Subject: [PATCH 02/12] gnu: python-pypa-build: Update to 0.3.0. * gnu/packages/python-build.scm (python-pypa-build): Update to 0.3.0. --- gnu/packages/python-build.scm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 140629ca43..92dbda314f 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -118,13 +118,13 @@ Language (TOML) configuration files.") (define-public python-pypa-build (package (name "python-pypa-build") - (version "0.1.0") + (version "0.3.0") (source (origin (method url-fetch) (uri (pypi-uri "build" version)) (sha256 (base32 - "1d6m21lijwm04g50nwgsgj7x3vhblzw7jv05ah8psqgzk20bbch8")))) + "1pazq66c35whrqd5b0zcydjvy2rmghi8riljkd67q3bpiln5pf8f")))) (build-system python-build-system) (arguments `(#:tests? #f ;to tests in the PyPI release -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0003-gnu-python-wheel-Install-entrypoint-scripts.patch" >From 00ef998413dc0fe6605653ae16f65d6377c2ec77 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:02:15 +0100 Subject: [PATCH 03/12] gnu: python-wheel: Install entrypoint scripts * gnu/packages/python-build.scm (pythont-wheel) [arguments]: Add phase 'patch-enable-entrypoints. --- gnu/packages/python-build.scm | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 92dbda314f..232e24f470 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -47,10 +47,17 @@ "0ii6f34rvpjg3nmw4bc2h7fhdsy38y1h93hghncfs5akfrldmj8h")))) (build-system python-build-system) (arguments - ;; FIXME: The test suite runs "python setup.py bdist_wheel", which in turn - ;; fails to find the newly-built bdist_wheel library, even though it is - ;; available on PYTHONPATH. What search path is consulted by setup.py? - '(#:tests? #f)) + `(#:phases + (modify-phases %standard-phases + (add-after 'unpack 'patch-enable-entrypoints + (lambda _ + ;; python-wheel tells setuptools to not install entry point scripts + ;; by default. Stop doing that, so wheels built contain all data + ;; required. + (substitute* "wheel/bdist_wheel.py" + (("(install_scripts\\.no_ep = )True" all assignment) + (string-append assignment "False"))) + #t))))) (home-page "https://bitbucket.org/pypa/wheel/") (synopsis "Format for built Python packages") (description -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0004-gnu-python-setuptools-Bootstrap-using-itself.patch" Content-Transfer-Encoding: 8bit >From 61313d8ddba30772e2587e3e16ca30d1565d3c7e Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:05:51 +0100 Subject: [PATCH 04/12] gnu: python-setuptools: Bootstrap using itself * gnu/packages/python-xyz.scm (python-setuptools) [arguments]: Add phase setting GUIX_PYTHONPATH to source directory. --- gnu/packages/python-xyz.scm | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index f8afa13f33..79d01f700a 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -1144,7 +1144,18 @@ other machines, such as over the network.") ;; FIXME: Tests require pytest, which itself relies on setuptools. ;; One could bootstrap with an internal untested setuptools. (arguments - `(#:tests? #f)) + `(#:tests? #f + #:python ,python-wrapper + #:phases (modify-phases %standard-phases + ;; Use this setuptools’ sources to bootstrap themselves. + (add-before 'build 'set-PYTHONPATH + (lambda _ + (format #t "current working dir ~s~%" (getcwd)) + (setenv "GUIX_PYTHONPATH" + (string-append ".:" (getenv "GUIX_PYTHONPATH"))) + #t))))) + ;; Not required when not building a wheel + ;(propagated-inputs `(("python-wheel" ,python-wheel))) (home-page "https://pypi.org/project/setuptools/") (synopsis "Library designed to facilitate packaging Python projects") -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="0005-python-build-Switch-to-PEP-517-based-build.patch" Content-Transfer-Encoding: 8bit >From 7a99aaa40e65fde58ee2e78ad7d3e0ccd6d169ae Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:08:58 +0100 Subject: [PATCH 05/12] python-build: Switch to PEP 517-based build * gnu/packages/python-commencement.scm: New file, containing python-toolchain. * gnu/local.mk: Add it. * gnu/packages/python.scm (python): Disable installing bundled pip/setuptools. * guix/build/python-build-system.scm: Rewrite using python-pypa-build. * guix/build-system/python.scm (default-python): Switch to python-toolchain (lower): Remove unused parameter. XXX: rationale --- gnu/local.mk | 1 + gnu/packages/python-commencement.scm | 175 +++++++++++++++++++ gnu/packages/python.scm | 2 +- guix/build-system/python.scm | 8 +- guix/build/python-build-system.scm | 249 ++++++++++++++++++--------- 5 files changed, 342 insertions(+), 93 deletions(-) create mode 100644 gnu/packages/python-commencement.scm diff --git a/gnu/local.mk b/gnu/local.mk index 202677fed1..ef2532cb5d 100644 --- a/gnu/local.mk +++ b/gnu/local.mk @@ -463,6 +463,7 @@ GNU_SYSTEM_MODULES = \ %D%/packages/python.scm \ %D%/packages/python-build.scm \ %D%/packages/python-check.scm \ + %D%/packages/python-commencement.scm \ %D%/packages/python-compression.scm \ %D%/packages/python-crypto.scm \ %D%/packages/python-science.scm \ diff --git a/gnu/packages/python-commencement.scm b/gnu/packages/python-commencement.scm new file mode 100644 index 0000000000..2ced3079bc --- /dev/null +++ b/gnu/packages/python-commencement.scm @@ -0,0 +1,175 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès +;;; Copyright © 2014 Andreas Enge +;;; Copyright © 2012 Nikita Karetnikov +;;; Copyright © 2014, 2015, 2017 Mark H Weaver +;;; Copyright © 2017, 2018, 2019, 2021 Efraim Flashner +;;; Copyright © 2018 Tobias Geerinckx-Rice +;;; Copyright © 2018, 2019, 2020 Jan (janneke) Nieuwenhuizen +;;; Copyright © 2019, 2020 Marius Bakke +;;; Copyright © 2020 Timothy Sample +;;; Copyright © 2020 Guy Fleury Iteriteka +;;; Copyright © 2021 Maxim Cournoyer +;;; Copyright © 2021 Lars-Dominik Braun +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (gnu packages python-commencement) + #:use-module ((guix licenses) #:prefix license:) + #:use-module (guix download) + #:use-module (guix packages) + #:use-module (guix build-system trivial) + #:use-module (guix build-system python) + #:use-module (gnu packages) + #:use-module (gnu packages python) + #:use-module (gnu packages python-build) + #:use-module (gnu packages python-xyz) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26)) + +;; Python toolchain and all packages required to bootstrap it. + +(define-public python-toolchain + (package + (name "python-toolchain") + (version (package-version python)) + (source #f) + (build-system trivial-build-system) + (arguments + '(#:modules ((guix build union)) + #:builder (begin + (use-modules (ice-9 match) + (srfi srfi-1) + (srfi srfi-26) + (guix build union)) + + (let ((out (assoc-ref %outputs "out"))) + (union-build out (filter-map (match-lambda + ((_ . directory) directory)) + %build-inputs)) + #t)))) + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools) + ("python-pip" ,python-pip))) ; XXX Maybe virtualenv/venv too? It kind of + ; defeats the purpose of guix, but is used + ; alot in local development. + (native-search-paths + (package-native-search-paths python)) + (search-paths + (package-search-paths python)) + (license (package-license python)) ; XXX + (synopsis "Python toolchain") + (description + "Python toolchain including Python itself, setuptools and pip. Use this +package if you need a fully-fledged Python toolchain instead of just the +interpreter.") + (home-page (package-home-page python)))) + +;; Python 3 toolchain for python-build-system. We cannot use python-toolchain +;; here, since we’d need to bootstrap python-pip somehow. +(define-public python-toolchain-for-build + (package + (inherit python-toolchain) + (name "python-toolchain-for-build") + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools) + ("python-pypa-build" ,python-pypa-build-from-setuptools))))) + +;; Python 3 toolchain to bootstrap python-pypa-build +(define-public python-toolchain-only-setuptools + (package + (inherit python-toolchain) + (name "python-toolchain-only-setuptools") + (inputs + `(("python" ,python-wrapper) + ("python-setuptools" ,python-setuptools))))) + +(define-public python-pypa-build-from-setuptools + (package + (inherit python-pypa-build) + (name "python-pypa-build-from-setuptools") + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools)) + (propagated-inputs + `(("python-pep517" ,python-pep517-from-setuptools) + ("python-packaging" ,python-packaging-from-setuptools))))) + +(define-public python-pep517-from-setuptools + (package + (inherit python-pep517-bootstrap) + (name "python-pep517-from-setuptools") + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools + #:phases (modify-phases %standard-phases + (add-after 'unpack 'patch + (lambda _ + (substitute* "setup.py" + (("distutils\\.core") "setuptools")) + #t))))) + (propagated-inputs + `(("python-toml" ,python-toml-from-setuptools) + ("python-wheel" ,python-wheel-from-setuptools))) + ;; Drop cyclic dependency. + (native-inputs '()))) + +(define-public python-toml-from-setuptools + (package + (inherit python-toml) + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools + ,@(package-arguments python-toml))))) + +(define-public python-packaging-from-setuptools + (package + (inherit python-packaging-bootstrap) + (name "python-packaging-from-setuptools") + (arguments + `(#:python ,python-toolchain-only-setuptools + ,@(package-arguments python-packaging-bootstrap))) + (propagated-inputs + `(("python-pyparsing" ,python-pyparsing-from-setuptools) + ("python-six" ,python-six-from-setuptools))))) + +(define-public python-pyparsing-from-setuptools + (package + (inherit python-pyparsing) + (name "python-pyparsing-from-setuptools") + (arguments + `(#:tests? #f + #:python ,python-toolchain-only-setuptools + ,@(package-arguments python-pyparsing))))) + +(define-public python-six-from-setuptools + (package + (inherit python-six-bootstrap) + (name "python-six-from-setuptools") + (arguments + `(#:python ,python-toolchain-only-setuptools + ,@(package-arguments python-six-bootstrap))))) + +(define-public python-wheel-from-setuptools + (package + (inherit python-wheel) + (name "python-wheel-from-setuptools") + (arguments + `(#:python ,python-toolchain-only-setuptools + ,@(package-arguments python-wheel))))) + diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm index 8e8f46467b..ca5ce667ef 100644 --- a/gnu/packages/python.scm +++ b/gnu/packages/python.scm @@ -182,7 +182,7 @@ (list "--enable-shared" ;allow embedding "--with-system-expat" ;for XML support "--with-system-ffi" ;build ctypes - "--with-ensurepip=install" ;install pip and setuptools + "--with-ensurepip=no" ;do not install pip and setuptools "--enable-unicode=ucs4" ;; Prevent the installed _sysconfigdata.py from retaining a reference diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm index 2bb6fa87ca..998ea9323d 100644 --- a/guix/build-system/python.scm +++ b/guix/build-system/python.scm @@ -65,8 +65,8 @@ extension, such as '.tar.gz'." (define (default-python) "Return the default Python package." ;; Lazily resolve the binding to avoid a circular dependency. - (let ((python (resolve-interface '(gnu packages python)))) - (module-ref python 'python-wrapper))) + (let ((python (resolve-interface '(gnu packages python-commencement)))) + (module-ref python 'python-toolchain-for-build))) (define (default-python2) "Return the default Python 2 package." @@ -172,8 +172,6 @@ pre-defined variants." (define* (python-build store name inputs #:key (tests? #t) - (test-target "test") - (use-setuptools? #t) (configure-flags ''()) (phases '(@ (guix build python-build-system) %standard-phases)) @@ -199,9 +197,7 @@ provides a 'setup.py' file as its build system." source)) #:configure-flags ,configure-flags #:system ,system - #:test-target ,test-target #:tests? ,tests? - #:use-setuptools? ,use-setuptools? #:phases ,phases #:outputs %outputs #:search-paths ',(map search-path-specification->sexp diff --git a/guix/build/python-build-system.scm b/guix/build/python-build-system.scm index 8ade1d5911..a5731511a9 100644 --- a/guix/build/python-build-system.scm +++ b/guix/build/python-build-system.scm @@ -34,6 +34,7 @@ #:use-module (ice-9 format) #:use-module (srfi srfi-1) #:use-module (srfi srfi-26) + #:use-module (srfi srfi-35) #:export (%standard-phases add-installed-pythonpath site-packages @@ -108,30 +109,17 @@ ;; "--single-version-externally-managed" is set, thus the .egg-info directory ;; and the scripts defined in entry-points will always be created. +;; Base error type. +(define-condition-type &python-build-error &error + python-build-error?) -(define setuptools-shim - ;; Run setup.py with "setuptools" being imported, which will patch - ;; "distutils". This is needed for packages using "distutils" instead of - ;; "setuptools" since the former does not understand the - ;; "--single-version-externally-managed" flag. - ;; Python code taken from pip 9.0.1 pip/utils/setuptools_build.py - (string-append - "import setuptools, tokenize;__file__='setup.py';" - "f=getattr(tokenize, 'open', open)(__file__);" - "code=f.read().replace('\\r\\n', '\\n');" - "f.close();" - "exec(compile(code, __file__, 'exec'))")) - -(define (call-setuppy command params use-setuptools?) - (if (file-exists? "setup.py") - (begin - (format #t "running \"python setup.py\" with command ~s and parameters ~s~%" - command params) - (if use-setuptools? - (apply invoke "python" "-c" setuptools-shim - command params) - (apply invoke "python" "./setup.py" command params))) - (error "no setup.py found"))) +;; Raised when 'check cannot find a valid test system in the inputs. +(define-condition-type &test-system-not-found &python-build-error + test-system-not-found?) + +;; Raised when multiple wheels are created by 'build. +(define-condition-type &cannot-extract-multiple-wheels &python-build-error + cannot-extract-multiple-wheels?) (define* (sanity-check #:key tests? inputs outputs #:allow-other-keys) "Ensure packages depending on this package via setuptools work properly, @@ -142,23 +130,51 @@ without errors." (with-directory-excursion "/tmp" (invoke "python" sanity-check.py (site-packages inputs outputs))))) -(define* (build #:key use-setuptools? #:allow-other-keys) +(define* (build #:key outputs #:allow-other-keys) "Build a given Python package." - (call-setuppy "build" '() use-setuptools?) + + (define pyproject-build (which "pyproject-build")) + + (define (build-pep517) + ;; XXX: should probably use a different path, outside of source directory, + ;; maybe secondary output “wheel”? + (mkdir-p "dist") + (invoke pyproject-build "--outdir" "dist" "--no-isolation" "--wheel" ".")) + + ;; XXX Would be nice, if we could use bdist_wheel here to remove extra + ;; code path in 'install, but that depends on python-wheel. + (define (build-setuptools) + (invoke "python" "setup.py" "build")) + + (if pyproject-build + (build-pep517) + (build-setuptools)) #t) -(define* (check #:key tests? test-target use-setuptools? #:allow-other-keys) +(define* (check #:key inputs outputs tests? #:allow-other-keys) "Run the test suite of a given Python package." (if tests? - ;; Running `setup.py test` creates an additional .egg-info directory in - ;; build/lib in some cases, e.g. if the source is in a sub-directory - ;; (given with `package_dir`). This will by copied to the output, too, - ;; so we need to remove. - (let ((before (find-files "build" "\\.egg-info$" #:directories? #t))) - (call-setuppy test-target '() use-setuptools?) - (let* ((after (find-files "build" "\\.egg-info$" #:directories? #t)) - (inter (lset-difference string=? after before))) - (for-each delete-file-recursively inter))) + ;; Unfortunately with PEP 517 there is no common method to specify test + ;; systems. Guess test system based on inputs instead. + (let ((pytest (which "pytest")) + (have-setup-py (file-exists? "setup.py"))) + ;; Prefer pytest + ;; XXX: support nose + (cond + (pytest + (begin + (format #t "using pytest~%") + (invoke pytest "-vv"))) ; XXX: support skipping tests based on name/extra arguments? + ;; But fall back to setup.py, which should work for most + ;; packages. XXX: would be nice not to depend on setup.py here? fails + ;; more often than not to find any tests at all. Maybe we can run + ;; `python -m unittest`? + (have-setup-py + (begin + (format #t "using setup.py~%") + (invoke "python" "setup.py" "test" "-v"))) + ;; The developer should explicitly disable tests in this case. + (#t (raise (condition (&test-system-not-found)))))) (format #t "test suite not run~%")) #t) @@ -195,31 +211,109 @@ running checks after installing the package." "/bin:" (getenv "PATH")))) -(define* (install #:key inputs outputs (configure-flags '()) use-setuptools? - #:allow-other-keys) - "Install a given Python package." - (let* ((out (python-output outputs)) - (python (assoc-ref inputs "python")) - (major-minor (map string->number - (take (string-split (python-version python) #\.) 2))) - (<3.7? (match major-minor - ((major minor) - (or (< major 3) (and (= major 3) (< minor 7)))))) - (params (append (list (string-append "--prefix=" out) - "--no-compile") - (if use-setuptools? - ;; distutils does not accept these flags - (list "--single-version-externally-managed" - "--root=/") - '()) - configure-flags))) - (call-setuppy "install" params use-setuptools?) - ;; Rather than produce potentially non-reproducible .pyc files on Pythons - ;; older than 3.7, whose 'compileall' module lacks the - ;; '--invalidation-mode' option, do not generate any. - (unless <3.7? - (invoke "python" "-m" "compileall" "--invalidation-mode=unchecked-hash" - out)))) +(define* (install #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Install a wheel file according to PEP 427" + ;; See https://www.python.org/dev/peps/pep-0427/#installing-a-wheel-distribution-1-0-py32-none-any-whl + (let* ((site-dir (site-packages inputs outputs)) + (out (assoc-ref outputs "out"))) + (define (extract file) + "Extract wheel (ZIP file) into site-packages directory" + ;; Use Python’s zipfile to avoid extra dependency + (invoke "python" "-m" "zipfile" "-e" file site-dir)) + + (define python-hashbang + (string-append "#!" (assoc-ref inputs "python") "/bin/python")) + + (define (move-data source destination) + (mkdir-p (dirname destination)) + (rename-file source destination)) + + (define (move-script source destination) + "Move executable script file from .data/scripts to out/bin and replace +temporary hashbang" + (move-data source destination) + ;; ZIP does not save/restore permissions, make executable + ;; XXX: might not be a file, but directory with subdirectories + (chmod destination #o755) + (substitute* destination (("#!python") python-hashbang))) + + ;; Python’s distutils.command.install defines this mapping from source to + ;; destination mapping. + (define install-schemes + `(("scripts" "bin" ,move-script) + ;; XXX: Why does Python not use share/ here? + ("data" "share" ,move-data))) + + (define (expand-data-directory directory) + "Move files from all .data subdirectories to their respective +destinations." + (for-each + (match-lambda ((source destination function) + (let ((source-path (string-append directory "/" source)) + (destination-path (string-append out "/" destination))) + (when (file-exists? source-path) + (begin + ;; This assumes only files exist in the scripts/ directory. + (for-each + (lambda (file) + (apply + function + (list + (string-append source-path "/" file) + (string-append destination-path "/" file)))) + (scandir source-path (negate (cut member <> '("." ".."))))) + (rmdir source-path)))))) + install-schemes)) + + (define pyproject-build (which "pyproject-build")) + + (define (list-directories base predicate) + ;; Cannot use find-files here, because it’s recursive. + (scandir + base + (lambda (name) + (let ((stat (lstat (string-append base "/" name)))) + (and + (not (member name '("." ".."))) + (eq? (stat:type stat) 'directory) + (predicate name stat)))))) + + (define (install-pep517) + "Install a wheel generated by a PEP 517-compatible builder." + (let ((wheels (find-files "dist" "\\.whl$"))) ; XXX: do not recurse + (when (> (length wheels) 1) ; This code does not support multiple wheels + ; yet, because their outputs would have to be + ; merged properly. + (raise (condition (&cannot-extract-multiple-wheels)))) + (for-each extract wheels)) + (let ((datadirs (map + (cut string-append site-dir "/" <>) + (list-directories site-dir (file-name-predicate "\\.data$"))))) + (for-each (lambda (directory) + (expand-data-directory directory) + (rmdir directory)) + datadirs))) + + (define (install-setuptools) + "Install using setuptools." + (let ((out (assoc-ref outputs "out"))) + (invoke "python" "setup.py" + "install" + "--prefix" out + "--single-version-externally-managed" + "--root=/"))) + + (if pyproject-build + (install-pep517) + (install-setuptools)) + #t)) + +(define* (compile-bytecode #:key inputs outputs (configure-flags '()) #:allow-other-keys) + "Compile installed byte-code in site-packages." + (let ((site-dir (site-packages inputs outputs))) + (invoke "python" "-m" "compileall" site-dir) + ;; XXX: We could compile with -O and -OO too here, at the cost of more space. + #t)) (define* (wrap #:key inputs outputs #:allow-other-keys) (define (list-of-files dir) @@ -243,29 +337,12 @@ running checks after installing the package." files))) bindirs))) -(define* (rename-pth-file #:key name inputs outputs #:allow-other-keys) - "Rename easy-install.pth to NAME.pth to avoid conflicts between packages -installed with setuptools." - ;; Even if the "easy-install.pth" is not longer created, we kept this phase. - ;; There still may be packages creating an "easy-install.pth" manually for - ;; some good reason. - (let* ((site-packages (site-packages inputs outputs)) - (easy-install-pth (string-append site-packages "/easy-install.pth")) - (new-pth (string-append site-packages "/" name ".pth"))) - (when (file-exists? easy-install-pth) - (rename-file easy-install-pth new-pth)))) - -(define* (ensure-no-mtimes-pre-1980 #:rest _) - "Ensure that there are no mtimes before 1980-01-02 in the source tree." - ;; Rationale: patch-and-repack creates tarballs with timestamps at the POSIX - ;; epoch, 1970-01-01 UTC. This causes problems with Python packages, - ;; because Python eggs are ZIP files, and the ZIP format does not support - ;; timestamps before 1980. - (let ((early-1980 315619200)) ; 1980-01-02 UTC - (ftw "." (lambda (file stat flag) - (unless (<= early-1980 (stat:mtime stat)) - (utime file early-1980 early-1980)) - #t)))) +(define* (set-SOURCE-DATE-EPOCH #:rest _) + "Set the 'SOURCE_DATE_EPOCH' environment variable. This is used by tools +that incorporate timestamps as a way to tell them to use a fixed timestamp. +See https://reproducible-builds.org/specs/source-date-epoch/." + (setenv "SOURCE_DATE_EPOCH" "315619200") ;; python-wheel respects this variable and sets pre-1980 times on files in zip files, which is unsupported + #t) (define* (enable-bytecode-determinism #:rest _) "Improve determinism of pyc files." @@ -292,11 +369,11 @@ by Cython." ;; prefix directory. The check phase is moved after the installation phase ;; to ease testing the built package. (modify-phases gnu:%standard-phases - (add-after 'unpack 'ensure-no-mtimes-pre-1980 ensure-no-mtimes-pre-1980) - (add-after 'ensure-no-mtimes-pre-1980 'enable-bytecode-determinism + (add-after 'unpack 'enable-bytecode-determinism enable-bytecode-determinism) (add-after 'enable-bytecode-determinism 'ensure-no-cythonized-files ensure-no-cythonized-files) + (replace 'set-SOURCE-DATE-EPOCH set-SOURCE-DATE-EPOCH) (delete 'bootstrap) (delete 'configure) ;not needed (replace 'build build) @@ -308,7 +385,7 @@ by Cython." (add-after 'add-install-to-path 'wrap wrap) (add-after 'wrap 'check check) (add-after 'check 'sanity-check sanity-check) - (add-before 'strip 'rename-pth-file rename-pth-file))) + (add-before 'check 'compile-bytecode compile-bytecode))) (define* (python-build #:key inputs (phases %standard-phases) #:allow-other-keys #:rest args) -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0006-gnu-Add-python-pytoml.patch" >From 14b70dea3d21684e7fdb66dd072031cef3214b07 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:17:10 +0100 Subject: [PATCH 06/12] gnu: Add python-pytoml. * gnu/packages/python-build.scm (python-pytoml): Add new variable. --- gnu/packages/python-build.scm | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index 232e24f470..a6a310177c 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -173,3 +173,20 @@ implementation developed for Poetry. This project is intended to be a light weight, fully compliant, self-contained package allowing PEP 517 compatible build front-ends to build Poetry managed projects.") (license license:expat))) + +(define-public python-pytoml + (package + (name "python-pytoml") + (version "0.1.21") + (source + (origin + (method url-fetch) + (uri (pypi-uri "pytoml" version)) + (sha256 + (base32 + "1rv1byiw82k7mj6aprcrqi2vdabs801y97xhfnrz7kxds34ggv4f")))) + (build-system python-build-system) + (home-page "https://github.com/avakar/pytoml") + (synopsis "A parser for TOML-0.4.0") + (description "A parser for TOML-0.4.0") + (license license:expat))) -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0007-gnu-Add-python-flit-core.patch" >From e8b1f40157b3d884bb2fc3e3d10bbfbd469c67b9 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:18:17 +0100 Subject: [PATCH 07/12] gnu: Add python-flit-core. * gnu/packages/python-build.scm (python-flit-core): New variable. --- gnu/packages/python-build.scm | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index a6a310177c..f767704a78 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -190,3 +190,30 @@ compatible build front-ends to build Poetry managed projects.") (synopsis "A parser for TOML-0.4.0") (description "A parser for TOML-0.4.0") (license license:expat))) + +(define-public python-flit-core + (package + (name "python-flit-core") + (version "3.0.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "flit_core" version)) + (sha256 + (base32 + "0bbw84r33gwi0xyp7m8dzp2dzpjs4harj3l5wrbxkmp2awh0ard4")))) + (build-system python-build-system) + (arguments + `(;; No tests. + #:tests? #f)) + (propagated-inputs + `(("python-pytoml" ,python-pytoml))) + (home-page "https://github.com/takluyver/flit") + (synopsis + "Simplified packaging of Python modules, distribution-building parts") + (description + "Flit is a simple way to put Python packages and modules on PyPI. It +tries to require less thought about packaging and help you avoid common +mistakes. Distribution-building parts of Flit.") + (license license:bsd-3))) + -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0008-gnu-python-pep517-bootstrap-Build-using-flit-core.patch" >From 316c25f310cad4dd8f0cb73a914a4776b1b1375c Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:20:48 +0100 Subject: [PATCH 08/12] gnu: python-pep517-bootstrap: Build using flit-core. * gnu/packages/python-build.scm (python-pep517-bootstrap) [arguments]: Relax dependency on flit-core version. [native-inputs]: Add flit-core. --- gnu/packages/python-build.scm | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/gnu/packages/python-build.scm b/gnu/packages/python-build.scm index f767704a78..f74a3ee49e 100644 --- a/gnu/packages/python-build.scm +++ b/gnu/packages/python-build.scm @@ -110,10 +110,21 @@ Language (TOML) configuration files.") "0zqidxah03qpnp6zkg3zd1kmd5f79hhdsfmlc0cldaniy80qddxf")))) (build-system python-build-system) (arguments - `(#:tests? #f)) ;to avoid circular dependencies + `(#:tests? #f ; To avoid circular dependencies. + #:phases + (modify-phases %standard-phases + (add-after 'unpack 'relax-dependency + (lambda _ + (substitute* "pyproject.toml" + (("flit_core >=2,<3") + "flit_core >=2,<4")) + #t))))) (propagated-inputs `(("python-toml" ,python-toml) ("python-wheel" ,python-wheel))) + (native-inputs + `(;; Build system. + ("python-flit-core" ,python-flit-core))) (home-page "https://github.com/pypa/pep517") (synopsis "Wrappers to build Python packages using PEP 517 hooks") (description -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0009-gnu-python-iniconfig-Add-missing-build-input.patch" >From c8898a6a282daaa2cceabfac96e417f7b09e8d5f Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Sun, 28 Feb 2021 13:23:53 +0100 Subject: [PATCH 09/12] gnu: python-iniconfig: Add missing build input. * gnu/packages/python-xyz.scm (python-iniconfig) [native-inputs] Add setuptools-scm. --- gnu/packages/python-xyz.scm | 1 + 1 file changed, 1 insertion(+) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index 79d01f700a..d598a380a9 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -15704,6 +15704,7 @@ in other versions.") (base32 "0ckzngs3scaa1mcfmsi1w40a1l8cxxnncscrxzjjwjyisx8z0fmw")))) (build-system python-build-system) + (native-inputs `(("python-setuptools-scm" ,python-setuptools-scm))) (home-page "https://github.com/RonnyPfannschmidt/iniconfig") (synopsis "Simple INI-file parser") (description "The @code{iniconfig} package provides a small and simple -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0010-gnu-Add-python-u-msgpack.patch" >From 8f5c9398ac1a6ac1871d5cf8a1efbe6a70ed4a1b Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 14:16:07 +0100 Subject: [PATCH 10/12] gnu: Add python-u-msgpack. * gnu/packages/python-xyz.scm (python-u-msgpack): New variable. The redundant -python postfix from the original package name has been removed. --- gnu/packages/python-xyz.scm | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm index d598a380a9..ab3c6d301e 100644 --- a/gnu/packages/python-xyz.scm +++ b/gnu/packages/python-xyz.scm @@ -23659,3 +23659,27 @@ Application Programming Interface based on the Open Inventor 2.1 API.") Crayons automatically wraps a given string in the foreground color and restores the original state after the string is printed.") (license license:expat))) + +(define-public python-u-msgpack + (package + (name "python-u-msgpack") + (version "2.7.1") + (source + (origin + (method url-fetch) + (uri (pypi-uri "u-msgpack-python" version)) + (sha256 + (base32 + "0lcmlr7gc4dydpxn6l5bdcq40c3ghf8mv1sjqyj72wdpr8rx9rxp")))) + (build-system python-build-system) + (home-page + "https://github.com/vsergeev/u-msgpack-python") + (synopsis + "Portable, lightweight MessagePack serializer and deserializer") + (description + "A portable, lightweight MessagePack serializer and deserializer written +in pure Python. u-msgpack-python is fully compliant with the latest MessagePack +specification. In particular, it supports the new binary, UTF-8 string, +application-defined ext, and timestamp types.") + (license license:expat))) + -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0011-gnu-Add-python-pytest-expect.patch" >From 39eef77658f400ccfceb65b6fcd3f4996ae90807 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 14:20:03 +0100 Subject: [PATCH 11/12] gnu: Add python-pytest-expect. * gnu/packages/python-check.scm (python-pytest-expect): New variable. --- gnu/packages/python-check.scm | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/gnu/packages/python-check.scm b/gnu/packages/python-check.scm index 9849b16685..4139f3cdde 100644 --- a/gnu/packages/python-check.scm +++ b/gnu/packages/python-check.scm @@ -1251,3 +1251,31 @@ help in debugging failures and optimizing the scheduler to improve speed.") (description "A pytest plugin for Sanic. It helps you to test your code asynchronously.") (license license:expat))) + +(define-public python-pytest-expect + (package + (name "python-pytest-expect") + (version "1.1.0") + (source + (origin + (method url-fetch) + (uri (pypi-uri "pytest-expect" version)) + (sha256 + (base32 + "0iyq3zd1g5ffaz2wv6mskjfn84sfbyh0j209glcrh1s50hkldd1n")))) + (build-system python-build-system) + (arguments `(#:tests? #f)) ; no tests via pypi + (propagated-inputs + `(("python-pytest" ,python-pytest) ; package declares this dependency + ("python-u-msgpack" ,python-u-msgpack))) + (home-page + "https://github.com/gsnedders/pytest-expect") + (synopsis + "Py.test plugin to store test expectations and mark tests based on them") + (description + "A py.test plugin that stores test expectations by saving the set of +failing tests, allowing them to be marked as xfail when running them in future. +The tests expectations are stored such that they can be distributed alongside +the tests.") + (license license:expat))) + -- 2.26.2 --9EzuTbC0hpH3Zvh5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0012-gnu-python-html5lib-Fix-tests-with-pytest-6.patch" >From 4ed7eba12c80dd33f20626ddb81abc34dd667ab3 Mon Sep 17 00:00:00 2001 From: Lars-Dominik Braun Date: Mon, 1 Mar 2021 14:23:07 +0100 Subject: [PATCH 12/12] gnu: python-html5lib: Fix tests with pytest 6. * gnu/packages/python-web.scm (python-html5lib) [source]: Add upstream patch. [native-inputs]: Add test dependencies. [arguments]: Remove unsupported #:test-target. --- gnu/packages/python-web.scm | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm index 947c200253..c484d7ba36 100644 --- a/gnu/packages/python-web.scm +++ b/gnu/packages/python-web.scm @@ -1020,13 +1020,27 @@ storage.") (uri (pypi-uri "html5lib" version)) (sha256 (base32 - "0vqlhk0hgbsfkh7ybmby93xhlx8dq6pr5blf356ka3z2c41b9rdj")))) + "0vqlhk0hgbsfkh7ybmby93xhlx8dq6pr5blf356ka3z2c41b9rdj")) + (patches + (list + ;; Adds Pytest 6 support. + (origin + (method url-fetch) + (uri (string-append + "https://github.com/html5lib/" + "html5lib-python/commit/" + "2c19b9899ab3a3e8bd0ca35e5d78544334204169.patch")) + (file-name "python-html5lib-support-pytest6.patch") + (sha256 + (base32 + "0jg2ry0439q8n7j1mf4p2hdq54i704pq9scv4wwa2pp3cwvb6dvg"))))))) (build-system python-build-system) (propagated-inputs `(("python-six" ,python-six) ("python-webencodings" ,python-webencodings))) - (arguments - `(#:test-target "check")) + (native-inputs + `(("python-pytest" ,python-pytest) + ("python-pytest-expect" ,python-pytest-expect))) (home-page "https://github.com/html5lib/html5lib-python") (synopsis -- 2.26.2 --9EzuTbC0hpH3Zvh5-- ------------=_1676024042-16198-1--