GNU bug report logs - #74445
[python-team] Respect --cores build argument.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 74445 in the body.
You can then email your comments to 74445 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Greg Hogan <code <at> greghogan.com>
To: bug-guix <at> gnu.org
Subject: Python respect --cores build argument
Date: Wed, 20 Nov 2024 10:10:27 -0500

#74436 fixed two instances of "--numprocesses" "auto" but a simple
grep shows 18 additional cases (one of which is commented out).

Message #8 received at 74445 <at> debbugs.gnu.org (full text, mbox):

From: Sharlatan Hellseher <sharlatanus <at> gmail.com>
To: 74445 <at> debbugs.gnu.org
Cc: Greg Hogan <code <at> greghogan.com>
Subject: Python respect --cores build argument
Date: Wed, 20 Nov 2024 18:47:45 +0000

[Message part 1 (text/plain, inline)]

Hi,

There a lot in astronomy module I guess ^.^

Would you like to send a patch adjusting them?

They could be "-n" "auto" as well.

Thanks,
Oleg

[Message part 2 (text/html, inline)]

Message #13 received at 74445 <at> debbugs.gnu.org (full text, mbox):

From: Sharlatan Hellseher <sharlatanus <at> gmail.com>
To: 74445 <at> debbugs.gnu.org
Cc: Sharlatan Hellseher <sharlatanus <at> gmail.com>
Subject: [PATCH 0/1] gnu: Respect --cores build argument in some python
 packages.
Date: Wed, 20 Nov 2024 20:54:08 +0000

Hi Greg,

Thanks for highlighting that, I was too much keen to make tests faster.

This patch adjust all "-n" or "--numprocesses" from "auto" to
"parallel-job-count" parameter.
This quite save to be pushed to master directly:

--8<---------------cut here---------------start------------->8---
Building the following 77 packages would ensure 180 dependent packages are
rebuilt: python-yt-astro-analysis <at> 1.1.3 variant-tools <at> 3.1.2 osc <at> 0.172.0
h-client <at> 0.0a0-1.e6c78b1 python-urlgrabber <at> 4.1.0 xapers <at> 0.8.2 ganeti <at> 3.0.2
awscli <at> 1.36.0 instrain <at> 1.5.4 python-osmnx <at> 1.9.3 conda <at> 22.9.0
python-awscrt <at> 0.23.0 awscli <at> 2.20.0 python-spacy <at> 3.5.3 snakemake <at> 6.15.5
python-scm-sr-ht <at> 0.22.23 awslogs <at> 0.14.0 python-tibanna <at> 5.4.3
python-hicexplorer <at> 3.7.4 python-vaex-core <at> 4.13.0 python-kanon <at> 0.6.6
python-zodipy <at> 1.1.0 python-coolest <at> 0.1.9 python-sncosmo <at> 2.11.1
python-halotools <at> 0.9.1 python-healpy <at> 1.16.6 scamp <at> 2.10.0
python-asdf-compression <at> 0.0.1-0.57cc7e7 python-asdf-zarr <at> 0.0.4
python-regularizepsf <at> 0.4.0 python-statmorph <at> 0.5.7 python-pyxsim <at> 4.4.2
python-viresclient <at> 0.12.0 python-naima <at> 0.10.0 python-aplpy <at> 2.1.0
python-astroalign <at> 2.6.0 ginga-qt5 <at> 5.1.0 python-poliastro <at> 0.17.0
python-astroplan <at> 0.10.1 python-bulkvis <at> 2.0.0-2.00a82a9 python-cooltools <at> 0.7.0
fanc <at> 0-1.354401e python-fanc <at> 0.9.25 python-pytorch-geometric <at> 2.4.0 cura <at> 4.13.1
python-sunkit-image <at> 0.5.1 python-radiospectra <at> 0.6.0 python-webbpsf <at> 1.4.0
python-acstools <at> 3.7.2 python-jwst-reffiles <at> 1.0.1 python-sbpy <at> 0.5.0
python-glue-qt <at> 0.3.1 python-glue-astronomy <at> 0.10.0 python-pyrodigal <at> 3.3.0
python-verde <at> 1.8.0 python-scikit-allel <at> 1.3.5 python-pandera <at> 0.18.0
python-modin <at> 0.15.1 python-cesium <at> 0.12.1 synapse <at> 1.29.0 python-sparse <at> 0.14.0
python-demuxem <at> 0.1.7 python-cellbender <at> 0.2.2 python-scikit-bio <at> 0.6.0
python-ikarus <at> 0.0.2 python-muon <at> 0.1.6 python-liana-py <at> 1.1.0
r-doubletcollection <at> 1.1.0-1.c0d62f1 python-episcanpy <at> 0.4.0
python-metacells <at> 0.9.4 python-celltypist <at> 1.6.2 python-hotspotsc <at> 1.1.1
python-multivelo <at> 0.1.2 python-baltica <at> 1.1.2 scregseg <at> 0.1.3
python-magic-impute <at> 1.2.1 python-multiscale-spatial-image <at> 1.0.0
--8<---------------cut here---------------end--------------->8---

Thanks,
Oleg

Sharlatan Hellseher (1):
  gnu: Respect --cores build argument in some python packages.

 gnu/packages/astronomy.scm  | 31 ++++++++++++++++---------------
 gnu/packages/check.scm      |  2 +-
 gnu/packages/databases.scm  |  2 +-
 gnu/packages/python-web.scm |  6 +++---
 gnu/packages/python-xyz.scm |  6 +++---
 5 files changed, 24 insertions(+), 23 deletions(-)


base-commit: da8afaa65fe30ae59e1bedbbb231490ad01c013c
-- 
2.46.0

Message #16 received at 74445 <at> debbugs.gnu.org (full text, mbox):

From: Sharlatan Hellseher <sharlatanus <at> gmail.com>
To: 74445 <at> debbugs.gnu.org
Cc: Sharlatan Hellseher <sharlatanus <at> gmail.com>
Subject: [PATCH 1/1] gnu: Respect --cores build argument in some python
 packages.
Date: Wed, 20 Nov 2024 20:54:24 +0000

Reported by Greg Hogan <code <at> greghogan.com> in #74445.

* gnu/packages/astronomy.scm (python-asdf-astropy, python-astropy,
python-photutils, python-poppy, python-regions, python-reproject,
python-sunpy, python-spectral-cube, python-stdatamodels, python-pysiaf,
python-sbpy, python-asdf-coordinates-schemas, python-roman-datamodels,
python-webbpsf, python-yt):
[arguments]<test-flags>: Adjust "-n" or "--numprocess" to respect
"--cores" build argument.

* gnu/packages/check.scm (python-crosshair): Likewise.

* gnu/packages/databases.scm (python-fastparquet, python-pycurl, awscli,
python-s3transfer): Likewise.

* gnu/packages/python-xyz.scm (python-glymur, python-zarr, python-dask):
Likewise.

Change-Id: Ifbc6435e4ad22b0ae822b485bccca41eaa165cc5
---
 gnu/packages/astronomy.scm  | 31 ++++++++++++++++---------------
 gnu/packages/check.scm      |  2 +-
 gnu/packages/databases.scm  |  2 +-
 gnu/packages/python-web.scm |  6 +++---
 gnu/packages/python-xyz.scm |  6 +++---
 5 files changed, 24 insertions(+), 23 deletions(-)

diff --git a/gnu/packages/astronomy.scm b/gnu/packages/astronomy.scm
index 6cc5142370..296ac44dbf 100644
--- a/gnu/packages/astronomy.scm
+++ b/gnu/packages/astronomy.scm
@@ -1653,7 +1653,7 @@ (define-public python-asdf-astropy
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases #~(modify-phases %standard-phases
                    (add-before 'check 'set-home-env
                      (lambda _ (setenv "HOME" "/tmp"))))))
@@ -3034,7 +3034,7 @@ (define-public python-astropy
      (list
       #:test-flags
       #~(list "--pyargs" "astropy"
-              "--numprocesses" "auto"
+              "--numprocesses" (number->string (parallel-job-count))
               "-k" (string-append
                     ;; Skip tests that need remote data.
                     "not remote_data"
@@ -3924,7 +3924,7 @@ (define-public python-photutils
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
          (add-after 'unpack 'relax-requirements
@@ -4085,7 +4085,7 @@ (define-public python-poppy
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto")))
+      #~(list "--numprocesses" (number->string (parallel-job-count)))))
     (propagated-inputs
      ;; XXX: With python-synphot (marked as optional) package added to the list
      ;; it tries to download from remote host during tests and fails. Overall
@@ -4191,7 +4191,7 @@ (define-public python-regions
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
           ;; setup.py was removed in 84c80a280431adda00641cda5264c7de18b43b2f
@@ -4295,7 +4295,7 @@ (define-public python-reproject
       #:test-flags
       #~(list "--arraydiff"
               "--arraydiff-default-format=fits"
-              "--numprocesses" "auto"
+              "--numprocesses" (number->string (parallel-job-count))
               "--pyargs" "reproject")
       #:phases
       #~(modify-phases %standard-phases
@@ -4399,7 +4399,7 @@ (define-public python-sunpy
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Requries SpicePy not packed in Guix yet.
               "--ignore=sunpy/coordinates/tests/test_spice.py")
       #:phases
@@ -4589,7 +4589,7 @@ (define-public python-spectral-cube
       ;; See <https://github.com/radio-astro-tools/radio-beam/issues/129>.
       #:tests? #f
       #:test-flags
-      #~(list "-n" "auto")))
+      #~(list "--numprocesses" (number->string (parallel-job-count)))))
     (propagated-inputs
      (list python-astropy
            ;; XXX: Currently failing in upstream as it's optional silent
@@ -4981,7 +4981,7 @@ (define-public python-stdatamodels
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Disable tests requiring access to CRDS servers to download
               ;; ~500MiB of data.
               "-k" "not test_crds_selectors_vs_datamodel")
@@ -5450,7 +5450,7 @@ (define-public python-pysiaf
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Disable 2 failing tests, see
               ;; <https://github.com/spacetelescope/pysiaf/issues/338>
               "-k" (string-append "not test_write_jwst_siaf_xlsx"
@@ -5543,7 +5543,8 @@ (define-public python-sbpy
     (build-system pyproject-build-system)
     (arguments
      (list
-      #:test-flags #~(list "--numprocesses" "auto")
+      #:test-flags
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
           (add-before 'check 'set-home-env
@@ -5835,7 +5836,7 @@ (define python-asdf-coordinates-schemas
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto")))
+      #~(list "--numprocesses" (number->string (parallel-job-count)))))
     (native-inputs
      (list python-pytest
            python-pytest-xdist
@@ -6137,7 +6138,7 @@ (define-public python-roman-datamodels
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
           (add-after 'unpack 'set-env
@@ -6295,7 +6296,7 @@ (define-public python-webbpsf
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               "-k" (string-append
                     ;; Test requiring network access
                     "not test_monthly_trending_plot_auto_opdtable"
@@ -6383,7 +6384,7 @@ (define-public python-yt
      (list
       #:build-backend "setuptools.build_meta"
       #:test-flags
-      #~(list "--numprocesses" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
          (add-after 'unpack 'relax-requirements
diff --git a/gnu/packages/check.scm b/gnu/packages/check.scm
index 6593eadf6a..ef1c08f8f3 100644
--- a/gnu/packages/check.scm
+++ b/gnu/packages/check.scm
@@ -2744,7 +2744,7 @@ (define-public python-crosshair
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; check_examples_test.py contains failing tests that
               ;; show what happens if a counterexample is found.
               "--ignore=crosshair/examples/check_examples_test.py"
diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index 5f762eb5be..142f9090c5 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -5203,7 +5203,7 @@ (define-public python-fastparquet
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto")
+      #~(list "--numprocesses" (number->string (parallel-job-count)))
       #:phases
       #~(modify-phases %standard-phases
           (add-after 'unpack 'relax-requirements
diff --git a/gnu/packages/python-web.scm b/gnu/packages/python-web.scm
index c030deeb12..cf3c15857c 100644
--- a/gnu/packages/python-web.scm
+++ b/gnu/packages/python-web.scm
@@ -2026,7 +2026,7 @@ (define-public python-pycurl
     (build-system pyproject-build-system)
     (arguments
      '(#:test-flags
-       (list "-n" "auto"
+       (list "--n" (number->string (parallel-job-count))
              "-k" (string-append
                    ;; Disable hanginging tests
                    "not test_multi_socket_select"
@@ -4235,7 +4235,7 @@ (define-public awscli
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Tests require networking.
               "--ignore" "tests/integration"
               ;; It strugles to set PYTHONPATH.
@@ -4909,7 +4909,7 @@ (define-public python-s3transfer
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Tests require networking.
               "--ignore" "tests/integration")))
     (native-inputs
diff --git a/gnu/packages/python-xyz.scm b/gnu/packages/python-xyz.scm
index ceb122e08d..3872b511a1 100644
--- a/gnu/packages/python-xyz.scm
+++ b/gnu/packages/python-xyz.scm
@@ -2296,7 +2296,7 @@ (define-public python-glymur
     (arguments
      (list
       #:test-flags
-      #~(list "--numprocesses" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; Failing test due to inability of ctypes.util.find_library()
               ;; to determine library path, which is patched above.
               "--ignore=tests/test_config.py")
@@ -28320,7 +28320,7 @@ (define-public python-zarr
     (arguments
      (list
       #:test-flags
-      #~(list "-n" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               ;; This tests are flaky.  The pass several times on my laptop
               ;; but occasionally fail.  They fail pretty reliably on the
               ;; build farm.
@@ -28845,7 +28845,7 @@ (define-public python-dask
      (list
       ;; Avoid coverage
       #:test-flags
-      #~(list "-n" "auto"
+      #~(list "--numprocesses" (number->string (parallel-job-count))
               "-m" "not gpu and not slow and not network"
               ;; These all fail with different hashes.  Doesn't seem
               ;; problematic.
-- 
2.46.0

Message #21 received at 74445-done <at> debbugs.gnu.org (full text, mbox):

From: Sharlatan Hellseher <sharlatanus <at> gmail.com>
To: 74445-done <at> debbugs.gnu.org
Subject: [python-team] Respect --cores build argument.
Date: Sat, 23 Nov 2024 14:54:43 +0000

[Message part 1 (text/plain, inline)]

Hi,

Pushed as 2d1a7ec92d91b8346d257e5352b65fdce35070a5 to master.

--
Thanks,
Oleg

[signature.asc (application/pgp-signature, inline)]

Message #24 received at 74445 <at> debbugs.gnu.org (full text, mbox):

From: Sharlatan Hellseher <sharlatanus <at> gmail.com>
To: 74445 <at> debbugs.gnu.org
Subject: [python-team] Respect --cores build argument.
Date: Sat, 23 Nov 2024 16:00:43 +0000

[Message part 1 (text/plain, inline)]

Correct commit: origin/master d3b010c38c449ea17bf7beba48de39acec26ae04

--
Oleg

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.