GNU bug report logs - #32484
Add Numba and Scanpy

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 32484 in the body.
You can then email your comments to 32484 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
To: <guix-patches <at> gnu.org>
Subject: Add Numba and Scanpy
Date: Mon, 20 Aug 2018 17:59:46 +0200

This patch series adds Scanpy, a toolkit for analyzing single-cell gene
expression data.  It depends on Numba, which requires llvmlite.
Unfortunately, llvmlite does not work with our regular llvm package.  It
needs a patched LLVM.

--
Ricardo

Message #8 received at 32484 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
To: <32484 <at> debbugs.gnu.org>
Cc: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
Subject: [PATCH 1/4] gnu: Add python-llvmlite.
Date: Mon, 20 Aug 2018 20:00:16 +0200

* gnu/packages/llvm.scm (python-llvmlite): New variable.
---
 gnu/packages/llvm.scm | 53 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 52 insertions(+), 1 deletion(-)

diff --git a/gnu/packages/llvm.scm b/gnu/packages/llvm.scm
index 98592ad09..e9b7065ab 100644
--- a/gnu/packages/llvm.scm
+++ b/gnu/packages/llvm.scm
@@ -3,7 +3,7 @@
 ;;; Copyright © 2015 Mark H Weaver <mhw <at> netris.org>
 ;;; Copyright © 2015, 2017, 2018 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2016 Dennis Mungai <dmngaie <at> gmail.com>
-;;; Copyright © 2016 Ricardo Wurmus <rekado <at> elephly.net>
+;;; Copyright © 2016, 2018 Ricardo Wurmus <rekado <at> elephly.net>
 ;;; Copyright © 2017 Roel Janssen <roel <at> gnu.org>
 ;;; Copyright © 2018 Marius Bakke <mbakke <at> fastmail.com>
 ;;;
@@ -29,6 +29,7 @@
   #:use-module (guix utils)
   #:use-module (guix build-system gnu)
   #:use-module (guix build-system cmake)
+  #:use-module (guix build-system python)
   #:use-module (gnu packages)
   #:use-module (gnu packages gcc)
   #:use-module (gnu packages bootstrap)           ;glibc-dynamic-linker
@@ -379,3 +380,53 @@ code analysis tools.")
        (patches (list (search-patch "llvm-for-extempore.patch")))))
     ;; Extempore refuses to build on architectures other than x86_64
     (supported-systems '("x86_64-linux"))))
+
+(define-public python-llvmlite
+  (package
+    (name "python-llvmlite")
+    (version "0.24.0")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "llvmlite" version))
+       (sha256
+        (base32
+         "01zwjlc3c5mhrwmv4b73zgbskwqps9ly0nrh54bbj1f1l72f839j"))))
+    (build-system python-build-system)
+    (inputs
+     `(("llvm"
+        ,(package
+           (inherit llvm)
+           (source (origin
+                     (inherit (package-source llvm))
+                     (patches
+                      (list
+                       (origin
+                         (method url-fetch)
+                         (uri (string-append "https://raw.githubusercontent.com/numba/"
+                                             "llvmlite/v" version "/conda-recipes/"
+                                             "D47188-svml.patch"))
+                         (sha256
+                          (base32
+                           "0mrj24jvkv3hjcmyg98zmvmyl1znlh2j63rdr69f6g7s96d2pfv1")))
+                       (origin
+                         (method url-fetch)
+                         (uri (string-append "https://raw.githubusercontent.com/numba/"
+                                             "llvmlite/v" version "/conda-recipes/"
+                                             "twine_cfg_undefined_behavior.patch"))
+                         (sha256
+                          (base32
+                           "07h71n2m1mn9zcfgw04zglffknplb233zqbcd6pckq0wygkrxflp")))
+                       (origin
+                         (method url-fetch)
+                         (uri (string-append "https://raw.githubusercontent.com/numba/"
+                                             "llvmlite/v" version "/conda-recipes/"
+                                             "0001-Transforms-Add-missing-header-for-InstructionCombini.patch"))
+                         (sha256
+                          (base32
+                           "1pp0z9696l6j4dwz7ypjrm4vvkj0d3mlf1g8zmiyk08akw5lz0cb")))))))))))
+    (home-page "http://llvmlite.pydata.org")
+    (synopsis "Wrapper around basic LLVM functionality")
+    (description
+     "This package provides a Python binding to LLVM for use in Numba.")
+    (license license:bsd-3)))
-- 
2.18.0

Message #11 received at 32484 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
To: <32484 <at> debbugs.gnu.org>
Cc: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
Subject: [PATCH 2/4] gnu: Add python-numba.
Date: Mon, 20 Aug 2018 20:00:17 +0200

* gnu/packages/python.scm (python-numba): New variable.
---
 gnu/packages/python.scm | 84 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 84 insertions(+)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 3db29e23f..7f343bdab 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -99,6 +99,7 @@
   #:use-module (gnu packages libevent)
   #:use-module (gnu packages libffi)
   #:use-module (gnu packages linux)
+  #:use-module (gnu packages llvm)
   #:use-module (gnu packages machine-learning)
   #:use-module (gnu packages man)
   #:use-module (gnu packages maths)
@@ -13889,3 +13890,86 @@ Let's Encrypt.")
 development that supports command line argument parsing, command string
 validation testing and application logic.")
     (license license:expat)))
+
+;; Make sure to upgrade python-llvmlite in (gnu packages llvm) together with
+;; python-numba.  They have a very unflexible relationship.
+(define-public python-numba
+  (package
+    (name "python-numba")
+    (version "0.39.0")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "numba" version))
+       (sha256
+        (base32
+         "1bibvkwga1v8293i9ivl469d8bzgabn3vgr2ig7c1i68v8frsx07"))))
+    (build-system python-build-system)
+    (arguments
+     `(#:modules ((guix build utils)
+                  (guix build python-build-system)
+                  (ice-9 ftw)
+                  (srfi srfi-1)
+                  (srfi srfi-26))
+       #:phases
+       (modify-phases %standard-phases
+         (add-after 'unpack 'disable-proprietary-features
+           (lambda _
+             (setenv "NUMBA_DISABLE_HSA" "1")
+             (setenv "NUMBA_DISABLE_CUDA" "1")
+             #t))
+         (add-after 'unpack 'remove-failing-tests
+           (lambda _
+             ;; FIXME: these tests fail for unknown reasons:
+             ;; test_non_writable_pycache, test_non_creatable_pycache, and
+             ;; test_frozen (all in numba.tests.test_dispatcher.TestCache).
+             (substitute* "numba/tests/test_dispatcher.py"
+               (("def test(_non_writable_pycache)" _ m)
+                (string-append "def guix_skip" m))
+               (("def test(_non_creatable_pycache)" _ m)
+                (string-append "def guix_skip" m))
+               (("def test(_frozen)" _ m)
+                (string-append "def guix_skip" m)))
+
+             ;; These tests fail because we don't run the tests from the build
+             ;; directory: test_setup_py_distutils, test_setup_py_setuptools
+             ;; They ar in numba.tests.test_pycc.TestDistutilsSupport.
+             (substitute* "numba/tests/test_pycc.py"
+               (("def test(_setup_py_distutils|_setup_py_setuptools)" _ m)
+                (string-append "def guix_skip" m)))
+             #t))
+         (replace 'check
+           (lambda _
+             (let ((cwd (getcwd)))
+               (setenv "PYTHONPATH"
+                       (string-append cwd "/build/"
+                                      (find (cut string-prefix? "lib" <>)
+                                            (scandir (string-append cwd "/build")))
+                                      ":"
+                                      (getenv "PYTHONPATH")))
+               ;; Something is wrong with the PYTHONPATH when running the
+               ;; tests from the build directory, as it complains about not being
+               ;; able to import certain modules.
+               (with-directory-excursion "/tmp"
+                 (invoke "python3" "-m" "numba.runtests" "-v" "-m")))
+             #t)))))
+    (propagated-inputs
+     `(("python-llvmlite" ,python-llvmlite)
+       ("python-numpy" ,python-numpy)
+       ("python-singledispatch" ,python-singledispatch)))
+    ;; Needed for tests.
+    (inputs
+     `(("python-jinja2" ,python-jinja2)
+       ("python-pygments" ,python-pygments)))
+    (home-page "https://numba.pydata.org")
+    (synopsis "Compile Python code using LLVM")
+    (description "Numba gives you the power to speed up your applications with
+high performance functions written directly in Python.  With a few
+annotations, array-oriented and math-heavy Python code can be just-in-time
+compiled to native machine instructions, similar in performance to C, C++ and
+Fortran, without having to switch languages or Python interpreters.
+
+Numba works by generating optimized machine code using the LLVM compiler
+infrastructure at import time, runtime, or statically (using the included pycc
+tool).")
+    (license license:bsd-3)))
-- 
2.18.0

Message #14 received at 32484 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
To: <32484 <at> debbugs.gnu.org>
Cc: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
Subject: [PATCH 3/4] gnu: Add python-anndata.
Date: Mon, 20 Aug 2018 20:00:18 +0200

* gnu/packages/python.scm (python-anndata): New variable.
---
 gnu/packages/python.scm | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 7f343bdab..2838d2ae5 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -13973,3 +13973,29 @@ Numba works by generating optimized machine code using the LLVM compiler
 infrastructure at import time, runtime, or statically (using the included pycc
 tool).")
     (license license:bsd-3)))
+
+(define-public python-anndata
+  (package
+    (name "python-anndata")
+    (version "0.6.9")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "anndata" version))
+       (sha256
+        (base32
+         "1fh461xyyc7pcrjfgd013bdc2alf53r46ss3gfw3431mbb1gappi"))))
+    (build-system python-build-system)
+    (propagated-inputs
+     `(("python-h5py" ,python-h5py)
+       ("python-natsort" ,python-natsort)
+       ("python-pandas" ,python-pandas)
+       ("python-scipy" ,python-scipy)))
+    (home-page "https://github.com/theislab/anndata")
+    (synopsis "Annotated data for data analysis pipelines")
+    (description "Anndata is a package for simple (functional) high-level APIs
+for data analysis pipelines.  In this context, it provides an efficient,
+scalable way of keeping track of data together with learned annotations and
+reduces the code overhead typically encountered when using a mostly
+object-oriented library such as @code{scikit-learn}.")
+    (license license:bsd-3)))
-- 
2.18.0

Message #17 received at 32484 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
To: <32484 <at> debbugs.gnu.org>
Cc: Ricardo Wurmus <ricardo.wurmus <at> mdc-berlin.de>
Subject: [PATCH 4/4] gnu: Add python-scanpy.
Date: Mon, 20 Aug 2018 20:00:19 +0200

* gnu/packages/bioinformatics.scm (python-scanpy): New variable.
---
 gnu/packages/bioinformatics.scm | 36 +++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/gnu/packages/bioinformatics.scm b/gnu/packages/bioinformatics.scm
index dce8ab62f..ef9d81435 100644
--- a/gnu/packages/bioinformatics.scm
+++ b/gnu/packages/bioinformatics.scm
@@ -13406,3 +13406,39 @@ conversions, region filtering, FASTA sequence extraction and more.")
       ;; gffread is under Expat, but gclib is under Artistic 2.0
       (license (list license:expat
                      license:artistic2.0)))))
+
+(define-public python-scanpy
+  (package
+    (name "python-scanpy")
+    (version "1.2.2")
+    (source
+     (origin
+       (method url-fetch)
+       (uri (pypi-uri "scanpy" version))
+       (sha256
+        (base32
+         "1ak7bxms5a0yvf65prppq2g38clkv7c7jnjbnfpkh3xxv7q512jz"))))
+    (build-system python-build-system)
+    (propagated-inputs
+     `(("python-anndata" ,python-anndata)
+       ("python-igraph" ,python-igraph)
+       ("python-numba" ,python-numba)
+       ("python-joblib" ,python-joblib)
+       ("python-natsort" ,python-natsort)
+       ("python-networkx" ,python-networkx)
+       ("python-statsmodels" ,python-statsmodels)
+       ("python-scikit-learn" ,python-scikit-learn)
+       ("python-matplotlib" ,python-matplotlib)
+       ("python-pandas" ,python-pandas)
+       ("python-scipy" ,python-scipy)
+       ("python-seaborn" ,python-seaborn)
+       ("python-h5py" ,python-h5py)
+       ("python-tables" ,python-tables)))
+    (home-page "http://github.com/theislab/scanpy")
+    (synopsis "Single-Cell Analysis in Python.")
+    (description "Scanpy is a scalable toolkit for analyzing single-cell gene
+expression data.  It includes preprocessing, visualization, clustering,
+pseudotime and trajectory inference and differential expression testing.  The
+Python-based implementation efficiently deals with datasets of more than one
+million cells.")
+    (license license:bsd-3)))
-- 
2.18.0

Message #22 received at 32484-done <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: 32484-done <at> debbugs.gnu.org
Subject: Re: [PATCH 4/4] gnu: Add python-scanpy.
Date: Wed, 29 Aug 2018 17:09:31 +0200

> * gnu/packages/bioinformatics.scm (python-scanpy): New variable.

I pushed this (and the other three patches) to the master branch with
commit c7fe888b4.

--
Ricardo

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.