Subject: bug#33046: pypi importer doesn't print the correct source

From: Maxim Cournoyer <[email-masked]>
Date: Sun, 31 Mar 2019 10:46:18 -0400

Hello!

Julien Lepiller <[email-masked]> writes:

> Hi, I tried to use the importer to refresh python-twisted, and got
> this source:
>
> (uri (pypi-uri "twisted" version))
>
> while the correct one should be
>
> (uri (pypi-uri "Twisted" version ".tar.bz2"))

I believe the attached patch fixes this :-). It's based on the patch set
attached to bug #24450 though, so that one should be reviewed/merged
first.

Here's the result, for a couple packages that were problematic in the
mixed case regards:

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix import pypi Twisted

Starting download of /tmp/guix-file.BdpmRc
>From https://files.pythonhosted.org/packages/5d/0e/a72d85a55761c2c3ff1cb968143a2fd5f360220779ed90e0fadf4106d4f2/Twisted-18.9.0.tar.bz2...
 …9.0.tar.bz2  2.9MiB                 1.5MiB/s 00:02 [##################] 100.0%
(package
  (name "python-twisted")
  (version "18.9.0")
  (source
    (origin
      (method url-fetch)
      (uri (pypi-uri "Twisted" version))
      (sha256
        (base32
          "15d3gmkrg8g27hyd6ihawv2y2dv5gnpyg67wy9npgbl4pz3f4jr9"))))
  (build-system python-build-system)
  (propagated-inputs
    `(("python-attrs" ,python-attrs)
      ("python-automat" ,python-automat)
      ("python-constantly" ,python-constantly)
      ("python-hyperlink" ,python-hyperlink)
      ("python-incremental" ,python-incremental)
      ("python-pyhamcrest" ,python-pyhamcrest)
      ("python-zope.interface" ,python-zope.interface)))
  (native-inputs
    `(("python-pydoctor" ,python-pydoctor)
      ("python-pyflakes" ,python-pyflakes)
      ("python-sphinx" ,python-sphinx)
      ("python-subunit" ,python-subunit)
      ("python-towncrier" ,python-towncrier)
      ("python-twisted-dev-tools"
       ,python-twisted-dev-tools)
      ("python-twistedchecker" ,python-twistedchecker)))
  (home-page "http://twistedmatrix.com/")
  (synopsis
    "An asynchronous networking framework written in Python")
  (description
    "An asynchronous networking framework written in Python")
  (license license:expat))
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix import pypi uWSGI

Starting download of /tmp/guix-file.TNSMx7
>From https://files.pythonhosted.org/packages/e7/1e/3dcca007f974fe4eb369bf1b8629d5e342bb3055e2001b2e5340aaefae7a/uwsgi-2.0.18.tar.gz...
 ….18.tar.gz  783KiB                  751KiB/s 00:01 [##################] 100.0%
guix import: warning: Cannot guess requirements from source archive: no requires.txt file found.
(package
  (name "python-uwsgi")
  (version "2.0.18")
  (source
    (origin
      (method url-fetch)
      (uri (pypi-uri "uwsgi" version))
      (sha256
        (base32
          "10zmk4npknigmbqcq1wmhd461dk93159px172112vyq0i19sqwj9"))))
  (build-system python-build-system)
  (home-page
    "https://uwsgi-docs.readthedocs.io/en/latest/")
  (synopsis "The uWSGI server")
  (description "The uWSGI server")
  (license #f))
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
./pre-inst-env guix import pypi Deprecated

Starting download of /tmp/guix-file.cHmIDG
>From https://files.pythonhosted.org/packages/2c/6a/f7c9d1db90d758a9dc21daaa5660a01b3ab22ca56533f6a62b24c0d113e6/Deprecated-1.2.5.tar.gz...
 …1.2.5.tar.gz  4.8MiB                1.2MiB/s 00:04 [##################] 100.0%

Starting download of /tmp/guix-file.Yj88Cr
>From https://files.pythonhosted.org/packages/9f/7a/003fa432f1e45625626549726c2fbb7a29baa764e9d1fdb2323a5d779f8a/Deprecated-1.2.5-py2.py3-none-any.whl...
 …y2.py3-none-any.whl  8KiB           2.1MiB/s 00:00 [##################] 100.0%
(package
  (name "python-deprecated")
  (version "1.2.5")
  (source
    (origin
      (method url-fetch)
      (uri (pypi-uri "Deprecated" version))
      (sha256
        (base32
          "0mw5hv0bci5lzzi7ip62vm1ivmxhqj7zwc557p7iyjz3xsq3wa9g"))))
  (build-system python-build-system)
  (propagated-inputs
    `(("python-wrapt" ,python-wrapt)))
  (native-inputs
    `(("python-bumpversion" ,python-bumpversion)
      ("python-pytest" ,python-pytest)
      ("python-pytest-cov" ,python-pytest-cov)
      ("python-sphinx" ,python-sphinx)
      ("python-tox" ,python-tox)))
  (home-page
    "https://github.com/tantale/deprecated")
  (synopsis
    "Python @deprecated decorator to deprecate old python classes, functions or methods.")
  (description
    "Python @deprecated decorator to deprecate old python classes, functions or methods.")
  (license license:expat))
--8<---------------cut here---------------end--------------->8---

>From 1290f9d1f0d594fdd4723d76b94116be25da9dd5 Mon Sep 17 00:00:00 2001
From: Maxim Cournoyer <[email-masked]>
Date: Sat, 30 Mar 2019 20:27:35 -0400
Subject: [PATCH] import: pypi: Preserve package name case when forming
 pypi-uri.

Fixes issue: #33046.

* guix/build-system/python.scm (pypi-uri): Update the host URI to
"files.pythonhosted.org".
* guix/import/pypi.scm (make-pypi-sexp): Preserve the package name case when
the source URL calls for it.
---
 guix/build-system/python.scm |  2 +-
 guix/import/pypi.scm         | 23 ++++++++++++++---------
 2 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm
index b753940bad..e39c06528e 100644
--- a/guix/build-system/python.scm
+++ b/guix/build-system/python.scm
@@ -50,7 +50,7 @@
   "Return a URI string for the Python package hosted on the Python Package
 Index (PyPI) corresponding to NAME and VERSION.  EXTENSION is the file name
 extension, such as '.tar.gz'."
-  (string-append "https://pypi.org/packages/source/"
+  (string-append "https://files.pythonhosted.org/packages/source/"
                  (string-take name 1) "/" name "/"
                  name "-" version extension))
 
diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm
index a2ce14b192..fecf95d0a7 100644
--- a/guix/import/pypi.scm
+++ b/guix/import/pypi.scm
@@ -371,15 +371,20 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
                 `(package
                    (name ,(python->package-name name))
                    (version ,version)
-                   (source (origin
-                             (method url-fetch)
-                             ;; Sometimes 'pypi-uri' doesn't quite work due to mixed
-                             ;; cases in NAME, for instance, as is the case with
-                             ;; "uwsgi".  In that case, fall back to a full URL.
-                             (uri (pypi-uri ,(string-downcase name) version))
-                             (sha256
-                              (base32
-                               ,(guix-hash-url temp)))))
+                   (source
+                    (origin
+                      (method url-fetch)
+                      ;; PyPI URL are case sensitive, but sometimes a project
+                      ;; named using mixed case has a URL using lower case, so
+                      ;; we must work around this inconsistency.  For actual
+                      ;; examples, compare the URLs of the "Deprecated" and
+                      ;; "uWSGI" PyPI packages.
+                      (uri ,(if (string-contains source-url name)
+                                `(pypi-uri ,name version)
+                                `(pypi-uri ,(string-downcase name) version)))
+                      (sha256
+                       (base32
+                        ,(guix-hash-url temp)))))
                    (build-system python-build-system)
                    ,@(maybe-inputs required-inputs 'propagated-inputs)
                    ,@(maybe-inputs test-inputs 'native-inputs)
-- 
2.20.1

Maxim