From debbugs-submit-bounces@debbugs.gnu.org Thu Jan 20 14:45:47 2022 Received: (at submit) by debbugs.gnu.org; 20 Jan 2022 19:45:47 +0000 Received: from localhost ([127.0.0.1]:59206 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nAdNS-0006c8-Qg for submit@debbugs.gnu.org; Thu, 20 Jan 2022 14:45:47 -0500 Received: from lists.gnu.org ([209.51.188.17]:58730) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nAdNR-0006c1-1C for submit@debbugs.gnu.org; Thu, 20 Jan 2022 14:45:45 -0500 Received: from eggs.gnu.org ([209.51.188.92]:39038) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nAdNO-0002KE-QB for guix-patches@gnu.org; Thu, 20 Jan 2022 14:45:44 -0500 Received: from [2a00:5881:4008:2810::309] (port=43526 helo=planete-kraus.eu) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_CHACHA20_POLY1305:256) (Exim 4.90_1) (envelope-from ) id 1nAdNM-0000Y8-7l for guix-patches@gnu.org; Thu, 20 Jan 2022 14:45:42 -0500 Received: from planete-kraus.eu (localhost.lan [127.0.0.1]) by planete-kraus.eu (OpenSMTPD) with ESMTP id 9c5185dc for ; Thu, 20 Jan 2022 19:45:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=planete-kraus.eu; h=from :to:subject:date:message-id:mime-version:content-type; s=dkim; bh=kl4Tk443jZSn3aAuTD8jcEDLnSg=; b=GW0YQWwG8wpmULEDvFLfr4Jt2vv/ XsJWegGOQB8VpGV6MJWNjATvLM22aTDUfpcheXOjKpfcNBpWFvN1rB53JvIPktDK o6Kmcy5fJK4yw4o+//P6FjCQGuyqyZV+xhfmHk5R2QWkKJWt5W8bIlbxcYkNQCn5 +bJi3LYKn+Ys1Cg= Received: by planete-kraus.eu (OpenSMTPD) with ESMTPSA id 60877e26 (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO); Thu, 20 Jan 2022 19:45:26 +0000 (UTC) From: Vivien Kraus To: guix-patches@gnu.org Subject: Fix pypi import for flake8-array-spacing Date: Thu, 20 Jan 2022 20:45:19 +0100 Message-ID: <87czkmi434.fsf@planete-kraus.eu> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Host-Lookup-Failed: Reverse DNS lookup failed for 2a00:5881:4008:2810::309 (failed) Received-SPF: pass client-ip=2a00:5881:4008:2810::309; envelope-from=vivien@planete-kraus.eu; helo=planete-kraus.eu X-Spam_score_int: -12 X-Spam_score: -1.3 X-Spam_bar: - X-Spam_report: (-1.3 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RDNS_NONE=0.793, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.4 (--) --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain Dear guix, flake8-array-spacing is a funny package because its download URI uses a base name of flake8_array_spacing. Apparently some other packages have similar features, appearing as the downcase version. What do you think? Best regards, Vivien --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-pypi-importer-Convert-to-_-in-pypi-urls-if-needed.patch Content-Transfer-Encoding: quoted-printable Content-Description: Fix the pypi importer From=20d8923c394fbe2e8eedf6fa548455d398f0caa022 Mon Sep 17 00:00:00 2001 From: Vivien Kraus Date: Thu, 20 Jan 2022 20:11:56 +0100 Subject: [PATCH] pypi importer: Convert - to _ in pypi urls if needed. * guix/import/pypi.scm (find-project-url): New function. (make-pypi-sexp): Use find-project-url. =2D-- guix/import/pypi.scm | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index b4284f5c33..fd176e65d5 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -418,6 +418,19 @@ (define process-requirements (values (map process-requirements dependencies) (concatenate dependencies)))) =20 +(define (find-project-url name pypi-url) + "Try different project name substitution until the result is found in +pypi-url. Downcase is required for \"Deprecated\" and \"uWSGI\", and +underscores are required for flake8-array-spacing." + (or (find (cut string-contains pypi-url <>) + (list name + (string-downcase name) + (string-replace-substring name "-" "_"))) + (begin + (warning (G_ "The project name `~a' does not appear in the pypi UR= L; you might need to fix the pypi-url declaration in the generated package.= The URL is: ~a~%") + name pypi-url) + name))) + (define (make-pypi-sexp name version source-url wheel-url home-page synops= is description license) "Return the `package' s-expression for a python package with the given N= AME, @@ -446,15 +459,7 @@ (define (maybe-upstream-name name) (origin (method url-fetch) (uri (pypi-uri =2D ;; PyPI URL are case sensitive, but sometim= es =2D ;; a project named using mixed case has a U= RL =2D ;; using lower case, so we must work around= this =2D ;; inconsistency. For actual examples, com= pare =2D ;; the URLs of the "Deprecated" and "uWSGI"= PyPI =2D ;; packages. =2D ,(if (string-contains source-url name) =2D name =2D (string-downcase name)) + ,(find-project-url name source-url) version ;; Some packages have been released as `.zip` ;; instead of the more common `.tar.gz`. For =2D-=20 2.34.0 --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAEBCAAdFiEEq4yIHjMvkliPpwQnO7C8EjLYuCwFAmHpu88ACgkQO7C8EjLY uCyfzwwAmJfEk+jF6rgzGBgUrYeDoNzc0nPoDCysNlOnQJ/NlgDRJhQ76FVgcaom /gYf9iToOqevo5+qsrw0Qldp2nzyREVf63WmU4fpANhM1e7OUxwkKtBJItEiqR9t ymva2WMkxL3/6BxIFJZiLkHK1T+odxtT2C391EJoxS92NlVHcWkFJr1T0PkrxTMo Iqdd8/JbO/+MEXunHtFnPkXfIqfhrvhP/QzBLj5N6RHgeL5mLAdQc5/sxr8B0NO1 4u2mkCaJoOgzGVttti4ZStYR+P8EwqXwYqosk9PK/FdC2iLEKwIhSGTRycyX6FRM vsqAtgjn+FoefNmqFoRWdB7/I3aEK18s4FWBQAZ88J6gunX+OmJNR0+YvlXzmBOr LFw2nUtw15t6YXhZdD02/gUGRjx4XZM+DeAadFKdMmSogdedCrMmaA2//s/1OieJ zBD3MEr69xOQ4nA/En6oMG9qECMlox1PTSj4BF+HTuKql6ZECA9diPDcUr/RCZRe 12FS3I/N =/Fqc -----END PGP SIGNATURE----- --==-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 24 09:05:57 2022 Received: (at 53395) by debbugs.gnu.org; 24 Jan 2022 14:05:57 +0000 Received: from localhost ([127.0.0.1]:42623 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nBzyn-0003Jn-1V for submit@debbugs.gnu.org; Mon, 24 Jan 2022 09:05:57 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:48788) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nBzyk-0003JU-1E for 53395@debbugs.gnu.org; Mon, 24 Jan 2022 09:05:55 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 5291B2D9; Mon, 24 Jan 2022 15:05:48 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id shBBr2w2YQyb; Mon, 24 Jan 2022 15:05:48 +0100 (CET) Received: from ribbon (91-160-117-201.subs.proxad.net [91.160.117.201]) by hera.aquilenet.fr (Postfix) with ESMTPSA id AC9068B; Mon, 24 Jan 2022 15:05:47 +0100 (CET) From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Vivien Kraus Subject: Re: bug#53395: Fix pypi import for flake8-array-spacing References: <87czkmi434.fsf@planete-kraus.eu> Date: Mon, 24 Jan 2022 15:05:47 +0100 In-Reply-To: <87czkmi434.fsf@planete-kraus.eu> (Vivien Kraus's message of "Thu, 20 Jan 2022 20:45:19 +0100") Message-ID: <87ee4xut38.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: / Authentication-Results: hera.aquilenet.fr; none X-Rspamd-Server: hera X-Rspamd-Queue-Id: 5291B2D9 X-Spamd-Result: default: False [-0.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 53395 Cc: 53395@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Hello! Vivien Kraus skribis: > From d8923c394fbe2e8eedf6fa548455d398f0caa022 Mon Sep 17 00:00:00 2001 > From: Vivien Kraus > Date: Thu, 20 Jan 2022 20:11:56 +0100 > Subject: [PATCH] pypi importer: Convert - to _ in pypi urls if needed. > > * guix/import/pypi.scm (find-project-url): New function. > (make-pypi-sexp): Use find-project-url. [...] > +(define (find-project-url name pypi-url) > + "Try different project name substitution until the result is found in > +pypi-url. Downcase is required for \"Deprecated\" and \"uWSGI\", and > +underscores are required for flake8-array-spacing." > + (or (find (cut string-contains pypi-url <>) > + (list name > + (string-downcase name) > + (string-replace-substring name "-" "_"))) > + (begin > + (warning (G_ "The project name `~a' does not appear in the pypi = URL; you might need to fix the pypi-url declaration in the generated packag= e. The URL is: ~a~%") > + name pypi-url) > + name))) As a rule of thumb, warnings are one-line messages (not sentences) describing the problem. Here you could emit such a warning, followed by a call to =E2=80=98display-hint=E2=80=99, which accepts Texinfo markup. Could you adjust accordingly? Also, what about adding a unit test? Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 24 17:23:11 2022 Received: (at 53395) by debbugs.gnu.org; 24 Jan 2022 22:23:12 +0000 Received: from localhost ([127.0.0.1]:46127 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nC7jz-0000be-8R for submit@debbugs.gnu.org; Mon, 24 Jan 2022 17:23:11 -0500 Received: from planete-kraus.eu ([89.234.140.182]:39032) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nC7jv-0000bP-U5 for 53395@debbugs.gnu.org; Mon, 24 Jan 2022 17:23:09 -0500 Received: from planete-kraus.eu (localhost.lan [127.0.0.1]) by planete-kraus.eu (OpenSMTPD) with ESMTP id c151ced7; Mon, 24 Jan 2022 22:22:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=planete-kraus.eu; h= references:from:to:cc:subject:date:in-reply-to:message-id :mime-version:content-type; s=dkim; bh=GIZo3yaCh6J3SWbjr9kmp3AT8 80=; b=iU1fbLVtusIXgSpBuLdXBAsOZzTO6EBJU707r/vTmIYRya53lybjENIPO kA8sqQl1Qxwsgz0GQM3WEJzA7BIM+YKlYOYMOqATQ52lU99v6k994UARnh4lL6OS AFDQ/7EF4G3yo+7nreDjmPU22QORSbdK0p4dsdxv4rk67pAebg= Received: by planete-kraus.eu (OpenSMTPD) with ESMTPSA id 10614eca (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO); Mon, 24 Jan 2022 22:21:51 +0000 (UTC) References: <87czkmi434.fsf@planete-kraus.eu> <87ee4xut38.fsf@gnu.org> User-agent: mu4e 1.6.10; emacs 27.2 From: Vivien Kraus To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#53395: Fix pypi import for flake8-array-spacing Date: Mon, 24 Jan 2022 23:07:46 +0100 In-reply-to: <87ee4xut38.fsf@gnu.org> Message-ID: <87mtjk22rn.fsf@planete-kraus.eu> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 53395 Cc: 53395@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi :) Ludovic Court=C3=A8s writes: > As a rule of thumb, warnings are one-line messages (not sentences) > describing the problem. Here you could emit such a warning, followed by > a call to =E2=80=98display-hint=E2=80=99, which accepts Texinfo markup. display-hint seems to always add \n\n at the end of the message, is it intended that way? Should I do one big display-hint instead of 3? > Also, what about adding a unit test? I added two by copying the "foo" example and pretending it=E2=80=99s named = "Foo" and "goo" respectively (fixing the package name in the json data), while keeping release URLs as /foo-=E2=80=A6 so the importer should in the first = case recognize "Foo" in the URL and in the second case it should not recognize "goo" in the URL so it should emit a warning. I didn=E2=80=99t te= st the dash-to-underscore transformation, because it would require test data with a package named "foo-something" but with release URIs containing "foo_something". That data point does not exist currently in the test module. Vivien --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-pypi-importer-Convert-to-_-in-pypi-urls-if-needed.patch Content-Transfer-Encoding: quoted-printable Content-Description: Better handling of rogue pypi packages From=203e7cfd8b2c3a76ab8aac82a7d2f2a9f3e1a5139d Mon Sep 17 00:00:00 2001 From: Vivien Kraus Date: Thu, 20 Jan 2022 20:11:56 +0100 Subject: [PATCH] pypi importer: Convert - to _ in pypi urls if needed. * guix/import/pypi.scm (find-project-url): New function. (make-pypi-sexp): Use find-project-url. * tests/pypi.scm ("Package Foo has a correct pypi-uri of foo"): New test. ("If the package goo is released as foo, the importer warns"): New test. =2D-- guix/import/pypi.scm | 30 ++++++--- tests/pypi.scm | 150 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 171 insertions(+), 9 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index b4284f5c33..3839c37a95 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -41,6 +41,7 @@ (define-module (guix import pypi) #:use-module (guix memoization) #:use-module (guix diagnostics) #:use-module (guix i18n) + #:use-module ((guix ui) #:select (display-hint)) #:use-module ((guix build utils) #:select ((package-name->name+version . hyphen-package-name->name+version) @@ -418,6 +419,25 @@ (define process-requirements (values (map process-requirements dependencies) (concatenate dependencies)))) =20 +(define (find-project-url name pypi-url) + "Try different project name substitution until the result is found in +pypi-uri. Downcase is required for \"uWSGI\", and +underscores are required for flake8-array-spacing." + (or (find (cut string-contains pypi-url <>) + (list name + (string-downcase name) + (string-replace-substring name "-" "_"))) + (begin + (warning + (G_ "The project name does not appear verbatim in the pypi URI~%"= )) + (display-hint + (format #f (G_ "The project name is: @strong{~a}") name)) + (display-hint + (format #f (G_ "The pypi URI is: @strong{@url{~a}}") pypi-url)) + (display-hint + (G_ "The pypi-uri declaration in the generated package might need= to be changed.")) + name))) + (define (make-pypi-sexp name version source-url wheel-url home-page synops= is description license) "Return the `package' s-expression for a python package with the given N= AME, @@ -446,15 +466,7 @@ (define (maybe-upstream-name name) (origin (method url-fetch) (uri (pypi-uri =2D ;; PyPI URL are case sensitive, but sometim= es =2D ;; a project named using mixed case has a U= RL =2D ;; using lower case, so we must work around= this =2D ;; inconsistency. For actual examples, com= pare =2D ;; the URLs of the "Deprecated" and "uWSGI"= PyPI =2D ;; packages. =2D ,(if (string-contains source-url name) =2D name =2D (string-downcase name)) + ,(find-project-url name source-url) version ;; Some packages have been released as `.zip` ;; instead of the more common `.tar.gz`. For diff --git a/tests/pypi.scm b/tests/pypi.scm index 1ea5f02643..b0aff2f0a1 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -23,10 +23,12 @@ (define-module (test-pypi) #:use-module (guix import pypi) #:use-module (guix base32) #:use-module (guix memoization) + #:use-module (guix utils) #:use-module (gcrypt hash) #:use-module (guix tests) #:use-module (guix build-system python) #:use-module ((guix build utils) #:select (delete-file-recursively which= mkdir-p)) + #:use-module ((guix diagnostics) #:select (guix-warning-port)) #:use-module (srfi srfi-64) #:use-module (ice-9 match)) =20 @@ -428,4 +430,152 @@ (define test-metadata-with-extras-jedi "\ (x (pk 'fail x #f)))))) =20 +(test-equal "Package Foo has a correct pypi-uri of foo" + "" + (call-with-output-string + (lambda (port) + (parameterize ((guix-warning-port port)) + ;; Replace network resources with sample data. + (mock ((guix import utils) url-fetch + (lambda (url file-name) + (match url + ("https://example.com/foo-1.0.0.tar.gz" + (begin + (mkdir-p "foo-1.0.0/foo.egg-info") + (with-output-to-file "foo-1.0.0/foo.egg-info/require= s.txt" + (lambda () + (display test-requires.txt))) + (parameterize ((current-output-port (%make-void-port= "rw+"))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) + (delete-file-recursively "foo-1.0.0"))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" + (begin + (mkdir "foo-1.0.0.dist-info") + (with-output-to-file "foo-1.0.0.dist-info/METADATA" + (lambda () + (display test-metadata))) + (let ((zip-file (string-append file-name ".zip"))) + ;; zip always adds a "zip" extension to the file i= t creates, + ;; so we need to rename it. + (system* "zip" "-q" zip-file "foo-1.0.0.dist-info/= METADATA") + (rename-file zip-file file-name)) + (delete-file-recursively "foo-1.0.0.dist-info"))) + (_ (error "Unexpected URL: " url))))) + (mock ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ;; Note the Foo capitalization in the project URL + ("https://pypi.org/pypi/Foo/json" + ;; The document advertises the package name as "= Foo" + ;; too + (let ((name-fixed + (string-replace-substring + test-json-1 + "\"name\": \"foo\"," + "\"name\": \"Foo\","))) + (values (open-input-string name-fixed) + (string-length name-fixed)))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.= whl" #f) + (_ (error "Unexpected URL: " url))))) + (match (pypi->guix-package "Foo") + (('package + ('name "python-foo") ;; Guix wants downcase packa= ge + ;; names + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri ('pypi-uri "foo" 'version)) + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'python-build-system) + ('propagated-inputs ('list 'python-bar 'python-ba= z)) + ('native-inputs ('list 'python-pytest)) + ('home-page "http://example.com") + ('synopsis "summary") + ('description "summary") + ('license 'license:lgpl2.0)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (error "Failed: ~S" x))))))))) + +(test-equal "If the package goo is released as foo, the importer warns" + "warning: The project name does not appear verbatim in the pypi URI +hint: The project name is: *goo* + +hint: The pypi URI is: *`https://example.com/foo-1.0.0.tar.gz'* + +hint: The pypi-uri declaration in the generated package might need to be c= hanged. + +" + (call-with-output-string + (lambda (port) + (parameterize ((guix-warning-port port) + (current-error-port port)) + ;; Replace network resources with sample data. + (mock ((guix import utils) url-fetch + (lambda (url file-name) + (match url + ("https://example.com/foo-1.0.0.tar.gz" + (begin + (mkdir-p "foo-1.0.0/foo.egg-info") + (with-output-to-file "foo-1.0.0/foo.egg-info/require= s.txt" + (lambda () + (display test-requires.txt))) + (parameterize ((current-output-port (%make-void-port= "rw+"))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) + (delete-file-recursively "foo-1.0.0"))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" + (begin + (mkdir "foo-1.0.0.dist-info") + (with-output-to-file "foo-1.0.0.dist-info/METADATA" + (lambda () + (display test-metadata))) + (let ((zip-file (string-append file-name ".zip"))) + ;; zip always adds a "zip" extension to the file i= t creates, + ;; so we need to rename it. + (system* "zip" "-q" zip-file "foo-1.0.0.dist-info/= METADATA") + (rename-file zip-file file-name)) + (delete-file-recursively "foo-1.0.0.dist-info"))) + (_ (error "Unexpected URL: " url))))) + (mock ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ("https://pypi.org/pypi/goo/json" + (let ((name-fixed ;; The true name of the packag= e is + ;; "goo", only the releases are + ;; named foo- + (string-replace-substring + test-json-1 + "\"name\": \"foo\"," + "\"name\": \"goo\","))) + (values (open-input-string name-fixed) + (string-length name-fixed)))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.= whl" #f) + (_ (error "Unexpected URL: " url))))) + (match (pypi->guix-package "goo") + (('package + ('name "python-goo") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri ('pypi-uri "goo" 'version)) + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'python-build-system) + ('propagated-inputs ('list 'python-bar 'python-ba= z)) + ('native-inputs ('list 'python-pytest)) + ('home-page "http://example.com") + ('synopsis "summary") + ('description "summary") + ('license 'license:lgpl2.0)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (error "Failed: ~S" x))))))))) + (test-end "pypi") =2D-=20 2.34.0 --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAEBCAAdFiEEq4yIHjMvkliPpwQnO7C8EjLYuCwFAmHvJnwACgkQO7C8EjLY uCwX8wv/eXkRyyEARa9SyDmi0kdRLydZT8JAL/O0n9t+mVNwK66fU2vZRJfDWT/o UzPnQx1qijpLa8+/6HU+y9/3k0VfOrnPg77aNnNhkGC4dn0vm9UV+1w8GP0ejqX2 jXabOX4hWGV7l3mZ/OXEPpDnVW+E8FGJ6bwHdqqdvSuEbjpeFNxJzabatjlvy8Nh mH0m8+J6giJcE50OvGLmVJ0inGqOCVDPPijt3M9ka1s8NH7Qrk5o1m8OYMzYyaI+ g4XGINy+CVurObmurcHyEzPuQDOpHCZUyUIMCR53uqW4V7/4/9UQrvA8/HxNjYBk jPBwxzAZO3V80YOlvDGdyv87NhRUmU8c7u5NX4ee7qXnDhUt9Ifo4DRNBx1P6rQV ebe0mejekujHZZXHZofBfdHP1nsnervtaqhEAWwI6rD/2NNuaASxAtX23TMTNlOn 438HHYRuHje5en1ECL1dOp3ElcTJYdkOf19fUTgtJPF0Raa/lr3vv8YeG0DNPTCl /tUqFBrp =m+G8 -----END PGP SIGNATURE----- --==-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 25 08:19:16 2022 Received: (at 53395) by debbugs.gnu.org; 25 Jan 2022 13:19:16 +0000 Received: from localhost ([127.0.0.1]:47397 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCLjA-0000W0-DT for submit@debbugs.gnu.org; Tue, 25 Jan 2022 08:19:16 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:55748) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCLj8-0000Vl-Cf for 53395@debbugs.gnu.org; Tue, 25 Jan 2022 08:19:15 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id D49281E8; Tue, 25 Jan 2022 14:19:07 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id a83Gxs0VoyoV; Tue, 25 Jan 2022 14:19:06 +0100 (CET) Received: from ribbon (91-160-117-201.subs.proxad.net [91.160.117.201]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 5373F9D; Tue, 25 Jan 2022 14:19:06 +0100 (CET) From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Vivien Kraus Subject: Re: bug#53395: Fix pypi import for flake8-array-spacing References: <87czkmi434.fsf@planete-kraus.eu> <87ee4xut38.fsf@gnu.org> <87mtjk22rn.fsf@planete-kraus.eu> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 6 =?utf-8?Q?Pluvi=C3=B4se?= an 230 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Tue, 25 Jan 2022 14:19:05 +0100 In-Reply-To: <87mtjk22rn.fsf@planete-kraus.eu> (Vivien Kraus's message of "Mon, 24 Jan 2022 23:07:46 +0100") Message-ID: <87y234lzqu.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: / Authentication-Results: hera.aquilenet.fr; none X-Rspamd-Server: hera X-Rspamd-Queue-Id: D49281E8 X-Spamd-Result: default: False [-0.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 53395 Cc: 53395@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Hi Vivien, Vivien Kraus skribis: > Ludovic Court=C3=A8s writes: >> As a rule of thumb, warnings are one-line messages (not sentences) >> describing the problem. Here you could emit such a warning, followed by >> a call to =E2=80=98display-hint=E2=80=99, which accepts Texinfo markup. > display-hint seems to always add \n\n at the end of the message, is it > intended that way? Should I do one big display-hint instead of 3? Yes, just one =E2=80=98display-hint=E2=80=99 call, using complete sentences= and paragraphs. The text should provide an explanation and more importantly a hint as to what can done, like =E2=80=9CThe generated package definition = might be wrong; please double-check =E2=80=A6=E2=80=9D. You can avoid @strong though, because it doesn=E2=80=99t do anything useful. > + (warning > + (G_ "The project name does not appear verbatim in the pypi URI~= %")) I=E2=80=99d make it: "project name ~a does not appear verbatim in the PyPI URI~%" >> Also, what about adding a unit test? [...] > +(test-equal "If the package goo is released as foo, the importer warns" > + "warning: The project name does not appear verbatim in the pypi URI > +hint: The project name is: *goo* > + > +hint: The pypi URI is: *`https://example.com/foo-1.0.0.tar.gz'* > + > +hint: The pypi-uri declaration in the generated package might need to be= changed. > + > +" > + (call-with-output-string > + (lambda (port) > + (parameterize ((guix-warning-port port) > + (current-error-port port)) > + ;; Replace network resources with sample data. > + (mock ((guix import utils) url-fetch > + (lambda (url file-name) > + (match url These two tests are really integration tests: they exercise the whole shebang. I was thinking about having a unit test of just =E2=80=98find-project-url=E2=80=99, which is less work (and maintenance :-)= ) and may be good enough. Perhaps you can also change one of the existing python-foo tests to exercise this bit, for instance by making it =E2=80=9CFoo=E2=80=9D. WDYT? Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 25 11:54:46 2022 Received: (at 53395) by debbugs.gnu.org; 25 Jan 2022 16:54:46 +0000 Received: from localhost ([127.0.0.1]:49915 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCP5h-0001Bs-Dv for submit@debbugs.gnu.org; Tue, 25 Jan 2022 11:54:46 -0500 Received: from planete-kraus.eu ([89.234.140.182]:39034) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCP5b-0001Bd-VZ for 53395@debbugs.gnu.org; Tue, 25 Jan 2022 11:54:44 -0500 Received: from planete-kraus.eu (localhost.lan [127.0.0.1]) by planete-kraus.eu (OpenSMTPD) with ESMTP id 6f6fd4c5; Tue, 25 Jan 2022 16:54:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=planete-kraus.eu; h= references:from:to:cc:subject:date:in-reply-to:message-id :mime-version:content-type; s=dkim; bh=xgEEppXEfvv1XHWCxJg/fmXGg 4k=; b=ZcR305VmoT4BAn1krWLKKWKKBgpFhbSXvrVOL8tkgW+YyPkBWdB65eGt2 KGNd1n+jeRnOviZfymNB+eNncFPGYKxL22q5w6Zns6UbiPx8SZyxpn+qMyI8K/a9 bU0dsiiq9PHE7l58G0cgRmWy5kXOSW6Pj0S7DLWbV2xOy/Iog0= Received: by planete-kraus.eu (OpenSMTPD) with ESMTPSA id 8505303b (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO); Tue, 25 Jan 2022 16:54:12 +0000 (UTC) References: <87czkmi434.fsf@planete-kraus.eu> <87ee4xut38.fsf@gnu.org> <87mtjk22rn.fsf@planete-kraus.eu> <87y234lzqu.fsf@gnu.org> User-agent: mu4e 1.6.10; emacs 27.2 From: Vivien Kraus To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#53395: Fix pypi import for flake8-array-spacing Date: Tue, 25 Jan 2022 17:39:54 +0100 In-reply-to: <87y234lzqu.fsf@gnu.org> Message-ID: <87ee4v21u6.fsf@planete-kraus.eu> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 53395 Cc: 53395@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Here we go! Ludovic Court=C3=A8s writes: > Yes, just one =E2=80=98display-hint=E2=80=99 call, using complete sentenc= es and > paragraphs. The text should provide an explanation and more importantly > a hint as to what can done, like =E2=80=9CThe generated package definitio= n might > be wrong; please double-check =E2=80=A6=E2=80=9D. I made a new version for that. > > You can avoid @strong though, because it doesn=E2=80=99t do > anything useful. Okay. > >> + (warning >> + (G_ "The project name does not appear verbatim in the pypi URI= ~%")) > > I=E2=80=99d make it: > > "project name ~a does not appear verbatim in the PyPI URI~%" Changed. > >>> Also, what about adding a unit test? > > [...] > >> +(test-equal "If the package goo is released as foo, the importer warns" >> + "warning: The project name does not appear verbatim in the pypi URI >> +hint: The project name is: *goo* >> + >> +hint: The pypi URI is: *`https://example.com/foo-1.0.0.tar.gz'* >> + >> +hint: The pypi-uri declaration in the generated package might need to b= e changed. >> + >> +" >> + (call-with-output-string >> + (lambda (port) >> + (parameterize ((guix-warning-port port) >> + (current-error-port port)) >> + ;; Replace network resources with sample data. >> + (mock ((guix import utils) url-fetch >> + (lambda (url file-name) >> + (match url > > These two tests are really integration tests: they exercise the whole > shebang. I was thinking about having a unit test of just > =E2=80=98find-project-url=E2=80=99, which is less work (and maintenance := -)) and may be > good enough. Let=E2=80=99s do both! If you don=E2=80=99t want the integration tests, jus= t delete them (now or when maintenance becomes too demanding). > > Perhaps you can also change one of the existing python-foo tests to > exercise this bit, for instance by making it =E2=80=9CFoo=E2=80=9D. I added a test function that can be used by all integration tests. It=E2=80= =99s not perfect (there=E2=80=99s still a lot of code copied from test to test, = and I=E2=80=99m not innocent there too) but I don=E2=80=99t want to change the = other tests beyond trivial things. Vivien --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: attachment; filename=0001-pypi-importer-Convert-to-_-in-pypi-urls-if-needed.patch Content-Transfer-Encoding: quoted-printable Content-Description: Fix an edge case in the pypi importer, and push more tests From=208792367617d288e584a703db38c82c749a067c26 Mon Sep 17 00:00:00 2001 From: Vivien Kraus Date: Thu, 20 Jan 2022 20:11:56 +0100 Subject: [PATCH] pypi importer: Convert - to _ in pypi urls if needed. * guix/import/pypi.scm (find-project-url): New function. (make-pypi-sexp): Use find-project-url. * tests/pypi.scm ("Package Foo has a correct pypi-uri of foo"): New test. ("If the package goo is released as foo, the importer warns"): New test. =2D-- guix/import/pypi.scm | 30 ++-- tests/pypi.scm | 337 ++++++++++++++++++++++++++++++++++++------- 2 files changed, 305 insertions(+), 62 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index b4284f5c33..8245ed6348 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -41,6 +41,7 @@ (define-module (guix import pypi) #:use-module (guix memoization) #:use-module (guix diagnostics) #:use-module (guix i18n) + #:use-module ((guix ui) #:select (display-hint)) #:use-module ((guix build utils) #:select ((package-name->name+version . hyphen-package-name->name+version) @@ -59,6 +60,7 @@ (define-module (guix import pypi) specification->requirement-name guix-package->pypi-name pypi-recursive-import + find-project-url pypi->guix-package %pypi-updater)) =20 @@ -418,6 +420,24 @@ (define process-requirements (values (map process-requirements dependencies) (concatenate dependencies)))) =20 +(define (find-project-url name pypi-url) + "Try different project name substitution until the result is found in +pypi-uri. Downcase is required for \"uWSGI\", and +underscores are required for flake8-array-spacing." + (or (find (cut string-contains pypi-url <>) + (list name + (string-downcase name) + (string-replace-substring name "-" "_"))) + (begin + (warning + (G_ "project name ~a does not appear verbatim in the PyPI URI~%") + name) + (display-hint + (format #f (G_ "The PyPI URI is: @url{~a}. You should review the +pypi-uri declaration in the generated package. You may need to replace ~s = with +a substring of the PyPI URI that identifies the package.") pypi-url name)) +name))) + (define (make-pypi-sexp name version source-url wheel-url home-page synops= is description license) "Return the `package' s-expression for a python package with the given N= AME, @@ -446,15 +466,7 @@ (define (maybe-upstream-name name) (origin (method url-fetch) (uri (pypi-uri =2D ;; PyPI URL are case sensitive, but sometim= es =2D ;; a project named using mixed case has a U= RL =2D ;; using lower case, so we must work around= this =2D ;; inconsistency. For actual examples, com= pare =2D ;; the URLs of the "Deprecated" and "uWSGI"= PyPI =2D ;; packages. =2D ,(if (string-contains source-url name) =2D name =2D (string-downcase name)) + ,(find-project-url name source-url) version ;; Some packages have been released as `.zip` ;; instead of the more common `.tar.gz`. For diff --git a/tests/pypi.scm b/tests/pypi.scm index 1ea5f02643..b1647ac0a7 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -23,68 +23,57 @@ (define-module (test-pypi) #:use-module (guix import pypi) #:use-module (guix base32) #:use-module (guix memoization) + #:use-module (guix utils) #:use-module (gcrypt hash) #:use-module (guix tests) #:use-module (guix build-system python) #:use-module ((guix build utils) #:select (delete-file-recursively which= mkdir-p)) + #:use-module ((guix diagnostics) #:select (guix-warning-port)) + #:use-module (json) + #:use-module (srfi srfi-26) #:use-module (srfi srfi-64) =2D #:use-module (ice-9 match)) + #:use-module (ice-9 match) + #:use-module (ice-9 optargs)) + +(define* (foo-json #:key (name "foo") (name-in-url #f)) + "Create a JSON description of an example pypi package, named @var{name}, +optionally using a different @var{name in its URL}." + (scm->json-string + `((info + . ((version . "1.0.0") + (name . ,name) + (license . "GNU LGPL") + (summary . "summary") + (home_page . "http://example.com") + (classifiers . #()) + (download_url . ""))) + (urls . #()) + (releases + . ((1.0.0 + . #(((url . ,(format #f "https://example.com/~a-1.0.0.egg" + (or name-in-url name))) + (packagetype . "bdist_egg")) + ((url . ,(format #f "https://example.com/~a-1.0.0.tar.gz" + (or name-in-url name))) + (packagetype . "sdist")) + ((url . ,(format #f "https://example.com/~a-1.0.0-py2.py3-no= ne-any.whl" + (or name-in-url name))) + (packagetype . "bdist_wheel"))))))))) =20 (define test-json-1 =2D "{ =2D \"info\": { =2D \"version\": \"1.0.0\", =2D \"name\": \"foo\", =2D \"license\": \"GNU LGPL\", =2D \"summary\": \"summary\", =2D \"home_page\": \"http://example.com\", =2D \"classifiers\": [], =2D \"download_url\": \"\" =2D }, =2D \"urls\": [], =2D \"releases\": { =2D \"1.0.0\": [ =2D { =2D \"url\": \"https://example.com/foo-1.0.0.egg\", =2D \"packagetype\": \"bdist_egg\" =2D }, { =2D \"url\": \"https://example.com/foo-1.0.0.tar.gz\", =2D \"packagetype\": \"sdist\" =2D }, { =2D \"url\": \"https://example.com/foo-1.0.0-py2.py3-none-any.whl\", =2D \"packagetype\": \"bdist_wheel\" =2D } =2D ] =2D } =2D}") + (foo-json)) =20 (define test-json-2 =2D "{ =2D \"info\": { =2D \"version\": \"1.0.0\", =2D \"name\": \"foo-99\", =2D \"license\": \"GNU LGPL\", =2D \"summary\": \"summary\", =2D \"home_page\": \"http://example.com\", =2D \"classifiers\": [], =2D \"download_url\": \"\" =2D }, =2D \"urls\": [], =2D \"releases\": { =2D \"1.0.0\": [ =2D { =2D \"url\": \"https://example.com/foo-99-1.0.0.egg\", =2D \"packagetype\": \"bdist_egg\" =2D }, { =2D \"url\": \"https://example.com/foo-99-1.0.0.tar.gz\", =2D \"packagetype\": \"sdist\" =2D }, { =2D \"url\": \"https://example.com/foo-99-1.0.0-py2.py3-none-any.whl= \", =2D \"packagetype\": \"bdist_wheel\" =2D } =2D ] =2D } =2D}") + (foo-json #:name "foo-99")) + +(define test-json-3 + (foo-json #:name "Foo" #:name-in-url "foo")) + +(define test-json-4 + (foo-json #:name "foo-bar" #:name-in-url "foo_bar")) + +(define test-json-5 + (foo-json #:name "foo" #:name-in-url "goo")) =20 (define test-source-hash "") @@ -211,6 +200,30 @@ (define test-metadata-with-extras-jedi "\ call-with-input-string) (parse-wheel-metadata test-metadata-with-extras-jedi))) =20 +(test-equal "find-project-url, with numpy" + "numpy" + (find-project-url + "numpy" + "https://files.pythonhosted.org/packages/0a/c8/a62767a6b374a0dfb02d2a04= 56e5f56a372cdd1689dbc6ffb6bf1ddedbc0/numpy-1.22.1.zip")) + +(test-equal "find-project-url, uWSGI" + "uwsgi" + (find-project-url + "uWSGI" + "https://files.pythonhosted.org/packages/24/fd/93851e4a076719199868d4c9= 18cc93a52742e68370188c1c570a6e42a54f/uwsgi-2.0.20.tar.gz")) + +(test-equal "find-project-url, flake8-array-spacing" + "flake8_array_spacing" + (find-project-url + "flake8-array-spacing" + "https://files.pythonhosted.org/packages/a4/21/ff29b901128b681b7de7a278= 7b3aeb3e1f3cba4a8c0cffa9712cbff016bc/flake8_array_spacing-0.2.0.tar.gz")) + +(test-equal "find-project-url, foo/goo" + "foo" + (find-project-url + "foo" + "https://files.pythonhosted.org/packages/f0/f00/goo-0.0.0.tar.gz")) + (test-assert "pypi->guix-package, no wheel" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch @@ -428,4 +441,222 @@ (define test-metadata-with-extras-jedi "\ (x (pk 'fail x #f)))))) =20 +(test-equal "pypi->guix-package, package name is capitalized but not in do= wnload URI (like uWSGI)" + ;; Checking that the importer doesn=E2=80=99t emit a warning + "" + (call-with-output-string + (lambda (port) + (parameterize ((guix-warning-port port)) + ;; Replace network resources with sample data. + (mock ((guix import utils) url-fetch + (lambda (url file-name) + (match url + ;; Package Foo has /foo- download URLs + ("https://example.com/foo-1.0.0.tar.gz" + (begin + (mkdir-p "foo-1.0.0/foo.egg-info") + (with-output-to-file "foo-1.0.0/foo.egg-info/require= s.txt" + (lambda () + (display test-requires.txt))) + (parameterize ((current-output-port (%make-void-port= "rw+"))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) + (delete-file-recursively "foo-1.0.0"))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" + (begin + (mkdir "foo-1.0.0.dist-info") + (with-output-to-file "foo-1.0.0.dist-info/METADATA" + (lambda () + (display test-metadata))) + (let ((zip-file (string-append file-name ".zip"))) + ;; zip always adds a "zip" extension to the file i= t creates, + ;; so we need to rename it. + (system* "zip" "-q" zip-file "foo-1.0.0.dist-info/= METADATA") + (rename-file zip-file file-name)) + (delete-file-recursively "foo-1.0.0.dist-info"))) + (_ (error "Unexpected URL: " url))))) + (mock ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ;; Note that the project name is Foo, so the proj= ect + ;; URL queried by the importer is /pypi/Foo/json + ("https://pypi.org/pypi/Foo/json" + (values (open-input-string test-json-3) + (string-length test-json-3))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.= whl" #f) + (_ (error "Unexpected URL: " url))))) + (match (pypi->guix-package "Foo") + (('package + ('name "python-foo") ;; The guix package name is + ;; computed from the pypi + ;; project name: Foo -> foo = -> + ;; python-foo + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ;; If the importer were too simple, it + ;; would have put '(pypi-uri "Foo" + ;; 'version) here, but as indicated by + ;; test-json-4, the package releases a= re + ;; usually published with /foo- URLs, = so + ;; we would get a 404 Not Found respon= se + ;; when building the package. + ('uri ('pypi-uri "foo" 'version)) + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'python-build-system) + ('propagated-inputs ('list 'python-bar 'python-ba= z)) + ('native-inputs ('list 'python-pytest)) + ('home-page "http://example.com") + ('synopsis "summary") + ('description "summary") + ('license 'license:lgpl2.0)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (error "Failed: ~S" x))))))))) + +(test-equal "pypi->guix-package, package name is kebab-case but the downlo= ad URI is snake_case (like flake8-array-spacing)" + ;; This test is similar to the previous one + "" + (call-with-output-string + (lambda (port) + (parameterize ((guix-warning-port port)) + ;; Replace network resources with sample data. + (mock ((guix import utils) url-fetch + (lambda (url file-name) + ;; Everything is foo_bar here: + (match url + ("https://example.com/foo_bar-1.0.0.tar.gz" + (begin + (mkdir-p "foo_bar-1.0.0/foo_bar.egg-info") + (with-output-to-file "foo_bar-1.0.0/foo_bar.egg-info= /requires.txt" + (lambda () + (display test-requires.txt))) + (parameterize ((current-output-port (%make-void-port= "rw+"))) + (system* "tar" "czvf" file-name "foo_bar-1.0.0/")) + (delete-file-recursively "foo_bar-1.0.0"))) + ("https://example.com/foo_bar-1.0.0-py2.py3-none-any.wh= l" + (begin + (mkdir "foo_bar-1.0.0.dist-info") + (with-output-to-file "foo_bar-1.0.0.dist-info/METADA= TA" + (lambda () + (display test-metadata))) + (let ((zip-file (string-append file-name ".zip"))) + ;; zip always adds a "zip" extension to the file i= t creates, + ;; so we need to rename it. + (system* "zip" "-q" zip-file "foo_bar-1.0.0.dist-i= nfo/METADATA") + (rename-file zip-file file-name)) + (delete-file-recursively "foo_bar-1.0.0.dist-info"))) + (_ (error "Unexpected URL: " url))))) + (mock ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ("https://pypi.org/pypi/foo-bar/json" ; (!) + (values (open-input-string test-json-4) + (string-length test-json-4))) + ("https://example.com/foo_bar-1.0.0-py2.py3-none-= any.whl" #f) + (_ (error "Unexpected URL: " url))))) + (match (pypi->guix-package "foo-bar") + (('package + ('name "python-foo-bar") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri ('pypi-uri "foo_bar" 'version)) + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'python-build-system) + ('propagated-inputs ('list 'python-bar 'python-ba= z)) + ('native-inputs ('list 'python-pytest)) + ('home-page "http://example.com") + ('synopsis "summary") + ('description "summary") + ('license 'license:lgpl2.0)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (error "Failed: ~S" x))))))))) + +(test-equal "pypi->guix-package, package name is foo but the download URI = uses goo" + ;; A warning should be emitted with a hint about the issue + "warning: project name foo does not appear verbatim in the PyPI URI +hint: The PyPI URI is: `https://example.com/goo-1.0.0.tar.gz'. You should= review the pypi-uri +declaration in the generated package. You may need to replace \"foo\" wit= h a substring of the PyPI +URI that identifies the package. + +" + (call-with-output-string + (lambda (port) + (parameterize ((guix-warning-port port) + (current-error-port port)) + ;; Replace network resources with sample data. + (mock ((guix import utils) url-fetch + (lambda (url file-name) + (match url + ("https://example.com/goo-1.0.0.tar.gz" + (begin + (mkdir-p "goo-1.0.0/goo.egg-info/") + (with-output-to-file "goo-1.0.0/goo.egg-info/require= s.txt" + (lambda () + (display "wrong data to make sure we're testing = wheels "))) + (parameterize ((current-output-port (%make-void-port= "rw+"))) + (system* "tar" "czvf" file-name "goo-1.0.0/")) + (delete-file-recursively "goo-1.0.0") + (set! test-source-hash + (call-with-input-file file-name port-sha256)))) + ("https://example.com/goo-1.0.0-py2.py3-none-any.whl" + (begin + (mkdir "goo-1.0.0.dist-info") + (with-output-to-file "goo-1.0.0.dist-info/METADATA" + (lambda () + (display test-metadata))) + (let ((zip-file (string-append file-name ".zip"))) + ;; zip always adds a "zip" extension to the file i= t creates, + ;; so we need to rename it. + (system* "zip" "-q" zip-file "goo-1.0.0.dist-info/= METADATA") + (rename-file zip-file file-name)) + (delete-file-recursively "goo-1.0.0.dist-info"))) + (_ (error "Unexpected URL: " url))))) + (mock ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ("https://pypi.org/pypi/foo/json" + (values (open-input-string test-json-5) + (string-length test-json-5))) + ("https://example.com/goo-1.0.0-py2.py3-none-any.= whl" #f) + (_ (error "Unexpected URL: " url))))) + ;; Not clearing the memoization cache here would mean = returning the value + ;; computed in the previous test. + (invalidate-memoization! pypi->guix-package) + (match (pypi->guix-package "foo") + (('package + ('name "python-foo") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ;; The importer can=E2=80=99t guess th= at we should + ;; replace with '(pypi-uri "goo" + ;; 'version), so it defaults to "foo" = with + ;; a warning. + ('uri ('pypi-uri "foo" 'version)) + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'python-build-system) + ('propagated-inputs ('list 'python-bar 'python-ba= z)) + ('native-inputs ('list 'python-pytest)) + ('home-page "http://example.com") + ('synopsis "summary") + ('description "summary") + ('license 'license:lgpl2.0)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (error "Failed: ~S" x))))))))) + (test-end "pypi") =2D-=20 2.34.0 --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGzBAEBCAAdFiEEq4yIHjMvkliPpwQnO7C8EjLYuCwFAmHwKzEACgkQO7C8EjLY uCyHywwAgxzKM/qvom+KBcAaV4Oe2s1DUHVY0CBMttUWdu4vcJVce8GSuLzRVIgi DZoZLFNT/WKe6Pw4EjphDyz9AzKgOAkgqF0I2w1NBHJS7BZ2Yuvt4mR0erAaArFs nttInBvGLOirfgbyKHcJ7pSBjm8RYVz9wV+m6Jl4Y+HsYu/yU/3TdLEYdPA7iVl5 L46UI3owFCTa+9QUTS2W69PSxnz2gwuZ83A6LrDo6DuQV+nBhDXnvzkSsCrfbOYv ifoLV+Jm+YwszF3BQ2adLDm2NTUAmxDbqxZXi5Z54ElgbiEXTsGqXiivE0c5ty+p bttmSKNRIMcujvn4jJlPypKyYwt96QSrtGU43VleqffXAuOsXmfdnbZVhyWa6nDY ne+D6Sx9EkX9XtgO4iGd9O7zJZuiaeBi/jCsXdJKGCyS4scOn2DjEY2inWjo+vW9 R7rNYfr05P9HQuMbjqlYtvOcY3dvRP5h3MxltSanfbRoXYfdlAbxMzGIAIWOPNZ+ Fu9mOBaD =YASa -----END PGP SIGNATURE----- --==-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 26 10:21:03 2022 Received: (at 53395-done) by debbugs.gnu.org; 26 Jan 2022 15:21:03 +0000 Received: from localhost ([127.0.0.1]:53191 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCk6Z-0004bW-3i for submit@debbugs.gnu.org; Wed, 26 Jan 2022 10:21:03 -0500 Received: from hera.aquilenet.fr ([185.233.100.1]:42096) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1nCk6X-0004aJ-2L for 53395-done@debbugs.gnu.org; Wed, 26 Jan 2022 10:21:01 -0500 Received: from localhost (localhost [127.0.0.1]) by hera.aquilenet.fr (Postfix) with ESMTP id 8D30045B; Wed, 26 Jan 2022 16:20:54 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at aquilenet.fr Received: from hera.aquilenet.fr ([127.0.0.1]) by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YgFipEARGZT2; Wed, 26 Jan 2022 16:20:53 +0100 (CET) Received: from ribbon (91-160-117-201.subs.proxad.net [91.160.117.201]) by hera.aquilenet.fr (Postfix) with ESMTPSA id 546709E; Wed, 26 Jan 2022 16:20:53 +0100 (CET) From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Vivien Kraus Subject: Re: bug#53395: Fix pypi import for flake8-array-spacing References: <87czkmi434.fsf@planete-kraus.eu> <87ee4xut38.fsf@gnu.org> <87mtjk22rn.fsf@planete-kraus.eu> <87y234lzqu.fsf@gnu.org> <87ee4v21u6.fsf@planete-kraus.eu> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 7 =?utf-8?Q?Pluvi=C3=B4se?= an 230 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Wed, 26 Jan 2022 16:20:52 +0100 In-Reply-To: <87ee4v21u6.fsf@planete-kraus.eu> (Vivien Kraus's message of "Tue, 25 Jan 2022 17:39:54 +0100") Message-ID: <87ee4uh6az.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: / Authentication-Results: hera.aquilenet.fr; none X-Rspamd-Server: hera X-Rspamd-Queue-Id: 8D30045B X-Spamd-Result: default: False [-0.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-Spam-Score: 1.0 (+) X-Debbugs-Envelope-To: 53395-done Cc: 53395-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Hi, Vivien Kraus skribis: > From 8792367617d288e584a703db38c82c749a067c26 Mon Sep 17 00:00:00 2001 > From: Vivien Kraus > Date: Thu, 20 Jan 2022 20:11:56 +0100 > Subject: [PATCH] pypi importer: Convert - to _ in pypi urls if needed. > > * guix/import/pypi.scm (find-project-url): New function. > (make-pypi-sexp): Use find-project-url. > * tests/pypi.scm ("Package Foo has a correct pypi-uri of foo"): New test. > ("If the package goo is released as foo, the importer warns"): New test. \o/ I took the liberty to remove the integration tests you had added, I think it=E2=80=99s preferable (especially since there=E2=80=99s a patch ref= actoring these bits that Maxime submitted recently=E2=80=A6). I mentioned the other changes to the commit log and added a copyright line for you. Let me know if I got anything wrong. Thank you! Ludo=E2=80=99. From unknown Tue Jun 17 22:30:50 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 24 Feb 2022 12:24:10 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator