From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 16 16:01:03 2016 Received: (at submit) by debbugs.gnu.org; 16 Sep 2016 20:01:03 +0000 Received: from localhost ([127.0.0.1]:33566 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bkzJm-0007I5-TD for submit@debbugs.gnu.org; Fri, 16 Sep 2016 16:01:03 -0400 Received: from eggs.gnu.org ([208.118.235.92]:52955) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bkzJl-0007HU-NF for submit@debbugs.gnu.org; Fri, 16 Sep 2016 16:01:02 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bkzJf-0004Pm-PU for submit@debbugs.gnu.org; Fri, 16 Sep 2016 16:00:56 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:37803) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bkzJf-0004PT-Hv for submit@debbugs.gnu.org; Fri, 16 Sep 2016 16:00:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:43906) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bkzJd-0002Nn-E4 for bug-guix@gnu.org; Fri, 16 Sep 2016 16:00:54 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bkzJY-0004Ng-6J for bug-guix@gnu.org; Fri, 16 Sep 2016 16:00:53 -0400 Received: from aibo.runbox.com ([91.220.196.211]:50329) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bkzJX-0004NM-VS for bug-guix@gnu.org; Fri, 16 Sep 2016 16:00:48 -0400 Received: from [10.9.9.210] (helo=mailfront10.runbox.com) by bars.runbox.com with esmtp (Exim 4.71) (envelope-from ) id 1bkzJV-0001uH-2M for bug-guix@gnu.org; Fri, 16 Sep 2016 22:00:45 +0200 Received: from xd9bb9d06.dyn.telefonica.de ([217.187.157.6] helo=localhost) by mailfront10.runbox.com with esmtpsa (uid:892961 ) (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) id 1bkzIp-0002xQ-KP for bug-guix@gnu.org; Fri, 16 Sep 2016 22:00:03 +0200 From: ng0 To: bug-guix@gnu.org Subject: pypi importer outputs strange character series in optional dependency case. Date: Fri, 16 Sep 2016 20:00:02 +0000 Message-ID: <87h99fipj1.fsf@we.make.ritual.n0.is> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -5.0 (-----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) I think this should not happen with pypi import: (inputs `(("python-certifi==2016.2.28" ,python-certifi==2016.2.28) ("python-dateutil==2.5.3" ,python-dateutil==2.5.3) ("python-flask-babel==0.11.1" ,python-flask-babel==0.11.1) ("python-flask==0.11.1" ,python-flask==0.11.1) ("python-lxml==3.6.0" ,python-lxml==3.6.0) ("python-ndg-httpsclient==0.4.1" ,python-ndg-httpsclient==0.4.1) ("python-pyasn1-modules==0.0.8" ,python-pyasn1-modules==0.0.8) ("python-pyasn1==0.1.9" ,python-pyasn1==0.1.9) ("python-pygments==2.1.3" ,python-pygments==2.1.3) ("python-pyopenssl==0.15.1" ,python-pyopenssl==0.15.1) ("python-pyyaml==3.11" ,python-pyyaml==3.11) ("python-requests[socks]==2.10.0" ,#{python-requests\x5b;socks\x5d;==2.10.0}#) ("python-setuptools" ,python-setuptools))) I can understand the version numbers, I can also understand the optional socks building/module of the python-requests, but why does it read like Gobbledygook? Can't we improve the output here? For version numbers, this is not a format which happened recently which is exclusive for python build system right? This is just bad formated because of the pypi query. I will first try and not pin the application to these version numbers, maybe itjustworks™. To reproduce: "guix import pypi searx" -- ng0 From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 28 00:32:27 2019 Received: (at control) by debbugs.gnu.org; 28 Mar 2019 04:32:27 +0000 Received: from localhost ([127.0.0.1]:33721 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9MiJ-0001uu-21 for submit@debbugs.gnu.org; Thu, 28 Mar 2019 00:32:27 -0400 Received: from mail-io1-f53.google.com ([209.85.166.53]:36177) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9MiG-0001uf-E8 for control@debbugs.gnu.org; Thu, 28 Mar 2019 00:32:25 -0400 Received: by mail-io1-f53.google.com with SMTP id f6so16082568iop.3 for ; Wed, 27 Mar 2019 21:32:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:date:message-id:to:subject; bh=tqL9PaCwEl5fTLggAew/TC1QwHc6x3/LpUbrL8cq93g=; b=PSIxhADH91HCMbcRVYImCM/KcOuc0YLmiantqTFFWBjcs5bafgQAOVQpoR025lXhsX pqumcc/91l07ecCI0IcUGuBs3HstG5mNAKsvzmTZ4p5UZPnhv2cbG0WPBws3PuPmGUL6 hR9smME3ICsmzKNzDGxZG8h15Ql3w1kVTTuRvmKXdqF9NahJTyBBqKWP/anZnK0ztTQu NdLVO2f9i339dXpd1SzObsagKHJAfqNawQt9RrFLokA3zJcfT4KSDy+tOpvXFpppuXxZ GXtgbzZkm3ksnzajNAhEGwdXpRitXkq0s8aUHIeSOncfxTFqvLAyb+rrKpOkpuXB+XG+ 5s7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:date:message-id:to:subject; bh=tqL9PaCwEl5fTLggAew/TC1QwHc6x3/LpUbrL8cq93g=; b=BgW+zdqSNzWZ7DGYPTsa9uAQ2HGRCURoJTd/8iGe/cFh8LYNVZKqcSEUhxuofn392f QpOlIwZiFrCJ3LMH6V2HMtMB9lKOjmv3Pb39B7I18MO03WxmrxD7uIV3POnVGaVeI9TI 3sLOhvl26FZ9LbuvgGoHJ/OEySMAcNEA+t0SQnRg595fr5yC9jzoUxs9y5T2cwgSD7VA W99r4qTkZLhWUe3MsaAp3yjQnheQi2oYob06QHWCcBhZ0Htb1NphYh2CP+eDgzlHcC5G cSttsIR9et3KAorVebcqARi1UBhIqYWWYuWIIU3ehtLrHzUxueCuxR+l1CbF3JZJJ3Dh UTtA== X-Gm-Message-State: APjAAAVu+mOhYT97i99Un13Jx1YO96ZJM0oHonrXEJfVQj3CDEKATd3k xvCXYLT+lSXX0PKbguge7fR/5177 X-Google-Smtp-Source: APXvYqyWX4+ufzC8+/XkbrJw1tC9ZhtMeZhixg0YxaMsT2AC65KmDNnnFpQVIfZ4fAo6a9Ia+q5y4w== X-Received: by 2002:a5e:d701:: with SMTP id v1mr27035992iom.276.1553747538482; Wed, 27 Mar 2019 21:32:18 -0700 (PDT) Received: from kwak ([2607:f2c0:94b4:fa00::235]) by smtp.gmail.com with ESMTPSA id g24sm1053710itk.14.2019.03.27.21.32.17 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 27 Mar 2019 21:32:17 -0700 (PDT) From: T460s laptop X-Google-Original-From: T460s laptop Date: Thu, 28 Mar 2019 00:32:15 -0400 Message-Id: <871s2r7h68.fsf@kwak.i-did-not-set--mail-host-address--so-tickle-me> To: control@debbugs.gnu.org Subject: control message for bug #33047 X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) merge 33047 24450 From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 28 00:37:11 2019 Received: (at control) by debbugs.gnu.org; 28 Mar 2019 04:37:12 +0000 Received: from localhost ([127.0.0.1]:33731 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9Mmt-00022b-KP for submit@debbugs.gnu.org; Thu, 28 Mar 2019 00:37:11 -0400 Received: from mail-it1-f179.google.com ([209.85.166.179]:54745) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9Mms-00022F-04 for control@debbugs.gnu.org; Thu, 28 Mar 2019 00:37:10 -0400 Received: by mail-it1-f179.google.com with SMTP id w18so3859751itj.4 for ; Wed, 27 Mar 2019 21:37:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:to:from:subject; bh=THyKsb0oOhvm//1kYxTGQKPAXqV+p5azepuJpttep3A=; b=KPnMkC6fK7hn34Dqvqu1CNhhhWUpfcahnNtoSKoiFu9d5yywWQl0F0exQTwLYa/qJh YepOz/SpYYESxs6WwdwHHBhCwxTlXQExVRyMlBcEHKX5l2El1FDjO/6qPinATYcaAS/F cQF4kPpnKlWZmn/EsmseOaGFXh8bxsPb0Pkux1RwDny5Px+Nx/miY+NrmqBpGxtdmf90 QxBtTUSM0llR37sLfdnWnQcnyawnobkkw39/wvxr5MVeyn5P9z1qxIVz687aLOBlG/YO rN+anClIJ4abq3pygWtMHwUJ78JKjIbERD7eQD+PTerKRwjyxzwvrloRekpA3kO70hu2 Y7Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:to:from:subject; bh=THyKsb0oOhvm//1kYxTGQKPAXqV+p5azepuJpttep3A=; b=UsDgVeRDBD8JXyQfVVNibAOoIvHnCJUvl5/byNAqBbaO+cpF+utT27XyHqKgqNkfmY loIk9tfvZVWfIe9mWBKgehdCNVPApQlD0yr6U3NcC6luu+BDlHYJZpfSPDc3O2UMrYY+ 5xlV9n3IE/DxrTxuHEmrz2t7HIv6YBdI4jRhzeHPAhzmRzUSEsBOCHc+Ui+rdpKV4yoT G4R3pvvkGzwGZhsMroj9RpITY+I29ZsRxV8WA5VTG3IQ2n4pIJ9E89o09ZVuw4T8/e95 TtbdU2cnJHYwP7ooUbjTEfe4SgcwD1vMdEqwt9aXSveEqB0wkQcwEXp7p/DepDacv+Ex tRJw== X-Gm-Message-State: APjAAAXV06YMSlNlYiDwRCgq+2nHeFTx9KhnEqR+IV+gPXK5JJ4wnOjA SNXiyAIn+f59uPiisU2nRADZaL6q X-Google-Smtp-Source: APXvYqx9es34w/+cZVsnL6I8nkpkmr3r+h2OwC0iOPJO+VAUMIyRUfJ35KoxzF6kvsBQ6QVvVtJdjw== X-Received: by 2002:a24:3f85:: with SMTP id d127mr6717462ita.170.1553747824300; Wed, 27 Mar 2019 21:37:04 -0700 (PDT) Received: from kwak ([2607:f2c0:94b4:fa00::235]) by smtp.gmail.com with ESMTPSA id g10sm3007532itc.36.2019.03.27.21.37.03 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 27 Mar 2019 21:37:03 -0700 (PDT) Date: Thu, 28 Mar 2019 00:37:02 -0400 Message-Id: <875zs362dt.fsf@gmail.com> To: control@debbugs.gnu.org From: Maxim Cournoyer Subject: control message for bug #33569 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) forcemerge 33569 24450 From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 29 00:19:22 2019 Received: (at 24450) by debbugs.gnu.org; 29 Mar 2019 04:19:22 +0000 Received: from localhost ([127.0.0.1]:34939 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9izC-0005n9-98 for submit@debbugs.gnu.org; Fri, 29 Mar 2019 00:19:22 -0400 Received: from mail-it1-f174.google.com ([209.85.166.174]:40504) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9izA-0005mq-E2; Fri, 29 Mar 2019 00:19:21 -0400 Received: by mail-it1-f174.google.com with SMTP id y63so1812056itb.5; Thu, 28 Mar 2019 21:19:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=H8z7PvZNCOiPhlqPYauL8U8reG8tvLXY+vlMB5BaQ9o=; b=uONJ/uZ++grtBT8Dhoj7GirM4ogB5yWePEz+hbhHkYmdyi3GXYTHAF9HCzhCc94TDk pgxzNj4gvsj21zesmZg2sTYjEYbzVbyXalxmy+IBwERiYm1RmW6EKPkxwLIB4TlHzKMQ mzN/w5x6UM+4KaVQJwF/3PH6Z3G00ULjV4psp73bMRkUZaIFsom85bojzzW6IefU1Crs TVpA6qIDg4bC6DzkTfjHhf7of/d9iQ78PMpUtw5VysAKlkUEsyI2Sm1dL2qgOebgDrz3 RkEWZwgQMXqkPRRz+9WmF0/jdaiiEZAsLeV9XikKoGtN2enxeXRWMvC3aPFLrsJrT48I zIbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=H8z7PvZNCOiPhlqPYauL8U8reG8tvLXY+vlMB5BaQ9o=; b=beeKPkHzMKUO+SSRenmcNTrXoz7nVK44tfWeA9eDI0L09cLGzbbEoL78+ubSxTT0GJ XMry6rElOgl/8t7HD0n7941wCKYN6vTpMzA2fVkgTkr81TDdZBH1vPuY6kBs3Z4I8wiw XmdsxLK+1COSUK5JKfh2Zdm1XdbpvWEDx0C2330V/eFMVgN7CIknmUo7UaTOZm2cpLx7 avUAGfLiMEYiQwxYdRSuZ4pLMzijQNeJjB5SyqwSZUyX4Xn1TjT5fX1tgw/LIiHnYFOR y51eSstUvmayTrr/QGoa6rtV5QkauodIefXM5bTprrobD7FtmPYqrWzDLPyJe1/1ngFd Jb7A== X-Gm-Message-State: APjAAAU/VTx/8dROHKKRVCvFPZG1qNjt4tqP+8EtwQBNyr3OITxThTEJ FOooXujGaig0DOJc+Jx8oL2gC9U4kQE= X-Google-Smtp-Source: APXvYqyRT0loXREb4DoefVGxnVLypjgEU6JcCW3XbuFTviqLhdcZLMF3AZcyWADMRdtsL7F32LFDWA== X-Received: by 2002:a02:8c4d:: with SMTP id j13mr35138401jal.48.1553833154577; Thu, 28 Mar 2019 21:19:14 -0700 (PDT) Received: from kwak ([2607:f2c0:94b4:fa00::235]) by smtp.gmail.com with ESMTPSA id r184sm9696440ita.3.2019.03.28.21.19.13 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 28 Mar 2019 21:19:13 -0700 (PDT) From: Maxim Cournoyer To: swedebugia Subject: Re: bug#34266: pypi importer cannot handle [ and ] correctly References: Date: Fri, 29 Mar 2019 00:19:12 -0400 In-Reply-To: (swedebugia's message of "Thu, 31 Jan 2019 19:25:12 +0100") Message-ID: <87va022tz3.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 34266@debbugs.gnu.org, 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) swedebugia writes: > $ ./pre-inst-env guix import pypi beaker > > following redirection to `https://pypi.org/pypi/Beaker/json'... > > Starting download of /tmp/guix-file.p15GJZ > From > https://files.pythonhosted.org/packages/c2/21/b052b2fbfee3def06670923d5d3= 4b0d353d4c278013e4a714c3fb663f150/Beaker-1.10.0.tar.gz... > ...0.0.tar.gz 40KiB 521KiB/s 00:00 > [##################] 100.0% > (package > (name "python-beaker") > (version "1.10.0") > (source > (origin > (method url-fetch) > (uri (pypi-uri "beaker" version)) > (sha256 > (base32 > "0l047yl3n9b3w7ba0wrqdb5fpww5y8pjy20kah2mlpr230lqjwk0")))) > (build-system python-build-system) > (propagated-inputs > `(("python-[crypto]" ,#{python-\x5b;crypto\x5d;}#) > ("python-[cryptography]" > ,#{python-\x5b;cryptography\x5d;}#) > ("python-[pycrypto]" > ,#{python-\x5b;pycrypto\x5d;}#) > ("python-[pycryptodome]" > ,#{python-\x5b;pycryptodome\x5d;}#) > ("python-[testsuite]" > ,#{python-\x5b;testsuite\x5d;}#) > ("python-coverage" ,python-coverage) > ("python-cryptography" ,python-cryptography) > ("python-cryptography" ,python-cryptography) > ("python-funcsigs" ,python-funcsigs) > ("python-memcached" ,python-memcached) > ("python-mock" ,python-mock) > ("python-nose" ,python-nose) > ("python-pycrypto" ,python-pycrypto) > ("python-pycryptodome" ,python-pycryptodome) > ("python-pycryptodome" ,python-pycryptodome) > ("python-pycryptopp" ,python-pycryptopp) > ("python-pylibmc" ,python-pylibmc) > ("python-pymongo" ,python-pymongo) > ("python-redis" ,python-redis) > ("python-sqlalchemy" ,python-sqlalchemy) > ("python-webtest" ,python-webtest))) > (home-page "https://beaker.readthedocs.io/") > (synopsis > "A Session and Caching library with WSGI Middleware") > (description > "A Session and Caching library with WSGI Middleware") > (license license:bsd-3)) Testing with my soon-to-be sent for review changes: --8<---------------cut here---------------start------------->8--- ./pre-inst-env guix import pypi beaker following redirection to `https://pypi.org/pypi/Beaker/json'... Starting download of /tmp/guix-file.0MWu4B >From https://files.pythonhosted.org/packages/76/87/ecc1a222f0caaa7ba7b89287= 37e89b2e91b8c22450c12b8a51ee625a4d87/Beaker-1.10.1.tar.gz... =E2=80=A60.1.tar.gz 40KiB 487KiB/s 00:00 [#############= #####] 100.0% (package (name "python-beaker") (version "1.10.1") (source (origin (method url-fetch) (uri (pypi-uri "beaker" version)) (sha256 (base32 "16zdjfl8v73yl1capph0n371vd26c7zpzb48n505ip32ffgmvc4f")))) (build-system python-build-system) (native-inputs `(("python-coverage" ,python-coverage) ("python-cryptography" ,python-cryptography) ("python-memcached" ,python-memcached) ("python-mock" ,python-mock) ("python-nose" ,python-nose) ("python-pycryptodome" ,python-pycryptodome) ("python-pylibmc" ,python-pylibmc) ("python-pymongo" ,python-pymongo) ("python-redis" ,python-redis) ("python-sqlalchemy" ,python-sqlalchemy) ("python-webtest" ,python-webtest))) (home-page "https://beaker.readthedocs.io/") (synopsis "A Session and Caching library with WSGI Middleware") (description "A Session and Caching library with WSGI Middleware") (license license:bsd-3)) --8<---------------cut here---------------end--------------->8--- Looking better? Maxim From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 29 00:21:03 2019 Received: (at 24450) by debbugs.gnu.org; 29 Mar 2019 04:21:03 +0000 Received: from localhost ([127.0.0.1]:34946 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9j0p-0005rV-Bq for submit@debbugs.gnu.org; Fri, 29 Mar 2019 00:21:03 -0400 Received: from mail-io1-f42.google.com ([209.85.166.42]:43104) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9j0n-0005qP-1Z; Fri, 29 Mar 2019 00:21:01 -0400 Received: by mail-io1-f42.google.com with SMTP id x3so672898iol.10; Thu, 28 Mar 2019 21:21:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=/45T5Zrchgzc5rzndzC+fgq5GVCdzqYhDsakWW1Wc28=; b=MrdGrKUMLsoHOZdPGmuzbEPmNG7V31QvFjGOSreEkmevg8DJzb02VGmr4LGRjHWza5 DfJfm7pUtv2x3g1BnZ4Q8uM4cINRZwJ+QCuEQxAIGdLb5MZLzPN636cuh78p+2pk8qH6 A87Eqp4FOgVCk+A8gdXmHhoCfnl/b3zlmXcg3CIHEGgVEj+bKd5WIsMKwADuVek0w5DK X15m0GUOXDWXxO/XY0f9bmnfw5qEsndcRnQMCYScvcWzKAqitiK2/88bij2wyox1JQC/ 9rCk/r5bkC67zVOoRR6javwki6Ib5/N4JUug3VwrMWAiniEqqGbp/NOcGhnRcvtqdaQE xFSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=/45T5Zrchgzc5rzndzC+fgq5GVCdzqYhDsakWW1Wc28=; b=cAWFNliUC7uLY1BuswlQYMdsaTaBmb7Gb4+L+SjjbBXGERVMwl5O3ekNnDdm4N1inN M5cqpCNZPmHFqaaj9dQ8H+YOCfOeOoEV5M7RLpm4ckn3zdRRwEk32Jy/D3OK1EkkGu98 LEmRauScd63RpbCAdIR8V3bRjPN1i2n0U2moCJj42LjdoC2zb1ZDYEuhV7FSzqRVE5bY YPujT08prSRcnwSqUMl9yu8cVSfPsl77N0OXfIU2qULF9Ns+w4JdHJJ3HJUtFai9r1jz SEZcq9tHU8vBL71HSSOqS/xHTgN+i2FBNf8Tjz3LglK9xxb1AvNqBjcdmXmRwRlNPDB7 FbKA== X-Gm-Message-State: APjAAAVH5aeMkOL4+ONWBVrGIeJ21c2fEv+sFGXQCqSX0EfgK274pwdG QXRsRZ0AjT+Ps4/hnWczNsESKHlqH4s= X-Google-Smtp-Source: APXvYqxAhhNaho89QTiivGtM6wMd9ALgdcJjS5y8YSJ1t4gV4OtB3/ewJB/bFeKTDeAFLNzocwHxsg== X-Received: by 2002:a6b:7804:: with SMTP id j4mr12887382iom.171.1553833254992; Thu, 28 Mar 2019 21:20:54 -0700 (PDT) Received: from kwak ([2607:f2c0:94b4:fa00::235]) by smtp.gmail.com with ESMTPSA id t68sm531752ita.4.2019.03.28.21.20.53 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 28 Mar 2019 21:20:54 -0700 (PDT) From: Maxim Cournoyer To: swedebugia Subject: Re: bug#33569: Missing sanitizing of '[]' in pypi-importer References: Date: Fri, 29 Mar 2019 00:20:53 -0400 In-Reply-To: (swedebugia's message of "Sun, 2 Dec 2018 01:32:02 +0100") Message-ID: <87r2aq2twa.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 33569@debbugs.gnu.org, 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) swedebugia writes: > E.g. > sdb@komputilo ~/guix-tree$ ~/guix-tree/pre-inst-env guix import pypi > snakemake > ... > (propagated-inputs > `(("python-[reports]" > ,#{python-\x5b;reports\x5d;}#) > ("python-appdirs" ,python-appdirs) > ... This one now gives (local branch): --8<---------------cut here---------------start------------->8--- ./pre-inst-env guix import pypi snakemake Starting download of /tmp/guix-file.4XvWMX >From https://files.pythonhosted.org/packages/4a/aa/aab1515d220be06fbdccf3c8= 9335d9585b08ac6be74b8e3c9e8c3c32798e/snakemake-5.4.4.tar.gz... =E2=80=A6.4.4.tar.gz 169KiB 723KiB/s 00:00 [#############= #####] 100.0% (package (name "python-snakemake") (version "5.4.4") (source (origin (method url-fetch) (uri (pypi-uri "snakemake" version)) (sha256 (base32 "0prpr5qajqwr8sh4gzggpj8l4np2rcm9nfdzvcp30d5yw7h26wqm")))) (build-system python-build-system) (propagated-inputs `(("python-appdirs" ,python-appdirs) ("python-configargparse" ,python-configargparse) ("python-datrie" ,python-datrie) ("python-docutils" ,python-docutils) ("python-gitpython" ,python-gitpython) ("python-jsonschema" ,python-jsonschema) ("python-pyyaml" ,python-pyyaml) ("python-ratelimiter" ,python-ratelimiter) ("python-requests" ,python-requests) ("python-wrapt" ,python-wrapt))) (home-page "http://snakemake.bitbucket.io") (synopsis "Snakemake is a workflow management system that aims to reduce the comp= lexity of creating workflows by providing a fast and comfortable execution = environment, together with a clean and modern specification language in pyt= hon style. Snakemake workflows are essentially Python scripts extended by d= eclarative code to define rules. Rules describe how to create output files = from input files.") (description "Snakemake is a workflow management system that aims to reduce the comp= lexity of creating workflows by providing a fast and comfortable execution = environment, together with a clean and modern specification language in pyt= hon style. Snakemake workflows are essentially Python scripts extended by d= eclarative code to define rules. Rules describe how to create output files = from input files.") (license license:expat)) --8<---------------cut here---------------end--------------->8--- From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 29 00:23:14 2019 Received: (at 24450) by debbugs.gnu.org; 29 Mar 2019 04:23:15 +0000 Received: from localhost ([127.0.0.1]:34953 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9j2w-0005wa-7h for submit@debbugs.gnu.org; Fri, 29 Mar 2019 00:23:14 -0400 Received: from mail-it1-f170.google.com ([209.85.166.170]:54156) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9j2t-0005wC-OM; Fri, 29 Mar 2019 00:23:12 -0400 Received: by mail-it1-f170.google.com with SMTP id y204so1798109itf.3; Thu, 28 Mar 2019 21:23:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=HQ7rZmIcfvilLf6um1+ebAf+xDRVm3uc8x0K915DRFs=; b=dBcUGhrRXYCND9EBrDhIOyxvooS7fp09ZBoqAS6vt52BYmCB8V+0YVI0awK3aZ6mnU 3NMdWgfO7jj0c24UI/BK+gi7wckk5IicwEsGh80nROyOEC4lg2zYW9VBwE1HpoNVICCk lRcNVNOLl+7V7jayLflZJV1IJag0bvA+H3CuHJ9uYli4SDaAXS8jP7Fdkqj6Zt0EyF8f vJHzhPx3hnQRNL0ojsekAKrnEML1PUbjygScSIF27NbwB2ge8d/QqQGuyguNHeekjGLv r8rQ88UeCe2exaky780nlFKaElijGfP0PwVomKjiFX1CDweC4n4Amidi1g/IiFczKIp2 VDpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=HQ7rZmIcfvilLf6um1+ebAf+xDRVm3uc8x0K915DRFs=; b=ImR4Oa6q+Gq/OTpp/AhMtHcKwRPWk5ZLTjGU/Ng2X/oCEnR36qqply4GEidFz8D1gK dj5vDdJtGozxSOstigHIhjamRVMf9T3zL79oWnqC/RK5OaxXlmZcFAN44hmevX3+0pux M6G6UJPjmeUq3yvwjG2lg1HwWgOu4O+0VT6z7L1g9aHA0bfUgFwQL7PC0gw8Y9z0AlHL IWkwJXOQhJKsoj4d+j9WqHaw2uBPp+Bxu+qyIVJ/Nlidlb7/KAVfF/GFIz4sbOyW4aFs 7vOUYQLYWcYdqSHU1NFmG1ExNCAloEQSboGTt+KcN2ymp6qD8BFmFcbs53n2I4TzLQ38 7fEQ== X-Gm-Message-State: APjAAAXhKxKofr6C4buCSX+6on/Rdjntz3bO77sh9FWXsi3hjQwWX38R 2H4GDA73ur7x4cDwKTFBUuDaVr00jnU= X-Google-Smtp-Source: APXvYqyx0e/wpvQf8iB163o1WLhS6ScRuaYaNkb/yAUHAu9GZWUESdu8bN0l2N3e9xjcxlTrOycBag== X-Received: by 2002:a24:a85:: with SMTP id 127mr2649853itw.40.1553833386007; Thu, 28 Mar 2019 21:23:06 -0700 (PDT) Received: from kwak ([2607:f2c0:94b4:fa00::235]) by smtp.gmail.com with ESMTPSA id d133sm607815ita.5.2019.03.28.21.23.04 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 28 Mar 2019 21:23:05 -0700 (PDT) From: Maxim Cournoyer To: Julien Lepiller Subject: Re: bug#33047: pypi importer uses incorrect package names References: Date: Fri, 29 Mar 2019 00:23:03 -0400 In-Reply-To: (Julien Lepiller's message of "Mon, 15 Oct 2018 15:43:02 +0200") Message-ID: <87mule2tso.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 33047@debbugs.gnu.org, 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Julien Lepiller writes: > Hi, > > I found that sometimes the pypi importer had trouble importing > packages correctly. For instance, running "guix import pypi txaio" > gave me this list of dependencies: > > (propagated-inputs > `(("python-[all]" ,#{python-\x5b;all\x5d;}#) > ("python-[asyncio]" > ,#{python-\x5b;asyncio\x5d;}#) > ...)) > > guix import pypi magic-wormhole had this: > > (propagated-inputs > ("python-autobahn[twisted]" > ,#{python-autobahn\x5b;twisted\x5d;}#) > ...)) > > Of course, they break the recursive importer, which makes it difficult > to import these packages correctly. Testing local branch: ./pre-inst-env guix import pypi txaio Starting download of /tmp/guix-file.jTNBQz >From https://files.pythonhosted.org/packages/c1/99/81de004578e9afe017bb1d4c= 8968088a33621c05449fe330bdd7016d5377/txaio-18.8.1.tar.gz... =E2=80=A68.1.tar.gz 50KiB 894KiB/s 00:00 [#############= #####] 100.0% Starting download of /tmp/guix-file.ZB3Q2n >From https://files.pythonhosted.org/packages/e9/6d/e1a6f7835cde86728e5bb1f5= 77be9b2d7d273fdb33c286e70b087d418ded/txaio-18.8.1-py2.py3-none-any.whl... =E2=80=A6.py3-none-any.whl 27KiB 746KiB/s 00:00 [#############= #####] 100.0% (package (name "python-txaio") (version "18.8.1") (source (origin (method url-fetch) (uri (pypi-uri "txaio" version)) (sha256 (base32 "1zmpdph6zddgrnkkcykh6qk5s46l7s5mzfqrh82m4b5iffn61qv7")))) (build-system python-build-system) (propagated-inputs `(("python-six" ,python-six))) (native-inputs `(("python-mock" ,python-mock) ("python-pep8" ,python-pep8) ("python-pyenchant" ,python-pyenchant) ("python-pytest" ,python-pytest) ("python-pytest-cov" ,python-pytest-cov) ("python-sphinx" ,python-sphinx) ("python-sphinx-rtd-theme" ,python-sphinx-rtd-theme) ("python-sphinxcontrib-spelling" ,python-sphinxcontrib-spelling) ("python-tox" ,python-tox) ("python-twine" ,python-twine) ("python-wheel" ,python-wheel))) (home-page "https://github.com/crossbario/txaio") (synopsis "Compatibility API between asyncio/Twisted/Trollius") (description "Compatibility API between asyncio/Twisted/Trollius") (license #f)) and ./pre-inst-env guix import pypi magic-wormhole Starting download of /tmp/guix-file.80RRqk >From https://files.pythonhosted.org/packages/77/15/9438290bab8146efc0213f7c= 3d9645d9bc5a2e885e4049477e7432e40336/magic-wormhole-0.11.2.tar.gz... =E2=80=A6-0.11.2.tar.gz 193KiB 911KiB/s 00:00 [#############= #####] 100.0% Starting download of /tmp/guix-file.mRGAx3 >From https://files.pythonhosted.org/packages/82/98/3e8d12fdb90457e8f3e1f5b8= 77ee27f5db58dbaf4a4fbe95f7287a568401/magic_wormhole-0.11.2-py2.py3-none-any= .whl... =E2=80=A6-py2.py3-none-any.whl 128KiB 1009KiB/s 00:00 [#############= #####] 100.0% (package (name "python-magic-wormhole") (version "0.11.2") (source (origin (method url-fetch) (uri (pypi-uri "magic-wormhole" version)) (sha256 (base32 "01fr4bi6kc6fz9n3c4qq892inrc3nf6p2djy65yvm7xkvdxncydf")))) (build-system python-build-system) (propagated-inputs `(("python-attrs" ,python-attrs) ("python-autobahn" ,python-autobahn) ("python-automat" ,python-automat) ("python-click" ,python-click) ("python-hkdf" ,python-hkdf) ("python-humanize" ,python-humanize) ("python-pynacl" ,python-pynacl) ("python-pywin32" ,python-pywin32) ("python-six" ,python-six) ("python-spake2" ,python-spake2) ("python-tqdm" ,python-tqdm) ("python-twisted" ,python-twisted) ("python-txtorcon" ,python-txtorcon))) (native-inputs `(("python-magic-wormhole-mailbox-server" ,python-magic-wormhole-mailbox-server) ("python-magic-wormhole-transit-relay" ,python-magic-wormhole-transit-relay) ("python-mock" ,python-mock) ("python-pyflakes" ,python-pyflakes) ("python-tox" ,python-tox))) (home-page "https://github.com/warner/magic-wormhole") (synopsis "Securely transfer data between computers") (description "Securely transfer data between computers") (license license:expat)) =20=20 From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 29 00:24:35 2019 Received: (at 24450) by debbugs.gnu.org; 29 Mar 2019 04:24:35 +0000 Received: from localhost ([127.0.0.1]:34960 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9j4E-0005zH-K0 for submit@debbugs.gnu.org; Fri, 29 Mar 2019 00:24:34 -0400 Received: from mail-io1-f50.google.com ([209.85.166.50]:40671) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9j4C-0005z2-4O for 24450@debbugs.gnu.org; Fri, 29 Mar 2019 00:24:33 -0400 Received: by mail-io1-f50.google.com with SMTP id d201so692341iof.7 for <24450@debbugs.gnu.org>; Thu, 28 Mar 2019 21:24:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=z5dv62qPuuOkTnSSRikJYv2S4fhzUYtvKYib860yK9I=; b=HHi+ve/cJWBCv+TsssBw3lCtzDph0H/VF8X2KNlZWpmHUO5Y0e4CSEbCV7RzgWxOLT 7OpRliS4l26ZGETMPNOIwcPyr4tXK6LKkyfZBpGoDEvOV7iz+w6KOPP2Rzd4Pv44WAnN aqECFwTUAM6g+MUXL36ezBb0WrY0/pT100RIj1nUfUaSd/kCt33w2VrGPT57rtD4AhlW SdptxmtVdnUESmpxGKhBw/YHR6gyY7i2xYMUHoMZXToQTZFiOsO2Flbon4mHrHZlatuF Xfv1Y83BlL2n9xAlKjo6xbrjG3gxTQlL67sTU2GI9tpVzLHRFEE3UkMKs6UWO9fnMO+E cQtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=z5dv62qPuuOkTnSSRikJYv2S4fhzUYtvKYib860yK9I=; b=dpFX7YC38cZKu9GuBZN+ohflmP8jX7pWLUkOoQoNdcIdVbn8rZ8bEJCr1il/v0JKcP f0tnbN0gkZLmMUsN9qG7QOWc+5Tdieucw5WTi4/al3kB82A6uXM7NLYdwPHV+O6NM021 RsJIdcuEIj2n+HC2Cl+vG8s4orHklZTVgBz+wnLjdKqhEjMSqf157P5XhPpFBPJFrgWM o0PHzEpeoMH45BKb6wVlVLsceYf3U260ZNy3KKqJTwl0TC4VPcBDzUSDS0bZrW8zg56k zY2HixYXq5T622e+fBlSTUVFmFaRp2GxgmiI5qDHOsPTyYefj80qViQiEl5K+xtJegtZ q6hA== X-Gm-Message-State: APjAAAU9yRS7ytp6IRfTQmi5UOEAqGJr6QCqo3aGs5W/zvydJrNHbmBk RKGXDOmoiQFJ5qGPAKitTn9L1MjsoK8= X-Google-Smtp-Source: APXvYqxfoaUVGgYoDv+0BlMedX4k+TjozDDJdIpAqyBl6/RwJmQHlY80QQEYKkp2juOXiTAfwY7JwA== X-Received: by 2002:a5e:c019:: with SMTP id u25mr32624713iol.104.1553833466372; Thu, 28 Mar 2019 21:24:26 -0700 (PDT) Received: from kwak ([2607:f2c0:94b4:fa00::235]) by smtp.gmail.com with ESMTPSA id q1sm530329itb.22.2019.03.28.21.24.25 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 28 Mar 2019 21:24:25 -0700 (PDT) From: Maxim Cournoyer To: ng0 Subject: Re: bug#24450: pypi importer outputs strange character series in optional dependency case. References: <87h99fipj1.fsf@we.make.ritual.n0.is> Date: Fri, 29 Mar 2019 00:24:24 -0400 In-Reply-To: <87h99fipj1.fsf@we.make.ritual.n0.is> (ng0's message of "Fri, 16 Sep 2016 20:00:02 +0000") Message-ID: <87imw22tqf.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) ng0 writes: > I think this should not happen with pypi import: > > (inputs > `(("python-certifi=3D=3D2016.2.28" > ,python-certifi=3D=3D2016.2.28) > ("python-dateutil=3D=3D2.5.3" > ,python-dateutil=3D=3D2.5.3) > ("python-flask-babel=3D=3D0.11.1" > ,python-flask-babel=3D=3D0.11.1) > ("python-flask=3D=3D0.11.1" ,python-flask=3D=3D0.11.1) > ("python-lxml=3D=3D3.6.0" ,python-lxml=3D=3D3.6.0) > ("python-ndg-httpsclient=3D=3D0.4.1" > ,python-ndg-httpsclient=3D=3D0.4.1) > ("python-pyasn1-modules=3D=3D0.0.8" > ,python-pyasn1-modules=3D=3D0.0.8) > ("python-pyasn1=3D=3D0.1.9" ,python-pyasn1=3D=3D0.1.9) > ("python-pygments=3D=3D2.1.3" > ,python-pygments=3D=3D2.1.3) > ("python-pyopenssl=3D=3D0.15.1" > ,python-pyopenssl=3D=3D0.15.1) > ("python-pyyaml=3D=3D3.11" ,python-pyyaml=3D=3D3.11) > ("python-requests[socks]=3D=3D2.10.0" > ,#{python-requests\x5b;socks\x5d;=3D=3D2.10.0}#) > ("python-setuptools" ,python-setuptools))) > > > I can understand the version numbers, I can also understand the optional > socks building/module of the python-requests, but why does it read like > Gobbledygook? Can't we improve the output here? > > For version numbers, this is not a format which happened recently which > is exclusive for python build system right? This is just bad formated > because of the pypi query. > I will first try and not pin the application to these version numbers, > maybe itjustworks=E2=84=A2. > > > To reproduce: "guix import pypi searx" This would now give (change to be sent for review soon): --8<---------------cut here---------------start------------->8--- ./pre-inst-env guix import pypi searx Starting download of /tmp/guix-file.1wD8K4 >From https://files.pythonhosted.org/packages/75/3f/5941ad2d500ff7cf6f8da102= 2c78013dcd2207941d533586a8e7bfe699d3/searx-0.15.0.tar.gz... =E2=80=A65.0.tar.gz 1.6MiB 729KiB/s 00:02 [#############= #####] 100.0% (package (name "python-searx") (version "0.15.0") (source (origin (method url-fetch) (uri (pypi-uri "searx" version)) (sha256 (base32 "1gmww73q7wydkvlyz73wnr3sybpjn40wha7avnz9ak9m365zcjxf")))) (build-system python-build-system) (propagated-inputs `(("python-certifi" ,python-certifi) ("python-dateutil" ,python-dateutil) ("python-flask" ,python-flask) ("python-flask-babel" ,python-flask-babel) ("python-idna" ,python-idna) ("python-lxml" ,python-lxml) ("python-pygments" ,python-pygments) ("python-pyopenssl" ,python-pyopenssl) ("python-pyyaml" ,python-pyyaml) ("python-requests" ,python-requests))) (native-inputs `(("python-babel" ,python-babel) ("python-cov-core" ,python-cov-core) ("python-mock" ,python-mock) ("python-nose2" ,python-nose2) ("python-pep8" ,python-pep8) ("python-plone.testing" ,python-plone.testing) ("python-selenium" ,python-selenium) ("python-splinter" ,python-splinter) ("python-transifex-client" ,python-transifex-client) ("python-unittest2" ,python-unittest2) ("python-zope.testrunner" ,python-zope.testrunner))) (home-page "https://github.com/asciimoo/searx") (synopsis "A privacy-respecting, hackable metasearch engine") (description "A privacy-respecting, hackable metasearch engine") (license #f)) --8<---------------cut here---------------end--------------->8--- From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 29 00:35:01 2019 Received: (at 24450) by debbugs.gnu.org; 29 Mar 2019 04:35:01 +0000 Received: from localhost ([127.0.0.1]:34964 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9jEF-0006JX-Ow for submit@debbugs.gnu.org; Fri, 29 Mar 2019 00:35:01 -0400 Received: from mail-io1-f53.google.com ([209.85.166.53]:37937) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1h9jEC-0006JD-1A for 24450@debbugs.gnu.org; Fri, 29 Mar 2019 00:34:54 -0400 Received: by mail-io1-f53.google.com with SMTP id v4so720376ioj.5 for <24450@debbugs.gnu.org>; Thu, 28 Mar 2019 21:34:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=6mzXAgxhQEFz3WJtxhSlcuJcz7t5auGz15CWS+yDmas=; b=sjO4EEMnfZzZaacfxMLXfCUq96B5sSGlNDF7Pj1gu+FWcFD/mvoh92A5+j+KB//8Qw cPs+lZapvEQP6MPZJ2TnsJVCNhr8vwBhkjXuo864u9qu6mVk8/wiYgpTzZfCMEpU7fgD N/Xj4khvTs73BPmS0/ccf/EO9QJ1x5ajb9DIxk1Va+PBRx5DiaQL2vReYOTUV9hOowlN qj1sKKV+dE4vUxDAzYmi+9PKH2v4d9+l/kuOJdKyWdEEt6V6dQwVGQ7wBu9om2NrNPv5 hGSmRvPqICpHVRkUPOnhLnbC8bbzNH+zBUrcUGYHTjeXxWQzyZSpZrk2UG8J0YVUJKx4 9+kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=6mzXAgxhQEFz3WJtxhSlcuJcz7t5auGz15CWS+yDmas=; b=AWf5ShgZXVHYMjmvFYnv0Nbbb8+udWTySLmdi1JT5X/5Wtlc9cIlh3QfVl2V2rYt+I JjRqvgsWqGxP4cVjKw/rxf2Yc+7aXmRbidjx/60tE8XDrSUGCV34fKmEZ0E1cx21UtjX hQxyNsnOg2Ibo5YReNE5imF7rPQlJJHSNkGkLv48MAFMUMvtAdLO3cKt3GvaawxqT0dh l/USxkliXgGWmshu4lfILMWFl38JU1KT4E6PSgTfvhFFzvwGE0bt+VEW96Y6Xm/Lk2Fe Gcu8UKTHmn/jbbK/WuiklhTdJG1F7+tHMoDejkkX/MTynmy4PqxdJeoGGei8cs/9OnFd EQjg== X-Gm-Message-State: APjAAAW6HVgp4l2MqMqtnbaDGAJZGNTFrObX5YAHqhetkiGqNjyLrYcM 7zCorCch5RdsthHriZxWRZI/xFSfhhE= X-Google-Smtp-Source: APXvYqxYBmqHGPCE5cB8PfRj8zcUadcObTJHrelEUvXiycq+I/tDpJdLYTQ9JxUHpeDr8YOtbAymhw== X-Received: by 2002:a6b:4a09:: with SMTP id w9mr31624350iob.79.1553834086245; Thu, 28 Mar 2019 21:34:46 -0700 (PDT) Received: from kwak ([2607:f2c0:94b4:fa00::235]) by smtp.gmail.com with ESMTPSA id y16sm450522ion.64.2019.03.28.21.34.44 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 28 Mar 2019 21:34:45 -0700 (PDT) From: Maxim Cournoyer To: ng0 Subject: [PATCH] bug#24450: pypi importer outputs strange character series in optional dependency case. References: <87h99fipj1.fsf@we.make.ritual.n0.is> Date: Fri, 29 Mar 2019 00:34:43 -0400 In-Reply-To: <87h99fipj1.fsf@we.make.ritual.n0.is> (ng0's message of "Fri, 16 Sep 2016 20:00:02 +0000") Message-ID: <87tvfm1eos.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello, ng0 writes: > I think this should not happen with pypi import: > > (inputs > `(("python-certifi=3D=3D2016.2.28" > ,python-certifi=3D=3D2016.2.28) > ("python-dateutil=3D=3D2.5.3" > ,python-dateutil=3D=3D2.5.3) > ("python-flask-babel=3D=3D0.11.1" > ,python-flask-babel=3D=3D0.11.1) > ("python-flask=3D=3D0.11.1" ,python-flask=3D=3D0.11.1) > ("python-lxml=3D=3D3.6.0" ,python-lxml=3D=3D3.6.0) > ("python-ndg-httpsclient=3D=3D0.4.1" > ,python-ndg-httpsclient=3D=3D0.4.1) > ("python-pyasn1-modules=3D=3D0.0.8" > ,python-pyasn1-modules=3D=3D0.0.8) > ("python-pyasn1=3D=3D0.1.9" ,python-pyasn1=3D=3D0.1.9) > ("python-pygments=3D=3D2.1.3" > ,python-pygments=3D=3D2.1.3) > ("python-pyopenssl=3D=3D0.15.1" > ,python-pyopenssl=3D=3D0.15.1) > ("python-pyyaml=3D=3D3.11" ,python-pyyaml=3D=3D3.11) > ("python-requests[socks]=3D=3D2.10.0" > ,#{python-requests\x5b;socks\x5d;=3D=3D2.10.0}#) > ("python-setuptools" ,python-setuptools))) > > > I can understand the version numbers, I can also understand the optional > socks building/module of the python-requests, but why does it read like > Gobbledygook? Can't we improve the output here? > > For version numbers, this is not a format which happened recently which > is exclusive for python build system right? This is just bad formated > because of the pypi query. > I will first try and not pin the application to these version numbers, > maybe itjustworks=E2=84=A2. > > > To reproduce: "guix import pypi searx" The following patches fix this, and more! --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-import-pypi-Do-not-consider-requirements.txt-files.patch Content-Transfer-Encoding: quoted-printable From=2054e44b7397f17910d95dbdb233d23e5c97c095aa Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:00 -0400 Subject: [PATCH 1/7] import: pypi: Do not consider requirements.txt files. * guix/import/pypi.scm (guess-requirements): Update comment. [guess-requirements-from-source]: Do not attempt to parse the file requirements.txt. Streamline logic. =2D-- guix/import/pypi.scm | 35 +++++++++++++---------------------- tests/pypi.scm | 23 +++++++++++------------ 2 files changed, 24 insertions(+), 34 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 3a20fc4b9b..8269aa61d7 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -206,35 +206,26 @@ cannot determine package dependencies")) (call-with-temporary-directory (lambda (dir) (let* ((pypi-name (string-take dirname (string-rindex dirname= #\-))) =2D (req-files (list (string-append dirname "/requiremen= ts.txt") =2D (string-append dirname "/" pypi-nam= e ".egg-info" =2D "/requires.txt"))) =2D (exit-codes (map (lambda (file-name) =2D (parameterize ((current-error-por= t (%make-void-port "rw+")) =2D (current-output-po= rt (%make-void-port "rw+"))) =2D (system* "tar" "xf" tarball "-C= " dir file-name))) =2D req-files))) =2D ;; Only one of these files needs to exist. =2D (if (any zero? exit-codes) =2D (match (find-files dir) =2D ((file . _) =2D (read-requirements file)) =2D (() =2D (warning (G_ "No requirements file found.\n")))) + (requires.txt (string-append dirname "/" pypi-name + ".egg-info" "/requires.tx= t")) + (exit-code (parameterize ((current-error-port (%make-v= oid-port "rw+")) + (current-output-port (%make-= void-port "rw+"))) + (system* "tar" "xf" tarball "-C" dir requ= ires.txt)))) + (if (zero? exit-code) + (read-requirements (string-append dir "/" requires.txt)) (begin =2D (warning (G_ "Failed to extract requirements files\= n")) + (warning + (G_ "Failed to extract file: ~a from source.~%") + requires.txt) '()))))) '()))) =20 =2D ;; First, try to compute the requirements using the wheel, since that = is the =2D ;; most reliable option. If a wheel is not provided for this package, = try =2D ;; getting them by reading either the "requirements.txt" file or the =2D ;; "requires.txt" from the egg-info directory from the source tarball.= Note =2D ;; that "requirements.txt" is not mandatory, so this is likely to fail. + ;; First, try to compute the requirements using the wheel, else, fallbac= k to + ;; reading the "requires.txt" from the egg-info directory from the source + ;; tarball. (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 =2D (define (compute-inputs source-url wheel-url tarball) "Given the SOURCE-URL of an already downloaded TARBALL, return a list of name/variable pairs describing the required inputs of this package. Also diff --git a/tests/pypi.scm b/tests/pypi.scm index 6daa44a6e7..335be42644 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -23,7 +23,7 @@ #:use-module (gcrypt hash) #:use-module (guix tests) #:use-module (guix build-system python) =2D #:use-module ((guix build utils) #:select (delete-file-recursively whi= ch)) + #:use-module ((guix build utils) #:select (delete-file-recursively which= mkdir-p)) #:use-module (srfi srfi-64) #:use-module (ice-9 match)) =20 @@ -55,11 +55,10 @@ (define test-source-hash "") =20 =2D(define test-requirements =2D"# A comment =2D # A comment after a space +(define test-requires.txt "\ bar =2Dbaz > 13.37") +baz > 13.37 +") =20 (define test-metadata "{ @@ -107,10 +106,10 @@ baz > 13.37") (match url ("https://example.com/foo-1.0.0.tar.gz" (begin =2D (mkdir "foo-1.0.0") =2D (with-output-to-file "foo-1.0.0/requirements.txt" + (mkdir-p "foo-1.0.0/foo.egg-info/") + (with-output-to-file "foo-1.0.0/foo.egg-info/requires.tx= t" (lambda () =2D (display test-requirements))) + (display test-requires.txt))) (system* "tar" "czvf" file-name "foo-1.0.0/") (delete-file-recursively "foo-1.0.0") (set! test-source-hash @@ -157,11 +156,11 @@ baz > 13.37") (lambda (url file-name) (match url ("https://example.com/foo-1.0.0.tar.gz" =2D (begin =2D (mkdir "foo-1.0.0") =2D (with-output-to-file "foo-1.0.0/requirements.txt" + (begin + (mkdir-p "foo-1.0.0/foo.egg-info/") + (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" (lambda () =2D (display test-requirements))) + (display test-requires.txt))) (system* "tar" "czvf" file-name "foo-1.0.0/") (delete-file-recursively "foo-1.0.0") (set! test-source-hash =2D-=20 2.20.1 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0002-import-pypi-Do-not-parse-optional-requirements-from-.patch Content-Transfer-Encoding: quoted-printable From=205f79b0502f62bd1dacc8ea143c1dbd9ef7cfc29d Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:00 -0400 Subject: [PATCH 2/7] import: pypi: Do not parse optional requirements from source. * guix/import/pypi.scm: Export PARSE-REQUIRES.TXT. (guess-requirements): Move the READ-REQUIREMENTS procedure to the top level, and rename it to PARSE-REQUIRES.TXT. Move the CLEAN-REQUIREMENT and COMMEN= T? functions inside the READ-REQUIREMENTS procedure. (parse-requires.txt): Add a SECTION-HEADER? predicate, and use it to prevent parsing optional requirements. * tests/pypi.scm (test-requires-with-sections): New variable. ("parse-requires.txt, with sections"): New test. ("pypi->guix-package"): Mute tar output to stdout. =2D-- guix/import/pypi.scm | 76 +++++++++++++++++++++++++++----------------- tests/pypi.scm | 21 ++++++++++-- 2 files changed, 65 insertions(+), 32 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 8269aa61d7..91e987e9f1 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -47,7 +47,8 @@ #:use-module (guix upstream) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix build-system python) =2D #:export (guix-package->pypi-name + #:export (parse-requires.txt + guix-package->pypi-name pypi-recursive-import pypi->guix-package %pypi-updater)) @@ -117,6 +118,49 @@ package definition." ((package-inputs ...) `((propagated-inputs (,'quasiquote ,package-inputs)))))) =20 +(define (clean-requirement s) + ;; Given a requirement LINE, as can be found in a setuptools requires.txt + ;; file, remove everything other than the actual name of the required + ;; package, and return it. + (string-take s (or (string-index s (lambda (chr) + (member chr '(#\space #\> #\=3D #\<= )))) + (string-length s)))) + +(define (parse-requires.txt requires.txt) + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of +requirement names." + ;; This is a very incomplete parser, which job is to select the non-opti= onal + ;; dependencies and strip them out of any version information. + ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) + ;; library and the requirements grammar defined by PEP-0508 + ;; (https://www.python.org/dev/peps/pep-0508/). + + (define (comment? line) + ;; Return #t if the given LINE is a comment, #f otherwise. + (eq? (string-ref (string-trim line) 0) #\#)) + + (define (section-header? line) + ;; Return #t if the given LINE is a section header, #f otherwise. + (let ((trimmed-line (string-trim line))) + (and (not (string-null? trimmed-line)) + (eq? (string-ref trimmed-line 0) #\[)))) + + (call-with-input-file requires.txt + (lambda (port) + (let loop ((result '())) + (let ((line (read-line port))) + ;; Stop when a section is encountered, as sections contains opti= onal + ;; (extra) requirements. Non-optional requirements must appear + ;; before any section is defined. + (if (or (eof-object? line) (section-header? line)) + (reverse result) + (cond + ((or (string-null? line) (comment? line)) + (loop result)) + (else + (loop (cons (clean-requirement line) + result)))))))))) + (define (guess-requirements source-url wheel-url tarball) "Given SOURCE-URL, WHEEL-URL and a TARBALL of the package, return a list of the required packages specified in the requirements.txt file. TARBALL = will @@ -139,34 +183,6 @@ be extracted in a temporary directory." cannot determine package dependencies")) #f))))) =20 =2D (define (clean-requirement s) =2D ;; Given a requirement LINE, as can be found in a Python requirement= s.txt =2D ;; file, remove everything other than the actual name of the required =2D ;; package, and return it. =2D (string-take s =2D (or (string-index s (lambda (chr) (member chr '(#\space #\> #\=3D = #\<)))) =2D (string-length s)))) =2D =2D (define (comment? line) =2D ;; Return #t if the given LINE is a comment, #f otherwise. =2D (eq? (string-ref (string-trim line) 0) #\#)) =2D =2D (define (read-requirements requirements-file) =2D ;; Given REQUIREMENTS-FILE, a Python requirements.txt file, return a= list =2D ;; of name/variable pairs describing the requirements. =2D (call-with-input-file requirements-file =2D (lambda (port) =2D (let loop ((result '())) =2D (let ((line (read-line port))) =2D (if (eof-object? line) =2D result =2D (cond =2D ((or (string-null? line) (comment? line)) =2D (loop result)) =2D (else =2D (loop (cons (clean-requirement line) =2D result)))))))))) =2D (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's ;; requirements. @@ -212,7 +228,7 @@ cannot determine package dependencies")) (current-output-port (%make-= void-port "rw+"))) (system* "tar" "xf" tarball "-C" dir requ= ires.txt)))) (if (zero? exit-code) =2D (read-requirements (string-append dir "/" requires.tx= t)) + (parse-requires.txt (string-append dir "/" requires.txt= )) (begin (warning (G_ "Failed to extract file: ~a from source.~%") diff --git a/tests/pypi.scm b/tests/pypi.scm index 335be42644..e4b7142311 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -60,6 +60,15 @@ bar baz > 13.37 ") =20 +(define test-requires-with-sections "\ +# A comment +foo ~=3D 3 +bar !=3D 2 + +[test] +pytest (>=3D2.5.0) +") + (define test-metadata "{ \"run_requires\": [ @@ -99,6 +108,12 @@ baz > 13.37 (uri (list "https://bitheap.org/cram/cram-0.7.tar.gz" (pypi-uri "cram" "0.7")))))))) =20 +(test-equal "parse-requires.txt, with sections" + '("foo" "bar") + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-requires.txt test-requires-with-sections))) + (test-assert "pypi->guix-package" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch @@ -110,7 +125,8 @@ baz > 13.37 (with-output-to-file "foo-1.0.0/foo.egg-info/requires.tx= t" (lambda () (display test-requires.txt))) =2D (system* "tar" "czvf" file-name "foo-1.0.0/") + (parameterize ((current-output-port (%make-void-port "rw= +"))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) (delete-file-recursively "foo-1.0.0") (set! test-source-hash (call-with-input-file file-name port-sha256)))) @@ -161,7 +177,8 @@ baz > 13.37 (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" (lambda () (display test-requires.txt))) =2D (system* "tar" "czvf" file-name "foo-1.0.0/") + (parameterize ((current-output-port (%make-void-port "rw+"= ))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) (delete-file-recursively "foo-1.0.0") (set! test-source-hash (call-with-input-file file-name port-sha256)))) =2D-=20 2.20.1 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0003-import-pypi-Improve-parsing-of-requirement-specifica.patch Content-Transfer-Encoding: quoted-printable From=200c62b541a3e8925b5ca31fe55dbe7536cf95151f Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:01 -0400 Subject: [PATCH 3/7] import: pypi: Improve parsing of requirement specifications. The previous solution was fragile and could leave unwanted characters in a requirement name, such as '[' or ']'. Partially fixes issue #33047 (see: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D33047). * guix/import/pypi.scm (use-modules): Export SPECIFICATION->REQUIREMENT-NAME (%requirement-name-regexp): New variable. (clean-requirement): Rename to... (specification->requirement-name): this, which now uses %requirement-name-regexp to select the requirement name from the requirement specification. (parse-requires.txt): Adapt. =2D-- guix/import/pypi.scm | 43 ++++++++++++++++++++++++++++++++++--------- tests/pypi.scm | 12 ++++++++++++ 2 files changed, 46 insertions(+), 9 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 91e987e9f1..efb5939c78 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -48,6 +48,7 @@ #:use-module ((guix licenses) #:prefix license:) #:use-module (guix build-system python) #:export (parse-requires.txt + specification->requirement-name guix-package->pypi-name pypi-recursive-import pypi->guix-package @@ -118,13 +119,37 @@ package definition." ((package-inputs ...) `((propagated-inputs (,'quasiquote ,package-inputs)))))) =20 =2D(define (clean-requirement s) =2D ;; Given a requirement LINE, as can be found in a setuptools requires.= txt =2D ;; file, remove everything other than the actual name of the required =2D ;; package, and return it. =2D (string-take s (or (string-index s (lambda (chr) =2D (member chr '(#\space #\> #\=3D #= \<)))) =2D (string-length s)))) +(define %requirement-name-regexp + ;; Regexp to match the requirement name in a requirement specification. + + ;; Some grammar, taken from PEP-0508 (see: + ;; https://www.python.org/dev/peps/pep-0508/). + + ;; The unified rule can be expressed as: + ;; specification =3D wsp* ( url_req | name_req ) wsp* + + ;; where url_req is: + ;; url_req =3D name wsp* extras? wsp* urlspec wsp+ quoted_marker? + + ;; and where name_req is: + ;; name_req =3D name wsp* extras? wsp* versionspec? wsp* quoted_marker? + + ;; Thus, we need only matching NAME, which is expressed as: + ;; identifer_end =3D letterOrDigit | (('-' | '_' | '.' )* letterOrDigit) + ;; identifier =3D letterOrDigit identifier_end* + ;; name =3D identifier + (let* ((letter-or-digit "[A-Za-z0-9]") + (identifier-end (string-append "(" letter-or-digit "|" + "[-_.]*" letter-or-digit ")")) + (identifier (string-append "^" letter-or-digit identifier-end "*"= )) + (name identifier)) + (make-regexp name))) + +(define (specification->requirement-name spec) + "Given a specification SPEC, return the requirement name." + (match:substring + (or (regexp-exec %requirement-name-regexp spec) + (error (G_ "Could not extract requirement name in spec:") spec)))) =20 (define (parse-requires.txt requires.txt) "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of @@ -158,7 +183,7 @@ requirement names." ((or (string-null? line) (comment? line)) (loop result)) (else =2D (loop (cons (clean-requirement line) + (loop (cons (specification->requirement-name line) result)))))))))) =20 (define (guess-requirements source-url wheel-url tarball) @@ -200,7 +225,7 @@ cannot determine package dependencies")) (hash-ref (list-ref run_requir= es 0) "requires") '()))) =2D (map clean-requirement requirements))))) + (map specification->requirement-name requirements))))) (lambda () (delete-file json-file) (rmdir dirname)))))) diff --git a/tests/pypi.scm b/tests/pypi.scm index e4b7142311..82d6bba8dd 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -55,6 +55,14 @@ (define test-source-hash "") =20 +(define test-specifications + '("Fizzy [foo, bar]" + "PickyThing<1.6,>1.9,!=3D1.9.6,<2.0a0,=3D=3D2.4c1" + "SomethingWithMarker[foo]>1.0;python_version<\"2.7\"" + "requests [security,tests] >=3D 2.8.1, =3D=3D 2.8.* ; python_version <= \"2.7\"" + "pip @ https://github.com/pypa/pip/archive/1.3.1.zip#\ +sha1=3Dda9234ee9982d4bbb3c72346a6de940a148ea686")) + (define test-requires.txt "\ bar baz > 13.37 @@ -108,6 +116,10 @@ pytest (>=3D2.5.0) (uri (list "https://bitheap.org/cram/cram-0.7.tar.gz" (pypi-uri "cram" "0.7")))))))) =20 +(test-equal "specification->requirement-name" + '("Fizzy" "PickyThing" "SomethingWithMarker" "requests" "pip") + (map specification->requirement-name test-specifications)) + (test-equal "parse-requires.txt, with sections" '("foo" "bar") (mock ((ice-9 ports) call-with-input-file =2D-=20 2.20.1 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0004-import-pypi-Deduplicate-requirements.patch Content-Transfer-Encoding: quoted-printable From=2076e4a3150f8126e0b952c6129b6e1371afba80c0 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:01 -0400 Subject: [PATCH 4/7] import: pypi: Deduplicate requirements. * guix/import/pypi.scm (parse-requires.txt): Remove potential duplicates. =2D-- guix/import/pypi.scm | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index efb5939c78..a90be67bb0 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -178,7 +178,11 @@ requirement names." ;; (extra) requirements. Non-optional requirements must appear ;; before any section is defined. (if (or (eof-object? line) (section-header? line)) =2D (reverse result) + ;; Duplicates can occur, since the same requirement can be + ;; listed multiple times with different conditional markers,= e.g. + ;; pytest >=3D 3 ; python_version >=3D "3.3" + ;; pytest < 3 ; python_version < "3.3" + (reverse (delete-duplicates result)) (cond ((or (string-null? line) (comment? line)) (loop result)) =2D-=20 2.20.1 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0005-import-pypi-Support-more-types-of-archives.patch Content-Transfer-Encoding: quoted-printable From=2073e27235cac1275ba7671fd2364325cf5788cb3c Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:02 -0400 Subject: [PATCH 5/7] import: pypi: Support more types of archives. This change enables the PyPI importer to look for requirements in a source archive of a different type than "tar.gz" or "tar.bz2". * guix/import/pypi.scm: (guess-requirements)[tarball-directory]: Rename to.= .. [archive-root-directory]: this. Use COMPRESSED-FILED? to determine if an archive is supported or not. [guess-requirements-from-source]: Adapt to use the new method, and use unzip to extract ZIP archives. (guess-requirements): Rename the TARBALL argument to ARCHIVE, to denote the archive format is no longer bound specifically to the Tar format. =2D-- guix/import/pypi.scm | 47 ++++++++++++++++++++++---------------------- 1 file changed, 24 insertions(+), 23 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index a90be67bb0..8e93653717 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -190,27 +190,24 @@ requirement names." (loop (cons (specification->requirement-name line) result)))))))))) =20 =2D(define (guess-requirements source-url wheel-url tarball) =2D "Given SOURCE-URL, WHEEL-URL and a TARBALL of the package, return a li= st =2Dof the required packages specified in the requirements.txt file. TARBAL= L will +(define (guess-requirements source-url wheel-url archive) + "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a list +of the required packages specified in the requirements.txt file. ARCHIVE = will be extracted in a temporary directory." =20 =2D (define (tarball-directory url) =2D ;; Given the URL of the package's tarball, return the name of the di= rectory + (define (archive-root-directory url) + ;; Given the URL of the package's archive, return the name of the dire= ctory ;; that will be created upon decompressing it. If the filetype is not ;; supported, return #f. =2D ;; TODO: Support more archive formats. =2D (let ((basename (substring url (+ 1 (string-rindex url #\/))))) =2D (cond =2D ((string-suffix? ".tar.gz" basename) =2D (string-drop-right basename 7)) =2D ((string-suffix? ".tar.bz2" basename) =2D (string-drop-right basename 8)) =2D (else + (if (compressed-file? url) + (let ((root-directory (file-sans-extension (basename url)))) + (if (string=3D? "tar" (file-extension root-directory)) + (file-sans-extension root-directory) + root-directory)) (begin =2D (warning (G_ "Unsupported archive format: \ =2Dcannot determine package dependencies")) =2D #f))))) + (warning (G_ "Unsupported archive format (~a): \ +cannot determine package dependencies") (file-extension url)) + #f))) =20 (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's @@ -246,16 +243,20 @@ cannot determine package dependencies")) =20 (define (guess-requirements-from-source) ;; Return the package's requirements by guessing them from the source. =2D (let ((dirname (tarball-directory source-url))) + (let ((dirname (archive-root-directory source-url)) + (extension (file-extension source-url))) (if (string? dirname) (call-with-temporary-directory (lambda (dir) (let* ((pypi-name (string-take dirname (string-rindex dirname= #\-))) (requires.txt (string-append dirname "/" pypi-name ".egg-info" "/requires.tx= t")) =2D (exit-code (parameterize ((current-error-port (%make= -void-port "rw+")) =2D (current-output-port (%mak= e-void-port "rw+"))) =2D (system* "tar" "xf" tarball "-C" dir re= quires.txt)))) + (exit-code + (parameterize ((current-error-port (%make-void-port "= rw+")) + (current-output-port (%make-void-port = "rw+"))) + (if (string=3D? "zip" extension) + (system* "unzip" archive "-d" dir requires.txt) + (system* "tar" "xf" archive "-C" dir requires.t= xt))))) (if (zero? exit-code) (parse-requires.txt (string-append dir "/" requires.txt= )) (begin @@ -271,13 +272,13 @@ cannot determine package dependencies")) (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 =2D(define (compute-inputs source-url wheel-url tarball) =2D "Given the SOURCE-URL of an already downloaded TARBALL, return a list = of +(define (compute-inputs source-url wheel-url archive) + "Given the SOURCE-URL of an already downloaded ARCHIVE, return a list of name/variable pairs describing the required inputs of this package. Also return the unaltered list of upstream dependency names." (let ((dependencies (remove (cut string=3D? "argparse" <>) =2D (guess-requirements source-url wheel-url tarball)))) + (guess-requirements source-url wheel-url archive)))) (values (sort (map (lambda (input) (let ((guix-name (python->package-name input))) =2D-=20 2.20.1 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0006-import-pypi-Parse-wheel-METADATA-instead-of-metadata.patch Content-Transfer-Encoding: quoted-printable From=20fb0547ef225103c0f8355a7eccc41e0d028f6563 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:03 -0400 Subject: [PATCH 6/7] import: pypi: Parse wheel METADATA instead of metadata.json. With newer Wheel releases, there is no more metadata.json file; the METADATA file should be used instead (see: https://github.com/pypa/wheel/issues/195). This change updates our PyPI importer so that it uses the later. * guix/import/pypi.scm (define-module): Remove unnecessary modules and expo= rt the PARSE-WHEEL-METADATA method. (parse-wheel-metadata): Add method. (guess-requirements): Use it. * tests/pypi.scm (test-metadata): Test it. =2D-- guix/import/pypi.scm | 66 +++++++++++++++++++++++++++++--------------- tests/pypi.scm | 60 ++++++++++++++++++++++++++++++---------- 2 files changed, 89 insertions(+), 37 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 8e93653717..c520213b6a 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -21,9 +21,7 @@ ;;; along with GNU Guix. If not, see . =20 (define-module (guix import pypi) =2D #:use-module (ice-9 binary-ports) #:use-module (ice-9 match) =2D #:use-module (ice-9 pretty-print) #:use-module (ice-9 regex) #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) @@ -31,9 +29,6 @@ #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) =2D #:use-module (rnrs bytevectors) =2D #:use-module (json) =2D #:use-module (web uri) #:use-module (guix ui) #:use-module (guix utils) #:use-module ((guix build utils) @@ -48,6 +43,7 @@ #:use-module ((guix licenses) #:prefix license:) #:use-module (guix build-system python) #:export (parse-requires.txt + parse-wheel-metadata specification->requirement-name guix-package->pypi-name pypi-recursive-import @@ -190,6 +186,37 @@ requirement names." (loop (cons (specification->requirement-name line) result)))))))))) =20 +(define (parse-wheel-metadata metadata) + "Given METADATA, a Wheel metadata file, return a list of requirement nam= es." + ;; METADATA is a RFC-2822-like, header based file. + + (define (requires-dist-header? line) + ;; Return #t if the given LINE is a Requires-Dist header. + (regexp-match? (string-match "^Requires-Dist: " line))) + + (define (requires-dist-value line) + (string-drop line (string-length "Requires-Dist: "))) + + (define (extra? line) + ;; Return #t if the given LINE is an "extra" requirement. + (regexp-match? (string-match "extra =3D=3D " line))) + + (call-with-input-file metadata + (lambda (port) + (let loop ((requirements '())) + (let ((line (read-line port))) + ;; Stop at the first 'Provides-Extra' section: the non-optional + ;; requirements appear before the optional ones. + (if (eof-object? line) + (reverse (delete-duplicates requirements)) + (cond + ((and (requires-dist-header? line) (not (extra? line))) + (loop (cons (specification->requirement-name + (requires-dist-value line)) + requirements))) + (else + (loop requirements))))))))) + (define (guess-requirements source-url wheel-url archive) "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a list of the required packages specified in the requirements.txt file. ARCHIVE = will @@ -211,25 +238,18 @@ cannot determine package dependencies") (file-extensi= on url)) =20 (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's =2D ;; requirements. + ;; requirements, or #f if the metadata file contained therein couldn't= be + ;; extracted. (let* ((dirname (wheel-url->extracted-directory wheel-url)) =2D (json-file (string-append dirname "/metadata.json"))) =2D (and (zero? (system* "unzip" "-q" wheel-archive json-file)) =2D (dynamic-wind =2D (const #t) =2D (lambda () =2D (call-with-input-file json-file =2D (lambda (port) =2D (let* ((metadata (json->scm port)) =2D (run_requires (hash-ref metadata "run_requires= ")) =2D (requirements (if run_requires =2D (hash-ref (list-ref run_requ= ires 0) =2D "requires") =2D '()))) =2D (map specification->requirement-name requirements))= ))) =2D (lambda () =2D (delete-file json-file) =2D (rmdir dirname)))))) + (metadata (string-append dirname "/METADATA"))) + (call-with-temporary-directory + (lambda (dir) + (if (zero? (system* "unzip" "-q" wheel-archive "-d" dir metadata)) + (parse-wheel-metadata (string-append dir "/" metadata)) + (begin + (warning + (G_ "Failed to extract file: ~a from wheel.~%") metadata) + #f)))))) =20 (define (guess-requirements-from-wheel) ;; Return the package's requirements using the wheel, or #f if an error diff --git a/tests/pypi.scm b/tests/pypi.scm index 82d6bba8dd..ca8cb5f6de 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -21,6 +21,7 @@ #:use-module (guix import pypi) #:use-module (guix base32) #:use-module (gcrypt hash) + #:use-module (guix memoization) #:use-module (guix tests) #:use-module (guix build-system python) #:use-module ((guix build utils) #:select (delete-file-recursively which= mkdir-p)) @@ -77,17 +78,33 @@ bar !=3D 2 pytest (>=3D2.5.0) ") =20 =2D(define test-metadata =2D "{ =2D \"run_requires\": [ =2D { =2D \"requires\": [ =2D \"bar\", =2D \"baz (>13.37)\" =2D ] =2D } =2D ] =2D}") +(define test-metadata "\ +Classifier: Programming Language :: Python :: 3.7 +Requires-Dist: baz ~=3D 3 +Requires-Dist: bar !=3D 2 +Provides-Extra: test +pytest (>=3D2.5.0) +") + +(define test-metadata-with-extras " +Classifier: Programming Language :: Python :: 3.7 +Requires-Python: >=3D2.7, !=3D3.0.*, !=3D3.1.*, !=3D3.2.*, !=3D3.3.* +Requires-Dist: wrapt (<2,>=3D1) +Requires-Dist: bar + +Provides-Extra: dev +Requires-Dist: tox ; extra =3D=3D 'dev' +Requires-Dist: bumpversion (<1) ; extra =3D=3D 'dev' +") + +;;; Provides-Extra can appear before Requires-Dist. +(define test-metadata-with-extras-jedi "\ +Requires-Python: >=3D2.7, !=3D3.0.*, !=3D3.1.*, !=3D3.2.*, !=3D3.3.* +Provides-Extra: testing +Requires-Dist: parso (>=3D0.3.0) +Provides-Extra: testing +Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testing' +") =20 (test-begin "pypi") =20 @@ -126,6 +143,18 @@ pytest (>=3D2.5.0) call-with-input-string) (parse-requires.txt test-requires-with-sections))) =20 +(test-equal "parse-wheel-metadata, with extras" + '("wrapt" "bar") + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-wheel-metadata test-metadata-with-extras))) + +(test-equal "parse-wheel-metadata, with extras - Jedi" + '("parso") + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-wheel-metadata test-metadata-with-extras-jedi))) + (test-assert "pypi->guix-package" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch @@ -188,7 +217,7 @@ pytest (>=3D2.5.0) (mkdir-p "foo-1.0.0/foo.egg-info/") (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" (lambda () =2D (display test-requires.txt))) + (display "wrong data to make sure we're testing wheel= s "))) (parameterize ((current-output-port (%make-void-port "rw+"= ))) (system* "tar" "czvf" file-name "foo-1.0.0/")) (delete-file-recursively "foo-1.0.0") @@ -197,13 +226,13 @@ pytest (>=3D2.5.0) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" (begin (mkdir "foo-1.0.0.dist-info") =2D (with-output-to-file "foo-1.0.0.dist-info/metadata.json" + (with-output-to-file "foo-1.0.0.dist-info/METADATA" (lambda () (display test-metadata))) (let ((zip-file (string-append file-name ".zip"))) ;; zip always adds a "zip" extension to the file it cre= ates, ;; so we need to rename it. =2D (system* "zip" zip-file "foo-1.0.0.dist-info/metadata= .json") + (system* "zip" zip-file "foo-1.0.0.dist-info/METADATA") (rename-file zip-file file-name)) (delete-file-recursively "foo-1.0.0.dist-info"))) (_ (error "Unexpected URL: " url))))) @@ -215,6 +244,9 @@ pytest (>=3D2.5.0) (string-length test-json))) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #= f) (_ (error "Unexpected URL: " url))))) + ;; Not clearing the memoization cache here would mean return= ing the value + ;; computed in the previous test. + (invalidate-memoization! pypi->guix-package) (match (pypi->guix-package "foo") (('package ('name "python-foo") =2D-=20 2.20.1 --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: attachment; filename=0007-import-pypi-Include-optional-test-inputs-as-native-i.patch Content-Transfer-Encoding: quoted-printable From=20ea0f24eb7b19c57ebb24ec48ba776b240bccfc99 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 23:12:26 -0400 Subject: [PATCH 7/7] import: pypi: Include optional test inputs as native-inputs. * guix/import/pypi.scm (maybe-inputs): Add INPUT-TYPE argument, and use it. (test-section?): New predicate. (parse-requires.txt): Collect the optional test inputs, and return them as = the second element of the returned list. (parse-wheel-metadata): Likewise. (guess-requirements): Adapt, and hide unzip output. (make-pypi-sexp): Likewise, and include the test inputs requirements as nat= ive inputs in the returned package expression. * tests/pypi.scm (test-requires.txt): Include a test section in the test-requires.txt data. (test-requires.txt-beaker): New variable. ("parse-requires.txt"): Adapt. ("parse-requires.txt - Beaker"): New test. ("parse-wheel-metadata, with extras"): Adapt. ("parse-wheel-metadata, with extras - Jedi"): Adapt. ("pypi->guix-package, no wheel"): Re-indent, and add the expected native-inputs. ("pypi->guix-package, wheels"): Likewise. ("pypi->guix-package, no usable requirement file."): New test. =2D-- guix/import/pypi.scm | 158 ++++++++++++++++++++++++++++--------------- tests/pypi.scm | 123 +++++++++++++++++++++++++-------- 2 files changed, 199 insertions(+), 82 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index c520213b6a..f84ad88e44 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -4,6 +4,7 @@ ;;; Copyright =C2=A9 2015, 2016, 2017 Ludovic Court=C3=A8s ;;; Copyright =C2=A9 2017 Mathieu Othacehe ;;; Copyright =C2=A9 2018 Ricardo Wurmus +;;; Copyright =C2=A9 2019 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -26,6 +27,7 @@ #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) @@ -106,14 +108,15 @@ package on PyPI." ((name version _ ...) (string-append name "-" version ".dist-info")))) =20 =2D(define (maybe-inputs package-inputs) +(define (maybe-inputs package-inputs input-type) "Given a list of PACKAGE-INPUTS, tries to generate the 'inputs' field of= a =2Dpackage definition." +package definition. INPUT-TYPE, a symbol, is used to populate the name of +the input field." (match package-inputs (() '()) ((package-inputs ...) =2D `((propagated-inputs (,'quasiquote ,package-inputs)))))) + `((,input-type (,'quasiquote ,package-inputs)))))) =20 (define %requirement-name-regexp ;; Regexp to match the requirement name in a requirement specification. @@ -147,11 +150,21 @@ package definition." (or (regexp-exec %requirement-name-regexp spec) (error (G_ "Could not extract requirement name in spec:") spec)))) =20 +(define (test-section? name) + "Return #t if the section name contains 'test' or 'dev'." + (any (cut string-contains-ci name <>) + '("test" "dev"))) + (define (parse-requires.txt requires.txt) =2D "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of =2Drequirement names." =2D ;; This is a very incomplete parser, which job is to select the non-op= tional =2D ;; dependencies and strip them out of any version information. + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a pair of re= quirements. + +The first element of the pair contains the required dependencies while the +second the optional test dependencies. Note that currently, optional, +non-test dependencies are omitted since these can be difficult or expensiv= e to +satisfy." + + ;; This is a very incomplete parser, which job is to read in the require= ment + ;; specification lines, and strip them out of any version information. ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) ;; library and the requirements grammar defined by PEP-0508 ;; (https://www.python.org/dev/peps/pep-0508/). @@ -168,57 +181,89 @@ requirement names." =20 (call-with-input-file requires.txt (lambda (port) =2D (let loop ((result '())) + (let loop ((required-deps '()) + (test-deps '()) + (inside-test-section? #f) + (optional? #f)) (let ((line (read-line port))) =2D ;; Stop when a section is encountered, as sections contains op= tional =2D ;; (extra) requirements. Non-optional requirements must appear =2D ;; before any section is defined. =2D (if (or (eof-object? line) (section-header? line)) + (if (eof-object? line) ;; Duplicates can occur, since the same requirement can be ;; listed multiple times with different conditional markers,= e.g. ;; pytest >=3D 3 ; python_version >=3D "3.3" ;; pytest < 3 ; python_version < "3.3" =2D (reverse (delete-duplicates result)) + (map (compose reverse delete-duplicates) + (list required-deps test-deps)) (cond ((or (string-null? line) (comment? line)) =2D (loop result)) =2D (else + (loop required-deps test-deps inside-test-section? optiona= l?)) + ((section-header? line) + ;; Encountering a section means that all the requirements + ;; listed below are optional. Since we want to pick only t= he + ;; test dependencies from the optional dependencies, we mu= st + ;; track those separately. + (loop required-deps test-deps (test-section? line) #t)) + (inside-test-section? + (loop required-deps + (cons (specification->requirement-name line) + test-deps) + inside-test-section? optional?)) + ((not optional?) (loop (cons (specification->requirement-name line) =2D result)))))))))) + required-deps) + test-deps inside-test-section? optional?)) + (optional? + ;; Skip optional items. + (loop required-deps test-deps inside-test-section? optiona= l?)) + (else + (warning (G_ "parse-requires.txt reached an unexpected \ +condition on line ~a~%") line))))))))) =20 (define (parse-wheel-metadata metadata) =2D "Given METADATA, a Wheel metadata file, return a list of requirement n= ames." + "Given METADATA, a Wheel metadata file, return a pair of requirements. + +The first element of the pair contains the required dependencies while the= second the optional +test dependencies. Note that currently, optional, non-test dependencies a= re +omitted since these can be difficult or expensive to satisfy." ;; METADATA is a RFC-2822-like, header based file. =20 (define (requires-dist-header? line) ;; Return #t if the given LINE is a Requires-Dist header. =2D (regexp-match? (string-match "^Requires-Dist: " line))) + (string-match "^Requires-Dist: " line)) =20 (define (requires-dist-value line) (string-drop line (string-length "Requires-Dist: "))) =20 (define (extra? line) ;; Return #t if the given LINE is an "extra" requirement. =2D (regexp-match? (string-match "extra =3D=3D " line))) + (string-match "extra =3D=3D '(.*)'" line)) + + (define (test-requirement? line) + (let ((extra-label (match:substring (extra? line) 1))) + (and extra-label (test-section? extra-label)))) =20 (call-with-input-file metadata (lambda (port) =2D (let loop ((requirements '())) + (let loop ((required-deps '()) + (test-deps '())) (let ((line (read-line port))) =2D ;; Stop at the first 'Provides-Extra' section: the non-optional =2D ;; requirements appear before the optional ones. (if (eof-object? line) =2D (reverse (delete-duplicates requirements)) + (map (compose reverse delete-duplicates) + (list required-deps test-deps)) (cond ((and (requires-dist-header? line) (not (extra? line))) (loop (cons (specification->requirement-name (requires-dist-value line)) =2D requirements))) + required-deps) + test-deps)) + ((and (requires-dist-header? line) (test-requirement? line)) + (loop required-deps + (cons (specification->requirement-name (requires-dis= t-value line)) + test-deps))) (else =2D (loop requirements))))))))) + (loop required-deps test-deps))))))))) ;skip line =20 (define (guess-requirements source-url wheel-url archive) =2D "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a li= st + "Given SOURCE-URL, WHEEL-URL and an ARCHIVE of the package, return a list of the required packages specified in the requirements.txt file. ARCHIVE = will be extracted in a temporary directory." =20 @@ -244,7 +289,10 @@ cannot determine package dependencies") (file-extensio= n url)) (metadata (string-append dirname "/METADATA"))) (call-with-temporary-directory (lambda (dir) =2D (if (zero? (system* "unzip" "-q" wheel-archive "-d" dir metadat= a)) + (if (zero? + (parameterize ((current-error-port (%make-void-port "rw+")) + (current-output-port (%make-void-port "rw+"))) + (system* "unzip" wheel-archive "-d" dir metadata))) (parse-wheel-metadata (string-append dir "/" metadata)) (begin (warning @@ -283,32 +331,38 @@ cannot determine package dependencies") (file-extensi= on url)) (warning (G_ "Failed to extract file: ~a from source.~%") requires.txt) =2D '()))))) =2D '()))) + (list '() '())))))) + (list '() '())))) =20 ;; First, try to compute the requirements using the wheel, else, fallbac= k to ;; reading the "requires.txt" from the egg-info directory from the source =2D ;; tarball. + ;; archive. (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 (define (compute-inputs source-url wheel-url archive) =2D "Given the SOURCE-URL of an already downloaded ARCHIVE, return a list = of =2Dname/variable pairs describing the required inputs of this package. Also =2Dreturn the unaltered list of upstream dependency names." =2D (let ((dependencies =2D (remove (cut string=3D? "argparse" <>) =2D (guess-requirements source-url wheel-url archive)))) =2D (values (sort =2D (map (lambda (input) =2D (let ((guix-name (python->package-name input))) =2D (list guix-name (list 'unquote (string->symbol gui= x-name))))) =2D dependencies) =2D (lambda args =2D (match args =2D (((a _ ...) (b _ ...)) =2D (string-ci) deps)) + + (define (requirement->package-name/sort deps) + (sort + (map (lambda (input) + (let ((guix-name (python->package-name input))) + (list guix-name (list 'unquote (string->symbol guix-name))))) + deps) + (lambda args + (match args + (((a _ ...) (b _ ...)) + (string-cipackage-name/sort strip-argparse)) + + (map process-requirements (guess-requirements source-url wheel-url archi= ve))) =20 (define (make-pypi-sexp name version source-url wheel-url home-page synops= is description license) @@ -317,15 +371,13 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION= , and LICENSE." (call-with-temporary-output-file (lambda (temp port) (and (url-fetch source-url temp) =2D (receive (input-package-names upstream-dependency-names) =2D (compute-inputs source-url wheel-url temp) =2D (values + (match (compute-inputs source-url wheel-url temp) + ((required-inputs test-inputs) `(package (name ,(python->package-name name)) (version ,version) (source (origin (method url-fetch) =2D ;; Sometimes 'pypi-uri' doesn't quite work due t= o mixed ;; cases in NAME, for instance, as is the case w= ith ;; "uwsgi". In that case, fall back to a full U= RL. @@ -334,12 +386,12 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION= , and LICENSE." (base32 ,(guix-hash-url temp))))) (build-system python-build-system) =2D ,@(maybe-inputs input-package-names) + ,@(maybe-inputs required-inputs 'propagated-inputs) + ,@(maybe-inputs test-inputs 'native-inputs) (home-page ,home-page) (synopsis ,synopsis) (description ,description) =2D (license ,(license->symbol license))) =2D upstream-dependency-names)))))) + (license ,(license->symbol license))))))))) =20 (define pypi->guix-package (memoize diff --git a/tests/pypi.scm b/tests/pypi.scm index ca8cb5f6de..aa08e2cb54 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2014 David Thompson ;;; Copyright =C2=A9 2016 Ricardo Wurmus +;;; Copyright =C2=A9 2019 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -65,11 +66,6 @@ sha1=3Dda9234ee9982d4bbb3c72346a6de940a148ea686")) =20 (define test-requires.txt "\ =2Dbar =2Dbaz > 13.37 =2D") =2D =2D(define test-requires-with-sections "\ # A comment foo ~=3D 3 bar !=3D 2 @@ -78,12 +74,25 @@ bar !=3D 2 pytest (>=3D2.5.0) ") =20 +;; Beaker contains only optional dependencies. +(define test-requires.txt-beaker "\ +[crypto] +pycryptopp>=3D0.5.12 + +[cryptography] +cryptography + +[testsuite] +Mock +coverage +") + (define test-metadata "\ Classifier: Programming Language :: Python :: 3.7 Requires-Dist: baz ~=3D 3 Requires-Dist: bar !=3D 2 Provides-Extra: test =2Dpytest (>=3D2.5.0) +Requires-Dist: pytest (>=3D2.5.0) ; extra =3D=3D 'test' ") =20 (define test-metadata-with-extras " @@ -137,25 +146,31 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'test= ing' '("Fizzy" "PickyThing" "SomethingWithMarker" "requests" "pip") (map specification->requirement-name test-specifications)) =20 =2D(test-equal "parse-requires.txt, with sections" =2D '("foo" "bar") +(test-equal "parse-requires.txt" + (list '("foo" "bar") '("pytest")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) =2D (parse-requires.txt test-requires-with-sections))) + (parse-requires.txt test-requires.txt))) + +(test-equal "parse-requires.txt - Beaker" + (list '() '("Mock" "coverage")) + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-requires.txt test-requires.txt-beaker))) =20 (test-equal "parse-wheel-metadata, with extras" =2D '("wrapt" "bar") + (list '("wrapt" "bar") '("tox" "bumpversion")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) (parse-wheel-metadata test-metadata-with-extras))) =20 (test-equal "parse-wheel-metadata, with extras - Jedi" =2D '("parso") + (list '("parso") '("pytest")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) (parse-wheel-metadata test-metadata-with-extras-jedi))) =20 =2D(test-assert "pypi->guix-package" +(test-assert "pypi->guix-package, no wheel" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch (lambda (url file-name) @@ -195,7 +210,10 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testi= ng' ('propagated-inputs ('quasiquote (("python-bar" ('unquote 'python-bar)) =2D ("python-baz" ('unquote 'python-baz))))) + ("python-foo" ('unquote 'python-foo))))) + ('native-inputs + ('quasiquote + (("python-pytest" ('unquote 'python-pytest))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary") @@ -216,25 +234,25 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'test= ing' (begin (mkdir-p "foo-1.0.0/foo.egg-info/") (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" =2D (lambda () =2D (display "wrong data to make sure we're testing whe= els "))) + (lambda () + (display "wrong data to make sure we're testing wheels= "))) (parameterize ((current-output-port (%make-void-port "rw+"= ))) (system* "tar" "czvf" file-name "foo-1.0.0/")) =2D (delete-file-recursively "foo-1.0.0") =2D (set! test-source-hash =2D (call-with-input-file file-name port-sha256)))) + (delete-file-recursively "foo-1.0.0") + (set! test-source-hash + (call-with-input-file file-name port-sha256)))) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" =2D (begin =2D (mkdir "foo-1.0.0.dist-info") =2D (with-output-to-file "foo-1.0.0.dist-info/METADATA" =2D (lambda () =2D (display test-metadata))) =2D (let ((zip-file (string-append file-name ".zip"))) =2D ;; zip always adds a "zip" extension to the file it c= reates, =2D ;; so we need to rename it. =2D (system* "zip" zip-file "foo-1.0.0.dist-info/METADATA= ") =2D (rename-file zip-file file-name)) =2D (delete-file-recursively "foo-1.0.0.dist-info"))) + (begin + (mkdir "foo-1.0.0.dist-info") + (with-output-to-file "foo-1.0.0.dist-info/METADATA" + (lambda () + (display test-metadata))) + (let ((zip-file (string-append file-name ".zip"))) + ;; zip always adds a "zip" extension to the file it crea= tes, + ;; so we need to rename it. + (system* "zip" "-q" zip-file "foo-1.0.0.dist-info/METADA= TA") + (rename-file zip-file file-name)) + (delete-file-recursively "foo-1.0.0.dist-info"))) (_ (error "Unexpected URL: " url))))) (mock ((guix http-client) http-fetch (lambda (url . rest) @@ -262,6 +280,9 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testin= g' ('quasiquote (("python-bar" ('unquote 'python-bar)) ("python-baz" ('unquote 'python-baz))))) + ('native-inputs + ('quasiquote + (("python-pytest" ('unquote 'python-pytest))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary") @@ -272,4 +293,48 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testi= ng' (x (pk 'fail x #f)))))) =20 +(test-assert "pypi->guix-package, no usable requirement file." + ;; Replace network resources with sample data. + (mock ((guix import utils) url-fetch + (lambda (url file-name) + (match url + ("https://example.com/foo-1.0.0.tar.gz" + (set! test-source-hash + (call-with-input-file file-name port-sha256)) + #t) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #t) + (_ (error "Unexpected URL: " url))))) + (mock ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ("https://pypi.org/pypi/foo/json" + (values (open-input-string test-json) + (string-length test-json))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #= f) + (_ (error "Unexpected URL: " url))))) + ;; Not clearing the memoization cache here would mean return= ing the value + ;; computed in the previous test. + (invalidate-memoization! pypi->guix-package) + (match (pypi->guix-package "foo") + (('package + ('name "python-foo") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri ('pypi-uri "foo" 'version)) + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'python-build-system) + ('home-page "http://example.com") + ('synopsis "summary") + ('description "summary") + ('license 'license:lgpl2.0)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (pk 'fail x #f))) + ))) + (test-end "pypi") =2D-=20 2.20.1 --=-=-= Content-Type: text/plain Thanks, Maxim --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEJ9WGpPiQCFQyn/CfEmDkZILmNWIFAlydoGMACgkQEmDkZILm NWJMOA/+Nk3rtALS/r5Su8aNXIZfC5z0p5Zk3WBc/FSOkZ0e+56qdiiLLQ6bEBrT xEmr23LqIiHfwU2H3H+mpZCKWDmqBb0edLW0u6cKRRO0eU5T55pEA71CyQj8c+mu UHNyFGm6y/hmaMKPVsyP8p3rhNl8TIPJe2vFXnN4LWaCJaj3E0X6Ge2fsnNTgzXk zpe5eU1vAFNfYOaf7uc658cfIPFzTEWgBfHASsfuKTLE34ZNB9HOxwskTNd1CvlN USjs3kSFRGQl2q0tXQzrxt9eNwQzcXWMk61bCYtDrZ4d87dhENKd2EAivMgGLGq/ 0poA4ijhb/zfmK7RNtHItivipzaZ+xAOpIEhcfwlmz2WxIZfV4eKkpLEB1q6QGAQ 0OKUmzAPHdYvNcHkhhAZIZl195JZ11HlrpTX5gUMqjL11d4ReuhlqNlAnynt8+tJ 8WgjVemJ7dhn8oZVLcXELBdFH01DZnbvF/8kheAyO6/dhTzKbxua+mPbNTpMn0w4 Ipe94BiF5DlaDffvWfmTebR9j+x7j8WSQeezc0GiNQuV8wOXKdVGsm+B66MdCpOH ioQ8B/62Tu/9ul4L+hkaZCBrVM/JI8W+SAeAn+CMQ0Zye20POpwn1xXK5SCn1FRU wXoFrEGE1L7yG+B406fRiABr711AItzTO9MvZQQmEquZBCN0qqI= =2rRk -----END PGP SIGNATURE----- --==-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 29 22:12:51 2019 Received: (at 24450) by debbugs.gnu.org; 30 Mar 2019 02:12:51 +0000 Received: from localhost ([127.0.0.1]:36098 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hA3UI-0001p6-2Q for submit@debbugs.gnu.org; Fri, 29 Mar 2019 22:12:50 -0400 Received: from mail-it1-f170.google.com ([209.85.166.170]:38884) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hA3UF-0001ot-8U for 24450@debbugs.gnu.org; Fri, 29 Mar 2019 22:12:48 -0400 Received: by mail-it1-f170.google.com with SMTP id f22so6909223ita.3 for <24450@debbugs.gnu.org>; Fri, 29 Mar 2019 19:12:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=EgbkHK8tXP2pXrN2Xadripcz517aU8UocDjP1CF46e0=; b=TAxh8ud45GfYrUIRyIkuBIlk7QXlEP7AGicuzjnimK0d50uuiKBhHdVmvNTS8YiIc1 UIgyBRIrjAtpNlsAO9aP1ggOM5Ymai8OVXbYfnfkV1ljTi43DrXMVKt1GeRElBgrlFBM MqARB04X7IyzDo+NcPglHdXDEw2yEp2tdSK3B/QtW0eba3OrwNpjgiyFjY4lV9bldZyY kwPnsT23NB+anT3A1AnIP674mUFp8yNxSJt2a2zedlIBZS/i6lk4l+jgHeSYdPM7it1+ wG78hLAWl4sos1NzyefZFJKkxiodjqK6/qSV0Z4ecaQFUKbiq2i2ZOPjJ7nQlk1o4JsM s0ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=EgbkHK8tXP2pXrN2Xadripcz517aU8UocDjP1CF46e0=; b=D6r2k2QhSacML7DGrG65A62nx7Zcjd15DgqiSXafviHrL32aXVy67MpxozFaqlK8nE RQ+YDnET4zGIUnWUVt9XNxM/T69ONxmy+rCcu+7MCjuwvHY1cJdqJflKYWsRfIZvW+FA 0aU0qGAs3OnmheHswLc2xsae3ZsqFRwci5TBcERqlhOji1i2pvO4+B1lBJaqRYuHlg8N ecRHMDjNwzPflLKT/Pn2Bx2iJ9hTVYqMC7GF2rFtmZhjmndfJzjICRTAXZpBLBDmUBZL MtuJ3E3WljE4gIK0sl3Xcxl6wbpENOWmaNXiu35O8VEQqer4aFOxhVVtYbBCRlBu+nok BiOg== X-Gm-Message-State: APjAAAWmHtbjGhjRcMuTpkRwT0efyvIlJ+dsJpB9ZFUoGX/TwzH8JYJG YGbiKzcFQnGNZfI79psMy+/kFMe7Ewc= X-Google-Smtp-Source: APXvYqyOPSgw/BDGn4hNSHnAfSraKno3Ok9bdqvqQiKH01HmFtWNSLUDn+KGzQtWaL7eXJc9rZOkDw== X-Received: by 2002:a24:c106:: with SMTP id e6mr7056521itg.21.1553911961563; Fri, 29 Mar 2019 19:12:41 -0700 (PDT) Received: from kwak ([2607:f2c0:94b4:fa00::235]) by smtp.gmail.com with ESMTPSA id e10sm1383385iok.85.2019.03.29.19.12.39 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 29 Mar 2019 19:12:40 -0700 (PDT) From: T460s laptop X-Google-Original-From: T460s laptop To: Maxim Cournoyer Subject: [PATCHv2] bug#24450: pypi importer outputs strange character series in optional dependency case. References: <87h99fipj1.fsf@we.make.ritual.n0.is> <87tvfm1eos.fsf@gmail.com> Date: Fri, 29 Mar 2019 22:12:38 -0400 In-Reply-To: <87tvfm1eos.fsf@gmail.com> (Maxim Cournoyer's message of "Fri, 29 Mar 2019 00:34:43 -0400") Message-ID: <877ech5cvd.fsf_-_@kwak.i-did-not-set--mail-host-address--so-tickle-me> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: ng0 , 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain The previous 0007 patch had broken the recursive importer. This reworked version fixes this. The rest of the patches stack sent in the previous message is still good as is. --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: attachment; filename=0007-import-pypi-Include-optional-test-inputs-as-native-i.patch Content-Transfer-Encoding: quoted-printable From=2037e499d5d5d5f690aa0a065c730e13f6a31dd30d Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 23:12:26 -0400 Subject: [PATCH] import: pypi: Include optional test inputs as native-input= s. * guix/import/pypi.scm (maybe-inputs): Add INPUT-TYPE argument, and use it. (test-section?): New predicate. (parse-requires.txt): Collect the optional test inputs, and return them as = the second element of the returned list. (parse-wheel-metadata): Likewise. (guess-requirements): Adapt, and hide unzip output. (make-pypi-sexp): Likewise, and include the test inputs requirements as nat= ive inputs in the returned package expression. * tests/pypi.scm (test-requires.txt): Include a test section in the test-requires.txt data. (test-requires.txt-beaker): New variable. ("parse-requires.txt"): Adapt. ("parse-requires.txt - Beaker"): New test. ("parse-wheel-metadata, with extras"): Adapt. ("parse-wheel-metadata, with extras - Jedi"): Adapt. ("pypi->guix-package, no wheel"): Re-indent, and add the expected native-inputs. ("pypi->guix-package, wheels"): Likewise. ("pypi->guix-package, no usable requirement file."): New test. =2D-- guix/import/pypi.scm | 195 ++++++++++++++++++++++++++++--------------- tests/pypi.scm | 123 ++++++++++++++++++++------- 2 files changed, 222 insertions(+), 96 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index c520213b6a..099768f0c8 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -4,6 +4,7 @@ ;;; Copyright =C2=A9 2015, 2016, 2017 Ludovic Court=C3=A8s ;;; Copyright =C2=A9 2017 Mathieu Othacehe ;;; Copyright =C2=A9 2018 Ricardo Wurmus +;;; Copyright =C2=A9 2019 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -26,6 +27,7 @@ #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) @@ -106,14 +108,15 @@ package on PyPI." ((name version _ ...) (string-append name "-" version ".dist-info")))) =20 =2D(define (maybe-inputs package-inputs) +(define (maybe-inputs package-inputs input-type) "Given a list of PACKAGE-INPUTS, tries to generate the 'inputs' field of= a =2Dpackage definition." +package definition. INPUT-TYPE, a symbol, is used to populate the name of +the input field." (match package-inputs (() '()) ((package-inputs ...) =2D `((propagated-inputs (,'quasiquote ,package-inputs)))))) + `((,input-type (,'quasiquote ,package-inputs)))))) =20 (define %requirement-name-regexp ;; Regexp to match the requirement name in a requirement specification. @@ -147,11 +150,21 @@ package definition." (or (regexp-exec %requirement-name-regexp spec) (error (G_ "Could not extract requirement name in spec:") spec)))) =20 +(define (test-section? name) + "Return #t if the section name contains 'test' or 'dev'." + (any (cut string-contains-ci name <>) + '("test" "dev"))) + (define (parse-requires.txt requires.txt) =2D "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of =2Drequirement names." =2D ;; This is a very incomplete parser, which job is to select the non-op= tional =2D ;; dependencies and strip them out of any version information. + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a pair of re= quirements. + +The first element of the pair contains the required dependencies while the +second the optional test dependencies. Note that currently, optional, +non-test dependencies are omitted since these can be difficult or expensiv= e to +satisfy." + + ;; This is a very incomplete parser, which job is to read in the require= ment + ;; specification lines, and strip them out of any version information. ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) ;; library and the requirements grammar defined by PEP-0508 ;; (https://www.python.org/dev/peps/pep-0508/). @@ -168,57 +181,89 @@ requirement names." =20 (call-with-input-file requires.txt (lambda (port) =2D (let loop ((result '())) + (let loop ((required-deps '()) + (test-deps '()) + (inside-test-section? #f) + (optional? #f)) (let ((line (read-line port))) =2D ;; Stop when a section is encountered, as sections contains op= tional =2D ;; (extra) requirements. Non-optional requirements must appear =2D ;; before any section is defined. =2D (if (or (eof-object? line) (section-header? line)) + (if (eof-object? line) ;; Duplicates can occur, since the same requirement can be ;; listed multiple times with different conditional markers,= e.g. ;; pytest >=3D 3 ; python_version >=3D "3.3" ;; pytest < 3 ; python_version < "3.3" =2D (reverse (delete-duplicates result)) + (map (compose reverse delete-duplicates) + (list required-deps test-deps)) (cond ((or (string-null? line) (comment? line)) =2D (loop result)) =2D (else + (loop required-deps test-deps inside-test-section? optiona= l?)) + ((section-header? line) + ;; Encountering a section means that all the requirements + ;; listed below are optional. Since we want to pick only t= he + ;; test dependencies from the optional dependencies, we mu= st + ;; track those separately. + (loop required-deps test-deps (test-section? line) #t)) + (inside-test-section? + (loop required-deps + (cons (specification->requirement-name line) + test-deps) + inside-test-section? optional?)) + ((not optional?) (loop (cons (specification->requirement-name line) =2D result)))))))))) + required-deps) + test-deps inside-test-section? optional?)) + (optional? + ;; Skip optional items. + (loop required-deps test-deps inside-test-section? optiona= l?)) + (else + (warning (G_ "parse-requires.txt reached an unexpected \ +condition on line ~a~%") line))))))))) =20 (define (parse-wheel-metadata metadata) =2D "Given METADATA, a Wheel metadata file, return a list of requirement n= ames." + "Given METADATA, a Wheel metadata file, return a pair of requirements. + +The first element of the pair contains the required dependencies while the= second the optional +test dependencies. Note that currently, optional, non-test dependencies a= re +omitted since these can be difficult or expensive to satisfy." ;; METADATA is a RFC-2822-like, header based file. =20 (define (requires-dist-header? line) ;; Return #t if the given LINE is a Requires-Dist header. =2D (regexp-match? (string-match "^Requires-Dist: " line))) + (string-match "^Requires-Dist: " line)) =20 (define (requires-dist-value line) (string-drop line (string-length "Requires-Dist: "))) =20 (define (extra? line) ;; Return #t if the given LINE is an "extra" requirement. =2D (regexp-match? (string-match "extra =3D=3D " line))) + (string-match "extra =3D=3D '(.*)'" line)) + + (define (test-requirement? line) + (let ((extra-label (match:substring (extra? line) 1))) + (and extra-label (test-section? extra-label)))) =20 (call-with-input-file metadata (lambda (port) =2D (let loop ((requirements '())) + (let loop ((required-deps '()) + (test-deps '())) (let ((line (read-line port))) =2D ;; Stop at the first 'Provides-Extra' section: the non-optional =2D ;; requirements appear before the optional ones. (if (eof-object? line) =2D (reverse (delete-duplicates requirements)) + (map (compose reverse delete-duplicates) + (list required-deps test-deps)) (cond ((and (requires-dist-header? line) (not (extra? line))) (loop (cons (specification->requirement-name (requires-dist-value line)) =2D requirements))) + required-deps) + test-deps)) + ((and (requires-dist-header? line) (test-requirement? line)) + (loop required-deps + (cons (specification->requirement-name (requires-dis= t-value line)) + test-deps))) (else =2D (loop requirements))))))))) + (loop required-deps test-deps))))))))) ;skip line =20 (define (guess-requirements source-url wheel-url archive) =2D "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a li= st + "Given SOURCE-URL, WHEEL-URL and an ARCHIVE of the package, return a list of the required packages specified in the requirements.txt file. ARCHIVE = will be extracted in a temporary directory." =20 @@ -244,7 +289,10 @@ cannot determine package dependencies") (file-extensio= n url)) (metadata (string-append dirname "/METADATA"))) (call-with-temporary-directory (lambda (dir) =2D (if (zero? (system* "unzip" "-q" wheel-archive "-d" dir metadat= a)) + (if (zero? + (parameterize ((current-error-port (%make-void-port "rw+")) + (current-output-port (%make-void-port "rw+"))) + (system* "unzip" wheel-archive "-d" dir metadata))) (parse-wheel-metadata (string-append dir "/" metadata)) (begin (warning @@ -283,32 +331,41 @@ cannot determine package dependencies") (file-extensi= on url)) (warning (G_ "Failed to extract file: ~a from source.~%") requires.txt) =2D '()))))) =2D '()))) + (list '() '())))))) + (list '() '())))) =20 ;; First, try to compute the requirements using the wheel, else, fallbac= k to ;; reading the "requires.txt" from the egg-info directory from the source =2D ;; tarball. + ;; archive. (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 (define (compute-inputs source-url wheel-url archive) =2D "Given the SOURCE-URL of an already downloaded ARCHIVE, return a list = of =2Dname/variable pairs describing the required inputs of this package. Also + "Given the SOURCE-URL and WHEEL-URL of an already downloaded ARCHIVE, re= turn +a pair of lists, each consisting of a list of name/variable pairs, for the +propagated inputs and the native inputs, respectively. Also return the unaltered list of upstream dependency names." =2D (let ((dependencies =2D (remove (cut string=3D? "argparse" <>) =2D (guess-requirements source-url wheel-url archive)))) =2D (values (sort =2D (map (lambda (input) =2D (let ((guix-name (python->package-name input))) =2D (list guix-name (list 'unquote (string->symbol gui= x-name))))) =2D dependencies) =2D (lambda args =2D (match args =2D (((a _ ...) (b _ ...)) =2D (string-ci) deps)) + + (define (requirement->package-name/sort deps) + (sort + (map (lambda (input) + (let ((guix-name (python->package-name input))) + (list guix-name (list 'unquote (string->symbol guix-name))))) + deps) + (lambda args + (match args + (((a _ ...) (b _ ...)) + (string-cipackage-name/sort strip-argparse)) + + (let ((dependencies (guess-requirements source-url wheel-url archive))) + (values (map process-requirements dependencies) + (concatenate dependencies)))) =20 (define (make-pypi-sexp name version source-url wheel-url home-page synops= is description license) @@ -317,29 +374,33 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION= , and LICENSE." (call-with-temporary-output-file (lambda (temp port) (and (url-fetch source-url temp) =2D (receive (input-package-names upstream-dependency-names) + (receive (guix-dependencies upstream-dependencies) (compute-inputs source-url wheel-url temp) =2D (values =2D `(package =2D (name ,(python->package-name name)) =2D (version ,version) =2D (source (origin =2D (method url-fetch) =2D =2D ;; Sometimes 'pypi-uri' doesn't quite work due= to mixed =2D ;; cases in NAME, for instance, as is the case= with =2D ;; "uwsgi". In that case, fall back to a full= URL. =2D (uri (pypi-uri ,(string-downcase name) version= )) =2D (sha256 =2D (base32 =2D ,(guix-hash-url temp))))) =2D (build-system python-build-system) =2D ,@(maybe-inputs input-package-names) =2D (home-page ,home-page) =2D (synopsis ,synopsis) =2D (description ,description) =2D (license ,(license->symbol license))) =2D upstream-dependency-names)))))) + (match guix-dependencies + ((required-inputs test-inputs) + (values + `(package + (name ,(python->package-name name)) + (version ,version) + (source (origin + (method url-fetch) + ;; Sometimes 'pypi-uri' doesn't quite work du= e to mixed + ;; cases in NAME, for instance, as is the cas= e with + ;; "uwsgi". In that case, fall back to a ful= l URL. + (uri (pypi-uri ,(string-downcase name) versio= n)) + (sha256 + (base32 + ,(guix-hash-url temp))))) + (build-system python-build-system) + ,@(maybe-inputs required-inputs 'propagated-inputs) + ,@(maybe-inputs test-inputs 'native-inputs) + (home-page ,home-page) + (synopsis ,synopsis) + (description ,description) + (license ,(license->symbol license))) + ;; Flatten the nested lists and return the upstream + ;; dependencies. + upstream-dependencies)))))))) =20 (define pypi->guix-package (memoize diff --git a/tests/pypi.scm b/tests/pypi.scm index ca8cb5f6de..aa08e2cb54 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2014 David Thompson ;;; Copyright =C2=A9 2016 Ricardo Wurmus +;;; Copyright =C2=A9 2019 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -65,11 +66,6 @@ sha1=3Dda9234ee9982d4bbb3c72346a6de940a148ea686")) =20 (define test-requires.txt "\ =2Dbar =2Dbaz > 13.37 =2D") =2D =2D(define test-requires-with-sections "\ # A comment foo ~=3D 3 bar !=3D 2 @@ -78,12 +74,25 @@ bar !=3D 2 pytest (>=3D2.5.0) ") =20 +;; Beaker contains only optional dependencies. +(define test-requires.txt-beaker "\ +[crypto] +pycryptopp>=3D0.5.12 + +[cryptography] +cryptography + +[testsuite] +Mock +coverage +") + (define test-metadata "\ Classifier: Programming Language :: Python :: 3.7 Requires-Dist: baz ~=3D 3 Requires-Dist: bar !=3D 2 Provides-Extra: test =2Dpytest (>=3D2.5.0) +Requires-Dist: pytest (>=3D2.5.0) ; extra =3D=3D 'test' ") =20 (define test-metadata-with-extras " @@ -137,25 +146,31 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'test= ing' '("Fizzy" "PickyThing" "SomethingWithMarker" "requests" "pip") (map specification->requirement-name test-specifications)) =20 =2D(test-equal "parse-requires.txt, with sections" =2D '("foo" "bar") +(test-equal "parse-requires.txt" + (list '("foo" "bar") '("pytest")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) =2D (parse-requires.txt test-requires-with-sections))) + (parse-requires.txt test-requires.txt))) + +(test-equal "parse-requires.txt - Beaker" + (list '() '("Mock" "coverage")) + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-requires.txt test-requires.txt-beaker))) =20 (test-equal "parse-wheel-metadata, with extras" =2D '("wrapt" "bar") + (list '("wrapt" "bar") '("tox" "bumpversion")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) (parse-wheel-metadata test-metadata-with-extras))) =20 (test-equal "parse-wheel-metadata, with extras - Jedi" =2D '("parso") + (list '("parso") '("pytest")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) (parse-wheel-metadata test-metadata-with-extras-jedi))) =20 =2D(test-assert "pypi->guix-package" +(test-assert "pypi->guix-package, no wheel" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch (lambda (url file-name) @@ -195,7 +210,10 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testi= ng' ('propagated-inputs ('quasiquote (("python-bar" ('unquote 'python-bar)) =2D ("python-baz" ('unquote 'python-baz))))) + ("python-foo" ('unquote 'python-foo))))) + ('native-inputs + ('quasiquote + (("python-pytest" ('unquote 'python-pytest))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary") @@ -216,25 +234,25 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'test= ing' (begin (mkdir-p "foo-1.0.0/foo.egg-info/") (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" =2D (lambda () =2D (display "wrong data to make sure we're testing whe= els "))) + (lambda () + (display "wrong data to make sure we're testing wheels= "))) (parameterize ((current-output-port (%make-void-port "rw+"= ))) (system* "tar" "czvf" file-name "foo-1.0.0/")) =2D (delete-file-recursively "foo-1.0.0") =2D (set! test-source-hash =2D (call-with-input-file file-name port-sha256)))) + (delete-file-recursively "foo-1.0.0") + (set! test-source-hash + (call-with-input-file file-name port-sha256)))) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" =2D (begin =2D (mkdir "foo-1.0.0.dist-info") =2D (with-output-to-file "foo-1.0.0.dist-info/METADATA" =2D (lambda () =2D (display test-metadata))) =2D (let ((zip-file (string-append file-name ".zip"))) =2D ;; zip always adds a "zip" extension to the file it c= reates, =2D ;; so we need to rename it. =2D (system* "zip" zip-file "foo-1.0.0.dist-info/METADATA= ") =2D (rename-file zip-file file-name)) =2D (delete-file-recursively "foo-1.0.0.dist-info"))) + (begin + (mkdir "foo-1.0.0.dist-info") + (with-output-to-file "foo-1.0.0.dist-info/METADATA" + (lambda () + (display test-metadata))) + (let ((zip-file (string-append file-name ".zip"))) + ;; zip always adds a "zip" extension to the file it crea= tes, + ;; so we need to rename it. + (system* "zip" "-q" zip-file "foo-1.0.0.dist-info/METADA= TA") + (rename-file zip-file file-name)) + (delete-file-recursively "foo-1.0.0.dist-info"))) (_ (error "Unexpected URL: " url))))) (mock ((guix http-client) http-fetch (lambda (url . rest) @@ -262,6 +280,9 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testin= g' ('quasiquote (("python-bar" ('unquote 'python-bar)) ("python-baz" ('unquote 'python-baz))))) + ('native-inputs + ('quasiquote + (("python-pytest" ('unquote 'python-pytest))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary") @@ -272,4 +293,48 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testi= ng' (x (pk 'fail x #f)))))) =20 +(test-assert "pypi->guix-package, no usable requirement file." + ;; Replace network resources with sample data. + (mock ((guix import utils) url-fetch + (lambda (url file-name) + (match url + ("https://example.com/foo-1.0.0.tar.gz" + (set! test-source-hash + (call-with-input-file file-name port-sha256)) + #t) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #t) + (_ (error "Unexpected URL: " url))))) + (mock ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ("https://pypi.org/pypi/foo/json" + (values (open-input-string test-json) + (string-length test-json))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #= f) + (_ (error "Unexpected URL: " url))))) + ;; Not clearing the memoization cache here would mean return= ing the value + ;; computed in the previous test. + (invalidate-memoization! pypi->guix-package) + (match (pypi->guix-package "foo") + (('package + ('name "python-foo") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri ('pypi-uri "foo" 'version)) + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'python-build-system) + ('home-page "http://example.com") + ('synopsis "summary") + ('description "summary") + ('license 'license:lgpl2.0)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (pk 'fail x #f))) + ))) + (test-end "pypi") =2D-=20 2.20.1 --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEJ9WGpPiQCFQyn/CfEmDkZILmNWIFAlye0JYACgkQEmDkZILm NWJGEg/9HElFyj+z1M2Z6ffboYKW3WJ0D7q+vV0fvkacd3X/8XkC6DPEjI8veFYf 3qiygovcK6tgI5sAVzRWZ4WJ7oW284YoUURGjfax1ilvJl3kGkzqNy45nD3CQo3A JK/rfW432Pu2UgC/Ewu+4V1NZqU5gCoHnz/rVNt0gRwWANBCLQG9fia/0u7zq8WK E0UIlKJZeOi6/1xr+QYcSYtMV9Nr3mb7OOC3TleHrz+lrT8YbmrsqdxXfZuJw3iA FY4jMX6SGAlfhKc7/+2F2z+iwkev7tTg1AAVhLWyqeR+zqYwBOCj5TOEAFMBcO8e FMUr6GONxdJfHwSapnzzEq6HcTqhx+MutMCiynap3hBVxONDA2uXmGTM/pe0zrZm Dqdr4XgFgaSJcaAtc3Ar/v6jwRy3muNKHBl6opQjd7zbxIju9hgzsiXvTbC/RUgg 9Oc7Yd2PT2kCBvJ1GgCk996EHcY1ACOeHHI0Yg0XHO6JCx1Dqi9i0vImGqcoOg/o OmA0cxRntaCgiCSyGaSyBzcyHjsfieKwZlRQk5DaeMfguzGHGFS9b6OHuJ7Ofdb7 QESgYGuD5YlQhAsNc64A0GTm6+JDh9v84MwUqyR0EcU3toPO3r2zAWeGoi8k3FDZ IaPuNC3qZ9XY2Pp7kHsHcTCRs+lXcoIDxZElkOgNMWCvW5wKpr4= =2WjY -----END PGP SIGNATURE----- --==-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Mar 29 22:16:00 2019 Received: (at control) by debbugs.gnu.org; 30 Mar 2019 02:16:00 +0000 Received: from localhost ([127.0.0.1]:36102 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hA3XM-0001uB-Ah for submit@debbugs.gnu.org; Fri, 29 Mar 2019 22:16:00 -0400 Received: from mail-io1-f50.google.com ([209.85.166.50]:35915) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hA3XK-0001tu-Tp for control@debbugs.gnu.org; Fri, 29 Mar 2019 22:15:59 -0400 Received: by mail-io1-f50.google.com with SMTP id f6so3342983iop.3 for ; Fri, 29 Mar 2019 19:15:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:date:message-id:to:subject; bh=3Y648C0P0HNEVcmBjEU/KhyWcn47LHXMEJq78zjZsjw=; b=JyV04Vmg0dHtG2XlAC7LKoAZGzxbbYMp/8lZ9q3F4h6xkbzjA9fvFGH8WhpU1Egobt 2rdwsCQpgOD5wPidmfdz9sRXnvopEr1ssKCvQgw+wSq8qG75V36JGycENceBQvs3LASe 4WvopHt72tYsDIV6/WIXu4qIQtg4/mikJwEGpf/5Zikja6Xlw3KqCvE+C2Lq1QTEeUqi VY19/HMMZlzXlrQ0pWIT+znEt5fXyPgQB+71VITWgQTSyRW5CK66R3UpAVI9gLNKUVm9 9UhWZ5et1Nd7tcGQ42+P32TQYzMa/lDxU/CgIN8PtL1U/fgse2wkATobDiiLVNaJaoet bJlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:date:message-id:to:subject; bh=3Y648C0P0HNEVcmBjEU/KhyWcn47LHXMEJq78zjZsjw=; b=Tc2kw8u5Q2R6j3CArZasZmVoXUlSD9dNNh4tGSDPBCO4jSQsiQ9+cwJsiCLkBU2QaI JiiTzptI2/sZy2uBdFWwiqChGEWVkdKI8ZSaKM0IHyno0iDRqtsc9D3VWpaFCQTVLx8+ 3mvfukexJQjtHStUmH0IFAQxayn7dfJ0w/cSPDxnqy3yErqu/jATTW2GhLFPmE1hN7uk 2Vs7S7XKg7Mf4er1FMId50Mz6kle7Fwya4A5ouV09GGRwuIZiIBw1CIi6KPFyoEkwmUh bpZFGKu8BStk/bAXan2aU5ou1TTMP6yhJdF1CA5ly4JoDQ/nm5rYog7QfX4AIlwH73pp TI9A== X-Gm-Message-State: APjAAAWnPPnIZ+EtA0eZP7Zemd6a91HfcqQkhIFA4i/8czQXWNTz9eek qHG+eKh/swtd1prIvaYma5PEQtVt X-Google-Smtp-Source: APXvYqyQrXRx7EQ8UZQMvxAPjGx+YI92G8fqfql3RjsCbTVA2ukK3rXl83lQgKaB0MU7hHSNGWMhIA== X-Received: by 2002:a6b:5118:: with SMTP id f24mr37278065iob.62.1553912153194; Fri, 29 Mar 2019 19:15:53 -0700 (PDT) Received: from kwak ([2607:f2c0:94b4:fa00::235]) by smtp.gmail.com with ESMTPSA id u204sm2169917ita.33.2019.03.29.19.15.52 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 29 Mar 2019 19:15:52 -0700 (PDT) From: T460s laptop X-Google-Original-From: T460s laptop Date: Fri, 29 Mar 2019 22:15:51 -0400 Message-Id: <875zs15cq0.fsf@kwak.i-did-not-set--mail-host-address--so-tickle-me> To: control@debbugs.gnu.org Subject: control message for bug #24450 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) forcemerge 24450 24557 From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 30 03:57:19 2019 Received: (at 24450) by debbugs.gnu.org; 30 Mar 2019 07:57:19 +0000 Received: from localhost ([127.0.0.1]:36219 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hA8rf-00082j-Ev for submit@debbugs.gnu.org; Sat, 30 Mar 2019 03:57:19 -0400 Received: from mx1.riseup.net ([198.252.153.129]:39846) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hA8rc-00082X-VN; Sat, 30 Mar 2019 03:57:17 -0400 Received: from capuchin.riseup.net (capuchin-pn.riseup.net [10.0.1.176]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "*.riseup.net", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.riseup.net (Postfix) with ESMTPS id 9D6DF1A04CC; Sat, 30 Mar 2019 00:57:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=riseup.net; s=squak; t=1553932635; bh=+c+whlUv6xmEB/A02sIIyLCNBnGlDBqd7p+pMl2SEMU=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=UjPMNuNooQDLMyOMo8czC4erYlKvDRSc+J6EuXGEtpEBPNUfDDKkyqXxrENv+Si4L /1dZ0vPnoo9JvBTKevJIIhC+5d2pSxjKl8mUjtp13CkhIFhN3jmbtzuyss9C1eI9k3 AIsuusR2jh/0JG+4BC8boqJPjLtiKbc/6teP/K9w= X-Riseup-User-ID: C37BD159BBAC4720BE3C07E4F3E98F87960E1CCC9A7021F4FF2DE41F17106A67 Received: from [127.0.0.1] (localhost [127.0.0.1]) by capuchin.riseup.net (Postfix) with ESMTPSA id 47427120527; Sat, 30 Mar 2019 00:57:13 -0700 (PDT) Subject: Re: bug#34266: pypi importer cannot handle [ and ] correctly To: Maxim Cournoyer References: <87va022tz3.fsf@gmail.com> From: swedebugia Message-ID: <991a432d-e5b8-88cd-ee47-bb60af1a8dbe@riseup.net> Date: Sat, 30 Mar 2019 08:59:12 +0100 MIME-Version: 1.0 In-Reply-To: <87va022tz3.fsf@gmail.com> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="D6uQrkeYOdfpFinkcF0h2Gd5DPiR87m3v" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 24450 Cc: 34266@debbugs.gnu.org, 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --D6uQrkeYOdfpFinkcF0h2Gd5DPiR87m3v Content-Type: multipart/mixed; boundary="jl6FJnS6n97xldJs8GtZBPzjbQChLA20K"; protected-headers="v1" From: swedebugia To: Maxim Cournoyer Cc: 34266@debbugs.gnu.org, 24450@debbugs.gnu.org Message-ID: <991a432d-e5b8-88cd-ee47-bb60af1a8dbe@riseup.net> Subject: Re: bug#34266: pypi importer cannot handle [ and ] correctly References: <87va022tz3.fsf@gmail.com> In-Reply-To: <87va022tz3.fsf@gmail.com> --jl6FJnS6n97xldJs8GtZBPzjbQChLA20K Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2019-03-29 05:19, Maxim Cournoyer wrote: > swedebugia writes: >=20 >> $ ./pre-inst-env guix import pypi beaker >> >> following redirection to `https://pypi.org/pypi/Beaker/json'... >> >> Starting download of /tmp/guix-file.p15GJZ >> From >> https://files.pythonhosted.org/packages/c2/21/b052b2fbfee3def06670923d= 5d34b0d353d4c278013e4a714c3fb663f150/Beaker-1.10.0.tar.gz... >> ...0.0.tar.gz 40KiB 521KiB/s 00:00 >> [##################] 100.0% >> (package >> (name "python-beaker") >> (version "1.10.0") >> (source >> (origin >> (method url-fetch) >> (uri (pypi-uri "beaker" version)) >> (sha256 >> (base32 >> "0l047yl3n9b3w7ba0wrqdb5fpww5y8pjy20kah2mlpr230lqjwk0")))) >> (build-system python-build-system) >> (propagated-inputs >> `(("python-[crypto]" ,#{python-\x5b;crypto\x5d;}#) >> ("python-[cryptography]" >> ,#{python-\x5b;cryptography\x5d;}#) >> ("python-[pycrypto]" >> ,#{python-\x5b;pycrypto\x5d;}#) >> ("python-[pycryptodome]" >> ,#{python-\x5b;pycryptodome\x5d;}#) >> ("python-[testsuite]" >> ,#{python-\x5b;testsuite\x5d;}#) >> ("python-coverage" ,python-coverage) >> ("python-cryptography" ,python-cryptography) >> ("python-cryptography" ,python-cryptography) >> ("python-funcsigs" ,python-funcsigs) >> ("python-memcached" ,python-memcached) >> ("python-mock" ,python-mock) >> ("python-nose" ,python-nose) >> ("python-pycrypto" ,python-pycrypto) >> ("python-pycryptodome" ,python-pycryptodome) >> ("python-pycryptodome" ,python-pycryptodome) >> ("python-pycryptopp" ,python-pycryptopp) >> ("python-pylibmc" ,python-pylibmc) >> ("python-pymongo" ,python-pymongo) >> ("python-redis" ,python-redis) >> ("python-sqlalchemy" ,python-sqlalchemy) >> ("python-webtest" ,python-webtest))) >> (home-page "https://beaker.readthedocs.io/") >> (synopsis >> "A Session and Caching library with WSGI Middleware") >> (description >> "A Session and Caching library with WSGI Middleware") >> (license license:bsd-3)) >=20 > Testing with my soon-to-be sent for review changes: >=20 > --8<---------------cut here---------------start------------->8--- > ./pre-inst-env guix import pypi beaker > following redirection to `https://pypi.org/pypi/Beaker/json'... >=20 > Starting download of /tmp/guix-file.0MWu4B > From https://files.pythonhosted.org/packages/76/87/ecc1a222f0caaa7ba7b8= 928737e89b2e91b8c22450c12b8a51ee625a4d87/Beaker-1.10.1.tar.gz... > =E2=80=A60.1.tar.gz 40KiB 487KiB/s 00:00 [#########= #########] 100.0% > (package > (name "python-beaker") > (version "1.10.1") > (source > (origin > (method url-fetch) > (uri (pypi-uri "beaker" version)) > (sha256 > (base32 > "16zdjfl8v73yl1capph0n371vd26c7zpzb48n505ip32ffgmvc4f")))) > (build-system python-build-system) > (native-inputs > `(("python-coverage" ,python-coverage) > ("python-cryptography" ,python-cryptography) > ("python-memcached" ,python-memcached) > ("python-mock" ,python-mock) > ("python-nose" ,python-nose) > ("python-pycryptodome" ,python-pycryptodome) > ("python-pylibmc" ,python-pylibmc) > ("python-pymongo" ,python-pymongo) > ("python-redis" ,python-redis) > ("python-sqlalchemy" ,python-sqlalchemy) > ("python-webtest" ,python-webtest))) > (home-page "https://beaker.readthedocs.io/") > (synopsis > "A Session and Caching library with WSGI Middleware") > (description > "A Session and Caching library with WSGI Middleware") > (license license:bsd-3)) > --8<---------------cut here---------------end--------------->8--- >=20 > Looking better? Perfect! --=20 Cheers Swedebugia --jl6FJnS6n97xldJs8GtZBPzjbQChLA20K-- --D6uQrkeYOdfpFinkcF0h2Gd5DPiR87m3v Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iHUEARYIAB0WIQR6IWNlOiLn9hD6a1XPzUNbKAts0gUCXJ8h0QAKCRDPzUNbKAts 0tuQAP9R2vpUsnHIQ6Pk0La+OsLBsQrgs9d0cC4P2OQqbJx27QD+MrOAJdhoebrd 7oBT8HcsUREpWzV60AgRQIXVFwVZWAk= =NPhy -----END PGP SIGNATURE----- --D6uQrkeYOdfpFinkcF0h2Gd5DPiR87m3v-- From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 31 10:40:46 2019 Received: (at 24450) by debbugs.gnu.org; 31 Mar 2019 14:40:47 +0000 Received: from localhost ([127.0.0.1]:38385 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hAbde-00082y-B7 for submit@debbugs.gnu.org; Sun, 31 Mar 2019 10:40:46 -0400 Received: from mail-it1-f177.google.com ([209.85.166.177]:56201) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hAbdb-00082k-Km for 24450@debbugs.gnu.org; Sun, 31 Mar 2019 10:40:44 -0400 Received: by mail-it1-f177.google.com with SMTP id z126so10707984itd.5 for <24450@debbugs.gnu.org>; Sun, 31 Mar 2019 07:40:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=0G0xcICAc43GZzoldViEH4c2GNH7/Cn3abH0qMFli20=; b=i4LGtZyC6lEFTkP6Tn9INjCNVsCYxZ4m+6Fo0cjJ4NyhVB82WDnnICufzj0W9EIPxk IrzTK8kNMDn5zt24IslpQG8kLtzzMZl8tUZuSUcWR+6Cac5QS/xgmBNt0kPYMidj5oRe LekJeVhLj/BCraqrcpY1eQWntizHCxItQxZMC8HNv0AOMbgBp01DYCDLl4D82zDRGlI8 9+PCjv4EN6MwHWJJXjQHX1N8QXodkix92g9Bak2gtMejz4hvZYc6puCEl1MF7qlgqm42 2bDnzRzQ/fxwWQG3Sh7uQlgrRUMRIuKZ7fO8UJovfGGS86aw/IEzhuQ7EozPHNDMeQ8z T1Rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=0G0xcICAc43GZzoldViEH4c2GNH7/Cn3abH0qMFli20=; b=PR2MNaOTn3FxLirvQhoGcdp07EdNC+Yc2knmNBLHAtlLH/tdmquppUDgRlGDuMed4b IqaGtupJ1LsFqdhzKpzU0kWTGaYY/ciKd/L8Et/JZ52j+RG/J5gYavHnxeUmcma25ECV iqCmrG0qTgfcaNvlPvWnAqD/VvdB0ISAzFkEzZo5Km5TxJBy/GwFEY2q2HXQ76jeyPz3 gZ+A82iWPXPHz96VA37DeYaaHnQpcXE+LQOH17JVP5LBjo3fnT90iBE8nKgC1Hhijy9L of3/ondd72m8n5IS13H7bls4I00oN2aZyVq1C/lBQbzr7q53DClOx9+oqyATggTtMJij 1yNg== X-Gm-Message-State: APjAAAXaZr9tZoOgJumfTl4h3XiTNYxVV3TuDnyaYSuQDiwBrNn8ZwWi wAekn+MNIc9HlTW2ta2wlSd8vQh6 X-Google-Smtp-Source: APXvYqyygj12BQYizOr2VYfk+yOF2hPhw62wEGkmz8uAVbJWfe6Rc/PL2G+zJJs87i7SUfHEAFCTUQ== X-Received: by 2002:a24:ad2:: with SMTP id 201mr11237869itw.26.1554043237593; Sun, 31 Mar 2019 07:40:37 -0700 (PDT) Received: from kwak ([2607:f2c0:94b4:fa00::235]) by smtp.gmail.com with ESMTPSA id v10sm5039307itv.8.2019.03.31.07.40.35 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 31 Mar 2019 07:40:36 -0700 (PDT) From: Maxim Cournoyer To: ng0 Subject: [PATCH] bug#24450: pypi importer outputs strange character series in optional dependency case. References: <87h99fipj1.fsf@we.make.ritual.n0.is> <87tvfm1eos.fsf@gmail.com> <877ech5cvd.fsf_-_@kwak.i-did-not-set--mail-host-address--so-tickle-me> Date: Sun, 31 Mar 2019 10:40:34 -0400 In-Reply-To: <877ech5cvd.fsf_-_@kwak.i-did-not-set--mail-host-address--so-tickle-me> (T460s laptop's message of "Fri, 29 Mar 2019 22:12:38 -0400") Message-ID: <87ftr32jkt.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain Hello! I've yet another commit to add on the top of the current stack of patches already sent. This one makes it so that the source archive is searched for the '[...].egg-info/requires.txt' file rather than check the more common, fixed location. This makes the importer more useful, as some packages such as robotframework-sshlibrary use a "src" directory in their hierarchy which would cause the previous scheme to not find requires.txt. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0008-import-pypi-Scan-source-archive-to-find-requires.txt.patch Content-Transfer-Encoding: quoted-printable From=20cfde6e09f8f8c692fe252d76ed27e8c50a9e5377 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Sat, 30 Mar 2019 23:13:26 -0400 Subject: [PATCH] import: pypi: Scan source archive to find requires.txt fil= e. * guix/import/pypi.scm (use-modules): Use invoke from (guix build utils). (guess-requirements)[archive-root-directory]: Remove procedure. [guess-requirements-from-wheel]: Re-ident. [guess-requirements-from-source]: Search for the requires.txt file in the archive instead of using a static, expected location. * tests/pypi.scm ("pypi->guix-package, no wheel"): Mock the requires.txt at= a non-standard location to test the new feature. ("pypi->guix-package, no usable requirement file."): Adapt. =2D-- guix/import/pypi.scm | 65 ++++++++++++++++++-------------------------- tests/pypi.scm | 17 +++++++----- 2 files changed, 37 insertions(+), 45 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 099768f0c8..a2ce14b192 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -36,7 +36,8 @@ #:use-module ((guix build utils) #:select ((package-name->name+version . hyphen-package-name->name+version) =2D find-files)) + find-files + invoke)) #:use-module (guix import utils) #:use-module ((guix download) #:prefix download:) #:use-module (guix import json) @@ -267,19 +268,6 @@ omitted since these can be difficult or expensive to s= atisfy." of the required packages specified in the requirements.txt file. ARCHIVE = will be extracted in a temporary directory." =20 =2D (define (archive-root-directory url) =2D ;; Given the URL of the package's archive, return the name of the di= rectory =2D ;; that will be created upon decompressing it. If the filetype is not =2D ;; supported, return #f. =2D (if (compressed-file? url) =2D (let ((root-directory (file-sans-extension (basename url)))) =2D (if (string=3D? "tar" (file-extension root-directory)) =2D (file-sans-extension root-directory) =2D root-directory)) =2D (begin =2D (warning (G_ "Unsupported archive format (~a): \ =2Dcannot determine package dependencies") (file-extension url)) =2D #f))) =20 (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's @@ -305,33 +293,34 @@ cannot determine package dependencies") (file-extensi= on url)) (call-with-temporary-output-file (lambda (temp port) (if wheel-url =2D (and (url-fetch wheel-url temp) =2D (read-wheel-metadata temp)) =2D #f)))) + (and (url-fetch wheel-url temp) + (read-wheel-metadata temp)) + #f)))) =20 (define (guess-requirements-from-source) ;; Return the package's requirements by guessing them from the source. =2D (let ((dirname (archive-root-directory source-url)) =2D (extension (file-extension source-url))) =2D (if (string? dirname) =2D (call-with-temporary-directory =2D (lambda (dir) =2D (let* ((pypi-name (string-take dirname (string-rindex dirna= me #\-))) =2D (requires.txt (string-append dirname "/" pypi-name =2D ".egg-info" "/requires.= txt")) =2D (exit-code =2D (parameterize ((current-error-port (%make-void-port= "rw+")) =2D (current-output-port (%make-void-por= t "rw+"))) =2D (if (string=3D? "zip" extension) =2D (system* "unzip" archive "-d" dir requires.tx= t) =2D (system* "tar" "xf" archive "-C" dir requires= .txt))))) =2D (if (zero? exit-code) =2D (parse-requires.txt (string-append dir "/" requires.t= xt)) =2D (begin =2D (warning =2D (G_ "Failed to extract file: ~a from source.~%") =2D requires.txt) =2D (list '() '())))))) + (if (compressed-file? source-url) + (call-with-temporary-directory + (lambda (dir) + (parameterize ((current-error-port (%make-void-port "rw+")) + (current-output-port (%make-void-port "rw+"))) + (if (string=3D? "zip" (file-extension source-url)) + (invoke "unzip" archive "-d" dir) + (invoke "tar" "xf" archive "-C" dir))) + (let ((requires.txt-files + (find-files dir (lambda (abs-file-name _) + (string-match "\\.egg-info/requires.txt$" + abs-file-name))))) + (if (> (length requires.txt-files) 0) + (begin + (parse-requires.txt (first requires.txt-files))) + (begin (warning (G_ "Cannot guess requirements from sourc= e archive:\ + no requires.txt file found.~%")) + (list '() '())))))) + (begin + (warning (G_ "Unsupported archive format; \ +cannot determine package dependencies from source archive: ~a~%") + (basename source-url)) (list '() '())))) =20 ;; First, try to compute the requirements using the wheel, else, fallbac= k to diff --git a/tests/pypi.scm b/tests/pypi.scm index aa08e2cb54..ad188df16c 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -177,8 +177,9 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testin= g' (match url ("https://example.com/foo-1.0.0.tar.gz" (begin =2D (mkdir-p "foo-1.0.0/foo.egg-info/") =2D (with-output-to-file "foo-1.0.0/foo.egg-info/requires.= txt" + ;; Unusual requires.txt location should still be found. + (mkdir-p "foo-1.0.0/src/bizarre.egg-info") + (with-output-to-file "foo-1.0.0/src/bizarre.egg-info/req= uires.txt" (lambda () (display test-requires.txt))) (parameterize ((current-output-port (%make-void-port "rw= +"))) @@ -299,10 +300,13 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'test= ing' (lambda (url file-name) (match url ("https://example.com/foo-1.0.0.tar.gz" + (mkdir-p "foo-1.0.0/foo.egg-info/") + (parameterize ((current-output-port (%make-void-port "rw+"))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) + (delete-file-recursively "foo-1.0.0") (set! test-source-hash =2D (call-with-input-file file-name port-sha256)) =2D #t) =2D ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #t) + (call-with-input-file file-name port-sha256))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f) (_ (error "Unexpected URL: " url))))) (mock ((guix http-client) http-fetch (lambda (url . rest) @@ -334,7 +338,6 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testin= g' test-source-hash) hash)) (x =2D (pk 'fail x #f))) =2D ))) + (pk 'fail x #f)))))) =20 (test-end "pypi") =2D-=20 2.20.1 --=-=-= Content-Type: text/plain Thanks, Maxim --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEJ9WGpPiQCFQyn/CfEmDkZILmNWIFAlyg0WIACgkQEmDkZILm NWIEMA/9HsMlrdzTwtllz/jPaAM13pPbyvI1sx1uwQi6AYessiZYBr/vGiFEBBTB FJJUQ9Sz0XQOTW3wOitgTRz1kr55c17cv80IvwWYhTZLGZ+O2F29hFpp38OucRS7 d4XeZSJwBSn9ccYgytebd0Xtkp340HHeLOHnNTnA600TLwTq/6YXP4hLi5hiU7p1 zIOHe3Nze9dk1B30tuivqwvIVkSMnnPQyCmPSE+1vhzczVQ+m+f5PxxmSbP6Kznx KF7Rlm+Ltjup0JfiKeglNiqLYYOKRpYuWM+oMzWSJgl1zA/MzDRZCnbs30TD/XKd dkixB0Hmazer2BBRJoNSgCKu/Mx8hZg6ML/NcgrbmJnUd4Sw4r/M2My/Iq1D6RLA Kkj7gzoqqowy+PcsU+/l+XrLdvT0wd6VR2x+DwY+y3Fiv8DSQRWI0mXxl9V4M4St gF27p1RhfyUhIQo801AtJ+Tin/N5cYik8orRnARHnWNXVy7Iqz1RGr4MIqGheS0l 8GMOMVeLskAqagLynhWY4fz0teGCnaGAjVyzda7R1tYBkWzsioLugzsDoIftpEA8 p0TBiVEu0/BAWlL/BYdWyOIbkPQ05xV+Yad+QvECS3uu4/MckAcEwLMQpPjqKP8O J5tRiRMRkEP1Dxl7oBuKC1IOHDhGvUOuIqFKqmyGRcBInwTC5Is= =H7lm -----END PGP SIGNATURE----- --==-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Mar 31 10:41:50 2019 Received: (at control) by debbugs.gnu.org; 31 Mar 2019 14:41:50 +0000 Received: from localhost ([127.0.0.1]:38389 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hAbef-00084m-W3 for submit@debbugs.gnu.org; Sun, 31 Mar 2019 10:41:50 -0400 Received: from mail-it1-f173.google.com ([209.85.166.173]:34745) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hAbee-00084W-PY for control@debbugs.gnu.org; Sun, 31 Mar 2019 10:41:49 -0400 Received: by mail-it1-f173.google.com with SMTP id z17so2917139itc.1 for ; Sun, 31 Mar 2019 07:41:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:to:from:subject; bh=08YkkqNe9rbOVLvYSxf0V1Oy5VOYOrj6mp5bwHEtdkk=; b=N1p8NilQ/hH1d4pSyyUoBveEvccR6tNhazEpWRKoYujGNkm7BWI0P8MNFBrXIVmYTI YrvwrUSsAbXj95toxvjfgJdjwuX+biFY/JrHNpfPU73Mcg6qrKLl1VRbuj10+cEduazY K2sJNqiGCpQKDzlfoMdafafF0JLveCR6dWmLkN0xYyNC1wHaR3BviJHocCG3IPxdnHra A1yQgLjaolaNzyGy+vw6QxfNJB818S1pGnVBOH4Vt6pHH8Zt6Bk7zu15QqNqX2xuAh+z CfVyx+0kDfQRnzAcSbwNG08mK/gHHpVQrPALPSLEv86wqgDGdli4rHhABb3Y5yTZigx+ OKdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:to:from:subject; bh=08YkkqNe9rbOVLvYSxf0V1Oy5VOYOrj6mp5bwHEtdkk=; b=n6Jb64PWMZ4k0ZUV6CAgabAvViOdwqDvcfIy4gCIlYYxox2n7vwUVb98OoVHnjMurF 4Z558ML3UmWjBoAzcEOJ9bNGmICw9phwby4OzywLe7Od/OtKs7+Kl+0ayyhiAc6xw1kl tvJCQIIlB2e9TytaJvTSbbKtDsNoRyt6Ppl4L5Fwx4HePNY6W8sEzdlwMSkkE+fFRT/V vCQi1AbmiGe+bqGrW3zc4GnPuMkUCU3kfN9rgRaR2fGqs+BY4qoIp6gRpUeTbE1fIFlu Ahu9ZsMb1C9sYzfq+BtDOHhhFNi09xAZ5ByJ/K7UkeHuG6eY5veoGOBImJtnDXzWGVqI 2BYw== X-Gm-Message-State: APjAAAXi2R+aAUg48nVL1zJ8U7fGC2WMwjtkiOiP7mASmbS3wJCR4OXe DRvvOZKjjyPnBzlEjCcaEPdVoPdI X-Google-Smtp-Source: APXvYqyHrfAE9xeHYpw7TJVPGwBfG/qEtp+UtCkYXueq7IfFk0jbu1oWYyWQcNttPRipzKiJj2anKA== X-Received: by 2002:a24:56d1:: with SMTP id o200mr12814345itb.111.1554043303021; Sun, 31 Mar 2019 07:41:43 -0700 (PDT) Received: from kwak ([2607:f2c0:94b4:fa00::235]) by smtp.gmail.com with ESMTPSA id u19sm3591480iol.16.2019.03.31.07.41.41 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 31 Mar 2019 07:41:42 -0700 (PDT) Date: Sun, 31 Mar 2019 10:41:40 -0400 Message-Id: <87ef6n2jiz.fsf@gmail.com> To: control@debbugs.gnu.org From: Maxim Cournoyer Subject: control message for bug #24450 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) tags 24450 patch From debbugs-submit-bounces@debbugs.gnu.org Mon Apr 01 11:28:56 2019 Received: (at 24450) by debbugs.gnu.org; 1 Apr 2019 15:28:56 +0000 Received: from localhost ([127.0.0.1]:39856 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hAyrn-0004L7-V3 for submit@debbugs.gnu.org; Mon, 01 Apr 2019 11:28:56 -0400 Received: from eggs.gnu.org ([209.51.188.92]:49646) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hAyrl-0004Ku-Q8 for 24450@debbugs.gnu.org; Mon, 01 Apr 2019 11:28:54 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48861) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hAyrf-00049E-Pd; Mon, 01 Apr 2019 11:28:47 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=60670 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hAyrf-000886-9P; Mon, 01 Apr 2019 11:28:47 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: T460s laptop Subject: Re: bug#24450: [PATCHv2] bug#24450: pypi importer outputs strange character series in optional dependency case. References: <87h99fipj1.fsf@we.make.ritual.n0.is> <87tvfm1eos.fsf@gmail.com> <877ech5cvd.fsf_-_@kwak.i-did-not-set--mail-host-address--so-tickle-me> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 12 Germinal an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 01 Apr 2019 17:28:45 +0200 In-Reply-To: <877ech5cvd.fsf_-_@kwak.i-did-not-set--mail-host-address--so-tickle-me> (T460s laptop's message of "Fri, 29 Mar 2019 22:12:38 -0400") Message-ID: <87sgv1lp76.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: ng0 , 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Maxim, T460s laptop skribis: > From 37e499d5d5d5f690aa0a065c730e13f6a31dd30d Mon Sep 17 00:00:00 2001 > From: Maxim Cournoyer > Date: Thu, 28 Mar 2019 23:12:26 -0400 > Subject: [PATCH] import: pypi: Include optional test inputs as native-inp= uts. > > * guix/import/pypi.scm (maybe-inputs): Add INPUT-TYPE argument, and use i= t. > (test-section?): New predicate. > (parse-requires.txt): Collect the optional test inputs, and return them a= s the > second element of the returned list. > (parse-wheel-metadata): Likewise. > (guess-requirements): Adapt, and hide unzip output. > (make-pypi-sexp): Likewise, and include the test inputs requirements as n= ative > inputs in the returned package expression. > > * tests/pypi.scm (test-requires.txt): Include a test section in the > test-requires.txt data. > (test-requires.txt-beaker): New variable. > ("parse-requires.txt"): Adapt. > ("parse-requires.txt - Beaker"): New test. > ("parse-wheel-metadata, with extras"): Adapt. > ("parse-wheel-metadata, with extras - Jedi"): Adapt. > ("pypi->guix-package, no wheel"): Re-indent, and add the expected > native-inputs. > ("pypi->guix-package, wheels"): Likewise. > ("pypi->guix-package, no usable requirement file."): New test. It seems that the patch does several unrelated things, such as silencing =E2=80=98unzip=E2=80=99, improving docstrings, handling inputs other than =E2=80=98propagated-inputs=E2=80=99, and correctly parsing wheels. Could you try to separate these as much as possible to simplify review? Thank you! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Wed May 15 07:06:45 2019 Received: (at 24450) by debbugs.gnu.org; 15 May 2019 11:06:45 +0000 Received: from localhost ([127.0.0.1]:51213 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hQrkD-0007TN-DR for submit@debbugs.gnu.org; Wed, 15 May 2019 07:06:45 -0400 Received: from a2062.mx.srv.dfn.de ([194.95.232.172]:57641) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hQrk9-0007TC-3e for 24450@debbugs.gnu.org; Wed, 15 May 2019 07:06:44 -0400 Received: from localhost (localhost [127.0.0.1]) by a2062.mx.srv.dfn.de (Postfix) with ESMTP id 0B7E1A0040; Wed, 15 May 2019 13:06:39 +0200 (CEST) Received: from a2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-han.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id q81pI5gS98Yv; Wed, 15 May 2019 13:06:38 +0200 (CEST) Received: from SW-IT-P-CAS1.mdc-berlin.net (mgw10-1.mdc-berlin.de [141.80.113.53]) by a2062.mx.srv.dfn.de (Postfix) with ESMTPS; Wed, 15 May 2019 13:06:38 +0200 (CEST) Received: from localhost (141.80.113.103) by SW-IT-P-CAS1.mdc-berlin.net (141.80.113.53) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 15 May 2019 13:06:37 +0200 User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Subject: pypi importer outputs strange character series in optional dependency case. X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Wed, 15 May 2019 13:06:37 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [141.80.113.103] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24614.006 X-TM-AS-Result: No-0.221700-8.000000-10 X-TMASE-MatchedRID: LW0QVZJomtQtJMbDWD8p3sGNvKPnBgOavsOKtKk5i8ibKItl61J/yfJv ocwUrWp7F/+bTgkwLgmxAvNp3ItVysRB0bsfrpPI0PU0TdJoUtdr1zy6nWVLiy99JXWpbu2MAXF 0F1M/6DuyS7ltI9uzcYqHCl76dtq4kX579PzRktSCO7HU4V6gix4khEYMlUzT2X7sU86hby5agL TzP9hRYpRMZUCEHkRt X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10-0.221700-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24614.006 X-TM-SNTS-SMTP: 958579B9A9FDC0F2324CF128070A7B4D0396BA70CD0B492A11F3B8008A8C94AC2000:9 X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Maxim, I would very much like to see your improvements to the pypi importer to be merged. Have you been able to separate the independent changes as suggested by Ludo? -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Mon May 20 00:06:19 2019 Received: (at 24450) by debbugs.gnu.org; 20 May 2019 04:06:19 +0000 Received: from localhost ([127.0.0.1]:35930 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hSZYz-0002rS-CC for submit@debbugs.gnu.org; Mon, 20 May 2019 00:06:19 -0400 Received: from mail-qt1-f170.google.com ([209.85.160.170]:38429) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hSZYt-0002rB-W5 for 24450@debbugs.gnu.org; Mon, 20 May 2019 00:06:10 -0400 Received: by mail-qt1-f170.google.com with SMTP id l3so3972847qtj.5 for <24450@debbugs.gnu.org>; Sun, 19 May 2019 21:06:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=9y/qCtQ3fqYVlpocNkBDY2Mz07C4QStpc06TE0O95L0=; b=ssaEOrp2wVz4ZptvM8Cl4FjhO8DCbdEhFf1VCXGi1Gr5YzokrNkbC4SavXqL+9GoQx uewIyUUlP78Y79FolyiwqvJz4KLCqvaR3owI9QKPlXN+beOXj7xdJg3byBbckzsVA/92 tBlW2ygIqs+xRXvqZDDbGPsHt0AaAEXaAot8JeW+z4KG4RxqjI6JQOX2zOIScqQiuI4+ hgmOZx2xfL/p5Rls3uCstL5q+8CxiD0rqa2LnY+z8FxrdxlUpVX6X/NkswcFFRKtBOYQ 4iOXYtMSy+7Va5BwiC4gapIA/t7mtGfzNqWu6DrxO7Bi/KIrkPTtzOPx9yw46Exe3/Ji FZiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=9y/qCtQ3fqYVlpocNkBDY2Mz07C4QStpc06TE0O95L0=; b=MWXyqdCnYW0JdpnMCUJXy0/4RW566DTxZiokfFq74pHx+3tUfgpJ/isrQrilD0KSJ5 zjgYO2A7vKvOmwJKjTQXpWgY5Eikz7fskmA1QnG/bEq22sQ6gb2jQmdXLOiQ/l4cGdz1 Z+EDF6c3Wv9HggMT3ITXnQLFjBLhkToi+/Rt+ivTYtQ+YqOA0luy3yNV3tECgVc05FeY JLn93jLmXh8hC7HVVLDoij1Uq039w5aFOzMaI1OVvFnp1sj9u+m5Hg9o0zCcYcdZqTlb eAiSZrSJipJOZEIdoxfu9FJzwqG5cg5DnHfWeV/T7Ui/UtRBWC5B5E0lhsA3w9dxo9OC lwCg== X-Gm-Message-State: APjAAAWIkdMGdIBDOMHMpEYbLkPTbMI7NAAxZ6FWK4iS8jyhoG9hJIvp 92PGCrsw27R6W/JzRJXXSoKSnaur X-Google-Smtp-Source: APXvYqz1WR45bXuo+oycun9h1Q/loqlBtFgQZTc6Tm9uDzXmLT5Waq1+//AeNQtSe59/gcb/ZaCmgg== X-Received: by 2002:a0c:d941:: with SMTP id t1mr58198518qvj.204.1558325162441; Sun, 19 May 2019 21:06:02 -0700 (PDT) Received: from kwak (dsl-152-210.b2b2c.ca. [66.158.152.210]) by smtp.gmail.com with ESMTPSA id s3sm9358892qtb.12.2019.05.19.21.06.00 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 19 May 2019 21:06:00 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: Date: Mon, 20 May 2019 00:05:59 -0400 In-Reply-To: (Ricardo Wurmus's message of "Wed, 15 May 2019 13:06:37 +0200") Message-ID: <87pnod7ot4.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain Hi Ricardo! Ricardo Wurmus writes: > Hi Maxim, > > I would very much like to see your improvements to the pypi importer to > be merged. Have you been able to separate the independent changes as > suggested by Ludo? I'm thrilled that someone has an interest in this :-) I took my time, but finally got around to restructure the changes a bit. I hope it'll be easier to review this time around! Thank you! Maxim --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-import-pypi-Do-not-consider-requirements.txt-files.patch Content-Transfer-Encoding: quoted-printable From=2054e44b7397f17910d95dbdb233d23e5c97c095aa Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:00 -0400 Subject: [PATCH 1/9] import: pypi: Do not consider requirements.txt files. * guix/import/pypi.scm (guess-requirements): Update comment. [guess-requirements-from-source]: Do not attempt to parse the file requirements.txt. Streamline logic. =2D-- guix/import/pypi.scm | 35 +++++++++++++---------------------- tests/pypi.scm | 23 +++++++++++------------ 2 files changed, 24 insertions(+), 34 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 3a20fc4b9b..8269aa61d7 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -206,35 +206,26 @@ cannot determine package dependencies")) (call-with-temporary-directory (lambda (dir) (let* ((pypi-name (string-take dirname (string-rindex dirname= #\-))) =2D (req-files (list (string-append dirname "/requiremen= ts.txt") =2D (string-append dirname "/" pypi-nam= e ".egg-info" =2D "/requires.txt"))) =2D (exit-codes (map (lambda (file-name) =2D (parameterize ((current-error-por= t (%make-void-port "rw+")) =2D (current-output-po= rt (%make-void-port "rw+"))) =2D (system* "tar" "xf" tarball "-C= " dir file-name))) =2D req-files))) =2D ;; Only one of these files needs to exist. =2D (if (any zero? exit-codes) =2D (match (find-files dir) =2D ((file . _) =2D (read-requirements file)) =2D (() =2D (warning (G_ "No requirements file found.\n")))) + (requires.txt (string-append dirname "/" pypi-name + ".egg-info" "/requires.tx= t")) + (exit-code (parameterize ((current-error-port (%make-v= oid-port "rw+")) + (current-output-port (%make-= void-port "rw+"))) + (system* "tar" "xf" tarball "-C" dir requ= ires.txt)))) + (if (zero? exit-code) + (read-requirements (string-append dir "/" requires.txt)) (begin =2D (warning (G_ "Failed to extract requirements files\= n")) + (warning + (G_ "Failed to extract file: ~a from source.~%") + requires.txt) '()))))) '()))) =20 =2D ;; First, try to compute the requirements using the wheel, since that = is the =2D ;; most reliable option. If a wheel is not provided for this package, = try =2D ;; getting them by reading either the "requirements.txt" file or the =2D ;; "requires.txt" from the egg-info directory from the source tarball.= Note =2D ;; that "requirements.txt" is not mandatory, so this is likely to fail. + ;; First, try to compute the requirements using the wheel, else, fallbac= k to + ;; reading the "requires.txt" from the egg-info directory from the source + ;; tarball. (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 =2D (define (compute-inputs source-url wheel-url tarball) "Given the SOURCE-URL of an already downloaded TARBALL, return a list of name/variable pairs describing the required inputs of this package. Also diff --git a/tests/pypi.scm b/tests/pypi.scm index 6daa44a6e7..335be42644 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -23,7 +23,7 @@ #:use-module (gcrypt hash) #:use-module (guix tests) #:use-module (guix build-system python) =2D #:use-module ((guix build utils) #:select (delete-file-recursively whi= ch)) + #:use-module ((guix build utils) #:select (delete-file-recursively which= mkdir-p)) #:use-module (srfi srfi-64) #:use-module (ice-9 match)) =20 @@ -55,11 +55,10 @@ (define test-source-hash "") =20 =2D(define test-requirements =2D"# A comment =2D # A comment after a space +(define test-requires.txt "\ bar =2Dbaz > 13.37") +baz > 13.37 +") =20 (define test-metadata "{ @@ -107,10 +106,10 @@ baz > 13.37") (match url ("https://example.com/foo-1.0.0.tar.gz" (begin =2D (mkdir "foo-1.0.0") =2D (with-output-to-file "foo-1.0.0/requirements.txt" + (mkdir-p "foo-1.0.0/foo.egg-info/") + (with-output-to-file "foo-1.0.0/foo.egg-info/requires.tx= t" (lambda () =2D (display test-requirements))) + (display test-requires.txt))) (system* "tar" "czvf" file-name "foo-1.0.0/") (delete-file-recursively "foo-1.0.0") (set! test-source-hash @@ -157,11 +156,11 @@ baz > 13.37") (lambda (url file-name) (match url ("https://example.com/foo-1.0.0.tar.gz" =2D (begin =2D (mkdir "foo-1.0.0") =2D (with-output-to-file "foo-1.0.0/requirements.txt" + (begin + (mkdir-p "foo-1.0.0/foo.egg-info/") + (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" (lambda () =2D (display test-requirements))) + (display test-requires.txt))) (system* "tar" "czvf" file-name "foo-1.0.0/") (delete-file-recursively "foo-1.0.0") (set! test-source-hash =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0002-import-pypi-Do-not-parse-optional-requirements-from-.patch Content-Transfer-Encoding: quoted-printable From=205f79b0502f62bd1dacc8ea143c1dbd9ef7cfc29d Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:00 -0400 Subject: [PATCH 2/9] import: pypi: Do not parse optional requirements from source. * guix/import/pypi.scm: Export PARSE-REQUIRES.TXT. (guess-requirements): Move the READ-REQUIREMENTS procedure to the top level, and rename it to PARSE-REQUIRES.TXT. Move the CLEAN-REQUIREMENT and COMMEN= T? functions inside the READ-REQUIREMENTS procedure. (parse-requires.txt): Add a SECTION-HEADER? predicate, and use it to prevent parsing optional requirements. * tests/pypi.scm (test-requires-with-sections): New variable. ("parse-requires.txt, with sections"): New test. ("pypi->guix-package"): Mute tar output to stdout. =2D-- guix/import/pypi.scm | 76 +++++++++++++++++++++++++++----------------- tests/pypi.scm | 21 ++++++++++-- 2 files changed, 65 insertions(+), 32 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 8269aa61d7..91e987e9f1 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -47,7 +47,8 @@ #:use-module (guix upstream) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix build-system python) =2D #:export (guix-package->pypi-name + #:export (parse-requires.txt + guix-package->pypi-name pypi-recursive-import pypi->guix-package %pypi-updater)) @@ -117,6 +118,49 @@ package definition." ((package-inputs ...) `((propagated-inputs (,'quasiquote ,package-inputs)))))) =20 +(define (clean-requirement s) + ;; Given a requirement LINE, as can be found in a setuptools requires.txt + ;; file, remove everything other than the actual name of the required + ;; package, and return it. + (string-take s (or (string-index s (lambda (chr) + (member chr '(#\space #\> #\=3D #\<= )))) + (string-length s)))) + +(define (parse-requires.txt requires.txt) + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of +requirement names." + ;; This is a very incomplete parser, which job is to select the non-opti= onal + ;; dependencies and strip them out of any version information. + ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) + ;; library and the requirements grammar defined by PEP-0508 + ;; (https://www.python.org/dev/peps/pep-0508/). + + (define (comment? line) + ;; Return #t if the given LINE is a comment, #f otherwise. + (eq? (string-ref (string-trim line) 0) #\#)) + + (define (section-header? line) + ;; Return #t if the given LINE is a section header, #f otherwise. + (let ((trimmed-line (string-trim line))) + (and (not (string-null? trimmed-line)) + (eq? (string-ref trimmed-line 0) #\[)))) + + (call-with-input-file requires.txt + (lambda (port) + (let loop ((result '())) + (let ((line (read-line port))) + ;; Stop when a section is encountered, as sections contains opti= onal + ;; (extra) requirements. Non-optional requirements must appear + ;; before any section is defined. + (if (or (eof-object? line) (section-header? line)) + (reverse result) + (cond + ((or (string-null? line) (comment? line)) + (loop result)) + (else + (loop (cons (clean-requirement line) + result)))))))))) + (define (guess-requirements source-url wheel-url tarball) "Given SOURCE-URL, WHEEL-URL and a TARBALL of the package, return a list of the required packages specified in the requirements.txt file. TARBALL = will @@ -139,34 +183,6 @@ be extracted in a temporary directory." cannot determine package dependencies")) #f))))) =20 =2D (define (clean-requirement s) =2D ;; Given a requirement LINE, as can be found in a Python requirement= s.txt =2D ;; file, remove everything other than the actual name of the required =2D ;; package, and return it. =2D (string-take s =2D (or (string-index s (lambda (chr) (member chr '(#\space #\> #\=3D = #\<)))) =2D (string-length s)))) =2D =2D (define (comment? line) =2D ;; Return #t if the given LINE is a comment, #f otherwise. =2D (eq? (string-ref (string-trim line) 0) #\#)) =2D =2D (define (read-requirements requirements-file) =2D ;; Given REQUIREMENTS-FILE, a Python requirements.txt file, return a= list =2D ;; of name/variable pairs describing the requirements. =2D (call-with-input-file requirements-file =2D (lambda (port) =2D (let loop ((result '())) =2D (let ((line (read-line port))) =2D (if (eof-object? line) =2D result =2D (cond =2D ((or (string-null? line) (comment? line)) =2D (loop result)) =2D (else =2D (loop (cons (clean-requirement line) =2D result)))))))))) =2D (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's ;; requirements. @@ -212,7 +228,7 @@ cannot determine package dependencies")) (current-output-port (%make-= void-port "rw+"))) (system* "tar" "xf" tarball "-C" dir requ= ires.txt)))) (if (zero? exit-code) =2D (read-requirements (string-append dir "/" requires.tx= t)) + (parse-requires.txt (string-append dir "/" requires.txt= )) (begin (warning (G_ "Failed to extract file: ~a from source.~%") diff --git a/tests/pypi.scm b/tests/pypi.scm index 335be42644..e4b7142311 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -60,6 +60,15 @@ bar baz > 13.37 ") =20 +(define test-requires-with-sections "\ +# A comment +foo ~=3D 3 +bar !=3D 2 + +[test] +pytest (>=3D2.5.0) +") + (define test-metadata "{ \"run_requires\": [ @@ -99,6 +108,12 @@ baz > 13.37 (uri (list "https://bitheap.org/cram/cram-0.7.tar.gz" (pypi-uri "cram" "0.7")))))))) =20 +(test-equal "parse-requires.txt, with sections" + '("foo" "bar") + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-requires.txt test-requires-with-sections))) + (test-assert "pypi->guix-package" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch @@ -110,7 +125,8 @@ baz > 13.37 (with-output-to-file "foo-1.0.0/foo.egg-info/requires.tx= t" (lambda () (display test-requires.txt))) =2D (system* "tar" "czvf" file-name "foo-1.0.0/") + (parameterize ((current-output-port (%make-void-port "rw= +"))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) (delete-file-recursively "foo-1.0.0") (set! test-source-hash (call-with-input-file file-name port-sha256)))) @@ -161,7 +177,8 @@ baz > 13.37 (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" (lambda () (display test-requires.txt))) =2D (system* "tar" "czvf" file-name "foo-1.0.0/") + (parameterize ((current-output-port (%make-void-port "rw+"= ))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) (delete-file-recursively "foo-1.0.0") (set! test-source-hash (call-with-input-file file-name port-sha256)))) =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0003-import-pypi-Improve-parsing-of-requirement-specifica.patch Content-Transfer-Encoding: quoted-printable From=200c62b541a3e8925b5ca31fe55dbe7536cf95151f Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:01 -0400 Subject: [PATCH 3/9] import: pypi: Improve parsing of requirement specifications. The previous solution was fragile and could leave unwanted characters in a requirement name, such as '[' or ']'. Partially fixes issue #33047 (see: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D33047). * guix/import/pypi.scm (use-modules): Export SPECIFICATION->REQUIREMENT-NAME (%requirement-name-regexp): New variable. (clean-requirement): Rename to... (specification->requirement-name): this, which now uses %requirement-name-regexp to select the requirement name from the requirement specification. (parse-requires.txt): Adapt. =2D-- guix/import/pypi.scm | 43 ++++++++++++++++++++++++++++++++++--------- tests/pypi.scm | 12 ++++++++++++ 2 files changed, 46 insertions(+), 9 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 91e987e9f1..efb5939c78 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -48,6 +48,7 @@ #:use-module ((guix licenses) #:prefix license:) #:use-module (guix build-system python) #:export (parse-requires.txt + specification->requirement-name guix-package->pypi-name pypi-recursive-import pypi->guix-package @@ -118,13 +119,37 @@ package definition." ((package-inputs ...) `((propagated-inputs (,'quasiquote ,package-inputs)))))) =20 =2D(define (clean-requirement s) =2D ;; Given a requirement LINE, as can be found in a setuptools requires.= txt =2D ;; file, remove everything other than the actual name of the required =2D ;; package, and return it. =2D (string-take s (or (string-index s (lambda (chr) =2D (member chr '(#\space #\> #\=3D #= \<)))) =2D (string-length s)))) +(define %requirement-name-regexp + ;; Regexp to match the requirement name in a requirement specification. + + ;; Some grammar, taken from PEP-0508 (see: + ;; https://www.python.org/dev/peps/pep-0508/). + + ;; The unified rule can be expressed as: + ;; specification =3D wsp* ( url_req | name_req ) wsp* + + ;; where url_req is: + ;; url_req =3D name wsp* extras? wsp* urlspec wsp+ quoted_marker? + + ;; and where name_req is: + ;; name_req =3D name wsp* extras? wsp* versionspec? wsp* quoted_marker? + + ;; Thus, we need only matching NAME, which is expressed as: + ;; identifer_end =3D letterOrDigit | (('-' | '_' | '.' )* letterOrDigit) + ;; identifier =3D letterOrDigit identifier_end* + ;; name =3D identifier + (let* ((letter-or-digit "[A-Za-z0-9]") + (identifier-end (string-append "(" letter-or-digit "|" + "[-_.]*" letter-or-digit ")")) + (identifier (string-append "^" letter-or-digit identifier-end "*"= )) + (name identifier)) + (make-regexp name))) + +(define (specification->requirement-name spec) + "Given a specification SPEC, return the requirement name." + (match:substring + (or (regexp-exec %requirement-name-regexp spec) + (error (G_ "Could not extract requirement name in spec:") spec)))) =20 (define (parse-requires.txt requires.txt) "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of @@ -158,7 +183,7 @@ requirement names." ((or (string-null? line) (comment? line)) (loop result)) (else =2D (loop (cons (clean-requirement line) + (loop (cons (specification->requirement-name line) result)))))))))) =20 (define (guess-requirements source-url wheel-url tarball) @@ -200,7 +225,7 @@ cannot determine package dependencies")) (hash-ref (list-ref run_requir= es 0) "requires") '()))) =2D (map clean-requirement requirements))))) + (map specification->requirement-name requirements))))) (lambda () (delete-file json-file) (rmdir dirname)))))) diff --git a/tests/pypi.scm b/tests/pypi.scm index e4b7142311..82d6bba8dd 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -55,6 +55,14 @@ (define test-source-hash "") =20 +(define test-specifications + '("Fizzy [foo, bar]" + "PickyThing<1.6,>1.9,!=3D1.9.6,<2.0a0,=3D=3D2.4c1" + "SomethingWithMarker[foo]>1.0;python_version<\"2.7\"" + "requests [security,tests] >=3D 2.8.1, =3D=3D 2.8.* ; python_version <= \"2.7\"" + "pip @ https://github.com/pypa/pip/archive/1.3.1.zip#\ +sha1=3Dda9234ee9982d4bbb3c72346a6de940a148ea686")) + (define test-requires.txt "\ bar baz > 13.37 @@ -108,6 +116,10 @@ pytest (>=3D2.5.0) (uri (list "https://bitheap.org/cram/cram-0.7.tar.gz" (pypi-uri "cram" "0.7")))))))) =20 +(test-equal "specification->requirement-name" + '("Fizzy" "PickyThing" "SomethingWithMarker" "requests" "pip") + (map specification->requirement-name test-specifications)) + (test-equal "parse-requires.txt, with sections" '("foo" "bar") (mock ((ice-9 ports) call-with-input-file =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0004-import-pypi-Deduplicate-requirements.patch Content-Transfer-Encoding: quoted-printable From=2076e4a3150f8126e0b952c6129b6e1371afba80c0 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:01 -0400 Subject: [PATCH 4/9] import: pypi: Deduplicate requirements. * guix/import/pypi.scm (parse-requires.txt): Remove potential duplicates. =2D-- guix/import/pypi.scm | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index efb5939c78..a90be67bb0 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -178,7 +178,11 @@ requirement names." ;; (extra) requirements. Non-optional requirements must appear ;; before any section is defined. (if (or (eof-object? line) (section-header? line)) =2D (reverse result) + ;; Duplicates can occur, since the same requirement can be + ;; listed multiple times with different conditional markers,= e.g. + ;; pytest >=3D 3 ; python_version >=3D "3.3" + ;; pytest < 3 ; python_version < "3.3" + (reverse (delete-duplicates result)) (cond ((or (string-null? line) (comment? line)) (loop result)) =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0005-import-pypi-Support-more-types-of-archives.patch Content-Transfer-Encoding: quoted-printable From=2073e27235cac1275ba7671fd2364325cf5788cb3c Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:02 -0400 Subject: [PATCH 5/9] import: pypi: Support more types of archives. This change enables the PyPI importer to look for requirements in a source archive of a different type than "tar.gz" or "tar.bz2". * guix/import/pypi.scm: (guess-requirements)[tarball-directory]: Rename to.= .. [archive-root-directory]: this. Use COMPRESSED-FILED? to determine if an archive is supported or not. [guess-requirements-from-source]: Adapt to use the new method, and use unzip to extract ZIP archives. (guess-requirements): Rename the TARBALL argument to ARCHIVE, to denote the archive format is no longer bound specifically to the Tar format. =2D-- guix/import/pypi.scm | 47 ++++++++++++++++++++++---------------------- 1 file changed, 24 insertions(+), 23 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index a90be67bb0..8e93653717 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -190,27 +190,24 @@ requirement names." (loop (cons (specification->requirement-name line) result)))))))))) =20 =2D(define (guess-requirements source-url wheel-url tarball) =2D "Given SOURCE-URL, WHEEL-URL and a TARBALL of the package, return a li= st =2Dof the required packages specified in the requirements.txt file. TARBAL= L will +(define (guess-requirements source-url wheel-url archive) + "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a list +of the required packages specified in the requirements.txt file. ARCHIVE = will be extracted in a temporary directory." =20 =2D (define (tarball-directory url) =2D ;; Given the URL of the package's tarball, return the name of the di= rectory + (define (archive-root-directory url) + ;; Given the URL of the package's archive, return the name of the dire= ctory ;; that will be created upon decompressing it. If the filetype is not ;; supported, return #f. =2D ;; TODO: Support more archive formats. =2D (let ((basename (substring url (+ 1 (string-rindex url #\/))))) =2D (cond =2D ((string-suffix? ".tar.gz" basename) =2D (string-drop-right basename 7)) =2D ((string-suffix? ".tar.bz2" basename) =2D (string-drop-right basename 8)) =2D (else + (if (compressed-file? url) + (let ((root-directory (file-sans-extension (basename url)))) + (if (string=3D? "tar" (file-extension root-directory)) + (file-sans-extension root-directory) + root-directory)) (begin =2D (warning (G_ "Unsupported archive format: \ =2Dcannot determine package dependencies")) =2D #f))))) + (warning (G_ "Unsupported archive format (~a): \ +cannot determine package dependencies") (file-extension url)) + #f))) =20 (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's @@ -246,16 +243,20 @@ cannot determine package dependencies")) =20 (define (guess-requirements-from-source) ;; Return the package's requirements by guessing them from the source. =2D (let ((dirname (tarball-directory source-url))) + (let ((dirname (archive-root-directory source-url)) + (extension (file-extension source-url))) (if (string? dirname) (call-with-temporary-directory (lambda (dir) (let* ((pypi-name (string-take dirname (string-rindex dirname= #\-))) (requires.txt (string-append dirname "/" pypi-name ".egg-info" "/requires.tx= t")) =2D (exit-code (parameterize ((current-error-port (%make= -void-port "rw+")) =2D (current-output-port (%mak= e-void-port "rw+"))) =2D (system* "tar" "xf" tarball "-C" dir re= quires.txt)))) + (exit-code + (parameterize ((current-error-port (%make-void-port "= rw+")) + (current-output-port (%make-void-port = "rw+"))) + (if (string=3D? "zip" extension) + (system* "unzip" archive "-d" dir requires.txt) + (system* "tar" "xf" archive "-C" dir requires.t= xt))))) (if (zero? exit-code) (parse-requires.txt (string-append dir "/" requires.txt= )) (begin @@ -271,13 +272,13 @@ cannot determine package dependencies")) (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 =2D(define (compute-inputs source-url wheel-url tarball) =2D "Given the SOURCE-URL of an already downloaded TARBALL, return a list = of +(define (compute-inputs source-url wheel-url archive) + "Given the SOURCE-URL of an already downloaded ARCHIVE, return a list of name/variable pairs describing the required inputs of this package. Also return the unaltered list of upstream dependency names." (let ((dependencies (remove (cut string=3D? "argparse" <>) =2D (guess-requirements source-url wheel-url tarball)))) + (guess-requirements source-url wheel-url archive)))) (values (sort (map (lambda (input) (let ((guix-name (python->package-name input))) =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0006-import-pypi-Parse-wheel-METADATA-instead-of-metadata.patch Content-Transfer-Encoding: quoted-printable From=20fb0547ef225103c0f8355a7eccc41e0d028f6563 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:03 -0400 Subject: [PATCH 6/9] import: pypi: Parse wheel METADATA instead of metadata.json. With newer Wheel releases, there is no more metadata.json file; the METADATA file should be used instead (see: https://github.com/pypa/wheel/issues/195). This change updates our PyPI importer so that it uses the later. * guix/import/pypi.scm (define-module): Remove unnecessary modules and expo= rt the PARSE-WHEEL-METADATA method. (parse-wheel-metadata): Add method. (guess-requirements): Use it. * tests/pypi.scm (test-metadata): Test it. =2D-- guix/import/pypi.scm | 66 +++++++++++++++++++++++++++++--------------- tests/pypi.scm | 60 ++++++++++++++++++++++++++++++---------- 2 files changed, 89 insertions(+), 37 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 8e93653717..c520213b6a 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -21,9 +21,7 @@ ;;; along with GNU Guix. If not, see . =20 (define-module (guix import pypi) =2D #:use-module (ice-9 binary-ports) #:use-module (ice-9 match) =2D #:use-module (ice-9 pretty-print) #:use-module (ice-9 regex) #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) @@ -31,9 +29,6 @@ #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) =2D #:use-module (rnrs bytevectors) =2D #:use-module (json) =2D #:use-module (web uri) #:use-module (guix ui) #:use-module (guix utils) #:use-module ((guix build utils) @@ -48,6 +43,7 @@ #:use-module ((guix licenses) #:prefix license:) #:use-module (guix build-system python) #:export (parse-requires.txt + parse-wheel-metadata specification->requirement-name guix-package->pypi-name pypi-recursive-import @@ -190,6 +186,37 @@ requirement names." (loop (cons (specification->requirement-name line) result)))))))))) =20 +(define (parse-wheel-metadata metadata) + "Given METADATA, a Wheel metadata file, return a list of requirement nam= es." + ;; METADATA is a RFC-2822-like, header based file. + + (define (requires-dist-header? line) + ;; Return #t if the given LINE is a Requires-Dist header. + (regexp-match? (string-match "^Requires-Dist: " line))) + + (define (requires-dist-value line) + (string-drop line (string-length "Requires-Dist: "))) + + (define (extra? line) + ;; Return #t if the given LINE is an "extra" requirement. + (regexp-match? (string-match "extra =3D=3D " line))) + + (call-with-input-file metadata + (lambda (port) + (let loop ((requirements '())) + (let ((line (read-line port))) + ;; Stop at the first 'Provides-Extra' section: the non-optional + ;; requirements appear before the optional ones. + (if (eof-object? line) + (reverse (delete-duplicates requirements)) + (cond + ((and (requires-dist-header? line) (not (extra? line))) + (loop (cons (specification->requirement-name + (requires-dist-value line)) + requirements))) + (else + (loop requirements))))))))) + (define (guess-requirements source-url wheel-url archive) "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a list of the required packages specified in the requirements.txt file. ARCHIVE = will @@ -211,25 +238,18 @@ cannot determine package dependencies") (file-extensi= on url)) =20 (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's =2D ;; requirements. + ;; requirements, or #f if the metadata file contained therein couldn't= be + ;; extracted. (let* ((dirname (wheel-url->extracted-directory wheel-url)) =2D (json-file (string-append dirname "/metadata.json"))) =2D (and (zero? (system* "unzip" "-q" wheel-archive json-file)) =2D (dynamic-wind =2D (const #t) =2D (lambda () =2D (call-with-input-file json-file =2D (lambda (port) =2D (let* ((metadata (json->scm port)) =2D (run_requires (hash-ref metadata "run_requires= ")) =2D (requirements (if run_requires =2D (hash-ref (list-ref run_requ= ires 0) =2D "requires") =2D '()))) =2D (map specification->requirement-name requirements))= ))) =2D (lambda () =2D (delete-file json-file) =2D (rmdir dirname)))))) + (metadata (string-append dirname "/METADATA"))) + (call-with-temporary-directory + (lambda (dir) + (if (zero? (system* "unzip" "-q" wheel-archive "-d" dir metadata)) + (parse-wheel-metadata (string-append dir "/" metadata)) + (begin + (warning + (G_ "Failed to extract file: ~a from wheel.~%") metadata) + #f)))))) =20 (define (guess-requirements-from-wheel) ;; Return the package's requirements using the wheel, or #f if an error diff --git a/tests/pypi.scm b/tests/pypi.scm index 82d6bba8dd..ca8cb5f6de 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -21,6 +21,7 @@ #:use-module (guix import pypi) #:use-module (guix base32) #:use-module (gcrypt hash) + #:use-module (guix memoization) #:use-module (guix tests) #:use-module (guix build-system python) #:use-module ((guix build utils) #:select (delete-file-recursively which= mkdir-p)) @@ -77,17 +78,33 @@ bar !=3D 2 pytest (>=3D2.5.0) ") =20 =2D(define test-metadata =2D "{ =2D \"run_requires\": [ =2D { =2D \"requires\": [ =2D \"bar\", =2D \"baz (>13.37)\" =2D ] =2D } =2D ] =2D}") +(define test-metadata "\ +Classifier: Programming Language :: Python :: 3.7 +Requires-Dist: baz ~=3D 3 +Requires-Dist: bar !=3D 2 +Provides-Extra: test +pytest (>=3D2.5.0) +") + +(define test-metadata-with-extras " +Classifier: Programming Language :: Python :: 3.7 +Requires-Python: >=3D2.7, !=3D3.0.*, !=3D3.1.*, !=3D3.2.*, !=3D3.3.* +Requires-Dist: wrapt (<2,>=3D1) +Requires-Dist: bar + +Provides-Extra: dev +Requires-Dist: tox ; extra =3D=3D 'dev' +Requires-Dist: bumpversion (<1) ; extra =3D=3D 'dev' +") + +;;; Provides-Extra can appear before Requires-Dist. +(define test-metadata-with-extras-jedi "\ +Requires-Python: >=3D2.7, !=3D3.0.*, !=3D3.1.*, !=3D3.2.*, !=3D3.3.* +Provides-Extra: testing +Requires-Dist: parso (>=3D0.3.0) +Provides-Extra: testing +Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testing' +") =20 (test-begin "pypi") =20 @@ -126,6 +143,18 @@ pytest (>=3D2.5.0) call-with-input-string) (parse-requires.txt test-requires-with-sections))) =20 +(test-equal "parse-wheel-metadata, with extras" + '("wrapt" "bar") + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-wheel-metadata test-metadata-with-extras))) + +(test-equal "parse-wheel-metadata, with extras - Jedi" + '("parso") + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-wheel-metadata test-metadata-with-extras-jedi))) + (test-assert "pypi->guix-package" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch @@ -188,7 +217,7 @@ pytest (>=3D2.5.0) (mkdir-p "foo-1.0.0/foo.egg-info/") (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" (lambda () =2D (display test-requires.txt))) + (display "wrong data to make sure we're testing wheel= s "))) (parameterize ((current-output-port (%make-void-port "rw+"= ))) (system* "tar" "czvf" file-name "foo-1.0.0/")) (delete-file-recursively "foo-1.0.0") @@ -197,13 +226,13 @@ pytest (>=3D2.5.0) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" (begin (mkdir "foo-1.0.0.dist-info") =2D (with-output-to-file "foo-1.0.0.dist-info/metadata.json" + (with-output-to-file "foo-1.0.0.dist-info/METADATA" (lambda () (display test-metadata))) (let ((zip-file (string-append file-name ".zip"))) ;; zip always adds a "zip" extension to the file it cre= ates, ;; so we need to rename it. =2D (system* "zip" zip-file "foo-1.0.0.dist-info/metadata= .json") + (system* "zip" zip-file "foo-1.0.0.dist-info/METADATA") (rename-file zip-file file-name)) (delete-file-recursively "foo-1.0.0.dist-info"))) (_ (error "Unexpected URL: " url))))) @@ -215,6 +244,9 @@ pytest (>=3D2.5.0) (string-length test-json))) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #= f) (_ (error "Unexpected URL: " url))))) + ;; Not clearing the memoization cache here would mean return= ing the value + ;; computed in the previous test. + (invalidate-memoization! pypi->guix-package) (match (pypi->guix-package "foo") (('package ('name "python-foo") =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: attachment; filename=0007-import-pypi-Include-optional-test-inputs-as-native-i.patch Content-Transfer-Encoding: quoted-printable From=2037e499d5d5d5f690aa0a065c730e13f6a31dd30d Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 23:12:26 -0400 Subject: [PATCH 7/9] import: pypi: Include optional test inputs as native-inputs. * guix/import/pypi.scm (maybe-inputs): Add INPUT-TYPE argument, and use it. (test-section?): New predicate. (parse-requires.txt): Collect the optional test inputs, and return them as = the second element of the returned list. (parse-wheel-metadata): Likewise. (guess-requirements): Adapt, and hide unzip output. (make-pypi-sexp): Likewise, and include the test inputs requirements as nat= ive inputs in the returned package expression. * tests/pypi.scm (test-requires.txt): Include a test section in the test-requires.txt data. (test-requires.txt-beaker): New variable. ("parse-requires.txt"): Adapt. ("parse-requires.txt - Beaker"): New test. ("parse-wheel-metadata, with extras"): Adapt. ("parse-wheel-metadata, with extras - Jedi"): Adapt. ("pypi->guix-package, no wheel"): Re-indent, and add the expected native-inputs. ("pypi->guix-package, wheels"): Likewise. ("pypi->guix-package, no usable requirement file."): New test. =2D-- guix/import/pypi.scm | 195 ++++++++++++++++++++++++++++--------------- tests/pypi.scm | 123 ++++++++++++++++++++------- 2 files changed, 222 insertions(+), 96 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index c520213b6a..099768f0c8 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -4,6 +4,7 @@ ;;; Copyright =C2=A9 2015, 2016, 2017 Ludovic Court=C3=A8s ;;; Copyright =C2=A9 2017 Mathieu Othacehe ;;; Copyright =C2=A9 2018 Ricardo Wurmus +;;; Copyright =C2=A9 2019 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -26,6 +27,7 @@ #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) @@ -106,14 +108,15 @@ package on PyPI." ((name version _ ...) (string-append name "-" version ".dist-info")))) =20 =2D(define (maybe-inputs package-inputs) +(define (maybe-inputs package-inputs input-type) "Given a list of PACKAGE-INPUTS, tries to generate the 'inputs' field of= a =2Dpackage definition." +package definition. INPUT-TYPE, a symbol, is used to populate the name of +the input field." (match package-inputs (() '()) ((package-inputs ...) =2D `((propagated-inputs (,'quasiquote ,package-inputs)))))) + `((,input-type (,'quasiquote ,package-inputs)))))) =20 (define %requirement-name-regexp ;; Regexp to match the requirement name in a requirement specification. @@ -147,11 +150,21 @@ package definition." (or (regexp-exec %requirement-name-regexp spec) (error (G_ "Could not extract requirement name in spec:") spec)))) =20 +(define (test-section? name) + "Return #t if the section name contains 'test' or 'dev'." + (any (cut string-contains-ci name <>) + '("test" "dev"))) + (define (parse-requires.txt requires.txt) =2D "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of =2Drequirement names." =2D ;; This is a very incomplete parser, which job is to select the non-op= tional =2D ;; dependencies and strip them out of any version information. + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a pair of re= quirements. + +The first element of the pair contains the required dependencies while the +second the optional test dependencies. Note that currently, optional, +non-test dependencies are omitted since these can be difficult or expensiv= e to +satisfy." + + ;; This is a very incomplete parser, which job is to read in the require= ment + ;; specification lines, and strip them out of any version information. ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) ;; library and the requirements grammar defined by PEP-0508 ;; (https://www.python.org/dev/peps/pep-0508/). @@ -168,57 +181,89 @@ requirement names." =20 (call-with-input-file requires.txt (lambda (port) =2D (let loop ((result '())) + (let loop ((required-deps '()) + (test-deps '()) + (inside-test-section? #f) + (optional? #f)) (let ((line (read-line port))) =2D ;; Stop when a section is encountered, as sections contains op= tional =2D ;; (extra) requirements. Non-optional requirements must appear =2D ;; before any section is defined. =2D (if (or (eof-object? line) (section-header? line)) + (if (eof-object? line) ;; Duplicates can occur, since the same requirement can be ;; listed multiple times with different conditional markers,= e.g. ;; pytest >=3D 3 ; python_version >=3D "3.3" ;; pytest < 3 ; python_version < "3.3" =2D (reverse (delete-duplicates result)) + (map (compose reverse delete-duplicates) + (list required-deps test-deps)) (cond ((or (string-null? line) (comment? line)) =2D (loop result)) =2D (else + (loop required-deps test-deps inside-test-section? optiona= l?)) + ((section-header? line) + ;; Encountering a section means that all the requirements + ;; listed below are optional. Since we want to pick only t= he + ;; test dependencies from the optional dependencies, we mu= st + ;; track those separately. + (loop required-deps test-deps (test-section? line) #t)) + (inside-test-section? + (loop required-deps + (cons (specification->requirement-name line) + test-deps) + inside-test-section? optional?)) + ((not optional?) (loop (cons (specification->requirement-name line) =2D result)))))))))) + required-deps) + test-deps inside-test-section? optional?)) + (optional? + ;; Skip optional items. + (loop required-deps test-deps inside-test-section? optiona= l?)) + (else + (warning (G_ "parse-requires.txt reached an unexpected \ +condition on line ~a~%") line))))))))) =20 (define (parse-wheel-metadata metadata) =2D "Given METADATA, a Wheel metadata file, return a list of requirement n= ames." + "Given METADATA, a Wheel metadata file, return a pair of requirements. + +The first element of the pair contains the required dependencies while the= second the optional +test dependencies. Note that currently, optional, non-test dependencies a= re +omitted since these can be difficult or expensive to satisfy." ;; METADATA is a RFC-2822-like, header based file. =20 (define (requires-dist-header? line) ;; Return #t if the given LINE is a Requires-Dist header. =2D (regexp-match? (string-match "^Requires-Dist: " line))) + (string-match "^Requires-Dist: " line)) =20 (define (requires-dist-value line) (string-drop line (string-length "Requires-Dist: "))) =20 (define (extra? line) ;; Return #t if the given LINE is an "extra" requirement. =2D (regexp-match? (string-match "extra =3D=3D " line))) + (string-match "extra =3D=3D '(.*)'" line)) + + (define (test-requirement? line) + (let ((extra-label (match:substring (extra? line) 1))) + (and extra-label (test-section? extra-label)))) =20 (call-with-input-file metadata (lambda (port) =2D (let loop ((requirements '())) + (let loop ((required-deps '()) + (test-deps '())) (let ((line (read-line port))) =2D ;; Stop at the first 'Provides-Extra' section: the non-optional =2D ;; requirements appear before the optional ones. (if (eof-object? line) =2D (reverse (delete-duplicates requirements)) + (map (compose reverse delete-duplicates) + (list required-deps test-deps)) (cond ((and (requires-dist-header? line) (not (extra? line))) (loop (cons (specification->requirement-name (requires-dist-value line)) =2D requirements))) + required-deps) + test-deps)) + ((and (requires-dist-header? line) (test-requirement? line)) + (loop required-deps + (cons (specification->requirement-name (requires-dis= t-value line)) + test-deps))) (else =2D (loop requirements))))))))) + (loop required-deps test-deps))))))))) ;skip line =20 (define (guess-requirements source-url wheel-url archive) =2D "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a li= st + "Given SOURCE-URL, WHEEL-URL and an ARCHIVE of the package, return a list of the required packages specified in the requirements.txt file. ARCHIVE = will be extracted in a temporary directory." =20 @@ -244,7 +289,10 @@ cannot determine package dependencies") (file-extensio= n url)) (metadata (string-append dirname "/METADATA"))) (call-with-temporary-directory (lambda (dir) =2D (if (zero? (system* "unzip" "-q" wheel-archive "-d" dir metadat= a)) + (if (zero? + (parameterize ((current-error-port (%make-void-port "rw+")) + (current-output-port (%make-void-port "rw+"))) + (system* "unzip" wheel-archive "-d" dir metadata))) (parse-wheel-metadata (string-append dir "/" metadata)) (begin (warning @@ -283,32 +331,41 @@ cannot determine package dependencies") (file-extensi= on url)) (warning (G_ "Failed to extract file: ~a from source.~%") requires.txt) =2D '()))))) =2D '()))) + (list '() '())))))) + (list '() '())))) =20 ;; First, try to compute the requirements using the wheel, else, fallbac= k to ;; reading the "requires.txt" from the egg-info directory from the source =2D ;; tarball. + ;; archive. (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 (define (compute-inputs source-url wheel-url archive) =2D "Given the SOURCE-URL of an already downloaded ARCHIVE, return a list = of =2Dname/variable pairs describing the required inputs of this package. Also + "Given the SOURCE-URL and WHEEL-URL of an already downloaded ARCHIVE, re= turn +a pair of lists, each consisting of a list of name/variable pairs, for the +propagated inputs and the native inputs, respectively. Also return the unaltered list of upstream dependency names." =2D (let ((dependencies =2D (remove (cut string=3D? "argparse" <>) =2D (guess-requirements source-url wheel-url archive)))) =2D (values (sort =2D (map (lambda (input) =2D (let ((guix-name (python->package-name input))) =2D (list guix-name (list 'unquote (string->symbol gui= x-name))))) =2D dependencies) =2D (lambda args =2D (match args =2D (((a _ ...) (b _ ...)) =2D (string-ci) deps)) + + (define (requirement->package-name/sort deps) + (sort + (map (lambda (input) + (let ((guix-name (python->package-name input))) + (list guix-name (list 'unquote (string->symbol guix-name))))) + deps) + (lambda args + (match args + (((a _ ...) (b _ ...)) + (string-cipackage-name/sort strip-argparse)) + + (let ((dependencies (guess-requirements source-url wheel-url archive))) + (values (map process-requirements dependencies) + (concatenate dependencies)))) =20 (define (make-pypi-sexp name version source-url wheel-url home-page synops= is description license) @@ -317,29 +374,33 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION= , and LICENSE." (call-with-temporary-output-file (lambda (temp port) (and (url-fetch source-url temp) =2D (receive (input-package-names upstream-dependency-names) + (receive (guix-dependencies upstream-dependencies) (compute-inputs source-url wheel-url temp) =2D (values =2D `(package =2D (name ,(python->package-name name)) =2D (version ,version) =2D (source (origin =2D (method url-fetch) =2D =2D ;; Sometimes 'pypi-uri' doesn't quite work due= to mixed =2D ;; cases in NAME, for instance, as is the case= with =2D ;; "uwsgi". In that case, fall back to a full= URL. =2D (uri (pypi-uri ,(string-downcase name) version= )) =2D (sha256 =2D (base32 =2D ,(guix-hash-url temp))))) =2D (build-system python-build-system) =2D ,@(maybe-inputs input-package-names) =2D (home-page ,home-page) =2D (synopsis ,synopsis) =2D (description ,description) =2D (license ,(license->symbol license))) =2D upstream-dependency-names)))))) + (match guix-dependencies + ((required-inputs test-inputs) + (values + `(package + (name ,(python->package-name name)) + (version ,version) + (source (origin + (method url-fetch) + ;; Sometimes 'pypi-uri' doesn't quite work du= e to mixed + ;; cases in NAME, for instance, as is the cas= e with + ;; "uwsgi". In that case, fall back to a ful= l URL. + (uri (pypi-uri ,(string-downcase name) versio= n)) + (sha256 + (base32 + ,(guix-hash-url temp))))) + (build-system python-build-system) + ,@(maybe-inputs required-inputs 'propagated-inputs) + ,@(maybe-inputs test-inputs 'native-inputs) + (home-page ,home-page) + (synopsis ,synopsis) + (description ,description) + (license ,(license->symbol license))) + ;; Flatten the nested lists and return the upstream + ;; dependencies. + upstream-dependencies)))))))) =20 (define pypi->guix-package (memoize diff --git a/tests/pypi.scm b/tests/pypi.scm index ca8cb5f6de..aa08e2cb54 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2014 David Thompson ;;; Copyright =C2=A9 2016 Ricardo Wurmus +;;; Copyright =C2=A9 2019 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -65,11 +66,6 @@ sha1=3Dda9234ee9982d4bbb3c72346a6de940a148ea686")) =20 (define test-requires.txt "\ =2Dbar =2Dbaz > 13.37 =2D") =2D =2D(define test-requires-with-sections "\ # A comment foo ~=3D 3 bar !=3D 2 @@ -78,12 +74,25 @@ bar !=3D 2 pytest (>=3D2.5.0) ") =20 +;; Beaker contains only optional dependencies. +(define test-requires.txt-beaker "\ +[crypto] +pycryptopp>=3D0.5.12 + +[cryptography] +cryptography + +[testsuite] +Mock +coverage +") + (define test-metadata "\ Classifier: Programming Language :: Python :: 3.7 Requires-Dist: baz ~=3D 3 Requires-Dist: bar !=3D 2 Provides-Extra: test =2Dpytest (>=3D2.5.0) +Requires-Dist: pytest (>=3D2.5.0) ; extra =3D=3D 'test' ") =20 (define test-metadata-with-extras " @@ -137,25 +146,31 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'test= ing' '("Fizzy" "PickyThing" "SomethingWithMarker" "requests" "pip") (map specification->requirement-name test-specifications)) =20 =2D(test-equal "parse-requires.txt, with sections" =2D '("foo" "bar") +(test-equal "parse-requires.txt" + (list '("foo" "bar") '("pytest")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) =2D (parse-requires.txt test-requires-with-sections))) + (parse-requires.txt test-requires.txt))) + +(test-equal "parse-requires.txt - Beaker" + (list '() '("Mock" "coverage")) + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-requires.txt test-requires.txt-beaker))) =20 (test-equal "parse-wheel-metadata, with extras" =2D '("wrapt" "bar") + (list '("wrapt" "bar") '("tox" "bumpversion")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) (parse-wheel-metadata test-metadata-with-extras))) =20 (test-equal "parse-wheel-metadata, with extras - Jedi" =2D '("parso") + (list '("parso") '("pytest")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) (parse-wheel-metadata test-metadata-with-extras-jedi))) =20 =2D(test-assert "pypi->guix-package" +(test-assert "pypi->guix-package, no wheel" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch (lambda (url file-name) @@ -195,7 +210,10 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testi= ng' ('propagated-inputs ('quasiquote (("python-bar" ('unquote 'python-bar)) =2D ("python-baz" ('unquote 'python-baz))))) + ("python-foo" ('unquote 'python-foo))))) + ('native-inputs + ('quasiquote + (("python-pytest" ('unquote 'python-pytest))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary") @@ -216,25 +234,25 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'test= ing' (begin (mkdir-p "foo-1.0.0/foo.egg-info/") (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" =2D (lambda () =2D (display "wrong data to make sure we're testing whe= els "))) + (lambda () + (display "wrong data to make sure we're testing wheels= "))) (parameterize ((current-output-port (%make-void-port "rw+"= ))) (system* "tar" "czvf" file-name "foo-1.0.0/")) =2D (delete-file-recursively "foo-1.0.0") =2D (set! test-source-hash =2D (call-with-input-file file-name port-sha256)))) + (delete-file-recursively "foo-1.0.0") + (set! test-source-hash + (call-with-input-file file-name port-sha256)))) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" =2D (begin =2D (mkdir "foo-1.0.0.dist-info") =2D (with-output-to-file "foo-1.0.0.dist-info/METADATA" =2D (lambda () =2D (display test-metadata))) =2D (let ((zip-file (string-append file-name ".zip"))) =2D ;; zip always adds a "zip" extension to the file it c= reates, =2D ;; so we need to rename it. =2D (system* "zip" zip-file "foo-1.0.0.dist-info/METADATA= ") =2D (rename-file zip-file file-name)) =2D (delete-file-recursively "foo-1.0.0.dist-info"))) + (begin + (mkdir "foo-1.0.0.dist-info") + (with-output-to-file "foo-1.0.0.dist-info/METADATA" + (lambda () + (display test-metadata))) + (let ((zip-file (string-append file-name ".zip"))) + ;; zip always adds a "zip" extension to the file it crea= tes, + ;; so we need to rename it. + (system* "zip" "-q" zip-file "foo-1.0.0.dist-info/METADA= TA") + (rename-file zip-file file-name)) + (delete-file-recursively "foo-1.0.0.dist-info"))) (_ (error "Unexpected URL: " url))))) (mock ((guix http-client) http-fetch (lambda (url . rest) @@ -262,6 +280,9 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testin= g' ('quasiquote (("python-bar" ('unquote 'python-bar)) ("python-baz" ('unquote 'python-baz))))) + ('native-inputs + ('quasiquote + (("python-pytest" ('unquote 'python-pytest))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary") @@ -272,4 +293,48 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testi= ng' (x (pk 'fail x #f)))))) =20 +(test-assert "pypi->guix-package, no usable requirement file." + ;; Replace network resources with sample data. + (mock ((guix import utils) url-fetch + (lambda (url file-name) + (match url + ("https://example.com/foo-1.0.0.tar.gz" + (set! test-source-hash + (call-with-input-file file-name port-sha256)) + #t) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #t) + (_ (error "Unexpected URL: " url))))) + (mock ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ("https://pypi.org/pypi/foo/json" + (values (open-input-string test-json) + (string-length test-json))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #= f) + (_ (error "Unexpected URL: " url))))) + ;; Not clearing the memoization cache here would mean return= ing the value + ;; computed in the previous test. + (invalidate-memoization! pypi->guix-package) + (match (pypi->guix-package "foo") + (('package + ('name "python-foo") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri ('pypi-uri "foo" 'version)) + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'python-build-system) + ('home-page "http://example.com") + ('synopsis "summary") + ('description "summary") + ('license 'license:lgpl2.0)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (pk 'fail x #f))) + ))) + (test-end "pypi") =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0008-import-pypi-Scan-source-archive-to-find-requires.txt.patch Content-Transfer-Encoding: quoted-printable From=20cfde6e09f8f8c692fe252d76ed27e8c50a9e5377 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Sat, 30 Mar 2019 23:13:26 -0400 Subject: [PATCH 8/9] import: pypi: Scan source archive to find requires.txt file. * guix/import/pypi.scm (use-modules): Use invoke from (guix build utils). (guess-requirements)[archive-root-directory]: Remove procedure. [guess-requirements-from-wheel]: Re-ident. [guess-requirements-from-source]: Search for the requires.txt file in the archive instead of using a static, expected location. * tests/pypi.scm ("pypi->guix-package, no wheel"): Mock the requires.txt at= a non-standard location to test the new feature. ("pypi->guix-package, no usable requirement file."): Adapt. =2D-- guix/import/pypi.scm | 65 ++++++++++++++++++-------------------------- tests/pypi.scm | 17 +++++++----- 2 files changed, 37 insertions(+), 45 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 099768f0c8..a2ce14b192 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -36,7 +36,8 @@ #:use-module ((guix build utils) #:select ((package-name->name+version . hyphen-package-name->name+version) =2D find-files)) + find-files + invoke)) #:use-module (guix import utils) #:use-module ((guix download) #:prefix download:) #:use-module (guix import json) @@ -267,19 +268,6 @@ omitted since these can be difficult or expensive to s= atisfy." of the required packages specified in the requirements.txt file. ARCHIVE = will be extracted in a temporary directory." =20 =2D (define (archive-root-directory url) =2D ;; Given the URL of the package's archive, return the name of the di= rectory =2D ;; that will be created upon decompressing it. If the filetype is not =2D ;; supported, return #f. =2D (if (compressed-file? url) =2D (let ((root-directory (file-sans-extension (basename url)))) =2D (if (string=3D? "tar" (file-extension root-directory)) =2D (file-sans-extension root-directory) =2D root-directory)) =2D (begin =2D (warning (G_ "Unsupported archive format (~a): \ =2Dcannot determine package dependencies") (file-extension url)) =2D #f))) =20 (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's @@ -305,33 +293,34 @@ cannot determine package dependencies") (file-extensi= on url)) (call-with-temporary-output-file (lambda (temp port) (if wheel-url =2D (and (url-fetch wheel-url temp) =2D (read-wheel-metadata temp)) =2D #f)))) + (and (url-fetch wheel-url temp) + (read-wheel-metadata temp)) + #f)))) =20 (define (guess-requirements-from-source) ;; Return the package's requirements by guessing them from the source. =2D (let ((dirname (archive-root-directory source-url)) =2D (extension (file-extension source-url))) =2D (if (string? dirname) =2D (call-with-temporary-directory =2D (lambda (dir) =2D (let* ((pypi-name (string-take dirname (string-rindex dirna= me #\-))) =2D (requires.txt (string-append dirname "/" pypi-name =2D ".egg-info" "/requires.= txt")) =2D (exit-code =2D (parameterize ((current-error-port (%make-void-port= "rw+")) =2D (current-output-port (%make-void-por= t "rw+"))) =2D (if (string=3D? "zip" extension) =2D (system* "unzip" archive "-d" dir requires.tx= t) =2D (system* "tar" "xf" archive "-C" dir requires= .txt))))) =2D (if (zero? exit-code) =2D (parse-requires.txt (string-append dir "/" requires.t= xt)) =2D (begin =2D (warning =2D (G_ "Failed to extract file: ~a from source.~%") =2D requires.txt) =2D (list '() '())))))) + (if (compressed-file? source-url) + (call-with-temporary-directory + (lambda (dir) + (parameterize ((current-error-port (%make-void-port "rw+")) + (current-output-port (%make-void-port "rw+"))) + (if (string=3D? "zip" (file-extension source-url)) + (invoke "unzip" archive "-d" dir) + (invoke "tar" "xf" archive "-C" dir))) + (let ((requires.txt-files + (find-files dir (lambda (abs-file-name _) + (string-match "\\.egg-info/requires.txt$" + abs-file-name))))) + (if (> (length requires.txt-files) 0) + (begin + (parse-requires.txt (first requires.txt-files))) + (begin (warning (G_ "Cannot guess requirements from sourc= e archive:\ + no requires.txt file found.~%")) + (list '() '())))))) + (begin + (warning (G_ "Unsupported archive format; \ +cannot determine package dependencies from source archive: ~a~%") + (basename source-url)) (list '() '())))) =20 ;; First, try to compute the requirements using the wheel, else, fallbac= k to diff --git a/tests/pypi.scm b/tests/pypi.scm index aa08e2cb54..ad188df16c 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -177,8 +177,9 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testin= g' (match url ("https://example.com/foo-1.0.0.tar.gz" (begin =2D (mkdir-p "foo-1.0.0/foo.egg-info/") =2D (with-output-to-file "foo-1.0.0/foo.egg-info/requires.= txt" + ;; Unusual requires.txt location should still be found. + (mkdir-p "foo-1.0.0/src/bizarre.egg-info") + (with-output-to-file "foo-1.0.0/src/bizarre.egg-info/req= uires.txt" (lambda () (display test-requires.txt))) (parameterize ((current-output-port (%make-void-port "rw= +"))) @@ -299,10 +300,13 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'test= ing' (lambda (url file-name) (match url ("https://example.com/foo-1.0.0.tar.gz" + (mkdir-p "foo-1.0.0/foo.egg-info/") + (parameterize ((current-output-port (%make-void-port "rw+"))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) + (delete-file-recursively "foo-1.0.0") (set! test-source-hash =2D (call-with-input-file file-name port-sha256)) =2D #t) =2D ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #t) + (call-with-input-file file-name port-sha256))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f) (_ (error "Unexpected URL: " url))))) (mock ((guix http-client) http-fetch (lambda (url . rest) @@ -334,7 +338,6 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testin= g' test-source-hash) hash)) (x =2D (pk 'fail x #f))) =2D ))) + (pk 'fail x #f)))))) =20 (test-end "pypi") =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0009-import-pypi-Preserve-package-name-case-when-forming-.patch Content-Transfer-Encoding: quoted-printable From=201290f9d1f0d594fdd4723d76b94116be25da9dd5 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Sat, 30 Mar 2019 20:27:35 -0400 Subject: [PATCH 9/9] import: pypi: Preserve package name case when forming pypi-uri. Fixes issue: #33046. * guix/build-system/python.scm (pypi-uri): Update the host URI to "files.pythonhosted.org". * guix/import/pypi.scm (make-pypi-sexp): Preserve the package name case when the source URL calls for it. =2D-- guix/build-system/python.scm | 2 +- guix/import/pypi.scm | 23 ++++++++++++++--------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm index b753940bad..e39c06528e 100644 =2D-- a/guix/build-system/python.scm +++ b/guix/build-system/python.scm @@ -50,7 +50,7 @@ "Return a URI string for the Python package hosted on the Python Package Index (PyPI) corresponding to NAME and VERSION. EXTENSION is the file name extension, such as '.tar.gz'." =2D (string-append "https://pypi.org/packages/source/" + (string-append "https://files.pythonhosted.org/packages/source/" (string-take name 1) "/" name "/" name "-" version extension)) =20 diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index a2ce14b192..fecf95d0a7 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -371,15 +371,20 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION= , and LICENSE." `(package (name ,(python->package-name name)) (version ,version) =2D (source (origin =2D (method url-fetch) =2D ;; Sometimes 'pypi-uri' doesn't quite work = due to mixed =2D ;; cases in NAME, for instance, as is the c= ase with =2D ;; "uwsgi". In that case, fall back to a f= ull URL. =2D (uri (pypi-uri ,(string-downcase name) vers= ion)) =2D (sha256 =2D (base32 =2D ,(guix-hash-url temp))))) + (source + (origin + (method url-fetch) + ;; PyPI URL are case sensitive, but sometimes a proj= ect + ;; named using mixed case has a URL using lower case= , so + ;; we must work around this inconsistency. For actu= al + ;; examples, compare the URLs of the "Deprecated" and + ;; "uWSGI" PyPI packages. + (uri ,(if (string-contains source-url name) + `(pypi-uri ,name version) + `(pypi-uri ,(string-downcase name) version= ))) + (sha256 + (base32 + ,(guix-hash-url temp))))) (build-system python-build-system) ,@(maybe-inputs required-inputs 'propagated-inputs) ,@(maybe-inputs test-inputs 'native-inputs) =2D-=20 2.21.0 --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEJ9WGpPiQCFQyn/CfEmDkZILmNWIFAlziJ6cACgkQEmDkZILm NWKgRA//WC0iRN9CECjP4VuWTah5C8jEWI8F9tjZqdpwHu0fwQ5Ip5I6Y1vxw4iv yCqPSsbLevqNrosl16k6KgIJoj9JKFm6egNKNZIuHSftZC+hC/xaN3z2rPTd0Vys CkpjSnjP1aJS/Zn5OdMuSIn/W+isbgaq4eQ8cR5sBP2QjFI79w5E+b+Ww3NXH46u UFrmILhQXjRZ3XOjHHDFychluhU1EHi97e35WUCAn+j8oSIwfsVkVLMPF8YvGVMq OkOk9q9BKv8eJH0a00edcflu1YwhU5xAV8g9lBxkZjGLO6lDKFTe+aq/03UDQ5eR vBHGQTMuykTdiVIOeNH7xLV2X12WIO22neEM6Cj+6IMSThqSoZKn99XDuC1rYqCO 1yRPf4TM2wXHwhaOatYeokwb+t3Ozztc2gT5RiZnnKNgYJlatoElJi0XuQj9Qrpc fh3fE/iccvszQuBF4ZQ0z9FoWQiDXblvZ1uy5/upQVnUMfal4I4td5C58laYef+Z zGzjw5Z/RWB2ssuztMQKOPrUoohx3Ne/s7QMndzhmBvEbfPqkQ7ROUXjO7Jf3kQl 1A6toSrO7ImsaiUsPJhM8dOqAoMXRvJfiG8viisPclN/aVrAItweiNZRH+oCc6iL lWYd8Lg+Yn6LUG9HS81bZ0Czi7fdYfY4T3rvctxv3fKcjc7YKZE= =cn9M -----END PGP SIGNATURE----- --==-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon May 20 11:06:11 2019 Received: (at 24450) by debbugs.gnu.org; 20 May 2019 15:06:11 +0000 Received: from localhost ([127.0.0.1]:37410 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hSjrf-0000iL-7j for submit@debbugs.gnu.org; Mon, 20 May 2019 11:06:11 -0400 Received: from eggs.gnu.org ([209.51.188.92]:60837) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hSjrd-0000i7-RA for 24450@debbugs.gnu.org; Mon, 20 May 2019 11:06:10 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:54377) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hSjrV-0006gZ-4A; Mon, 20 May 2019 11:06:01 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=50526 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hSjrU-0006Sp-LS; Mon, 20 May 2019 11:06:00 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxim Cournoyer Subject: Re: bug#24450: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: <87pnod7ot4.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 1 Prairial an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 20 May 2019 17:05:58 +0200 In-Reply-To: <87pnod7ot4.fsf@gmail.com> (Maxim Cournoyer's message of "Mon, 20 May 2019 00:05:59 -0400") Message-ID: <87k1elrwrt.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 24450 Cc: Ricardo Wurmus , 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello! Maxim Cournoyer skribis: > Ricardo Wurmus writes: > >> Hi Maxim, >> >> I would very much like to see your improvements to the pypi importer to >> be merged. Have you been able to separate the independent changes as >> suggested by Ludo? > > I'm thrilled that someone has an interest in this :-) > > I took my time, but finally got around to restructure the changes a > bit. I hope it'll be easier to review this time around! I=E2=80=99ll let Ricardo comment. For my part, I see that it has tests, and that=E2=80=99s enough to make me happy. :-) Thanks for improving the importer! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue May 21 21:13:34 2019 Received: (at 24450) by debbugs.gnu.org; 22 May 2019 01:13:34 +0000 Received: from localhost ([127.0.0.1]:41280 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hTFoz-0004dS-KC for submit@debbugs.gnu.org; Tue, 21 May 2019 21:13:33 -0400 Received: from mail-qt1-f173.google.com ([209.85.160.173]:44579) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hTFoy-0004dC-2N for 24450@debbugs.gnu.org; Tue, 21 May 2019 21:13:32 -0400 Received: by mail-qt1-f173.google.com with SMTP id f24so424224qtk.11 for <24450@debbugs.gnu.org>; Tue, 21 May 2019 18:13:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=hgkCV2xFUcFm/qUT4i6puLvLvvWMfIRPX2cWob7S8og=; b=ZHX+JDGH6RN6it2U43A9xecBfKuWVQ+MZIZQ1WaHYrAe1l475DDCqpokPvAgGdIGOx fsZEFpbmQcgjrCeQsyHf2Ql0vIvYIWF/wfsDcGg9PZZsfiDQ+RdFRGQGa0Wm7fNT5pj2 7kK61oG24lbidhE6owjo+Gwjb+tX5Lcdjf4eBRrbzGYc8I05QIQp8epo2JBp0pKfO5Ue ABsNPPBqRq+K5/F4nkLU7Fvp4gC0DVtsXFaAKgmtHak6TfrPuXTsjDf9+d9TIGhcYZsj i68IKUfYnmo65ggWKgxGnep+XBTluCjS0s1TxkZRHc9Y99e9NeXpi9c74DFz5HfFlnZh JmvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=hgkCV2xFUcFm/qUT4i6puLvLvvWMfIRPX2cWob7S8og=; b=i4WE+9vdJ9OLdhX0kDFYk+wSmAEzSNWAHLbwmJ5cfyAU0fHL4AyTtErl8HsD8DLPTb eYKeAU8vHxR4h3vM7SPlcSxN9UywY03+VIx4NaemeiWo3JrZhLQU42xMCyD7dXlDZ9F5 8ew18bqMxJbz2aQ9LCiil1WJobciR8yW2J7uhhpz4hKd6foCqEXJhBmLqauiAfzkDqe+ BHX38XZRJrCZKo1GcobQd4gCOJBvwc1kWtWWjTmlm55GZ9xdnL9eJLq/iRsc1BQwSMLS 8oCiZcHmDMPDpXBItI0kCG6AF7RUWAtMdqeivVOsN0CqSzhhCVX+XQJQIzKGcbGJzI39 Ig2Q== X-Gm-Message-State: APjAAAWVlLvCooA3TIrB3yrkn0epf6PPbAtaI1W1Pji3bSdJ1avQdHSf fudgUznoHUz0y56fQ1wjyoWYytX3 X-Google-Smtp-Source: APXvYqzd6EMp5hPTrLqeO3/lcdlxP7KPfZNvp+TyFnit5c9W5kLjV0+41lR44RDdF0aDD196F+A3Zw== X-Received: by 2002:ac8:1115:: with SMTP id c21mr70423079qtj.155.1558487606405; Tue, 21 May 2019 18:13:26 -0700 (PDT) Received: from kwak (dsl-157-210.b2b2c.ca. [66.158.157.210]) by smtp.gmail.com with ESMTPSA id g20sm10098575qki.52.2019.05.21.18.13.25 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 21 May 2019 18:13:25 -0700 (PDT) From: Maxim Cournoyer To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#24450: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: <87pnod7ot4.fsf@gmail.com> <87k1elrwrt.fsf@gnu.org> Date: Tue, 21 May 2019 21:13:24 -0400 In-Reply-To: <87k1elrwrt.fsf@gnu.org> ("Ludovic \=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\= \=\?utf-8\?Q\?s\?\= message of "Mon, 20 May 2019 17:05:58 +0200") Message-ID: <87lfyz5m17.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: Ricardo Wurmus , 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Greetings! Ludovic Court=C3=A8s writes: > Hello! > > Maxim Cournoyer skribis: > >> Ricardo Wurmus writes: >> >>> Hi Maxim, >>> >>> I would very much like to see your improvements to the pypi importer to >>> be merged. Have you been able to separate the independent changes as >>> suggested by Ludo? >> >> I'm thrilled that someone has an interest in this :-) >> >> I took my time, but finally got around to restructure the changes a >> bit. I hope it'll be easier to review this time around! > > I=E2=80=99ll let Ricardo comment. For my part, I see that it has tests, = and > that=E2=80=99s enough to make me happy. :-) OK! > Thanks for improving the importer! My pleasure! One less itch to scratch ;-) Maxim From debbugs-submit-bounces@debbugs.gnu.org Mon May 27 10:48:55 2019 Received: (at 24450) by debbugs.gnu.org; 27 May 2019 14:48:55 +0000 Received: from localhost ([127.0.0.1]:53766 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVGvm-00082i-QW for submit@debbugs.gnu.org; Mon, 27 May 2019 10:48:54 -0400 Received: from b2062.mx.srv.dfn.de ([194.95.234.172]:40455) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVGvk-00082X-HD for 24450@debbugs.gnu.org; Mon, 27 May 2019 10:48:53 -0400 Received: from localhost (localhost [127.0.0.1]) by b2062.mx.srv.dfn.de (Postfix) with ESMTP id 2890E16004F; Mon, 27 May 2019 16:48:44 +0200 (CEST) Received: from b2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-tub.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id l5IAVsJaufWg; Mon, 27 May 2019 16:48:43 +0200 (CEST) Received: from SW-IT-P-CAS4.mdc-berlin.net (mgw10-4.mdc-berlin.de [141.80.113.59]) by b2062.mx.srv.dfn.de (Postfix) with ESMTPS; Mon, 27 May 2019 16:48:43 +0200 (CEST) Received: from localhost (141.80.113.103) by SW-IT-P-CAS4.mdc-berlin.net (141.80.113.59) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 27 May 2019 16:48:43 +0200 References: <87pnod7ot4.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87pnod7ot4.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 27 May 2019 16:48:43 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [141.80.113.103] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24642.000 X-TM-AS-Result: No-9.524500-8.000000-10 X-TMASE-MatchedRID: 9zTThWtzIms4HKI/yaqRm5VRzPxemJL0APiR4btCEeYGmHr1eMxt2XJ4 YYfr3pskNGN2QO49D7NirfDD/q34cbSmTEY5EtKkEhGH3CRdKUX17lAMINbrD5LBwsVwuMFlkUC 9nJY4BcKF1UGHNi9w/j3lvbv1AogjTI7bsVk48a9LJhhSkbJGJ84Vo5svdK3bmyiLZetSf8kir3 kOMJmHTBQabjOuIvShC24oEZ6SpSk6XEE7Yhw4FrVm+A+BcDL0F+M75rJrk5gkD9hASP4Qsl3Xp BUNWtS3oE3DXDY2aZJDDKa3G4nrLQ== X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--9.524500-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24642.000 X-TM-SNTS-SMTP: DFA70782A3E73A861C7CF5C64837616B73341718A1CCA009E2AC5C10C0B911142000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Maxim, > Subject: [PATCH 1/9] import: pypi: Do not consider requirements.txt files. > > * guix/import/pypi.scm (guess-requirements): Update comment. > [guess-requirements-from-source]: Do not attempt to parse the file > requirements.txt. Streamline logic. Why remove the handling of the requirements.txt? Is it no longer popular enough to expect its availability in the source archives? Please also mention in the commit message that and how you adjusted the tests. You removed the comments from the example requires.txt =E2=80=94 are comments no longer permitted in these files? If they are, please don=E2=80= =99t include those changes. -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Mon May 27 11:11:43 2019 Received: (at 24450) by debbugs.gnu.org; 27 May 2019 15:11:43 +0000 Received: from localhost ([127.0.0.1]:53795 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVHHq-0000LL-Jp for submit@debbugs.gnu.org; Mon, 27 May 2019 11:11:42 -0400 Received: from a2062.mx.srv.dfn.de ([194.95.232.172]:46617) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVHHo-0000L7-AO for 24450@debbugs.gnu.org; Mon, 27 May 2019 11:11:41 -0400 Received: from localhost (localhost [127.0.0.1]) by a2062.mx.srv.dfn.de (Postfix) with ESMTP id 56860A0031; Mon, 27 May 2019 17:11:34 +0200 (CEST) Received: from a2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-han.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id qtm8TRXGyzuQ; Mon, 27 May 2019 17:11:33 +0200 (CEST) Received: from SW-IT-P-CAS4.mdc-berlin.net (mgw10-4.mdc-berlin.de [141.80.113.59]) by a2062.mx.srv.dfn.de (Postfix) with ESMTPS; Mon, 27 May 2019 17:11:33 +0200 (CEST) Received: from localhost (141.80.113.103) by SW-IT-P-CAS4.mdc-berlin.net (141.80.113.59) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 27 May 2019 17:11:33 +0200 References: <87pnod7ot4.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87pnod7ot4.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 27 May 2019 17:11:33 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [141.80.113.103] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24642.000 X-TM-AS-Result: No-13.419900-8.000000-10 X-TMASE-MatchedRID: rYpa/RC+czG2uvwZYhrIwMn9tWHiLD2GC/ExpXrHizzKFcLT6pRLbxJr vzJ/haMzF9rmBJnMwdDR9unqdQeoV/hDfjwsp9c4LIrMljt3adsAaovTveFt1W+fXVEQ/fGevf2 yHAohazkZ+rtnsjV8qxdbug1FfrKlOKmGqBRU0hAyByMiwk6+3qKRkGOW2z9ymJBe2bRXwlPNYU CWCSRufZvWsDuCVeeF1gNssQoB+JrUBIEM4OQ6eIS/TV9k6ppAm19A9Bm2BPNrRM6wvXgDabkp9 AXrzV+or9QfHPpB/z/ic+2pUF+sfgx8yLMFzjjQWTGejGdB9VL17lqbebntfdWb/MXO2XdLUZGE 0qUrJUW/KDFGIELrl69jYtkBPLyHSw7z8qjvveGInASnzB5VfIsMpRzKbrUkd6RFRE7vKfE5Q3C XMp7+HC1/HX6casAVTJY+8VEMeF6Dyz0cQfyB7yiWKOSuD9akVoopVBvm9s24zlF1pkNzlpXBzA slgCOzgudRl/9qZRsIdcRMkIJ5B1vLEDLz4/18ngIgpj8eDcBZDL1gLmoa/PoA9r2LThYYKrauX d3MZDWXf5sC39gVVKEipT4UdZy2sA84ejOv81qn0VhNgspui1gtnG0mcSFISwwcGKLTYEc= X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--13.419900-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24642.000 X-TM-SNTS-SMTP: 5F777502C8F8156577518B219BD2E33D6FD9F625F5AE00CE3AA57B5756AA5CD72000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Maxim, on to patch number 2! > From 5f79b0502f62bd1dacc8ea143c1dbd9ef7cfc29d Mon Sep 17 00:00:00 2001 > From: Maxim Cournoyer > Date: Thu, 28 Mar 2019 00:26:00 -0400 > Subject: [PATCH 2/9] import: pypi: Do not parse optional requirements from > source. > > * guix/import/pypi.scm: Export PARSE-REQUIRES.TXT. > (guess-requirements): Move the READ-REQUIREMENTS procedure to the top lev= el, > and rename it to PARSE-REQUIRES.TXT. Move the CLEAN-REQUIREMENT and COMM= ENT? > functions inside the READ-REQUIREMENTS procedure. > (parse-requires.txt): Add a SECTION-HEADER? predicate, and use it to prev= ent > parsing optional requirements. > > * tests/pypi.scm (test-requires-with-sections): New variable. > ("parse-requires.txt, with sections"): New test. > ("pypi->guix-package"): Mute tar output to stdout. The commit log does not match the changes. CLEAN-REQUIREMENT is now a top-level procedure, not a local procedure inside of READ-REQUIREMENTS as reported in the commit message. Which is correct? > + (call-with-input-file requires.txt > + (lambda (port) > + (let loop ((result '())) > + (let ((line (read-line port))) > + ;; Stop when a section is encountered, as sections contains op= tional Should be =E2=80=9Ccontain=E2=80=9D. > + ;; (extra) requirements. Non-optional requirements must appear > + ;; before any section is defined. > + (if (or (eof-object? line) (section-header? line)) > + (reverse result) > + (cond > + ((or (string-null? line) (comment? line)) > + (loop result)) > + (else > + (loop (cons (clean-requirement line) > + result)))))))))) > + I think it would be better to use =E2=80=9Cmatch=E2=80=9D here instead of n= ested =E2=80=9Clet=E2=80=9D, =E2=80=9Cif=E2=80=9D and =E2=80=9Ccond=E2=80=9D. At least you can drop the= =E2=80=9Cif=E2=80=9D and just use cond. The loop let and the inner let can be merged. > +(define (parse-requires.txt requires.txt) > + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of > +requirement names." > + ;; This is a very incomplete parser, which job is to select the non-op= tional =E2=80=9Cwhich=E2=80=9D =E2=80=93> =E2=80=9Cwhose=E2=80=9D > + ;; dependencies and strip them out of any version information. > + ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) > + ;; library and the requirements grammar defined by PEP-0508 > + ;; (https://www.python.org/dev/peps/pep-0508/). Let=E2=80=99s remove the sentence starting with =E2=80=9CAlternatively=E2= =80=A6=E2=80=9D. We could do that but we didn=E2=80=99t :) > + (define (section-header? line) > + ;; Return #t if the given LINE is a section header, #f otherwise. > + (let ((trimmed-line (string-trim line))) > + (and (not (string-null? trimmed-line)) > + (eq? (string-ref trimmed-line 0) #\[)))) > + How about using string-prefix? instead? This looks more complicated than it deserves. You can get rid of string-null? and eq? and string-ref and all that. Same here: > + (define (comment? line) > + ;; Return #t if the given LINE is a comment, #f otherwise. > + (eq? (string-ref (string-trim line) 0) #\#)) I=E2=80=99d just use string-prefix? here. > +(define (clean-requirement s) > + ;; Given a requirement LINE, as can be found in a setuptools requires.= txt > + ;; file, remove everything other than the actual name of the required > + ;; package, and return it. > + (string-take s (or (string-index s (lambda (chr) > + (member chr '(#\space #\> #\=3D #= \<)))) > + (string-length s)))) =E2=80=9Cstring-take=E2=80=9D with =E2=80=9Cstring-length=E2=80=9D is not v= ery elegant. The char predicate in string-index could better be a char set: --8<---------------cut here---------------start------------->8--- (define (clean-requirement s) (cond ((string-index s (char-set #\space #\> #\=3D #\<)) =3D> (cut string-take = s <>)) (else s))) --8<---------------cut here---------------end--------------->8--- > ("pypi->guix-package"): Mute tar output to stdout. Finally, I think it would be better to keep this separate because it=E2=80= =99s really orthogonal to the other changes in this patch. What do you think? --=20 Ricardo From debbugs-submit-bounces@debbugs.gnu.org Mon May 27 11:54:51 2019 Received: (at 24450) by debbugs.gnu.org; 27 May 2019 15:54:51 +0000 Received: from localhost ([127.0.0.1]:53838 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVHxa-0001ca-SW for submit@debbugs.gnu.org; Mon, 27 May 2019 11:54:51 -0400 Received: from b2062.mx.srv.dfn.de ([194.95.234.172]:59941) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVHxY-0001cM-4M for 24450@debbugs.gnu.org; Mon, 27 May 2019 11:54:49 -0400 Received: from localhost (localhost [127.0.0.1]) by b2062.mx.srv.dfn.de (Postfix) with ESMTP id F10F216004D; Mon, 27 May 2019 17:54:40 +0200 (CEST) Received: from b2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-tub.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id 5C9fI-_9wW4L; Mon, 27 May 2019 17:54:40 +0200 (CEST) Received: from SW-IT-P-CAS4.mdc-berlin.net (mgw10-4.mdc-berlin.de [141.80.113.59]) by b2062.mx.srv.dfn.de (Postfix) with ESMTPS; Mon, 27 May 2019 17:54:40 +0200 (CEST) Received: from localhost (141.80.113.103) by SW-IT-P-CAS4.mdc-berlin.net (141.80.113.59) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 27 May 2019 17:54:39 +0200 References: <87pnod7ot4.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87pnod7ot4.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 27 May 2019 17:54:39 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [141.80.113.103] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24642.000 X-TM-AS-Result: No-4.990700-8.000000-10 X-TMASE-MatchedRID: u7Yf2n7Ca/22uvwZYhrIwMn9tWHiLD2GC/ExpXrHizzKFcLT6pRLbxJr vzJ/haMzr6dMYfBuaBG3BZugHYBXou9L8Ou1dTn4boe6sMfg+k/x8ID3XObXhslzeqSoxZXHByy VimjmJJPP/pvZogcnjEB9KMwL71MVCuYbw9+K9RX0VCHd+VQiHvioIsi7Sa0gPILl10bYlyCg4X cMtyNB8LTGAnbIcXh+zy7SgyfekIqJ0bdBezCzBkjBb8q+S/OC/Eo10sZXT1OA6UrbM3j3qaiCj aOYCBaz2HSvkwz+7aR1t3KenjpIujcpdZ3fQiLdgxsfzkNRlfLaf1N18C+ZAhe9CQaLe2PP9xS3 mVzWUuCMx6OO8+QGvvfyGaLx1L90Wqk8y3be4x1LfE2IG+G6NbkfZkEl6bsgCeH/SKCQhr2JF5M RLADHKkae6l1qVK/dcJWNL1wvyiKtxmtEiJOVBpF+e/T80ZLUgjux1OFeoIseJIRGDJVM09l+7F POoW8uVlxr1FJij9s= X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--4.990700-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24642.000 X-TM-SNTS-SMTP: 220A456E4BAA1FE5BE9D857B4DDD2CDAA1921459C0C0B2DBE9BED274C4557A7E2000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Patch number 3! > From 0c62b541a3e8925b5ca31fe55dbe7536cf95151f Mon Sep 17 00:00:00 2001 > From: Maxim Cournoyer > Date: Thu, 28 Mar 2019 00:26:01 -0400 > Subject: [PATCH 3/9] import: pypi: Improve parsing of requirement > specifications. > > The previous solution was fragile and could leave unwanted characters in a > requirement name, such as '[' or ']'. Wouldn=E2=80=99t it be sufficient to add [ and ] to the list of forbidden characters? The tests pass with this implementation of clean-requirements: (define (clean-requirements s) (cond ((string-index s (char-set #\space #\> #\=3D #\< #\[ #\])) =3D> (cut stri= ng-take s <>)) (else s))) > +(define %requirement-name-regexp > + ;; Regexp to match the requirement name in a requirement specification. > + > + ;; Some grammar, taken from PEP-0508 (see: > + ;; https://www.python.org/dev/peps/pep-0508/). > + > + ;; The unified rule can be expressed as: > + ;; specification =3D wsp* ( url_req | name_req ) wsp* > + > + ;; where url_req is: > + ;; url_req =3D name wsp* extras? wsp* urlspec wsp+ quoted_marker? > + > + ;; and where name_req is: > + ;; name_req =3D name wsp* extras? wsp* versionspec? wsp* quoted_marker? > + > + ;; Thus, we need only matching NAME, which is expressed as: > + ;; identifer_end =3D letterOrDigit | (('-' | '_' | '.' )* letterOrDigi= t) > + ;; identifier =3D letterOrDigit identifier_end* > + ;; name =3D identifier > + (let* ((letter-or-digit "[A-Za-z0-9]") > + (identifier-end (string-append "(" letter-or-digit "|" > + "[-_.]*" letter-or-digit ")")) > + (identifier (string-append "^" letter-or-digit identifier-end "= *")) > + (name identifier)) > + (make-regexp name))) This seems a little too complicated. Translating a grammar into a regexp is probably not a good idea in general. Since we don=E2=80=99t care about anything other than the name it seems easier to just chop off the string tail as soon as we find an invalid character. --=20 Ricardo From debbugs-submit-bounces@debbugs.gnu.org Mon May 27 11:58:55 2019 Received: (at 24450) by debbugs.gnu.org; 27 May 2019 15:58:55 +0000 Received: from localhost ([127.0.0.1]:53842 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVI1X-0001jj-FZ for submit@debbugs.gnu.org; Mon, 27 May 2019 11:58:55 -0400 Received: from a2062.mx.srv.dfn.de ([194.95.232.172]:34041) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVI1V-0001ja-GH for 24450@debbugs.gnu.org; Mon, 27 May 2019 11:58:54 -0400 Received: from localhost (localhost [127.0.0.1]) by a2062.mx.srv.dfn.de (Postfix) with ESMTP id 13D4FA0040; Mon, 27 May 2019 17:58:49 +0200 (CEST) Received: from a2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-han.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id BT6dMlmkZQU7; Mon, 27 May 2019 17:58:48 +0200 (CEST) Received: from SW-IT-P-CAS4.mdc-berlin.net (mgw10-4.mdc-berlin.de [141.80.113.59]) by a2062.mx.srv.dfn.de (Postfix) with ESMTPS; Mon, 27 May 2019 17:58:48 +0200 (CEST) Received: from localhost (141.80.113.103) by SW-IT-P-CAS4.mdc-berlin.net (141.80.113.59) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 27 May 2019 17:58:47 +0200 References: <87pnod7ot4.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87pnod7ot4.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 27 May 2019 17:58:47 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [141.80.113.103] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24642.000 X-TM-AS-Result: No-5.832500-8.000000-10 X-TMASE-MatchedRID: 5+1rHnqhWUQ4HKI/yaqRmycRqaHJVb+nwdpr2Md246y1eX0jEQ9c6gPh SovijboimxlFMGczjUHAh3C751qB/ZY5EC27socWHTcjQ7W4FBGhp756/rM6Uk2YHSC4sMKRo8W MkQWv6iUCY+lsYFiWG+TCMddcL/gjymsk/wUE4hqOVWU4EfaweoNU1q2iAjPosGWpvwYBdvMBv8 605wxzep29oUZ5FTUkwL6SxPpr1/I= X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--5.832500-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24642.000 X-TM-SNTS-SMTP: 0931D5F0F5C9064E499603F125F8C20C174A8A48F03F899D8C465AC4937FEF392000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) > From 76e4a3150f8126e0b952c6129b6e1371afba80c0 Mon Sep 17 00:00:00 2001 > From: Maxim Cournoyer > Date: Thu, 28 Mar 2019 00:26:01 -0400 > Subject: [PATCH 4/9] import: pypi: Deduplicate requirements. > > * guix/import/pypi.scm (parse-requires.txt): Remove potential duplicates. This looks fine to me, but it is subject to changes that I requested to the procedure in my comments to an earlier patch. -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Tue May 28 06:24:01 2019 Received: (at 24450) by debbugs.gnu.org; 28 May 2019 10:24:01 +0000 Received: from localhost ([127.0.0.1]:55138 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVZGz-0006cR-4P for submit@debbugs.gnu.org; Tue, 28 May 2019 06:24:01 -0400 Received: from a2062.mx.srv.dfn.de ([194.95.232.172]:50057) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVZGx-0006cH-5V for 24450@debbugs.gnu.org; Tue, 28 May 2019 06:24:00 -0400 Received: from localhost (localhost [127.0.0.1]) by a2062.mx.srv.dfn.de (Postfix) with ESMTP id E0381A0042; Tue, 28 May 2019 12:23:54 +0200 (CEST) Received: from a2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-han.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id dZqK9cpfzH0V; Tue, 28 May 2019 12:23:54 +0200 (CEST) Received: from SW-IT-P-CAS4.mdc-berlin.net (mgw10-4.mdc-berlin.de [141.80.113.59]) by a2062.mx.srv.dfn.de (Postfix) with ESMTPS; Tue, 28 May 2019 12:23:54 +0200 (CEST) Received: from localhost (141.80.113.103) by SW-IT-P-CAS4.mdc-berlin.net (141.80.113.59) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 28 May 2019 12:23:53 +0200 References: <87pnod7ot4.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87pnod7ot4.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Tue, 28 May 2019 12:23:53 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [141.80.113.103] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24642.006 X-TM-AS-Result: No-11.409100-8.000000-10 X-TMASE-MatchedRID: gzVbiXtWD9s4HKI/yaqRmycRqaHJVb+nwdpr2Md246y1eX0jEQ9c6nv6 cG7t9uXql5W6lX6TdkUzTS3TZ/2KdnOZoMTSn0AKihCjnGX2dsSR9FWwkcR8XxMp09xmUg5vaUX s6FguVy1uzwujYBLHjyEXkRzrpr4MYw1f/0r5B96cxB01DrjF9wBqi9O94W3VdEE+9XN9VT+XzC wUPmVFK7iW7TGfzwPN/+SSq+G/75Dmg5/Pezso5KEhJ1MFZG4pI0tTv9t0pTZtNdeXlKR4VzukO keAvkA3OkfoCUthhUXWO++Oj87jh8wuqW6oB1DUSZJFFtJz2zciJN3aXuV/oVbWMtvHf+u6ChnZ 62X3DzSGfy5HRhd+rrx4dT6OoWrN0KaUpJQo+cL4v4Ch2tjAq7/I3arxTrviUo7/xeDfVHc4nW4 A0bLKcVkvX5/Fh8kHjoLZF60AVZCKRxk9I3roRPRR4zaLaXcQfS0Ip2eEHnzUHQeTVDUrIuB8J9 aWc1v/VymkLM+r7VTKayT/BQTiGgwWxr7XDKH8WUjHty/nWZW/YuGIFhgN4xFrGYz1pSPKujnyQ WR1/BarSnNy7QIs8Q== X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--11.409100-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24642.006 X-TM-SNTS-SMTP: A307A1B310F1D5164BCD97D88C27901913A83559976BBAEC077F18356974F0B92000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On to the next: > From 73e27235cac1275ba7671fd2364325cf5788cb3c Mon Sep 17 00:00:00 2001 > From: Maxim Cournoyer > Date: Thu, 28 Mar 2019 00:26:02 -0400 > Subject: [PATCH 5/9] import: pypi: Support more types of archives. > > This change enables the PyPI importer to look for requirements in a source > archive of a different type than "tar.gz" or "tar.bz2". Okay. > * guix/import/pypi.scm: (guess-requirements)[tarball-directory]: Rename t= o... > [archive-root-directory]: this. Use COMPRESSED-FILED? to determine if an > archive is supported or not. Nitpick: please use =E2=80=9C...this.=E2=80=9D and leave two spaces between= sentences. Typo: it should be COMPRESSED-FILE? > [guess-requirements-from-source]: Adapt to use the new method, and use un= zip > to extract ZIP archives. s/method/procedure/ Please also mention that =E2=80=9Ccompute-inputs=E2=80=9D has been adjusted. > - (define (tarball-directory url) > - ;; Given the URL of the package's tarball, return the name of the di= rectory > + (define (archive-root-directory url) > + ;; Given the URL of the package's archive, return the name of the di= rectory > ;; that will be created upon decompressing it. If the filetype is not > ;; supported, return #f. > - ;; TODO: Support more archive formats. > - (let ((basename (substring url (+ 1 (string-rindex url #\/))))) > - (cond > - ((string-suffix? ".tar.gz" basename) > - (string-drop-right basename 7)) > - ((string-suffix? ".tar.bz2" basename) > - (string-drop-right basename 8)) > - (else > + (if (compressed-file? url) > + (let ((root-directory (file-sans-extension (basename url)))) > + (if (string=3D? "tar" (file-extension root-directory)) > + (file-sans-extension root-directory) > + root-directory)) > (begin > - (warning (G_ "Unsupported archive format: \ > -cannot determine package dependencies")) > - #f))))) > + (warning (G_ "Unsupported archive format (~a): \ > +cannot determine package dependencies") (file-extension url)) > + #f))) I think the double application of file-sans-extension and the intermediate variable name =E2=80=9Croot-directory=E2=80=9D for something t= hat is a file is a little confusing, but I don=E2=80=99t have a better proposal (other th= an to replace file-extension and file-sans-extension with a match expression). > (define (read-wheel-metadata wheel-archive) > ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the packa= ge's > @@ -246,16 +243,20 @@ cannot determine package dependencies")) > (define (guess-requirements-from-source) > ;; Return the package's requirements by guessing them from the sourc= e. > - (let ((dirname (tarball-directory source-url))) > + (let ((dirname (archive-root-directory source-url)) > + (extension (file-extension source-url))) > (if (string? dirname) > (call-with-temporary-directory > (lambda (dir) > (let* ((pypi-name (string-take dirname (string-rindex dirna= me #\-))) > (requires.txt (string-append dirname "/" pypi-name > ".egg-info" "/requires.= txt")) > - (exit-code (parameterize ((current-error-port (%make= -void-port "rw+")) > - (current-output-port (%mak= e-void-port "rw+"))) > - (system* "tar" "xf" tarball "-C" dir re= quires.txt)))) > + (exit-code > + (parameterize ((current-error-port (%make-void-port= "rw+")) > + (current-output-port (%make-void-por= t "rw+"))) > + (if (string=3D? "zip" extension) > + (system* "unzip" archive "-d" dir requires.tx= t) > + (system* "tar" "xf" archive "-C" dir requires= .txt))))) I guess this is why I=E2=80=99m not too happy with this: we=E2=80=99re chec= king in multiple places if the format is supported but then forget about this again until the next time we need to do something to the file. I wonder if we could do better and answer the question just once. -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Tue May 28 07:04:59 2019 Received: (at 24450) by debbugs.gnu.org; 28 May 2019 11:04:59 +0000 Received: from localhost ([127.0.0.1]:55156 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVZuc-0007jm-JR for submit@debbugs.gnu.org; Tue, 28 May 2019 07:04:58 -0400 Received: from c2062.mx.srv.dfn.de ([194.95.238.172]:40203) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVZuW-0007jZ-1x for 24450@debbugs.gnu.org; Tue, 28 May 2019 07:04:56 -0400 Received: from localhost (localhost [127.0.0.1]) by c2062.mx.srv.dfn.de (Postfix) with ESMTP id AAA49300059; Tue, 28 May 2019 13:04:45 +0200 (CEST) Received: from c2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-erl.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id tuIza6h_M4oa; Tue, 28 May 2019 13:04:45 +0200 (CEST) Received: from SW-IT-P-CAS4.mdc-berlin.net (mgw10-4.mdc-berlin.de [141.80.113.59]) by c2062.mx.srv.dfn.de (Postfix) with ESMTPS; Tue, 28 May 2019 13:04:45 +0200 (CEST) Received: from localhost (141.80.113.103) by SW-IT-P-CAS4.mdc-berlin.net (141.80.113.59) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 28 May 2019 13:04:44 +0200 References: <87pnod7ot4.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87pnod7ot4.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Tue, 28 May 2019 13:04:44 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [141.80.113.103] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24642.006 X-TM-AS-Result: No-14.499100-8.000000-10 X-TMASE-MatchedRID: rYpa/RC+czG2uvwZYhrIwMn9tWHiLD2GC/ExpXrHizzKFcLT6pRLbxJr vzJ/haMz5qduK012SP4WrARxf8B3UjdAaNocqFZO4h8r8l3l4eb4qCLIu0mtIIPLx3vY4vNiy46 qzTSht7cl7D4SzcScH3Ty4RCE61JIjyK/VqbtGFwinwwiyYyq+5tcYV58cc5iu3lyBdlO6yv9h5 nHGTspyJRbIQqydwCLNBdWOwPv8a5Gx4fe0Y2YyJK9FvwQx1hFH7bpDOhZpjb5LkL/TyFZzToWC leGOMNFB+95tf7lFhRrCGYp+C2dpaeM1IOL++RXamejoqUad3p+CWCcHScOEyoZHTCp3F51Ozkc lRoxuAdmK9aztaE0Lv5+IdwrByKqOWS6eomMSUn1lJyvCKVZlMCsQ5ltAUImIFBEE5CFomLB5XO pRrOwbHT9nSaK1JjPtRIHGOmNS/TiVYVlX2l3kvXG/YkcGRwfyiCNl30ibNnjDtHs/gjS0klwFe 8piRSDQ0EvG5WLGkRkR3arAlEEHUkjllSXrjtQgxsfzkNRlfLaf1N18C+ZAhe9CQaLe2PP9xS3m VzWUuCgZHIBpyeFpnlK9aroYUxYTztWwxLLEeK2KYX4EwbkcxHktPmmcqBAftwZ3X11IV0= X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--14.499100-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24642.006 X-TM-SNTS-SMTP: 938C0B69BA9CA11023ACAE7F3AB7B886A26A4DD6A205D8013B4AE78004B31BCC2000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Patch number 6: > From fb0547ef225103c0f8355a7eccc41e0d028f6563 Mon Sep 17 00:00:00 2001 > From: Maxim Cournoyer > Date: Thu, 28 Mar 2019 00:26:03 -0400 > Subject: [PATCH 6/9] import: pypi: Parse wheel METADATA instead of > metadata.json. > With newer Wheel releases, there is no more metadata.json file; the METAD= ATA > file should be used instead (see: https://github.com/pypa/wheel/issues/19= 5). > This change updates our PyPI importer so that it uses the later. Typo: should be =E2=80=9Clatter=E2=80=9D instead of =E2=80=9Clater=E2=80=9D. > * guix/import/pypi.scm (define-module): Remove unnecessary modules and ex= port > the PARSE-WHEEL-METADATA method. Please remove the indentation here. Also, please don=E2=80=99t use =E2=80= =9Cmethod=E2=80=9D (because it=E2=80=99s not); use =E2=80=9Cprocedure=E2=80=9D instead. > (parse-wheel-metadata): Add method. Same here. > + (define (requires-dist-header? line) > + ;; Return #t if the given LINE is a Requires-Dist header. > + (regexp-match? (string-match "^Requires-Dist: " line))) > + > + (define (requires-dist-value line) > + (string-drop line (string-length "Requires-Dist: "))) > + > + (define (extra? line) > + ;; Return #t if the given LINE is an "extra" requirement. > + (regexp-match? (string-match "extra =3D=3D " line))) The use of =E2=80=9Cregexp-match?=E2=80=9D here isn=E2=80=99t strictly nece= ssary as the return value is true-ish anyway. > + (call-with-input-file metadata > + (lambda (port) > + (let loop ((requirements '())) > + (let ((line (read-line port))) > + ;; Stop at the first 'Provides-Extra' section: the non-optional > + ;; requirements appear before the optional ones. > + (if (eof-object? line) > + (reverse (delete-duplicates requirements)) > + (cond > + ((and (requires-dist-header? line) (not (extra? line))) > + (loop (cons (specification->requirement-name > + (requires-dist-value line)) > + requirements))) > + (else > + (loop requirements))))))))) > + As before you can simplify the nested let and merge =E2=80=9Cif=E2=80=9D an= d "cond=E2=80=9C. > (define (read-wheel-metadata wheel-archive) > ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the packa= ge's > - ;; requirements. > + ;; requirements, or #f if the metadata file contained therein couldn= 't be > + ;; extracted. > (let* ((dirname (wheel-url->extracted-directory wheel-url)) > - (json-file (string-append dirname "/metadata.json"))) > - (and (zero? (system* "unzip" "-q" wheel-archive json-file)) > - (dynamic-wind > - (const #t) > - (lambda () > - (call-with-input-file json-file > - (lambda (port) > - (let* ((metadata (json->scm port)) > - (run_requires (hash-ref metadata "run_requires= ")) > - (requirements (if run_requires > - (hash-ref (list-ref run_requ= ires 0) > - "requires") > - '()))) > - (map specification->requirement-name requirements))= ))) > - (lambda () > - (delete-file json-file) > - (rmdir dirname)))))) > + (metadata (string-append dirname "/METADATA"))) > + (call-with-temporary-directory > + (lambda (dir) > + (if (zero? (system* "unzip" "-q" wheel-archive "-d" dir metadat= a)) > + (parse-wheel-metadata (string-append dir "/" metadata)) > + (begin > + (warning > + (G_ "Failed to extract file: ~a from wheel.~%") metadata) > + #f)))))) The old approach took care of removing the unpacked archive no matter what happened. The new code doesn=E2=80=99t do that. > --- a/tests/pypi.scm > +++ b/tests/pypi.scm Thanks for the tests! -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Tue May 28 09:22:03 2019 Received: (at 24450) by debbugs.gnu.org; 28 May 2019 13:22:03 +0000 Received: from localhost ([127.0.0.1]:55307 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVc3G-0005JG-L6 for submit@debbugs.gnu.org; Tue, 28 May 2019 09:22:03 -0400 Received: from a2062.mx.srv.dfn.de ([194.95.232.172]:60109) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVc3E-0005Ip-5t for 24450@debbugs.gnu.org; Tue, 28 May 2019 09:22:01 -0400 Received: from localhost (localhost [127.0.0.1]) by a2062.mx.srv.dfn.de (Postfix) with ESMTP id 1363EA003C; Tue, 28 May 2019 15:21:58 +0200 (CEST) Received: from a2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-han.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id fM6YSXybgnM8; Tue, 28 May 2019 15:21:57 +0200 (CEST) Received: from SW-IT-P-CAS3.mdc-berlin.net (mgw10-3.mdc-berlin.de [141.80.113.58]) by a2062.mx.srv.dfn.de (Postfix) with ESMTPS; Tue, 28 May 2019 15:21:57 +0200 (CEST) Received: from localhost (141.80.113.103) by SW-IT-P-CAS3.mdc-berlin.net (141.80.113.58) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 28 May 2019 15:21:56 +0200 References: <87pnod7ot4.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87pnod7ot4.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Tue, 28 May 2019 15:21:56 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [141.80.113.103] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24642.007 X-TM-AS-Result: No-18.147600-8.000000-10 X-TMASE-MatchedRID: yebcs53SkkCeI04EoFwic/HkpkyUphL9hhy6s2hQl4R6J6k0q7HhedlM ASzPGs9UuTZLG6KdNeBfTa2GSyZtFDfXobiuOEu9boe6sMfg+k8iyyRgkoGSpH5Isu006IGGNvu +iBOiThKqx9XwNIYDmsQXwQMp2WWppulgThH5e9n4AIbmhYnu0qld1ZriBTRLuM5RdaZDc5aMBa f5zCPrtigACeNwiI6E7divN/tLk8xkHsVpDopD38K1Ib9JAALxi1ak2O8ganhrRM6wvXgDabkp9 AXrzV+or9QfHPpB/z/ic+2pUF+sfq6C0TivDfkyWTGejGdB9VIweLVh3LZYSC5hVkDKEqeBAFQy Ftt36WwktfZHgqAHlkqohIki/gPv5ngIfDCwywJusn2BWc6xnXe5jVG6a6tKSX8n1Gj4wAEh1OH 4Dg24m2j4Fd8xTVXv+/GYl971aouFqCjfEeLPM07CajSVoNFsoae+ev6zOlIjRiu1AuxJTHbP6u FvXWg/qOL9cwECOzHixmXu3vAjfzvPek3k5ArHrKAvSPiudyHsfG+/NIT6/qlTFDGZMPhCoFRzR /o0ovW0UPas4iSWEm+djeIRXiJ0WU/fB/XFmJwFZev51JnVPTm1fUEi5stba73+XlYDLuxGXW1U nyxu3cBA9x21fwFc1YrEAX+BI+q1DfGM6db7XwPZZctd3P4BIMZPqhT2JBu4yGjDjCUdsOcMSMs oUYnjS1bXI0nmofMGOBgRhBzamBuwdhbbhxE7zNIobH2DzGF+jSWdx0YUU/STUVc1uWcJCe8AXH 4vSUR3n+s7CNjQiexsL9XDoOGTv1l2Uvx6idpWdFebWIc3VsRB0bsfrpPIFT1PBs+MVY2l/MtrT wS4UOWTFOCbTDvXwY2rQsZMN+h+ZMjpDjg3k2ubq1NsFHDF5ZjKDMI4PWc= X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--18.147600-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24642.007 X-TM-SNTS-SMTP: E9F8ACAB84538E6E0E9CF55EF2BDE5A5FF24B872BE272240B04B017B5755031B2000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Next up: Seven of Nine, tertiary adjunct of unimatrix zero one: > From 37e499d5d5d5f690aa0a065c730e13f6a31dd30d Mon Sep 17 00:00:00 2001 > From: Maxim Cournoyer > Date: Thu, 28 Mar 2019 23:12:26 -0400 > Subject: [PATCH 7/9] import: pypi: Include optional test inputs as > native-inputs. > > * guix/import/pypi.scm (maybe-inputs): Add INPUT-TYPE argument, and use i= t. > (test-section?): New predicate. > (parse-requires.txt): Collect the optional test inputs, and return them a= s the > second element of the returned list. AFAICT parse-requires.txt now returns a list of pairs, but used to return a plain list before. Is this correct? > (define (parse-requires.txt requires.txt) > - "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of > -requirement names." > - ;; This is a very incomplete parser, which job is to select the non-op= tional > - ;; dependencies and strip them out of any version information. > + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a pair of = requirements. > + > +The first element of the pair contains the required dependencies while t= he > +second the optional test dependencies. Note that currently, optional, > +non-test dependencies are omitted since these can be difficult or expens= ive to > +satisfy." > + > + ;; This is a very incomplete parser, which job is to read in the requi= rement > + ;; specification lines, and strip them out of any version information. > ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) > ;; library and the requirements grammar defined by PEP-0508 > ;; (https://www.python.org/dev/peps/pep-0508/). Does it really return a pair? Or a list of pairs? Or is it a two-element list of lists? > (call-with-input-file requires.txt > (lambda (port) > - (let loop ((result '())) > + (let loop ((required-deps '()) > + (test-deps '()) > + (inside-test-section? #f) > + (optional? #f)) > (let ((line (read-line port))) > - ;; Stop when a section is encountered, as sections contains op= tional > - ;; (extra) requirements. Non-optional requirements must appear > - ;; before any section is defined. > - (if (or (eof-object? line) (section-header? line)) > + (if (eof-object? line) > ;; Duplicates can occur, since the same requirement can be > ;; listed multiple times with different conditional marker= s, e.g. > ;; pytest >=3D 3 ; python_version >=3D "3.3" > ;; pytest < 3 ; python_version < "3.3" > - (reverse (delete-duplicates result)) > + (map (compose reverse delete-duplicates) > + (list required-deps test-deps)) Looks like a list of lists to me. =E2=80=9Cdelete-duplicates=E2=80=9D now = won=E2=80=99t delete a name that is in both =E2=80=9Crequired-deps=E2=80=9D as well as in =E2=80= =9Ctest-deps=E2=80=9D. Is this acceptable? Personally, I=E2=80=99m not a fan of using data structures for returning multiple values, because we can simply return multiple values. Or we could have more than just strings. The meaning of these strings is provided by the bin into which they are thrown =E2=80=94 either =E2=80=9Crequired-deps=E2=80=9D or =E2=80=9Ctest-deps=E2=80=9D. It could b= e an option to collect tagged values instead and have the caller deal with filtering. > (define (parse-wheel-metadata metadata) > - "Given METADATA, a Wheel metadata file, return a list of requirement n= ames." > + "Given METADATA, a Wheel metadata file, return a pair of requirements. > + > +The first element of the pair contains the required dependencies while t= he second the optional > +test dependencies. Note that currently, optional, non-test dependencies= are > +omitted since these can be difficult or expensive to satisfy." > ;; METADATA is a RFC-2822-like, header based file. This sounds like this is going to duplicate the previous procedures. > (define (requires-dist-header? line) > ;; Return #t if the given LINE is a Requires-Dist header. > - (regexp-match? (string-match "^Requires-Dist: " line))) > + (string-match "^Requires-Dist: " line)) > > (define (requires-dist-value line) > (string-drop line (string-length "Requires-Dist: "))) > > (define (extra? line) > ;; Return #t if the given LINE is an "extra" requirement. > - (regexp-match? (string-match "extra =3D=3D " line))) > + (string-match "extra =3D=3D '(.*)'" line)) These hunks should be part of the previous patch where they were introduced. (See my comments there about =E2=80=9Cregexp-match?=E2=80=9D.) > + (define (test-requirement? line) > + (let ((extra-label (match:substring (extra? line) 1))) > + (and extra-label (test-section? extra-label)))) You can use =E2=80=9Cand=3D>=E2=80=9D instead of binding a name: (and=3D> (match:substring (extra? line) 1) test-section?) > (call-with-input-file metadata > (lambda (port) > - (let loop ((requirements '())) > + (let loop ((required-deps '()) > + (test-deps '())) > (let ((line (read-line port))) > - ;; Stop at the first 'Provides-Extra' section: the non-optional > - ;; requirements appear before the optional ones. > (if (eof-object? line) > - (reverse (delete-duplicates requirements)) > + (map (compose reverse delete-duplicates) > + (list required-deps test-deps)) > (cond > ((and (requires-dist-header? line) (not (extra? line))) > (loop (cons (specification->requirement-name > (requires-dist-value line)) > - requirements))) > + required-deps) > + test-deps)) > + ((and (requires-dist-header? line) (test-requirement? lin= e)) > + (loop required-deps > + (cons (specification->requirement-name (requires-d= ist-value line)) > + test-deps))) > (else > - (loop requirements))))))))) > + (loop required-deps test-deps))))))))) ;skip line Could you refactor this so that the other parser can be reused? If not, the same comments about =E2=80=9Cif=E2=80=9D and =E2=80=9Ccond=E2=80=9D and= the use of pairs/lists instead of =E2=80=9Cvalues=E2=80=9D applies here. > (define (guess-requirements source-url wheel-url archive) > - "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a li= st > + "Given SOURCE-URL, WHEEL-URL and an ARCHIVE of the package, return a l= ist > of the required packages specified in the requirements.txt file. ARCHIV= E will > be extracted in a temporary directory." > > @@ -244,7 +289,10 @@ cannot determine package dependencies") (file-extens= ion url)) > (metadata (string-append dirname "/METADATA"))) > (call-with-temporary-directory > (lambda (dir) > - (if (zero? (system* "unzip" "-q" wheel-archive "-d" dir metadat= a)) > + (if (zero? > + (parameterize ((current-error-port (%make-void-port "rw+")) > + (current-output-port (%make-void-port "rw+"= ))) > + (system* "unzip" wheel-archive "-d" dir metadata))) > (parse-wheel-metadata (string-append dir "/" metadata)) > (begin > (warning > @@ -283,32 +331,41 @@ cannot determine package dependencies") (file-exten= sion url)) > (warning > (G_ "Failed to extract file: ~a from source.~%") > requires.txt) > - '()))))) > - '()))) > + (list '() '())))))) > + (list '() '())))) I would like to see cosmetic changes like these three hunks in separate commits. > (define (compute-inputs source-url wheel-url archive) > - "Given the SOURCE-URL of an already downloaded ARCHIVE, return a list = of > -name/variable pairs describing the required inputs of this package. Also > + "Given the SOURCE-URL and WHEEL-URL of an already downloaded ARCHIVE, = return > +a pair of lists, each consisting of a list of name/variable pairs, for t= he > +propagated inputs and the native inputs, respectively. Also > return the unaltered list of upstream dependency names." > - (let ((dependencies > - (remove (cut string=3D? "argparse" <>) > - (guess-requirements source-url wheel-url archive)))) > - (values (sort > - (map (lambda (input) > - (let ((guix-name (python->package-name input))) > - (list guix-name (list 'unquote (string->symbol gui= x-name))))) > - dependencies) > - (lambda args > - (match args > - (((a _ ...) (b _ ...)) > - (string-ci - dependencies))) > + > + (define (strip-argparse deps) > + (remove (cut string=3D? "argparse" <>) deps)) > + > + (define (requirement->package-name/sort deps) > + (sort > + (map (lambda (input) > + (let ((guix-name (python->package-name input))) > + (list guix-name (list 'unquote (string->symbol guix-name))= ))) > + deps) > + (lambda args > + (match args > + (((a _ ...) (b _ ...)) > + (string-ci + > + (define process-requirements > + (compose requirement->package-name/sort strip-argparse)) > + > + (let ((dependencies (guess-requirements source-url wheel-url archive))) > + (values (map process-requirements dependencies) > + (concatenate dependencies)))) Giving names to these processing steps is fine and improves clarity, but I=E2=80=99m not so comfortable about returning ad-hoc pairs *and* multiple values (like above). > + (match guix-dependencies > + ((required-inputs test-inputs) > + (values > + `(package > + (name ,(python->package-name name)) > + (version ,version) > + (source (origin > + (method url-fetch) > + ;; Sometimes 'pypi-uri' doesn't quite work = due to mixed > + ;; cases in NAME, for instance, as is the c= ase with > + ;; "uwsgi". In that case, fall back to a f= ull URL. > + (uri (pypi-uri ,(string-downcase name) vers= ion)) > + (sha256 > + (base32 > + ,(guix-hash-url temp))))) > + (build-system python-build-system) > + ,@(maybe-inputs required-inputs 'propagated-inputs) > + ,@(maybe-inputs test-inputs 'native-inputs) > + (home-page ,home-page) > + (synopsis ,synopsis) > + (description ,description) > + (license ,(license->symbol license))) > + ;; Flatten the nested lists and return the upstream > + ;; dependencies. > + upstream-dependencies)))))))) I don=E2=80=99t see anything being flattened here? -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Tue May 28 10:49:04 2019 Received: (at 24450) by debbugs.gnu.org; 28 May 2019 14:49:04 +0000 Received: from localhost ([127.0.0.1]:56806 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVdPT-0001zH-UW for submit@debbugs.gnu.org; Tue, 28 May 2019 10:49:04 -0400 Received: from a2062.mx.srv.dfn.de ([194.95.232.172]:42723) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVdPR-0001ym-3f for 24450@debbugs.gnu.org; Tue, 28 May 2019 10:49:02 -0400 Received: from localhost (localhost [127.0.0.1]) by a2062.mx.srv.dfn.de (Postfix) with ESMTP id DF9BAA0031; Tue, 28 May 2019 16:48:58 +0200 (CEST) Received: from a2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-han.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id VSoCkgIm3V4v; Tue, 28 May 2019 16:48:58 +0200 (CEST) Received: from SW-IT-P-CAS4.mdc-berlin.net (mgw10-4.mdc-berlin.de [141.80.113.59]) by a2062.mx.srv.dfn.de (Postfix) with ESMTPS; Tue, 28 May 2019 16:48:58 +0200 (CEST) Received: from localhost (141.80.113.103) by SW-IT-P-CAS4.mdc-berlin.net (141.80.113.59) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 28 May 2019 16:48:57 +0200 References: <87pnod7ot4.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87pnod7ot4.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Tue, 28 May 2019 16:48:57 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [141.80.113.103] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24644.000 X-TM-AS-Result: No-8.006500-8.000000-10 X-TMASE-MatchedRID: eVEkOcJu0F44HKI/yaqRmycRqaHJVb+nGbJMFqqIm9z5+tteD5RzhUby mcrhw7V6rCGDncGt7gr+V96edmhD/kX2eeYt6mYTEhGH3CRdKUUAaovTveFt1eKBVNW3+kNcEWX AmFVxJU77zimhwE/anUAL2GJYFq0wyZVJ0dEjeFYUUThwRwumkx+26QzoWaY2+S5C/08hWc3Qt0 uI1DQUF0nB7EMnenbEaaUnfwx7Kvns7aQkqkpFyr6EJGSqPePTVy6w8Jlyem+8LlXh8TWXyHHEa /+2Su8egSA7jvhGUu7X6cuSGZUoH4PLPRxB/IHvHch4gZ8olb8XaMhZSeVFDOyuaeea5tPzCT2G AWrPhQbnzlXMYw4XMAGLeSok4rrZbdTuPa9VRGvEQdG7H66TyND1NE3SaFLXlSBmsMk+d8G0FOu 4+5eHgiqAHGBzAju2d+2VMTYavEgUvTmGAdl9l10SYGmo9FVEXceYKC9vSP2s/Lhmj69Rq9DOGx Oy82DLD/EiZwqh/WBl2xuA1t/ICypPUAInKdYv9avTT1aTf3c8mxHOwOUqug== X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--8.006500-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24644.000 X-TM-SNTS-SMTP: 6B79292751917CFF5BFE8E454CF1BAEF36A591DC01E33BE30C03733F941596CC2000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) > From cfde6e09f8f8c692fe252d76ed27e8c50a9e5377 Mon Sep 17 00:00:00 2001 > From: Maxim Cournoyer > Date: Sat, 30 Mar 2019 23:13:26 -0400 > Subject: [PATCH 8/9] import: pypi: Scan source archive to find requires.t= xt > file. > * guix/import/pypi.scm (use-modules): Use invoke from (guix build utils). > (guess-requirements)[archive-root-directory]: Remove procedure. Oh, I guess I reviewed this procedure in vain :( Please modify the commits so that added procedures are not removed in later commits. This is easier on the reviewer and makes for a clearer commit history. > (define (guess-requirements-from-source) > ;; Return the package's requirements by guessing them from the sourc= e. > - (let ((dirname (archive-root-directory source-url)) > - (extension (file-extension source-url))) > - (if (string? dirname) > - (call-with-temporary-directory > - (lambda (dir) > - (let* ((pypi-name (string-take dirname (string-rindex dirna= me #\-))) > - (requires.txt (string-append dirname "/" pypi-name > - ".egg-info" "/requires.= txt")) > - (exit-code > - (parameterize ((current-error-port (%make-void-port= "rw+")) > - (current-output-port (%make-void-por= t "rw+"))) > - (if (string=3D? "zip" extension) > - (system* "unzip" archive "-d" dir requires.tx= t) > - (system* "tar" "xf" archive "-C" dir requires= .txt))))) > - (if (zero? exit-code) > - (parse-requires.txt (string-append dir "/" requires.t= xt)) > - (begin > - (warning > - (G_ "Failed to extract file: ~a from source.~%") > - requires.txt) > - (list '() '())))))) > + (if (compressed-file? source-url) > + (call-with-temporary-directory > + (lambda (dir) > + (parameterize ((current-error-port (%make-void-port "rw+")) > + (current-output-port (%make-void-port "rw+"))) > + (if (string=3D? "zip" (file-extension source-url)) > + (invoke "unzip" archive "-d" dir) > + (invoke "tar" "xf" archive "-C" dir))) > + (let ((requires.txt-files > + (find-files dir (lambda (abs-file-name _) > + (string-match "\\.egg-info/requires.txt$" > + abs-file-name))))) > + (if (> (length requires.txt-files) 0) Let=E2=80=99s work on the empty list directly. Here =E2=80=9Cmatch=E2=80= =9D would be better. > + (begin > + (parse-requires.txt (first requires.txt-files))) No need for =E2=80=9Cbegin=E2=80=9D here. > + (begin (warning (G_ "Cannot guess requirements from sou= rce archive:\ > + no requires.txt file found.~%")) > + (list '() '())))))) I know that this is from an earlier commit, but I don=E2=80=99t like the lo= ok of =E2=80=9C(list '() '())=E2=80=9D at all :) > + (begin > + (warning (G_ "Unsupported archive format; \ > +cannot determine package dependencies from source archive: ~a~%") > + (basename source-url)) > (list '() '())))) Same here. Certainly there=E2=80=99s a better return value. -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Tue May 28 10:53:24 2019 Received: (at 24450) by debbugs.gnu.org; 28 May 2019 14:53:24 +0000 Received: from localhost ([127.0.0.1]:56810 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVdTf-00027K-Pn for submit@debbugs.gnu.org; Tue, 28 May 2019 10:53:23 -0400 Received: from c2062.mx.srv.dfn.de ([194.95.238.172]:45329) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hVdTe-00027A-6J for 24450@debbugs.gnu.org; Tue, 28 May 2019 10:53:22 -0400 Received: from localhost (localhost [127.0.0.1]) by c2062.mx.srv.dfn.de (Postfix) with ESMTP id 429A9300059; Tue, 28 May 2019 16:53:20 +0200 (CEST) Received: from c2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-erl.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id qUTPpK5QUDqR; Tue, 28 May 2019 16:53:18 +0200 (CEST) Received: from SW-IT-P-CAS4.mdc-berlin.net (mgw10-4.mdc-berlin.de [141.80.113.59]) by c2062.mx.srv.dfn.de (Postfix) with ESMTPS; Tue, 28 May 2019 16:53:18 +0200 (CEST) Received: from localhost (141.80.113.103) by SW-IT-P-CAS4.mdc-berlin.net (141.80.113.59) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 28 May 2019 16:53:17 +0200 References: <87pnod7ot4.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87pnod7ot4.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Tue, 28 May 2019 16:53:17 +0200 Message-ID: MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [141.80.113.103] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24644.000 X-TM-AS-Result: No-6.695700-8.000000-10 X-TMASE-MatchedRID: zGP2F0O7j/vlMOLj7fx09vHkpkyUphL9FBBqKQc8RSe1eX0jEQ9c6vsv oRyxTO1b9wfh7XcCXjRjUDQcCP59W5GlJ5QKyUiCmvnKSb020hw6En2bnefhoB6uJ+0YWzZ5Y2i R7K8WcsxXsKIAGSi8OjrkkPO1QqP53tSwkMiqVsesoC9I+K53ITmKihe1K2IezP9LEqj2Yng4UL AZwl+oC+LzNWBegCW2Yck91jY6aVLQLWxBF9DMQcRB0bsfrpPIFT1PBs+MVY3LLrRfL5Fh4Sie6 UL+8iPHCmlK8jNDU/TELBrsVt/GrX+la2h0hakWeXpC4wGnh8Q= X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--6.695700-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24644.000 X-TM-SNTS-SMTP: A258435FF57D0BAEABC860E412B828D6436901272C5C6B07B065AC2A6C9E51FE2000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) And finally: Number 9! > From 1290f9d1f0d594fdd4723d76b94116be25da9dd5 Mon Sep 17 00:00:00 2001 > From: Maxim Cournoyer > Date: Sat, 30 Mar 2019 20:27:35 -0400 > Subject: [PATCH 9/9] import: pypi: Preserve package name case when forming > pypi-uri. > > Fixes issue: #33046. Please change this to: Fixes . > * guix/build-system/python.scm (pypi-uri): Update the host URI to > "files.pythonhosted.org". > * guix/import/pypi.scm (make-pypi-sexp): Preserve the package name case when > the source URL calls for it. Is the first change to use files.pythonhosted.org required to fix this? Or is this unrelated? If it is required this looks fine to me. Thank you! -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Wed May 29 22:24:23 2019 Received: (at 24450) by debbugs.gnu.org; 30 May 2019 02:24:23 +0000 Received: from localhost ([127.0.0.1]:60489 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hWAju-0004zM-SG for submit@debbugs.gnu.org; Wed, 29 May 2019 22:24:23 -0400 Received: from mail-qk1-f177.google.com ([209.85.222.177]:34196) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hWAjt-0004z5-2M for 24450@debbugs.gnu.org; Wed, 29 May 2019 22:24:21 -0400 Received: by mail-qk1-f177.google.com with SMTP id t64so2916836qkh.1 for <24450@debbugs.gnu.org>; Wed, 29 May 2019 19:24:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=DCXlxtPNOdT1itQbJKFvRFfzYk8cNrTGkLq4rp73uD0=; b=l0oPy36kCBpmEBx/OjrGA+6zJwP+y6Sz+1uXIg9D3XVf2iE7WbgJB6X919eDMurUQs /fsPEC9dxb7mKzf7kbOgjSZ0hCyFMDcP4BKFbrdgWefEPW7M/qZUCFtuhvpfpZUpGWfC pgs64CZXPDUy9l5WM+LdxqPSmdRDINO0Biaxw3I5A/S74UKWairYdOr2MeS+80fBcoEH NOtnbx0+r7xo1/8lEaIQKxeTp1Da5LX1Xr8g5Qd05Y28gEBONJbSP2z2z3hRDYf2p29x zRPnOKRevzxRJ5hQgXL9FbeToaHkubNlimrcUkAtgF9+st/aUB+9c8mGFYcCw/o9b93K IKAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=DCXlxtPNOdT1itQbJKFvRFfzYk8cNrTGkLq4rp73uD0=; b=frAJPqnL0EizwTcuOGcqFGySiMHrTzU7KuVTpMtTVXfjdb1Lq3cfb2+Edtx7sfPYpG Y0dlnzpSbqojA3Y967o/DNfU4rBa+NcN9ecWKa8FpzerSiqqBuy1bJeaaMjkXusvjgW5 7JoXjTYkcbx41RKrjhq7NXqekVS4ZknCmrt0usj5HHmEMYkC7sWjKTveAvjILFBOFVcM Pn8dTXLMWff5HR/RvBbBeh/wAq19OUDdwt/Xe1iNnvS0gqxBdUC8OTMnS3KDkCRR2F57 DAR2wk3pj0cm6fG/hu79+lQyuSYV30KqnJRzJaPUoDx9DskWZXkW/nSKsnznHMULKZFr qoag== X-Gm-Message-State: APjAAAVPUIHWBFMIRefaWfftvHKjJgCZP2tykHk097UkyDdyEHfq4oZi PbYNxTfs6TM33puE+5ufra/SmI2T X-Google-Smtp-Source: APXvYqz3iIyMQkvxEKrrgKha0JQAerYAckHvoR1nfokuRyZH3cuwTgvvDMximdQFux3q4giGQgtGvQ== X-Received: by 2002:ae9:e903:: with SMTP id x3mr992634qkf.128.1559183055369; Wed, 29 May 2019 19:24:15 -0700 (PDT) Received: from kwak (dsl-10-134-136.b2b2c.ca. [72.10.134.136]) by smtp.gmail.com with ESMTPSA id u185sm726918qkf.67.2019.05.29.19.24.13 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 29 May 2019 19:24:14 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: <87pnod7ot4.fsf@gmail.com> Date: Wed, 29 May 2019 22:24:13 -0400 In-Reply-To: (Ricardo Wurmus's message of "Tue, 28 May 2019 16:53:17 +0200") Message-ID: <87pno03cj6.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.3 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.3 (-) Hello Ricardo! Ricardo Wurmus writes: > And finally: Number 9! > >> From 1290f9d1f0d594fdd4723d76b94116be25da9dd5 Mon Sep 17 00:00:00 2001 >> From: Maxim Cournoyer >> Date: Sat, 30 Mar 2019 20:27:35 -0400 >> Subject: [PATCH 9/9] import: pypi: Preserve package name case when forming >> pypi-uri. >> >> Fixes issue: #33046. > > Please change this to: > > Fixes . > >> * guix/build-system/python.scm (pypi-uri): Update the host URI to >> "files.pythonhosted.org". >> * guix/import/pypi.scm (make-pypi-sexp): Preserve the package name case when >> the source URL calls for it. > > Is the first change to use files.pythonhosted.org required to fix this? > Or is this unrelated? > > If it is required this looks fine to me. > > Thank you! Thank you for this thorough review! I'll need some time to go through it, and an upcoming travel will delay it some more, so don't fret if you don't hear back from me before a couple days. Sorry! I'll keep you posted. Thanks a bunch! Maxim From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 09 22:11:12 2019 Received: (at 24450) by debbugs.gnu.org; 10 Jun 2019 02:11:12 +0000 Received: from localhost ([127.0.0.1]:55710 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ha9mC-0002QP-91 for submit@debbugs.gnu.org; Sun, 09 Jun 2019 22:11:12 -0400 Received: from mail-pf1-f181.google.com ([209.85.210.181]:40457) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ha9m9-0002QB-N3 for 24450@debbugs.gnu.org; Sun, 09 Jun 2019 22:11:10 -0400 Received: by mail-pf1-f181.google.com with SMTP id p184so1036442pfp.7 for <24450@debbugs.gnu.org>; Sun, 09 Jun 2019 19:11:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=1Wnq+UCZU0q4EXOWlI0zG/gaMMsxleQvtmHhDiu+bzA=; b=BdNROotnc+1q8vM7s5nxqaDvk9cZRZABHFu/qOsVEHVNAa11QfxxuF3bI8BQwJLyMh 9EMBPqnUpwixm7O352oO4uXFET0Rux2B5XLOpVlPG2TfiBxP4uQ3C7VAua6PSG92f3ZF aE7rN2JSdRfciRTacCfEj27s/FUMlHw1YWS/DXL0aaZ6qnq3TKKvXiDKcIxXQxFQa32h 9XM6vCmhFbtLRxKX6/IOsJvLXolmVOo7h2aDWXl2nPwm3q2VthMKY/ccRmxuuiPspI86 idzr/S5BKximCcBh8p3lDpdvzX+PU6UJAHmsguOw7l3yO+OKwrHzXr4nlyASmpSC5chV 80hA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=1Wnq+UCZU0q4EXOWlI0zG/gaMMsxleQvtmHhDiu+bzA=; b=ES+123A6xvl/GZnjEI3qsOkDvkb2x4MRGHNqX/gBX5iNY9lo+i21SovCNOSM4DVayt lgVoyrogE3TqC+laOBNPZMBVLE9Eeqb4QwCaM0sNWwd4c8HroDJ7L+2hFu1C26Gqc80e GEWx2TRA9wAI8RFMKUqzodR13m3NHDabdS8SPGqwDe3g2gLOo/4VzJySV5BDZAMS5xM+ s5WkNOJ1DHudh3KOmEpHmCJ1eacJPfYR+xs8uPwNxSzICU1EjEVXWesVEUqYkkJPVDm6 EE1SsoJcR2qx40dW8fo6fLgESBtRQ5se4f/9xNLF7kpu3LlrFy4cv100LY5ykvvlVjIH uAmw== X-Gm-Message-State: APjAAAUA3Vjn2L8BaPQtu9rgzTs2UaUq/EULG2BM71P10txOT5Jq9e89 PZeC+OaFGkiibAGQjT+1wFREEWemUUA= X-Google-Smtp-Source: APXvYqzXcjihKu7vMeOCcSGS74UQMjW0zIaDxYet6z7cEb5YUzX6GTbRV3csIhDB1T8F4zV56lbt+A== X-Received: by 2002:a62:bd0e:: with SMTP id a14mr71997552pff.44.1560132663677; Sun, 09 Jun 2019 19:11:03 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id x7sm8662751pfm.82.2019.06.09.19.11.01 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sun, 09 Jun 2019 19:11:02 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: <87pnod7ot4.fsf@gmail.com> Date: Mon, 10 Jun 2019 11:10:58 +0900 In-Reply-To: (Ricardo Wurmus's message of "Mon, 27 May 2019 16:48:43 +0200") Message-ID: <87sgsi5gwd.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello Ricardo! Ricardo Wurmus writes: > Hi Maxim, > >> Subject: [PATCH 1/9] import: pypi: Do not consider requirements.txt file= s. >> >> * guix/import/pypi.scm (guess-requirements): Update comment. >> [guess-requirements-from-source]: Do not attempt to parse the file >> requirements.txt. Streamline logic. > > Why remove the handling of the requirements.txt? Is it no longer > popular enough to expect its availability in the source archives? > > Please also mention in the commit message that and how you adjusted the > tests. The commit message now explains the above: import: pypi: Do not consider requirements.txt files. PyPI packages are mandated to have a setup.py file, which contains a li= sting of the required dependencies. The setuptools/distutils machinery embed metadata in the archives they produce, which contains this information.= There is no need nor gain to collect the requirements from a "requirements.tx= t" file, as it is not the true record of dependencies for PyPI packages an= d may contain extraneous requirements or not exist at all. * guix/import/pypi.scm (guess-requirements): Update comment. [guess-requirements-from-source]: Do not attempt to parse the file requirements.txt. Streamline logic. * tests/pypi.scm (test-requires.txt): Rename from test-requirements, to= hint at the file being tested. ("pypi->guix-package"): Adapt so that the fake package contains a requi= res.txt file rather than a requirements.txt file. ("pypi->guix-package, wheels"): Likewise. > You removed the comments from the example requires.txt =E2=80=94 are > comments no longer permitted in these files? If they are, please don=E2= =80=99t > include those changes. The comments are now preserved. Thank you! Maxim From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 09 23:31:00 2019 Received: (at 24450) by debbugs.gnu.org; 10 Jun 2019 03:31:00 +0000 Received: from localhost ([127.0.0.1]:55782 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haB1Q-0006S4-3a for submit@debbugs.gnu.org; Sun, 09 Jun 2019 23:31:00 -0400 Received: from mail-pg1-f193.google.com ([209.85.215.193]:39771) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haB1O-0006Rs-9e for 24450@debbugs.gnu.org; Sun, 09 Jun 2019 23:30:58 -0400 Received: by mail-pg1-f193.google.com with SMTP id 196so4246058pgc.6 for <24450@debbugs.gnu.org>; Sun, 09 Jun 2019 20:30:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=xQtHcSmud7ZUGytQLBEaNVZ9MyGcN/uhxeyRcdXkNjM=; b=sBzrnPLw3GwE4iid109C/KVusESFa+UwEzXXCpOkanwdmh8hXoexTDCCXs0LRR1m5h L8fL8j1MEcMAi0/Tb006udHi8PNZv5dS22/AArGzi+e3LOrlO4Kj22Q+IPbeoPrNGHrx a2aX2V7SeA2waMA7oHgOD5ibHoSARmSvnN0noaSa0tFzs87BUMMSrQU3ufTsFEOXb9R8 gXkcXRvXLJboXt9Rkcb4gRa8w0pj/Z3XPDprbeYs+3F2KlZlm5boQpcC1TSSf7RD3axe ADA/LtB5emyu6uuaPa7V+YFWNk2W7cYWEx3sqQw+aFdeKn3RVjToIDyobnQqyfNtprDE m2vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=xQtHcSmud7ZUGytQLBEaNVZ9MyGcN/uhxeyRcdXkNjM=; b=DPTak81cZjVbQc5zDYuFi8a2Ekf7/8dT70tXYAIRKi7rYK5sUouhmTatqPSksjmyXF h7DcYei7bPmOab79VXkVumIRdM4Kut7IFrH9XgAMcGKS36eErptMNwaDx1l6Mk/wxGCO ddu9Imj0ts/snpQyyz0qKyJ9S4Bh/YjEToqo5jxRsM9Msih79ol9nu8nluIZULZs9Tno scPvEgoDtG+UEosb6FlW79D5/njdw07yZgJXG08QM4aEZ7PeHEf/GGpzD14nC7LLkShA yWb6FVZdB+4lOGQ9XukzWYBBPQZCv2pX0317zeKD+WAld8Wx5cGopElxpZxxalkPRff4 PhsA== X-Gm-Message-State: APjAAAUb1LLsl/e+XnNmcIkSDdeeQ4BAp/e5nDrPqIinSgWFYzhq4afh u8zZjnvUb1VGIWkBQOxXJDR6lUYE4tU= X-Google-Smtp-Source: APXvYqwEBdFh/t9JuzKGwPMNtrv27MqMy7M96H4qBQQ50v+s9oBbkYa0Reo2vngGv01DEM+us9jP4A== X-Received: by 2002:a65:4c07:: with SMTP id u7mr13379720pgq.93.1560137451941; Sun, 09 Jun 2019 20:30:51 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id t5sm8581715pgh.46.2019.06.09.20.30.50 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sun, 09 Jun 2019 20:30:51 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: <87pnod7ot4.fsf@gmail.com> Date: Mon, 10 Jun 2019 12:30:47 +0900 In-Reply-To: (Ricardo Wurmus's message of "Mon, 27 May 2019 17:11:33 +0200") Message-ID: <87muiq5d7c.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello again! Ricardo Wurmus writes: > Hi Maxim, > > on to patch number 2! Yay! >> From 5f79b0502f62bd1dacc8ea143c1dbd9ef7cfc29d Mon Sep 17 00:00:00 2001 >> From: Maxim Cournoyer >> Date: Thu, 28 Mar 2019 00:26:00 -0400 >> Subject: [PATCH 2/9] import: pypi: Do not parse optional requirements fr= om >> source. >> >> * guix/import/pypi.scm: Export PARSE-REQUIRES.TXT. >> (guess-requirements): Move the READ-REQUIREMENTS procedure to the top le= vel, >> and rename it to PARSE-REQUIRES.TXT. Move the CLEAN-REQUIREMENT and COM= MENT? >> functions inside the READ-REQUIREMENTS procedure. >> (parse-requires.txt): Add a SECTION-HEADER? predicate, and use it to pre= vent >> parsing optional requirements. >> >> * tests/pypi.scm (test-requires-with-sections): New variable. >> ("parse-requires.txt, with sections"): New test. >> ("pypi->guix-package"): Mute tar output to stdout. > > The commit log does not match the changes. CLEAN-REQUIREMENT is now a > top-level procedure, not a local procedure inside of READ-REQUIREMENTS > as reported in the commit message. Which is correct? Fixed. >> + (call-with-input-file requires.txt >> + (lambda (port) >> + (let loop ((result '())) >> + (let ((line (read-line port))) >> + ;; Stop when a section is encountered, as sections contains o= ptional > > Should be =E2=80=9Ccontain=E2=80=9D. Fixed. >> + ;; (extra) requirements. Non-optional requirements must appe= ar >> + ;; before any section is defined. >> + (if (or (eof-object? line) (section-header? line)) >> + (reverse result) >> + (cond >> + ((or (string-null? line) (comment? line)) >> + (loop result)) >> + (else >> + (loop (cons (clean-requirement line) >> + result)))))))))) >> + > > I think it would be better to use =E2=80=9Cmatch=E2=80=9D here instead of= nested =E2=80=9Clet=E2=80=9D, > =E2=80=9Cif=E2=80=9D and =E2=80=9Ccond=E2=80=9D. At least you can drop t= he =E2=80=9Cif=E2=80=9D and just use cond. > > The loop let and the inner let can be merged. I'm not sure I understand; wouldn't merging the named let with the plain let mean adding an extra LINE argument to my LOOP procedure? I don't want that. Also, how could the above code be expressed using "match"? I'm using predicates which tests for (special) characters in a string; I don't see how the more primitive pattern language of "match" will enable me to do the same. >> +(define (parse-requires.txt requires.txt) >> + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of >> +requirement names." >> + ;; This is a very incomplete parser, which job is to select the non-o= ptional > > =E2=80=9Cwhich=E2=80=9D =E2=80=93> =E2=80=9Cwhose=E2=80=9D Fixed, with due diligence reading on English grammar ;-) >> + ;; dependencies and strip them out of any version information. >> + ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) >> + ;; library and the requirements grammar defined by PEP-0508 >> + ;; (https://www.python.org/dev/peps/pep-0508/). > > Let=E2=80=99s remove the sentence starting with =E2=80=9CAlternatively=E2= =80=A6=E2=80=9D. We could do > that but we didn=E2=80=99t :) Alright; done! >> + (define (section-header? line) >> + ;; Return #t if the given LINE is a section header, #f otherwise. >> + (let ((trimmed-line (string-trim line))) >> + (and (not (string-null? trimmed-line)) >> + (eq? (string-ref trimmed-line 0) #\[)))) >> + > > How about using string-prefix? instead? This looks more complicated > than it deserves. You can get rid of string-null? and eq? and string-ref > and all that. > > Same here: > >> + (define (comment? line) >> + ;; Return #t if the given LINE is a comment, #f otherwise. >> + (eq? (string-ref (string-trim line) 0) #\#)) > > I=E2=80=99d just use string-prefix? here. Neat! Adjusted, for both counts. >> +(define (clean-requirement s) >> + ;; Given a requirement LINE, as can be found in a setuptools requires= .txt >> + ;; file, remove everything other than the actual name of the required >> + ;; package, and return it. >> + (string-take s (or (string-index s (lambda (chr) >> + (member chr '(#\space #\> #\=3D = #\<)))) >> + (string-length s)))) > > =E2=80=9Cstring-take=E2=80=9D with =E2=80=9Cstring-length=E2=80=9D is not= very elegant. The char > predicate in string-index could better be a char set: > > (define (clean-requirement s) > (cond > ((string-index s (char-set #\space #\> #\=3D #\<)) =3D> (cut string-tak= e s <>)) > (else s))) That's nicer, thanks! >> ("pypi->guix-package"): Mute tar output to stdout. > > Finally, I think it would be better to keep this separate because it=E2= =80=99s > really orthogonal to the other changes in this patch. OK, done! > What do you think? All good points; I just don't understand the one about using a match and/or merging the regular "let" with the named "let". Thanks, Maxim From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 10 04:32:15 2019 Received: (at 24450) by debbugs.gnu.org; 10 Jun 2019 08:32:15 +0000 Received: from localhost ([127.0.0.1]:55948 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haFix-0005WP-5Z for submit@debbugs.gnu.org; Mon, 10 Jun 2019 04:32:15 -0400 Received: from mail-pf1-f180.google.com ([209.85.210.180]:46681) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haFiu-0005WA-SJ for 24450@debbugs.gnu.org; Mon, 10 Jun 2019 04:32:13 -0400 Received: by mail-pf1-f180.google.com with SMTP id 81so4841552pfy.13 for <24450@debbugs.gnu.org>; Mon, 10 Jun 2019 01:32:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=KPV0Q2IcskXF3aAPyZlRruXlm/hM3X8yQ6GL6EVvR7k=; b=Xk1TCjdkC7i1D+UOSwu047fX5HF/gA5sMIm8KkZpvWHtWOPsjys9h0l9dpT6x5elCm y3Vgelpkz+fPEXbQ3QuQCdVDx1s5sojizIzaVV/4JpEcq1i1aueABnAPjrnsuGq+6y3y fsnJRnVtDg+79bRWBt9yDfejEjQRZsVjOAGWg8Y7hkZpdR/d6O5N03gz98xLR7NXijM+ gZnz63fvZr2MVvfKoYaGPJsYHhwm8Pk9HCDq8ESFknFLNdwzntCFHeiOosteYreemrpv f6rE/Ff54KvC42F5kbSnUDdC+mZ23xM4cd3HHdxqI2WHTsZLVDtTXYz5fvO0AzqDlCZ/ vo3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=KPV0Q2IcskXF3aAPyZlRruXlm/hM3X8yQ6GL6EVvR7k=; b=oycJymj0VOqh/RsmQoTXYtGLmMKn1XkMWr5FZxKqp7NKPwHO0wz/C5m8X38m5W5FbS t4xQVLTLYUe9l08XJFzhpWJgdVdL8nStl2Pp+/7i+hi1ty5p5t3vZZf5ETDTPsy+4ml6 DksGzBbOD6bHRgoRKRAI/YTKySin7WeY/SkVwonIbjiqTz40usm1/2GL+9kDJI2TEvJ6 2rys9jrdziaZgwwWgnRHVUX3tl0j4LcT0yTa7NmAokR0jYoc6t4R8YB84v8qykb6RaR0 raaBOFdd8dvAEzjMeNbIPAdN7oEUUmDntDOTXJwo7puk9yfqBsdxObsKPHeu9roG4MXL BNrg== X-Gm-Message-State: APjAAAXWTZJgg1VJgEwdpqtlmr7Y6zF3h9D0BeJw3JaZVWSCRJWx0Rjw 3RfrRZKdRlDJ8pqjtx2ZbOgYhBL8BH8= X-Google-Smtp-Source: APXvYqyIQ9kLu/P5T3xSxETKGZbA3RzBAPGnSiV7iGFAjtWAg9W2rLVOPTJa9+eE0AUg9VW/Q97qcQ== X-Received: by 2002:a17:90a:37e9:: with SMTP id v96mr19680584pjb.10.1560155526492; Mon, 10 Jun 2019 01:32:06 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id u12sm9163016pjn.24.2019.06.10.01.32.04 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 01:32:05 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: <87pnod7ot4.fsf@gmail.com> Date: Mon, 10 Jun 2019 17:32:01 +0900 In-Reply-To: (Ricardo Wurmus's message of "Mon, 27 May 2019 17:54:39 +0200") Message-ID: <87imtd6dtq.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello! Ricardo Wurmus writes: > Patch number 3! Yay! >> From 0c62b541a3e8925b5ca31fe55dbe7536cf95151f Mon Sep 17 00:00:00 2001 >> From: Maxim Cournoyer >> Date: Thu, 28 Mar 2019 00:26:01 -0400 >> Subject: [PATCH 3/9] import: pypi: Improve parsing of requirement >> specifications. >> >> The previous solution was fragile and could leave unwanted characters in= a >> requirement name, such as '[' or ']'. > > Wouldn=E2=80=99t it be sufficient to add [ and ] to the list of forbidden > characters? The tests pass with this implementation of > clean-requirements: > > (define (clean-requirements s) > (cond > ((string-index s (char-set #\space #\> #\=3D #\< #\[ #\])) =3D> (cut st= ring-take s <>)) > (else s))) Indeed this would be sufficient to make the tests pass, but the tests don't cover all the cases; as an example, consider: --8<---------------cut here---------------start------------->8--- argparse;python_version<"2.7" --8<---------------cut here---------------end--------------->8--- While we could make it work with the current logic by adding more invalid characters (such as ';' here) to the character set, it seems less error prone to use the upstream provided regex to match a package name. [0] >> +(define %requirement-name-regexp >> + ;; Regexp to match the requirement name in a requirement specificatio= n. >> + >> + ;; Some grammar, taken from PEP-0508 (see: >> + ;; https://www.python.org/dev/peps/pep-0508/). >> + >> + ;; The unified rule can be expressed as: >> + ;; specification =3D wsp* ( url_req | name_req ) wsp* >> + >> + ;; where url_req is: >> + ;; url_req =3D name wsp* extras? wsp* urlspec wsp+ quoted_marker? >> + >> + ;; and where name_req is: >> + ;; name_req =3D name wsp* extras? wsp* versionspec? wsp* quoted_marke= r? >> + >> + ;; Thus, we need only matching NAME, which is expressed as: >> + ;; identifer_end =3D letterOrDigit | (('-' | '_' | '.' )* letterOrDig= it) >> + ;; identifier =3D letterOrDigit identifier_end* >> + ;; name =3D identifier >> + (let* ((letter-or-digit "[A-Za-z0-9]") >> + (identifier-end (string-append "(" letter-or-digit "|" >> + "[-_.]*" letter-or-digit ")")) >> + (identifier (string-append "^" letter-or-digit identifier-end = "*")) >> + (name identifier)) >> + (make-regexp name))) > > This seems a little too complicated. Translating a grammar into a > regexp is probably not a good idea in general. Since we don=E2=80=99t ca= re > about anything other than the name it seems easier to just chop off > the string tail as soon as we find an invalid character. While I agree that a regexp is a bigger hammer than basic string manipulation, I see some merit to it here: 1) We can be assured of conformance with upstream, again, per PEP-0508. 2) It is easier to extend; we might want to add parsing for the version spec in order to disregard dependencies specified for Python < 3, for example. The use of the PEP-0508 grammar to define the regexp is useful to detail in a more human-friendly language the components of the regexp. We could have otherwise used the more cryptic regexp for Python distribution names: --8<---------------cut here---------------start------------->8--- ^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$ --8<---------------cut here---------------end--------------->8--- So I guess that what I'm saying is that I prefer this approach to using string-index with invalid characters, for the reasons above. [0] https://www.python.org/dev/peps/pep-0508/ Thanks! Maxim From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 10 05:12:07 2019 Received: (at 24450) by debbugs.gnu.org; 10 Jun 2019 09:12:07 +0000 Received: from localhost ([127.0.0.1]:55989 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haGLX-0006Vd-FK for submit@debbugs.gnu.org; Mon, 10 Jun 2019 05:12:07 -0400 Received: from a2062.mx.srv.dfn.de ([194.95.232.172]:41061) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haGLW-0006VV-D4 for 24450@debbugs.gnu.org; Mon, 10 Jun 2019 05:12:07 -0400 Received: from localhost (localhost [127.0.0.1]) by a2062.mx.srv.dfn.de (Postfix) with ESMTP id 2E2C0A0065; Mon, 10 Jun 2019 11:12:05 +0200 (CEST) Received: from a2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-han.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id y5iUxp6lpClD; Mon, 10 Jun 2019 11:12:04 +0200 (CEST) Received: from SW-IT-P-CAS3.mdc-berlin.net (mgw10-3.mdc-berlin.de [141.80.113.58]) by a2062.mx.srv.dfn.de (Postfix) with ESMTPS; Mon, 10 Jun 2019 11:12:04 +0200 (CEST) Received: from localhost (84.173.76.17) by SW-IT-P-CAS3.mdc-berlin.net (141.80.113.58) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 10 Jun 2019 11:12:04 +0200 References: <87pnod7ot4.fsf@gmail.com> <87imtd6dtq.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87imtd6dtq.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 10 Jun 2019 11:12:03 +0200 Message-ID: <87d0jldct8.fsf@mdc-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [84.173.76.17] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24664.003 X-TM-AS-Result: No-9.559600-8.000000-10 X-TMASE-MatchedRID: VfovoVrt/obOH9fyebH9T/HkpkyUphL9aVixrzjDaFN7pZMzAYF2Japh 1iI3bpfXaUkHgTSDEhvS0DwYk1xlFymx5D2997OSDdN6n3Qne5ACC8zqHvcG2l46k3j1xInd7N6 Kq6/qW8rNjdj+CIVISBrtg4gJZD1Ys3EVKhtczkcktnKC/kfM5RfbPFE2GHrVUCgEErrUGFxI7I 0LlNJFKJM8ZhG18NAhV4ysb0UgFCFdInhzedP5B8K1Ib9JAALxOhJ9m53n4aDzlv7FEwWOy1wz7 9OZizNh2AfP/R+DknNftuJwrFEhTbew1twePJJB3QfwsVk0UbvWxQsAIHSpEj9HIf1/E6cxCq3L 81mXxWp86hR7GnOS9vLHTNV7s0kPBVXG0Hf6Z7qUTGVAhB5EbQ== X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--9.559600-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24664.003 X-TM-SNTS-SMTP: FC8A31E93D438F90ADED51F1EBB2EE6010D81B0790CB6ED94780430D7C5FC0652000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Maxim Cournoyer writes: > While I agree that a regexp is a bigger hammer than basic string > manipulation, I see some merit to it here: > > 1) We can be assured of conformance with upstream, again, per PEP-0508. > 2) It is easier to extend; we might want to add parsing for the version > spec in order to disregard dependencies specified for Python < 3, for > example. > > The use of the PEP-0508 grammar to define the regexp is useful to detail > in a more human-friendly language the components of the regexp. We > could have otherwise used the more cryptic regexp for Python > distribution names: > > --8<---------------cut here---------------start------------->8--- > ^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$ > --8<---------------cut here---------------end--------------->8--- > > So I guess that what I'm saying is that I prefer this approach to using > string-index with invalid characters, for the reasons above. > > [0] https://www.python.org/dev/peps/pep-0508/ Okay, sounds good. Please make sure to note this in a comment, so that I won=E2=80=99t be asking myself this same question in a year :) -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 10 05:23:15 2019 Received: (at 24450) by debbugs.gnu.org; 10 Jun 2019 09:23:16 +0000 Received: from localhost ([127.0.0.1]:56001 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haGWJ-0006mw-II for submit@debbugs.gnu.org; Mon, 10 Jun 2019 05:23:15 -0400 Received: from b2062.mx.srv.dfn.de ([194.95.234.172]:46029) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haGWG-0006mn-Uv for 24450@debbugs.gnu.org; Mon, 10 Jun 2019 05:23:14 -0400 Received: from localhost (localhost [127.0.0.1]) by b2062.mx.srv.dfn.de (Postfix) with ESMTP id 6BCED160060; Mon, 10 Jun 2019 11:23:11 +0200 (CEST) Received: from b2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-tub.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id Vw8mYGptlIRn; Mon, 10 Jun 2019 11:23:10 +0200 (CEST) Received: from SW-IT-P-CAS3.mdc-berlin.net (mgw10-3.mdc-berlin.de [141.80.113.58]) by b2062.mx.srv.dfn.de (Postfix) with ESMTPS; Mon, 10 Jun 2019 11:23:10 +0200 (CEST) Received: from localhost (84.173.76.17) by SW-IT-P-CAS3.mdc-berlin.net (141.80.113.58) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 10 Jun 2019 11:23:10 +0200 References: <87pnod7ot4.fsf@gmail.com> <87muiq5d7c.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87muiq5d7c.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 10 Jun 2019 11:23:10 +0200 Message-ID: <87blz5dcap.fsf@mdc-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [84.173.76.17] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24664.003 X-TM-AS-Result: No-6.803700-8.000000-10 X-TMASE-MatchedRID: 8+bhjh9TQnHOH9fyebH9T/HkpkyUphL9fglgnB0nDhOsHCH3SeE3CTde 18J/6zdzNZTWYEhFabqoYal/MKaO3s5IBlUNZPHsD3uYMxd01bdM8zV+hbhmAw+XMlIFkG/VCK1 n+RaAKL9FhtKHemq9omWnn1PcTo7cwl/Y7CtAgSS2DJSILb1B3nFa/hQHt1A1hOKxyaUqiJkCsG czoaBAubQNmdi08xgouyDZtkmyijHLkQcFIcXJTVHmrymVJ0uQ4yuES3aMEKP+7KZICEbEsnxJl 9UIBUPmngb0nVfuaRoQYVPuzGa4HUJQic2Ku7fACuDAUX+yO6ZE9JRrX2yv6WrUzQFwpcryo8WM kQWv6iWhMIDkR/KfwI2j49Ftap9Eymsk/wUE4hoMFsa+1wyh/H6wm764g3HaEjXzCq1dguwrCNP jTcFECANS2W03P0LMaHtnAH8C6gm9zB58BMCU7NOWlLwfKqgVNsgMNYLU/+8Gfccr8JfVDhx8YT XRzzSfhQKVx56dv6vHvLA9QBmaR0HS10HyazBfJwtHltZ5lW2UTGVAhB5EbQ== X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--6.803700-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24664.003 X-TM-SNTS-SMTP: BD1C2B77CDDC555C26C21DA63BA625D2B2B518BA67495768E0264041C47E18C62000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Maxim Cournoyer writes: >>> + ;; (extra) requirements. Non-optional requirements must app= ear >>> + ;; before any section is defined. >>> + (if (or (eof-object? line) (section-header? line)) >>> + (reverse result) >>> + (cond >>> + ((or (string-null? line) (comment? line)) >>> + (loop result)) >>> + (else >>> + (loop (cons (clean-requirement line) >>> + result)))))))))) >>> + >> >> I think it would be better to use =E2=80=9Cmatch=E2=80=9D here instead o= f nested =E2=80=9Clet=E2=80=9D, >> =E2=80=9Cif=E2=80=9D and =E2=80=9Ccond=E2=80=9D. At least you can drop = the =E2=80=9Cif=E2=80=9D and just use cond. >> >> The loop let and the inner let can be merged. > > I'm not sure I understand; wouldn't merging the named let with the plain > let mean adding an extra LINE argument to my LOOP procedure? I don't > want that. Let=E2=80=99s forget about merging the nested =E2=80=9Clet=E2=80=9D, becaus= e you would indeed need to change a few more things. It=E2=80=99s fine to keep that as it is.= But (if =E2=80=A6 (cond =E2=80=A6)) is not pretty. At least it could be done i= n one =E2=80=9Ccond=E2=80=9D: (cond ((or (eof-object? line) (section-header? line)) (reverse result)) ((or (string-null? line) (comment? line)) (loop result)) (else (loop (cons (clean-requirement line) result)))) > Also, how could the above code be expressed using "match"? I'm using > predicates which tests for (special) characters in a string; I don't see > how the more primitive pattern language of "match" will enable me to do > the same. =E2=80=9Cmatch=E2=80=9D has support for predicates, so you could do somethi= ng like this: (match line ((or (eof-object) (? section-header?)) (reverse result)) ((or '() (? comment?)) (loop result)) (_ (loop (cons (clean-requirement line) result)))) This allows you to match =E2=80=9Ceof-object=E2=80=9D and '() directly. Wh= enever I see =E2=80=9Cstring-null?=E2=80=9D I think it might be better to =E2=80=9Cmatch= =E2=80=9D on the empty list directly. But really, that=E2=80=99s up to you. I only feel strongly about avoiding = =E2=80=9C(if =E2=80=A6 (cond =E2=80=A6))=E2=80=9D. -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 10 09:31:03 2019 Received: (at 24450) by debbugs.gnu.org; 10 Jun 2019 13:31:03 +0000 Received: from localhost ([127.0.0.1]:56169 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haKO6-0000q3-Dh for submit@debbugs.gnu.org; Mon, 10 Jun 2019 09:31:03 -0400 Received: from mail-pg1-f174.google.com ([209.85.215.174]:38661) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haKO4-0000j0-LB for 24450@debbugs.gnu.org; Mon, 10 Jun 2019 09:31:01 -0400 Received: by mail-pg1-f174.google.com with SMTP id v11so5055357pgl.5 for <24450@debbugs.gnu.org>; Mon, 10 Jun 2019 06:31:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=QAmMa7Xn2/4088hjoZ/1E+qBf3STrH9k3n1vfvLWRJQ=; b=VbF355E6c2YnJV00CRO/2m/5vbJ60C66N6Kjy+Hqc6pu6PU1z0LpdDVbh6I3BhA5N8 cWWf5fKNGo6NLZoJJ402ECiJzmovPE6zg+01Re6LtA2e/YHA1hNc6XlZqmeqrKD77a0B TG463v9eQxE0oFQTH4Yt/stzPi/7n1a5oXqsUoIUr2A/yrhcAkZ3Si/GfKFF96J14vDZ yAUG+jfp4zh9tHMX9oedtdT1QMOq0g04efU3JKdcwXlpCi1o4wDPLCxegZBaGUxQjNm8 KA1LNn0WMtGoRP/NRY1wrHL352F/K037HYF9s80auSPsBlH4YNClmrnGgO6QGOiyVZXR 6saA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=QAmMa7Xn2/4088hjoZ/1E+qBf3STrH9k3n1vfvLWRJQ=; b=iI3hKvWlWlFCRyeG7qi7DDeauRVD9xWOuJp+p22JM+oXf6Q7rSf2fHdeic55kYwosO Ua9o5yYKFrud6UkNvh9lgW2NpJpPD34jSMqJDkXPska2D8MbKrZllc9xKxV8hIRmdne+ Gzxu3li3tAPDENuyRsrG5RVsT7rk8EBSo0eFXvBCZBQsYrE5ZBirM9eGmv3U5iWc9X1H cGHDUCQjUKUMlxKiGhkf8zBMzMNMCwONabK1QPnIFcA+w1ODMOl9RvXCJdsd9GrvEwAb mEX16g59MI820WAsCBc7Wuqn9uGnvkHokF/Awj40ItfJ0UvJTKtFfhHWT6PfqGmeFdUe e0Cw== X-Gm-Message-State: APjAAAVRkIWj/Fx93Wb+c4W3huhNrpmC2FeQwA9BMXKJ5LsC1Ngr56XI HN797BzgbTV/BT8DQOi5l4caVLbIKrQ= X-Google-Smtp-Source: APXvYqx/ArWA4DUS+Kv9KYiZF2hU0Jh+8tTxGgHZvThdTHdfEB50fNaL3rVSiolNfydCAh8hiuxM5Q== X-Received: by 2002:aa7:9f8b:: with SMTP id z11mr7814655pfr.121.1560173453062; Mon, 10 Jun 2019 06:30:53 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id m6sm10966073pgr.18.2019.06.10.06.30.49 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 06:30:50 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: <87pnod7ot4.fsf@gmail.com> Date: Mon, 10 Jun 2019 22:30:45 +0900 In-Reply-To: (Ricardo Wurmus's message of "Tue, 28 May 2019 12:23:53 +0200") Message-ID: <87blz55zzu.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello again! Ricardo Wurmus writes: > On to the next: > >> From 73e27235cac1275ba7671fd2364325cf5788cb3c Mon Sep 17 00:00:00 2001 >> From: Maxim Cournoyer >> Date: Thu, 28 Mar 2019 00:26:02 -0400 >> Subject: [PATCH 5/9] import: pypi: Support more types of archives. >> >> This change enables the PyPI importer to look for requirements in a sour= ce >> archive of a different type than "tar.gz" or "tar.bz2". > > Okay. > >> * guix/import/pypi.scm: (guess-requirements)[tarball-directory]: Rename = to... >> [archive-root-directory]: this. Use COMPRESSED-FILED? to determine if an >> archive is supported or not. > > Nitpick: please use =E2=80=9C...this.=E2=80=9D and leave two spaces betwe= en sentences. Done. > Typo: it should be COMPRESSED-FILE? Fixed. >> [guess-requirements-from-source]: Adapt to use the new method, and use u= nzip >> to extract ZIP archives. > > s/method/procedure/ Done. > Please also mention that =E2=80=9Ccompute-inputs=E2=80=9D has been adjust= ed. Done. >> - (define (tarball-directory url) >> - ;; Given the URL of the package's tarball, return the name of the d= irectory >> + (define (archive-root-directory url) >> + ;; Given the URL of the package's archive, return the name of the d= irectory >> ;; that will be created upon decompressing it. If the filetype is n= ot >> ;; supported, return #f. >> - ;; TODO: Support more archive formats. >> - (let ((basename (substring url (+ 1 (string-rindex url #\/))))) >> - (cond >> - ((string-suffix? ".tar.gz" basename) >> - (string-drop-right basename 7)) >> - ((string-suffix? ".tar.bz2" basename) >> - (string-drop-right basename 8)) >> - (else >> + (if (compressed-file? url) >> + (let ((root-directory (file-sans-extension (basename url)))) >> + (if (string=3D? "tar" (file-extension root-directory)) >> + (file-sans-extension root-directory) >> + root-directory)) >> (begin >> - (warning (G_ "Unsupported archive format: \ >> -cannot determine package dependencies")) >> - #f))))) >> + (warning (G_ "Unsupported archive format (~a): \ >> +cannot determine package dependencies") (file-extension url)) >> + #f))) > > I think the double application of file-sans-extension and the > intermediate variable name =E2=80=9Croot-directory=E2=80=9D for something= that is a file > is a little confusing, but I don=E2=80=99t have a better proposal (other = than to > replace file-extension and file-sans-extension with a match expression). Done, w.r.t. using "match": --8<---------------cut here---------------start------------->8--- @@ -198,10 +198,12 @@ be extracted in a temporary directory." ;; that will be created upon decompressing it. If the filetype is not ;; supported, return #f. (if (compressed-file? url) - (let ((root-directory (file-sans-extension (basename url)))) - (if (string=3D? "tar" (file-extension root-directory)) - (file-sans-extension root-directory) - root-directory)) + (match (file-sans-extension (basename url)) + (root-directory + (match (file-extension root-directory) + ("tar" + (file-sans-extension root-directory)) + (_ root-directory)))) (begin (warning (G_ "Unsupported archive format (~a): \ cannot determine package dependencies") (file-extension url)) --8<---------------cut here---------------end--------------->8--- >> (define (read-wheel-metadata wheel-archive) >> ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the pack= age's >> @@ -246,16 +243,20 @@ cannot determine package dependencies")) > >> (define (guess-requirements-from-source) >> ;; Return the package's requirements by guessing them from the sour= ce. >> - (let ((dirname (tarball-directory source-url))) >> + (let ((dirname (archive-root-directory source-url)) >> + (extension (file-extension source-url))) >> (if (string? dirname) >> (call-with-temporary-directory >> (lambda (dir) >> (let* ((pypi-name (string-take dirname (string-rindex dirn= ame #\-))) >> (requires.txt (string-append dirname "/" pypi-name >> ".egg-info" "/requires= .txt")) >> - (exit-code (parameterize ((current-error-port (%mak= e-void-port "rw+")) >> - (current-output-port (%ma= ke-void-port "rw+"))) >> - (system* "tar" "xf" tarball "-C" dir r= equires.txt)))) >> + (exit-code >> + (parameterize ((current-error-port (%make-void-por= t "rw+")) >> + (current-output-port (%make-void-po= rt "rw+"))) >> + (if (string=3D? "zip" extension) >> + (system* "unzip" archive "-d" dir requires.t= xt) >> + (system* "tar" "xf" archive "-C" dir require= s.txt))))) > > I guess this is why I=E2=80=99m not too happy with this: we=E2=80=99re ch= ecking in > multiple places if the format is supported but then forget about this > again until the next time we need to do something to the file. I don't see much of a problem with the current design since there are two questions being answered: 1) What should be the directory name of the extracted package (retrieved from the base name of the archive). 2) What extractor should be used (zip vs tar). These two questions are orthogonal, and that the same primitive get used to answer both is an implementation, or rather, an optimization detail. > I wonder if we could do better and answer the question just once. The questions are different :-). We could optimize, but that would be at the price of expressiveness (squash the two questions into one solving space). What do you think? Maxim From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 10 16:13:47 2019 Received: (at 24450) by debbugs.gnu.org; 10 Jun 2019 20:13:47 +0000 Received: from localhost ([127.0.0.1]:57518 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haQfr-0003p8-1k for submit@debbugs.gnu.org; Mon, 10 Jun 2019 16:13:47 -0400 Received: from b2062.mx.srv.dfn.de ([194.95.234.172]:32773) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haQfo-0003on-5c for 24450@debbugs.gnu.org; Mon, 10 Jun 2019 16:13:45 -0400 Received: from localhost (localhost [127.0.0.1]) by b2062.mx.srv.dfn.de (Postfix) with ESMTP id 39C34160060; Mon, 10 Jun 2019 22:13:40 +0200 (CEST) Received: from b2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-tub.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id FTbkKdauUqnd; Mon, 10 Jun 2019 22:13:39 +0200 (CEST) Received: from SW-IT-P-CAS4.mdc-berlin.net (mgw10-4.mdc-berlin.de [141.80.113.59]) by b2062.mx.srv.dfn.de (Postfix) with ESMTPS; Mon, 10 Jun 2019 22:13:39 +0200 (CEST) Received: from localhost (84.173.76.17) by SW-IT-P-CAS4.mdc-berlin.net (141.80.113.59) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 10 Jun 2019 22:13:38 +0200 References: <87pnod7ot4.fsf@gmail.com> <87blz55zzu.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87blz55zzu.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 10 Jun 2019 22:13:38 +0200 Message-ID: <878su9ci6l.fsf@mdc-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [84.173.76.17] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24668.002 X-TM-AS-Result: No-11.543300-8.000000-10 X-TMASE-MatchedRID: fE0JoqABJp2WFjFUJMrS40sh+mzT1Unb9r9tEcSw8jcgcEd8uJSjxML/ Q3OQ+itwcdBQzl/1Fy820VDQd/tDLMrtH6VyE5Q0ec1y1wrvN8UfkDOlTQgzYY+l4fZNSHjuNEX BQbfAPCO85g2ZLV37tr/lEO1BJo6nupdRnX5g3CbkKCFOKwAEzLnPOu2nfL2j+/ud05za4ySRj7 cnKMMR7QZsHW9WrR2gHk67DmnJRhG3vOhE1YtMejdfT4zyWoZSNGx8ao+Wz5VzNvsuxUguvTeMa vhj0dcxJs64/ww1vI5i6WJ9nAfY3JeBZPz0jQyUd6ArnIiGOBBiSlK/b4FRPRn4299x0aHBbJjk wgpFDQP8T6p7z6MdccqpVltckF9XEPsEBK7W2y3ytAowAS7YFf9PtgEFRqMQmyiLZetSf8my5/t FZu9S3Ku6xVHLhqfxwrbXMGDYqV8f/PIv6AGqLDp0DRr1w0hIXbnqaijFab9NdSlDzYnR77EabP oteqr6 X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--11.543300-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24668.002 X-TM-SNTS-SMTP: D21E5145027A63C1B3A103DB25309DCF381BAD60AC063A9D44DF7B695CC57CCA2000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Maxim, thanks for your patience in addressing my comments. I appreciate it! >> I think the double application of file-sans-extension and the >> intermediate variable name =E2=80=9Croot-directory=E2=80=9D for somethin= g that is a file >> is a little confusing, but I don=E2=80=99t have a better proposal (other= than to >> replace file-extension and file-sans-extension with a match expression). > > Done, w.r.t. using "match": > > --8<---------------cut here---------------start------------->8--- > @@ -198,10 +198,12 @@ be extracted in a temporary directory." > ;; that will be created upon decompressing it. If the filetype is not > ;; supported, return #f. > (if (compressed-file? url) > - (let ((root-directory (file-sans-extension (basename url)))) > - (if (string=3D? "tar" (file-extension root-directory)) > - (file-sans-extension root-directory) > - root-directory)) > + (match (file-sans-extension (basename url)) > + (root-directory > + (match (file-extension root-directory) > + ("tar" > + (file-sans-extension root-directory)) > + (_ root-directory)))) > (begin > (warning (G_ "Unsupported archive format (~a): \ > cannot determine package dependencies") (file-extension url)) > --8<---------------cut here---------------end--------------->8--- The first application of =E2=80=9Cmatch=E2=80=9D matches anything. What I = had in mind was really a slightly different approach, namely to split up the =E2=80=9Cu= rl=E2=80=9D string at dots and then match the resulting list of strings. Something like this: (match (string-split "hello.tar.gz" #\.) ((base "tar" (or "bz2" "gz")) base) ((base ext) base)) > I don't see much of a problem with the current design since there are > two questions being answered: > > 1) What should be the directory name of the extracted package (retrieved > from the base name of the archive). > 2) What extractor should be used (zip vs tar). > > These two questions are orthogonal, and that the same primitive get used > to answer both is an implementation, or rather, an optimization detail. > >> I wonder if we could do better and answer the question just once. > > The questions are different :-). We could optimize, but that would be at > the price of expressiveness (squash the two questions into one solving > space). Okay. I guess I=E2=80=99m too picky :) I=E2=80=99d be happy if we could move the checks to tiny named procedures, = but it=E2=80=99s probably fine the way it is. Thanks! -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Mon Jun 10 20:40:03 2019 Received: (at 24450) by debbugs.gnu.org; 11 Jun 2019 00:40:03 +0000 Received: from localhost ([127.0.0.1]:57706 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haUpW-0004Eo-NN for submit@debbugs.gnu.org; Mon, 10 Jun 2019 20:40:03 -0400 Received: from mail-pg1-f173.google.com ([209.85.215.173]:39210) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1haUpU-0004EE-2k for 24450@debbugs.gnu.org; Mon, 10 Jun 2019 20:40:00 -0400 Received: by mail-pg1-f173.google.com with SMTP id 196so5910857pgc.6 for <24450@debbugs.gnu.org>; Mon, 10 Jun 2019 17:39:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=Rbmi/2AhrBXh0VqRIOt5CvCfyEQ239im2oYnXCKK+oI=; b=JEf0tf4ucG7NJMLEGNlwvvcW32mcuLweflDYPoqnSLExhL4wisyUCTwOE19Oek7s5t VKfV2TgtRvAAQCvkUnDqHwSFR6Qwmt/qsvb21XLwSAypQcCimHMztAqHoVjB0U8mXUVN Di4gJAIq+Cbhu62zkYMe68Gt+P7slIVjdNZbII4hquiOpYu51LZXPOnwJzPTmp7WdXK5 aoqqiHArcd71wgaypZ9YGaoXNbpFFWDctsLtFEh5NvH/C5aq6vhrKLaT97LeKaT0m9c0 LNZSfYLKpHcP4ca9XKeePMge23EPBxi98euwHNn8reHPvJtWuHRXAyQMaJjVJ9MKCxcx Y5Qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=Rbmi/2AhrBXh0VqRIOt5CvCfyEQ239im2oYnXCKK+oI=; b=C3+0Qnc+FUQ6lMcHc/EVTlKd1J3YouwJtQ522FN2sgljXWKYW0LwyKctsVD0E84JpN pxUtZaBWQYdwja7iZwY8tD1JAI+P6ToIlFjD2IMzvYCZSIth+mZkogb9h9J0EbHS8Qy5 9Z29laSMxTx2RFsOekTAN5c1VFnjJyawSb5dNdVHVslwUZQ9DTNv6O/s/N0WJuIlY7OW GRwNhc6lbyiPm/2f/U45swdJaliuBzXL/yBT701TW3BwvZydDGHhYsbvywSwlCf+DwDd gQemYc3+zHG0rzYI7f/HX5YW8QuUMOJZYZZiHhN9Eb0pnBWJQtzI2jDfznGNFkKZP1GJ sgaQ== X-Gm-Message-State: APjAAAXpGfACwm+BtvxGglm6zdxc3XQpD55f5NO3v2v5hVPvfi3iTulv pbbMyNeoBPVlDOiZgTvW2HjCuBk3HkY= X-Google-Smtp-Source: APXvYqxafVpJ7O8Fj9n9o5X7UjMH9PR7otLPRDJR5jqATBW+zN8NMWEL83QQevZhX5w0bn0T+8AGcw== X-Received: by 2002:a62:1ec1:: with SMTP id e184mr77792377pfe.185.1560213593701; Mon, 10 Jun 2019 17:39:53 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id w1sm15295798pfg.51.2019.06.10.17.39.52 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 10 Jun 2019 17:39:53 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: <87pnod7ot4.fsf@gmail.com> Date: Tue, 11 Jun 2019 09:39:48 +0900 In-Reply-To: (Ricardo Wurmus's message of "Tue, 28 May 2019 13:04:44 +0200") Message-ID: <874l4x550r.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello! Ricardo Wurmus writes: > Patch number 6: > >> From fb0547ef225103c0f8355a7eccc41e0d028f6563 Mon Sep 17 00:00:00 2001 >> From: Maxim Cournoyer >> Date: Thu, 28 Mar 2019 00:26:03 -0400 >> Subject: [PATCH 6/9] import: pypi: Parse wheel METADATA instead of >> metadata.json. > >> With newer Wheel releases, there is no more metadata.json file; the META= DATA >> file should be used instead (see: https://github.com/pypa/wheel/issues/1= 95). > >> This change updates our PyPI importer so that it uses the later. > > Typo: should be =E2=80=9Clatter=E2=80=9D instead of =E2=80=9Clater=E2=80= =9D. Fixed. >> * guix/import/pypi.scm (define-module): Remove unnecessary modules and e= xport >> the PARSE-WHEEL-METADATA method. > > Please remove the indentation here. Also, please don=E2=80=99t use =E2= =80=9Cmethod=E2=80=9D > (because it=E2=80=99s not); use =E2=80=9Cprocedure=E2=80=9D instead. Done. Thanks for fixing my terminology :-). >> (parse-wheel-metadata): Add method. > > Same here. Done. >> + (define (requires-dist-header? line) >> + ;; Return #t if the given LINE is a Requires-Dist header. >> + (regexp-match? (string-match "^Requires-Dist: " line))) >> + >> + (define (requires-dist-value line) >> + (string-drop line (string-length "Requires-Dist: "))) >> + >> + (define (extra? line) >> + ;; Return #t if the given LINE is an "extra" requirement. >> + (regexp-match? (string-match "extra =3D=3D " line))) > > The use of =E2=80=9Cregexp-match?=E2=80=9D here isn=E2=80=99t strictly ne= cessary as the return > value is true-ish anyway. Done. >> + (call-with-input-file metadata >> + (lambda (port) >> + (let loop ((requirements '())) >> + (let ((line (read-line port))) >> + ;; Stop at the first 'Provides-Extra' section: the non-option= al >> + ;; requirements appear before the optional ones. >> + (if (eof-object? line) >> + (reverse (delete-duplicates requirements)) >> + (cond >> + ((and (requires-dist-header? line) (not (extra? line))) >> + (loop (cons (specification->requirement-name >> + (requires-dist-value line)) >> + requirements))) >> + (else >> + (loop requirements))))))))) >> + > > As before you can simplify the nested let and merge =E2=80=9Cif=E2=80=9D = and "cond=E2=80=9C. Oh, I get it now, I think: --8<---------------cut here---------------start------------->8--- =20 (call-with-input-file metadata (lambda (port) (let loop ((requirements '())) - (let ((line (read-line port))) - ;; Stop at the first 'Provides-Extra' section: the non-optional - ;; requirements appear before the optional ones. - (if (eof-object? line) - (reverse (delete-duplicates requirements)) - (cond - ((and (requires-dist-header? line) (not (extra? line))) - (loop (cons (specification->requirement-name - (requires-dist-value line)) - requirements))) - (else - (loop requirements))))))))) + (match (read-line port) + (line + ;; Stop at the first 'Provides-Extra' section: the non-optional + ;; requirements appear before the optional ones. + (cond + ((eof-object? line) + (reverse (delete-duplicates requirements))) + ((and (requires-dist-header? line) (not (extra? line))) + (loop (cons (specification->requirement-name + (requires-dist-value line)) + requirements))) + (else + (loop requirements))))))))) =20 (define (guess-requirements source-url wheel-url archive) "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a list --8<---------------cut here---------------end--------------->8--- >> (define (read-wheel-metadata wheel-archive) >> ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the pack= age's >> - ;; requirements. >> + ;; requirements, or #f if the metadata file contained therein could= n't be >> + ;; extracted. >> (let* ((dirname (wheel-url->extracted-directory wheel-url)) >> - (json-file (string-append dirname "/metadata.json"))) >> - (and (zero? (system* "unzip" "-q" wheel-archive json-file)) >> - (dynamic-wind >> - (const #t) >> - (lambda () >> - (call-with-input-file json-file >> - (lambda (port) >> - (let* ((metadata (json->scm port)) >> - (run_requires (hash-ref metadata "run_require= s")) >> - (requirements (if run_requires >> - (hash-ref (list-ref run_req= uires 0) >> - "requires") >> - '()))) >> - (map specification->requirement-name requirements)= )))) >> - (lambda () >> - (delete-file json-file) >> - (rmdir dirname)))))) >> + (metadata (string-append dirname "/METADATA"))) >> + (call-with-temporary-directory >> + (lambda (dir) >> + (if (zero? (system* "unzip" "-q" wheel-archive "-d" dir metada= ta)) >> + (parse-wheel-metadata (string-append dir "/" metadata)) >> + (begin >> + (warning >> + (G_ "Failed to extract file: ~a from wheel.~%") metadat= a) >> + #f)))))) > > The old approach took care of removing the unpacked archive no matter > what happened. The new code doesn=E2=80=99t do that. The temporary directory where the archive is unpacked should be cleared when leaving upon leaving its scope; the docstring of "call-with-temporary-directory" says: "Call PROC with a name of a temporary directory; close the directory and delete it when leaving the dynamic extent of this call." >> --- a/tests/pypi.scm >> +++ b/tests/pypi.scm > > Thanks for the tests! :-) Maxim From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 11 07:56:58 2019 Received: (at 24450) by debbugs.gnu.org; 11 Jun 2019 11:56:59 +0000 Received: from localhost ([127.0.0.1]:58114 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hafOc-0008V6-Jk for submit@debbugs.gnu.org; Tue, 11 Jun 2019 07:56:58 -0400 Received: from b2062.mx.srv.dfn.de ([194.95.234.172]:60955) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hafOa-0008Uw-1K for 24450@debbugs.gnu.org; Tue, 11 Jun 2019 07:56:57 -0400 Received: from localhost (localhost [127.0.0.1]) by b2062.mx.srv.dfn.de (Postfix) with ESMTP id BE591160045; Tue, 11 Jun 2019 13:56:47 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mdc-berlin.de; h= content-transfer-encoding:content-type:content-type:mime-version :message-id:date:date:in-reply-to:subject:subject:from:from :user-agent:references:received:received:received; s=mdc; t= 1560254207; x=1562068608; bh=i/eL4dXwmRyHmabDraB0jGRX8SCLbFLaYln kn7y6Ka0=; b=gW1IM8TcF01sPCAC6+1OA4EXmBWy7tmUGtY/Kj2svTBvijoDS2V /fZYjyK0CJwats02KenyU+8xOnI6BGat0UtfKUOEgrkE6HSuijASIOxUMbIRVEzN QaKjvxatgtHLWX0IMTzP9/1WcxmZtc+/qvSUKnnnBzvlN6i508zLMgiE= Received: from b2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-tub.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id G2okmTiXF-w7; Tue, 11 Jun 2019 13:56:47 +0200 (CEST) Received: from SW-IT-P-CAS4.mdc-berlin.net (mgw10-4.mdc-berlin.de [141.80.113.59]) by b2062.mx.srv.dfn.de (Postfix) with ESMTPS; Tue, 11 Jun 2019 13:56:47 +0200 (CEST) Received: from localhost (141.80.247.204) by SW-IT-P-CAS4.mdc-berlin.net (141.80.113.59) with Microsoft SMTP Server (TLS) id 14.3.439.0; Tue, 11 Jun 2019 13:56:46 +0200 References: <87pnod7ot4.fsf@gmail.com> <874l4x550r.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <874l4x550r.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Tue, 11 Jun 2019 13:56:46 +0200 Message-ID: <87v9xc9vy9.fsf@mdc-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [141.80.247.204] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24668.007 X-TM-AS-Result: No-2.922400-8.000000-10 X-TMASE-MatchedRID: HXSqh3WYKfs4HKI/yaqRm4S/TV9k6ppAm19A9Bm2BPNrRM6wvXgDaRQt LIUOz+UC/05c43DMzih5LAkNM0VvMRuf8+qwj2wBYJF0zuzVwRRU+YqtAU+F2S5hVkDKEqeBJDl ZUZrKI635NLTowdvTKPO1tnuJuTiIcKE8cHrXBCP0hv/rD7WVZNs4MOUU/wa2+/ud05za4yS+t6 W5tO2WXucZzTixRiV9nauO56UjN5qoOZlmXWpufMzSKGx9g8xhfS0Ip2eEHnzUHQeTVDUrItRnE QCUU+jz9xS3mVzWUuCMx6OO8+QGvu6gVOTN7T76m9YxW+DEw9PUNiX/YHfve9M8froAaeOKKNX1 yUJf9rU4PGx8T0iHYxKEe2C5sh457wXtU5PLR+k0uh+qc8w9xZF+e/T80ZLUgjux1OFeoIseJIR GDJVM09l+7FPOoW8uVlxr1FJij9s= X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--2.922400-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24668.007 X-TM-SNTS-SMTP: 2919631E99F02CC1CF10B4BF961F93E25329F3430A3C62E06FF471051C0B49AA2000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Maxim, >>> + (call-with-input-file metadata >>> + (lambda (port) >>> + (let loop ((requirements '())) >>> + (let ((line (read-line port))) >>> + ;; Stop at the first 'Provides-Extra' section: the non-optio= nal >>> + ;; requirements appear before the optional ones. >>> + (if (eof-object? line) >>> + (reverse (delete-duplicates requirements)) >>> + (cond >>> + ((and (requires-dist-header? line) (not (extra? line))) >>> + (loop (cons (specification->requirement-name >>> + (requires-dist-value line)) >>> + requirements))) >>> + (else >>> + (loop requirements))))))))) >>> + >> >> As before you can simplify the nested let and merge =E2=80=9Cif=E2=80=9D= and "cond=E2=80=9C. > > Oh, I get it now, I think: > > --8<---------------cut here---------------start------------->8--- > > (call-with-input-file metadata > (lambda (port) > (let loop ((requirements '())) > - (let ((line (read-line port))) > - ;; Stop at the first 'Provides-Extra' section: the non-optional > - ;; requirements appear before the optional ones. > - (if (eof-object? line) > - (reverse (delete-duplicates requirements)) > - (cond > - ((and (requires-dist-header? line) (not (extra? line))) > - (loop (cons (specification->requirement-name > - (requires-dist-value line)) > - requirements))) > - (else > - (loop requirements))))))))) > + (match (read-line port) > + (line > + ;; Stop at the first 'Provides-Extra' section: the non-option= al > + ;; requirements appear before the optional ones. > + (cond > + ((eof-object? line) > + (reverse (delete-duplicates requirements))) > + ((and (requires-dist-header? line) (not (extra? line))) > + (loop (cons (specification->requirement-name > + (requires-dist-value line)) > + requirements))) > + (else > + (loop requirements))))))))) > > (define (guess-requirements source-url wheel-url archive) > "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a li= st > --8<---------------cut here---------------end--------------->8--- Not quite. Your =E2=80=98match=E2=80=99 expression here doesn=E2=80=99t do= anything that a =E2=80=98let=E2=80=99 wouldn=E2=80=99t have done. It really just binds the= return value of (read-line port) to =E2=80=98line=E2=80=99; that=E2=80=99s the same as (let= ((line (read-line port))) =E2=80=A6). I gave a match example using predicate matchers in a previous reply. In any case, using =E2=80=98cond=E2=80=99 inside of a let would be just fine. = If you wanted to go with =E2=80=98match=E2=80=99, though, you=E2=80=99d replace th= e =E2=80=98cond=E2=80=99. -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 11 23:01:05 2019 Received: (at 24450) by debbugs.gnu.org; 12 Jun 2019 03:01:05 +0000 Received: from localhost ([127.0.0.1]:60478 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hatVZ-0001Fc-43 for submit@debbugs.gnu.org; Tue, 11 Jun 2019 23:01:05 -0400 Received: from mail-pg1-f182.google.com ([209.85.215.182]:36817) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hatVV-0001F1-Ve for 24450@debbugs.gnu.org; Tue, 11 Jun 2019 23:01:03 -0400 Received: by mail-pg1-f182.google.com with SMTP id f21so2058142pgi.3 for <24450@debbugs.gnu.org>; Tue, 11 Jun 2019 20:01:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=vbPnAmJhy5HJ7tNrbIqpep7edceNaI/hdtYqxuxcKCM=; b=oenxhHG0tYeiYyXFVEKRfJRYaYxtInC74qMjv65BwhVuCzRpPofSJkw8LWKTMPNmXp qN3zxJrywNpjAPoVgoLKEHmU39joo8pKj0Hh/h9Lmf2+OuklH8POnQSha5yfJ51yaEYE MENf9T4o6sou7zKa+2DBlYuRkkL2aqTqP+7hYPvNltiib5MzCYuCD1x0PlZP1T+hhimA gkHJtoWNpS6YwOTgszCug0yySv9XnS+1lZ05MBXaGDAW+2HcaLbC6cERStEvoGjkSihY MriyaLC73wUDkfxbreZDtz+9oADlqgDCBW9fWzAQElD/lyeyS/N144n75dtwOsX12tVL 5qmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=vbPnAmJhy5HJ7tNrbIqpep7edceNaI/hdtYqxuxcKCM=; b=I3GSiRct6K0DMdfQWClml26MQ0yKRUyfVlOm/eK2lwuazSMVLzshs4VYU1/VuVl4uE /m+FIfc7QFSP0YYWUbTDWAbMwOAwUFthkMIZCM3ebqLtjkeGxfzOpFG4aOOQ3MZ3YhAw kdvNOgp9tJWi+xtBPBa/pzeklTMYAs6VXrV9DezCRgpOER/Ft8LXPjxQCEmQ4PpAcs+O UaebyPC11On8Dl696WLU2HXoWIaIUgV2NVWtWBe4NIDpi++pleXDB6ryn2wYU0BqeX04 qjGOLJrw+AbCLdRnNNyn01lNbOZ3Irxhu37WaVAuaqUu3/c095q15dJeJrOU0rjgM/MS AVxA== X-Gm-Message-State: APjAAAUjaKsCn2NQngWrIZ+zhS0FzrxnXp7SoVQMQUYgzV6FJ+z89nqS lKSGPl5SKxKsmbdr9vuA6k3AptW2tvM= X-Google-Smtp-Source: APXvYqzeHwkSOyyfDgBv8RJWD141fNrx2dW8hv8plCoB0eP0HHwpki1EPdczG4igpa/RSCyXv7o3Uw== X-Received: by 2002:aa7:80ce:: with SMTP id a14mr52391236pfn.249.1560308455136; Tue, 11 Jun 2019 20:00:55 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id x24sm3778529pjq.27.2019.06.11.20.00.53 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Tue, 11 Jun 2019 20:00:54 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. Date: Wed, 12 Jun 2019 12:00:50 +0900 Message-ID: <87imtb33tp.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ricardo Wurmus writes: > Next up: Seven of Nine, tertiary adjunct of unimatrix zero one: Ehe! I had to look up the reference; I'm not much of a Star Trek fan obviously :-P. >> From 37e499d5d5d5f690aa0a065c730e13f6a31dd30d Mon Sep 17 00:00:00 2001 >> From: Maxim Cournoyer >> Date: Thu, 28 Mar 2019 23:12:26 -0400 >> Subject: [PATCH 7/9] import: pypi: Include optional test inputs as >> native-inputs. >> >> * guix/import/pypi.scm (maybe-inputs): Add INPUT-TYPE argument, and use = it. >> (test-section?): New predicate. >> (parse-requires.txt): Collect the optional test inputs, and return them = as the >> second element of the returned list. > > AFAICT parse-requires.txt now returns a list of pairs, but used to > return a plain list before. Is this correct? Right, a list of two lists to be technically correct. >> (define (parse-requires.txt requires.txt) >> - "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of >> -requirement names." >> - ;; This is a very incomplete parser, which job is to select the non-o= ptional >> - ;; dependencies and strip them out of any version information. >> + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a pair of= requirements. >> + >> +The first element of the pair contains the required dependencies while = the >> +second the optional test dependencies. Note that currently, optional, >> +non-test dependencies are omitted since these can be difficult or expen= sive to >> +satisfy." >> + >> + ;; This is a very incomplete parser, which job is to read in the requ= irement >> + ;; specification lines, and strip them out of any version information. >> ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) >> ;; library and the requirements grammar defined by PEP-0508 >> ;; (https://www.python.org/dev/peps/pep-0508/). > > Does it really return a pair? Or a list of pairs? Or is it a > two-element list of lists? The latter! I've fixed the docstring accordingly. >> (call-with-input-file requires.txt >> (lambda (port) >> - (let loop ((result '())) >> + (let loop ((required-deps '()) >> + (test-deps '()) >> + (inside-test-section? #f) >> + (optional? #f)) >> (let ((line (read-line port))) >> - ;; Stop when a section is encountered, as sections contains o= ptional >> - ;; (extra) requirements. Non-optional requirements must appe= ar >> - ;; before any section is defined. >> - (if (or (eof-object? line) (section-header? line)) >> + (if (eof-object? line) >> ;; Duplicates can occur, since the same requirement can be >> ;; listed multiple times with different conditional marke= rs, e.g. >> ;; pytest >=3D 3 ; python_version >=3D "3.3" >> ;; pytest < 3 ; python_version < "3.3" >> - (reverse (delete-duplicates result)) >> + (map (compose reverse delete-duplicates) >> + (list required-deps test-deps)) > > Looks like a list of lists to me. =E2=80=9Cdelete-duplicates=E2=80=9D no= w won=E2=80=99t delete > a name that is in both =E2=80=9Crequired-deps=E2=80=9D as well as in =E2= =80=9Ctest-deps=E2=80=9D. Is > this acceptable? It is acceptable, as this corner case cannot exist given the current code (a requirement can exist in either required-deps or test-deps, but never in both). It also doesn't make sense that a run time requirement would also be listed as a test requirement, so that corner case is not likely to exist in the future either. > Personally, I=E2=80=99m not a fan of using data structures for returning > multiple values, because we can simply return multiple values. I thought the Guile supported multiple values return value would be great here as well, but I've found that for this specific case here, a list of lists worked better, since the two lists contain requirements to be processed the same, which "map" can readily do (i.e. less ceremony is required). > Or we could have more than just strings. The meaning of these strings > is provided by the bin into which they are thrown =E2=80=94 either > =E2=80=9Crequired-deps=E2=80=9D or =E2=80=9Ctest-deps=E2=80=9D. It could= be an option to collect tagged > values instead and have the caller deal with filtering. Sounds neat, but I'd rather punt on this one for now. >> (define (parse-wheel-metadata metadata) >> - "Given METADATA, a Wheel metadata file, return a list of requirement = names." >> + "Given METADATA, a Wheel metadata file, return a pair of requirements. >> + >> +The first element of the pair contains the required dependencies while = the second the optional >> +test dependencies. Note that currently, optional, non-test dependencie= s are >> +omitted since these can be difficult or expensive to satisfy." >> ;; METADATA is a RFC-2822-like, header based file. > > This sounds like this is going to duplicate the previous procedures. Instead of duplicating the docstring, I'm now referring to that of PARSE-REQUIRES.TXT for PARSE-WHEEL-METADATA. The two procedures share the same interface, but implement a different parser, which consist of mostly a loop + conditional branches. IMHO, it's not worth, or even, desirable to try to merge those two parsers into one; I prefer to keep the logic of two distinct parsers separate. >> (define (requires-dist-header? line) >> ;; Return #t if the given LINE is a Requires-Dist header. >> - (regexp-match? (string-match "^Requires-Dist: " line))) >> + (string-match "^Requires-Dist: " line)) >> >> (define (requires-dist-value line) >> (string-drop line (string-length "Requires-Dist: "))) >> >> (define (extra? line) >> ;; Return #t if the given LINE is an "extra" requirement. >> - (regexp-match? (string-match "extra =3D=3D " line))) >> + (string-match "extra =3D=3D '(.*)'" line)) > > These hunks should be part of the previous patch where they were > introduced. (See my comments there about =E2=80=9Cregexp-match?=E2=80=9D= .) Done. >> + (define (test-requirement? line) >> + (let ((extra-label (match:substring (extra? line) 1))) >> + (and extra-label (test-section? extra-label)))) > > You can use =E2=80=9Cand=3D>=E2=80=9D instead of binding a name: > > (and=3D> (match:substring (extra? line) 1) test-section?) Neat! I still don't have the reflex to use "and=3D>", thanks for reminding me about it! >> (call-with-input-file metadata >> (lambda (port) >> - (let loop ((requirements '())) >> + (let loop ((required-deps '()) >> + (test-deps '())) >> (let ((line (read-line port))) >> - ;; Stop at the first 'Provides-Extra' section: the non-option= al >> - ;; requirements appear before the optional ones. >> (if (eof-object? line) >> - (reverse (delete-duplicates requirements)) >> + (map (compose reverse delete-duplicates) >> + (list required-deps test-deps)) >> (cond >> ((and (requires-dist-header? line) (not (extra? line))) >> (loop (cons (specification->requirement-name >> (requires-dist-value line)) >> - requirements))) >> + required-deps) >> + test-deps)) >> + ((and (requires-dist-header? line) (test-requirement? li= ne)) >> + (loop required-deps >> + (cons (specification->requirement-name (requires-= dist-value line)) >> + test-deps))) >> (else >> - (loop requirements))))))))) >> + (loop required-deps test-deps))))))))) ;skip line > > Could you refactor this so that the other parser can be reused? If not, > the same comments about =E2=80=9Cif=E2=80=9D and =E2=80=9Ccond=E2=80=9D a= nd the use of pairs/lists > instead of =E2=80=9Cvalues=E2=80=9D applies here. Done w.r.t. using only "cond" rather than "if" + "cond". As explained above, the docstring's body now refer to the documentation of PARSE-REQUIRES.TXT; otherwise I prefer to keep the parsers separate. >> (define (guess-requirements source-url wheel-url archive) >> - "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a l= ist >> + "Given SOURCE-URL, WHEEL-URL and an ARCHIVE of the package, return a = list >> of the required packages specified in the requirements.txt file. ARCHI= VE will >> be extracted in a temporary directory." >> >> @@ -244,7 +289,10 @@ cannot determine package dependencies") (file-exten= sion url)) >> (metadata (string-append dirname "/METADATA"))) >> (call-with-temporary-directory >> (lambda (dir) >> - (if (zero? (system* "unzip" "-q" wheel-archive "-d" dir metada= ta)) >> + (if (zero? >> + (parameterize ((current-error-port (%make-void-port "rw+"= )) >> + (current-output-port (%make-void-port "rw+= "))) >> + (system* "unzip" wheel-archive "-d" dir metadata))) >> (parse-wheel-metadata (string-append dir "/" metadata)) >> (begin >> (warning >> @@ -283,32 +331,41 @@ cannot determine package dependencies") (file-exte= nsion url)) >> (warning >> (G_ "Failed to extract file: ~a from source.~%") >> requires.txt) >> - '()))))) >> - '()))) >> + (list '() '())))))) >> + (list '() '())))) > > I would like to see cosmetic changes like these three hunks in separate > commits. Done for the first two hunks; the last one isn't cosmetic; it changes the default return from an empty list to a list of two empty lists. >> (define (compute-inputs source-url wheel-url archive) >> - "Given the SOURCE-URL of an already downloaded ARCHIVE, return a list= of >> -name/variable pairs describing the required inputs of this package. Al= so >> + "Given the SOURCE-URL and WHEEL-URL of an already downloaded ARCHIVE,= return >> +a pair of lists, each consisting of a list of name/variable pairs, for = the >> +propagated inputs and the native inputs, respectively. Also >> return the unaltered list of upstream dependency names." >> - (let ((dependencies >> - (remove (cut string=3D? "argparse" <>) >> - (guess-requirements source-url wheel-url archive)))) >> - (values (sort >> - (map (lambda (input) >> - (let ((guix-name (python->package-name input))) >> - (list guix-name (list 'unquote (string->symbol gu= ix-name))))) >> - dependencies) >> - (lambda args >> - (match args >> - (((a _ ...) (b _ ...)) >> - (string-ci> - dependencies))) >> + >> + (define (strip-argparse deps) >> + (remove (cut string=3D? "argparse" <>) deps)) >> + >> + (define (requirement->package-name/sort deps) >> + (sort >> + (map (lambda (input) >> + (let ((guix-name (python->package-name input))) >> + (list guix-name (list 'unquote (string->symbol guix-name)= )))) >> + deps) >> + (lambda args >> + (match args >> + (((a _ ...) (b _ ...)) >> + (string-ci> + >> + (define process-requirements >> + (compose requirement->package-name/sort strip-argparse)) >> + >> + (let ((dependencies (guess-requirements source-url wheel-url archive)= )) >> + (values (map process-requirements dependencies) >> + (concatenate dependencies)))) > > Giving names to these processing steps is fine and improves clarity, but > I=E2=80=99m not so comfortable about returning ad-hoc pairs *and* multiple > values (like above). Using "values" is required by the API of the recursive importer. >> + (match guix-dependencies >> + ((required-inputs test-inputs) >> + (values >> + `(package >> + (name ,(python->package-name name)) >> + (version ,version) >> + (source (origin >> + (method url-fetch) >> + ;; Sometimes 'pypi-uri' doesn't quite work= due to mixed >> + ;; cases in NAME, for instance, as is the = case with >> + ;; "uwsgi". In that case, fall back to a = full URL. >> + (uri (pypi-uri ,(string-downcase name) ver= sion)) >> + (sha256 >> + (base32 >> + ,(guix-hash-url temp))))) >> + (build-system python-build-system) >> + ,@(maybe-inputs required-inputs 'propagated-inputs) >> + ,@(maybe-inputs test-inputs 'native-inputs) >> + (home-page ,home-page) >> + (synopsis ,synopsis) >> + (description ,description) >> + (license ,(license->symbol license))) >> + ;; Flatten the nested lists and return the upstream >> + ;; dependencies. >> + upstream-dependencies)))))))) > > I don=E2=80=99t see anything being flattened here? Good catch! Seems a remnant of something that is now gone :-). Thanks for the review. Maxim From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 12 02:39:23 2019 Received: (at 24450) by debbugs.gnu.org; 12 Jun 2019 06:39:23 +0000 Received: from localhost ([127.0.0.1]:60543 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hawup-0007hr-5s for submit@debbugs.gnu.org; Wed, 12 Jun 2019 02:39:23 -0400 Received: from a2062.mx.srv.dfn.de ([194.95.232.172]:47169) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hawum-0007hf-FK for 24450@debbugs.gnu.org; Wed, 12 Jun 2019 02:39:21 -0400 Received: from localhost (localhost [127.0.0.1]) by a2062.mx.srv.dfn.de (Postfix) with ESMTP id 791A1A003C; Wed, 12 Jun 2019 08:39:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mdc-berlin.de; h= content-transfer-encoding:content-type:content-type:mime-version :message-id:date:date:in-reply-to:subject:subject:from:from :user-agent:references:received:received:received; s=mdc; t= 1560321554; x=1562135955; bh=LvuRhPGRsh09StmIxAeIP4onagfqILM6eSm uzQHWKBw=; b=DqhNV5G/hPbdF8M08sJE8XUmk/I5n5Lqjjr4FQa4HRd65JZaczt oH+lIy1IoZuOQiWg5lHZSPa6c9uqe5d94YSyB6eoMa0uK/yuugcxpelSB3AWV4vX tgsqVjPxi0O4nrEyid+2gsJr74Cbm/au0KlAfqz9XYcXrAeA0ML4wZlk= Received: from a2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-han.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id SskZOhzQHVVt; Wed, 12 Jun 2019 08:39:14 +0200 (CEST) Received: from SW-IT-P-CAS3.mdc-berlin.net (mgw10-3.mdc-berlin.de [141.80.113.58]) by a2062.mx.srv.dfn.de (Postfix) with ESMTPS; Wed, 12 Jun 2019 08:39:13 +0200 (CEST) Received: from localhost (84.173.75.197) by SW-IT-P-CAS3.mdc-berlin.net (141.80.113.58) with Microsoft SMTP Server (TLS) id 14.3.439.0; Wed, 12 Jun 2019 08:39:13 +0200 References: <87imtb33tp.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87imtb33tp.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Wed, 12 Jun 2019 08:39:12 +0200 Message-ID: <87imtb9ujz.fsf@mdc-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [84.173.75.197] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24670.005 X-TM-AS-Result: No-12.154300-8.000000-10 X-TMASE-MatchedRID: X4bcv0S75Kk4HKI/yaqRm4S/TV9k6ppAm19A9Bm2BPNrRM6wvXgDaRQt LIUOz+UCgLSfrGCOKznTdBk0FPP/O6/fa3S3RxMoHPYwOJi6PLkGchEhVwJY34VnMfYE2VRDzWF Algkkbn06lGcxRgoBgffYx5EY7aRj/m+YyVVvvBgmFvfzyF2ijytW5wfuEjEybJq68ZV0A7rdoR NmoLpVBEPLEeKdMFLoUprcJdSjBlPFBmMwxidC2czSKGx9g8xhNACXtweanwbvRQsknAWmGmlzJ grTfsrkVx/fMeJDT14XHNc0tKkmkPGU4m5A0eV9/03t7eXCTBtpv5k9epk1ApuQSXVIw/hVdodz ZVKMVFyh9bMmqe83MeF6spR+t5u7Nyl1nd9CIt2DGx/OQ1GV8rHlqZYrZqdI+gtHj7OwNO2BSJy 8ngwKGcNwwTHu5qtjcL8kB9j6tW/Ly6+x/Hf9XtiSPO7Ga6A3BgTlRvbBsxg= X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--12.154300-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24670.005 X-TM-SNTS-SMTP: 25B7827EF38A537817B70A8B9087FE2EE6CACF9A8853F3BC045FAE8B84C62EE12000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Maxim, >>> (call-with-input-file requires.txt >>> (lambda (port) >>> - (let loop ((result '())) >>> + (let loop ((required-deps '()) >>> + (test-deps '()) >>> + (inside-test-section? #f) >>> + (optional? #f)) >>> (let ((line (read-line port))) >>> - ;; Stop when a section is encountered, as sections contains = optional >>> - ;; (extra) requirements. Non-optional requirements must app= ear >>> - ;; before any section is defined. >>> - (if (or (eof-object? line) (section-header? line)) >>> + (if (eof-object? line) >>> ;; Duplicates can occur, since the same requirement can = be >>> ;; listed multiple times with different conditional mark= ers, e.g. >>> ;; pytest >=3D 3 ; python_version >=3D "3.3" >>> ;; pytest < 3 ; python_version < "3.3" >>> - (reverse (delete-duplicates result)) >>> + (map (compose reverse delete-duplicates) >>> + (list required-deps test-deps)) >> >> Looks like a list of lists to me. =E2=80=9Cdelete-duplicates=E2=80=9D n= ow won=E2=80=99t delete >> a name that is in both =E2=80=9Crequired-deps=E2=80=9D as well as in =E2= =80=9Ctest-deps=E2=80=9D. Is >> this acceptable? > > It is acceptable, as this corner case cannot exist given the current > code (a requirement can exist in either required-deps or test-deps, but > never in both). It also doesn't make sense that a run time requirement > would also be listed as a test requirement, so that corner case is not > likely to exist in the future either. I mentioned it because I believe I=E2=80=99ve seen this in the past where t= he importer would return some of the same inputs as both regular inputs and test dependencies. >> Personally, I=E2=80=99m not a fan of using data structures for returning >> multiple values, because we can simply return multiple values. > > I thought the Guile supported multiple values return value would be > great here as well, but I've found that for this specific case here, a > list of lists worked better, since the two lists contain requirements to > be processed the same, which "map" can readily do (i.e. less ceremony is > required). =E2=80=9Cmap=E2=80=9D can also operate on more than one list at a time: (call-with-values (lambda () (values (list 1 2 3) (list 9 8 7))) (lambda (a b) (map + a b))) =3D> (10 10 10) Of course, it would be simpler to just use a single list of tagged items. -- Ricardo From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 16 01:11:09 2019 Received: (at 24450) by debbugs.gnu.org; 16 Jun 2019 05:11:09 +0000 Received: from localhost ([127.0.0.1]:39544 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcNRc-00022x-RN for submit@debbugs.gnu.org; Sun, 16 Jun 2019 01:11:09 -0400 Received: from mail-pl1-f182.google.com ([209.85.214.182]:37357) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcNRZ-00022N-Ks for 24450@debbugs.gnu.org; Sun, 16 Jun 2019 01:11:06 -0400 Received: by mail-pl1-f182.google.com with SMTP id bh12so2725221plb.4 for <24450@debbugs.gnu.org>; Sat, 15 Jun 2019 22:11:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=HDtv7niB3TvgJDFFbv4SwXQns2hs5SJoFstmBQ1VqH8=; b=kZ4f5QHn70OTsuTbcKq880XiSkER1ebpFcMV4+JhomvkjNwjJ3M4UToQIDg1KIZ6CT PNLzdk0p5cuUOtE4gw1TUIhe5ysXbFzf2mSzcQ0HqCX2b8DnatDZxdMCFDWln9jJRUrA f6sIJeH3anN4XgZLmDoKzXDx910UAU8R0qoEfV5YYe0Eh4/qRngMFRAphYxGvFXomj28 DKuF/C5J7bPxhGh3wkfHVjkU9fBfDfhh8GMJnQBDSENdv9znLXy0LbvvcgPFmQjypnPD sjKNDH1OqEpUcLv+TfKMtldR75iSURlzgo1bgyZqL8nE5URCT3YX5sH9Fz3n3MQIzxmw /Dug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=HDtv7niB3TvgJDFFbv4SwXQns2hs5SJoFstmBQ1VqH8=; b=h3sG0yMDn97kpNI3DjGM7uBuehDmCGtdxob9J8PgMey33VXskRD8cuJRRH/q81EbLx y9DY+lQ4hqRSEbhX1/YXEEd0XWTINvyQXu6qZ0+BLzDQ0BhdqIC/1YCGa0UeMAtPDK8c haTfM9+fIf2DmEoSH349N0nC0oNj3IL3hfDxmHi8P8RdF9jxCVnaQhc7S8Xfr0llFOG9 nTZ3JMnLcwYnY9dS4FLifQM0W2DHROxRJDl6eiMS4KMPeB/DubTxNPglkeRNw2LGVeoj MPEZfp4LDAcwRtVrRMg46u0Eqj9rJY2upAo4l8OdEk0LvPhJMdquT10VzokcWm8Km5OT Sg0A== X-Gm-Message-State: APjAAAV2aCtcaonW/fMRhk2X6GzPh7YnXcN9eUR4BNynomkiv/nX6UdC C6EcvlmupsSwt8MtF7EoL7IawYnSkpg= X-Google-Smtp-Source: APXvYqwbLjm8JAQlXwD34Q1gI9RZ0n7fpRKMsxSUTanU8SjaTfdKZ0wLi2CO+lxoklSDHp/OA8jmWg== X-Received: by 2002:a17:902:b695:: with SMTP id c21mr48867722pls.160.1560661859402; Sat, 15 Jun 2019 22:10:59 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id 85sm8532698pgb.52.2019.06.15.22.10.57 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 15 Jun 2019 22:10:58 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: <87pnod7ot4.fsf@gmail.com> Date: Sun, 16 Jun 2019 14:10:52 +0900 In-Reply-To: (Ricardo Wurmus's message of "Tue, 28 May 2019 16:48:57 +0200") Message-ID: <878su22jz7.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello Ricardo! Ricardo Wurmus writes: >> From cfde6e09f8f8c692fe252d76ed27e8c50a9e5377 Mon Sep 17 00:00:00 2001 >> From: Maxim Cournoyer >> Date: Sat, 30 Mar 2019 23:13:26 -0400 >> Subject: [PATCH 8/9] import: pypi: Scan source archive to find requires.= txt >> file. > >> * guix/import/pypi.scm (use-modules): Use invoke from (guix build utils). >> (guess-requirements)[archive-root-directory]: Remove procedure. > > Oh, I guess I reviewed this procedure in vain :( > > Please modify the commits so that added procedures are not removed in > later commits. This is easier on the reviewer and makes for a clearer > commit history. Indeed; I'll be more careful about this is the future; sorry! I've squashed this commit along with the one enabling more archive types support, as this is where the modified (and later removed) procedure originated. >> (define (guess-requirements-from-source) >> ;; Return the package's requirements by guessing them from the sour= ce. >> - (let ((dirname (archive-root-directory source-url)) >> - (extension (file-extension source-url))) >> - (if (string? dirname) >> - (call-with-temporary-directory >> - (lambda (dir) >> - (let* ((pypi-name (string-take dirname (string-rindex dirn= ame #\-))) >> - (requires.txt (string-append dirname "/" pypi-name >> - ".egg-info" "/requires= .txt")) >> - (exit-code >> - (parameterize ((current-error-port (%make-void-por= t "rw+")) >> - (current-output-port (%make-void-po= rt "rw+"))) >> - (if (string=3D? "zip" extension) >> - (system* "unzip" archive "-d" dir requires.t= xt) >> - (system* "tar" "xf" archive "-C" dir require= s.txt))))) >> - (if (zero? exit-code) >> - (parse-requires.txt (string-append dir "/" requires.= txt)) >> - (begin >> - (warning >> - (G_ "Failed to extract file: ~a from source.~%") >> - requires.txt) >> - (list '() '())))))) >> + (if (compressed-file? source-url) >> + (call-with-temporary-directory >> + (lambda (dir) >> + (parameterize ((current-error-port (%make-void-port "rw+")) >> + (current-output-port (%make-void-port "rw+"))) >> + (if (string=3D? "zip" (file-extension source-url)) >> + (invoke "unzip" archive "-d" dir) >> + (invoke "tar" "xf" archive "-C" dir))) >> + (let ((requires.txt-files >> + (find-files dir (lambda (abs-file-name _) >> + (string-match "\\.egg-info/requires.txt$" >> + abs-file-name))))) >> + (if (> (length requires.txt-files) 0) > > Let=E2=80=99s work on the empty list directly. Here =E2=80=9Cmatch=E2=80= =9D would be better. Done, like this: --8<---------------cut here---------------start------------->8--- - (if (> (length requires.txt-files) 0) - (parse-requires.txt (first requires.txt-files)) - (begin (warning (G_ "Cannot guess requirements from sourc= e archive:\ + (match requires.txt-files + (() + (warning (G_ "Cannot guess requirements from source archiv= e:\ no requires.txt file found.~%")) - '()))))) + '()) + (else (parse-requires.txt (first requires.txt-files))))))) --8<---------------cut here---------------end--------------->8--- >> + (begin >> + (parse-requires.txt (first requires.txt-files))) > > No need for =E2=80=9Cbegin=E2=80=9D here. Fixed. >> + (begin (warning (G_ "Cannot guess requirements from so= urce archive:\ >> + no requires.txt file found.~%")) >> + (list '() '())))))) > > I know that this is from an earlier commit, but I don=E2=80=99t like the = look of > =E2=80=9C(list '() '())=E2=80=9D at all :) > >> + (begin >> + (warning (G_ "Unsupported archive format; \ >> +cannot determine package dependencies from source archive: ~a~%") >> + (basename source-url)) >> (list '() '())))) > > Same here. Certainly there=E2=80=99s a better return value. This might look ugly, but I can't think of a better return value, since using anything else would mean having to introduce extra logic in the callers, while it is now a correct value that needs no special case. Maxim From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 16 01:53:49 2019 Received: (at 24450) by debbugs.gnu.org; 16 Jun 2019 05:53:49 +0000 Received: from localhost ([127.0.0.1]:39555 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcO6v-0003O3-CW for submit@debbugs.gnu.org; Sun, 16 Jun 2019 01:53:49 -0400 Received: from mail-pl1-f170.google.com ([209.85.214.170]:35261) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcO6r-0003Nj-01 for 24450@debbugs.gnu.org; Sun, 16 Jun 2019 01:53:45 -0400 Received: by mail-pl1-f170.google.com with SMTP id p1so2746345plo.2 for <24450@debbugs.gnu.org>; Sat, 15 Jun 2019 22:53:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=8v9GNAWx1N6NbU4zvb/wHRhFa2Day5rJ+Nz151jBEuA=; b=QvUXbVIPyB8b3kJJ2N4eVjQI4k3HxGXaaG2V/iqClnHA0AnxNpBg90whkzoDf3FM0c lAjkP4RAzoVpwPD3tqDh4DRQjRnLGv75a7Tf0pvPDkrzvK8bUeLg31AYBfZ6F8osAsr1 giVnbdacdg0y3iuY8sS0u6lnxxThJ+NeUhYR7zURG/2iZ3Ha2lFxURrF57d16895N+31 Ku+Gqtw53lKr76A5CfD+eHlt3s6z70ySrNcIZLdwKXuOD9oU4zIpFcukOehkl7n7Qb6P 0cTOLeY7XYmCvpLfJFb0qVT5yhRgbRtWJng9tt0nUG97ckSpFfdqHjRanV3p1ch0uvJQ vnCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=8v9GNAWx1N6NbU4zvb/wHRhFa2Day5rJ+Nz151jBEuA=; b=ZIReoOLzo3PZ+JDYuWkq0hQIHkVb0q8Wc1G5PoM0vWhtFJ+7ZPKnJnGb8nOd5tZuEs 51LtaR/WMh5q/qlhJADWoGJuOv/0y2ual382bBsqJwhnoiUecoLLdwcXnEiFChM+I290 o71S15f9UJ6nIdT/pJIHKhc5g1l1ce5DF4P07Eekjq+I7OU3rJBwV5RczI5+fViWEDwY dGpVn4S6gMNaJCkgnnBGz9UjNS0vxmnFRbH9UUUZLIdOW94R/gOxcTz3WXQUYidmBZhH 4wcRhWIMrUkxdjP/1g/wHduxxIOOmGUxitQmjCVYzoc6u+M/ldCNIDrC5AZC6TfQANQe c/4Q== X-Gm-Message-State: APjAAAWguovBjvGRyhRrkwMQeMQJ/z6bMCYzsaMXF8PEh5o8lYFmWfsQ JPJZYJMrAMnPfbmvZkcY0iq1yollM8M= X-Google-Smtp-Source: APXvYqx7m+wS9KdjOEWYCfh5Ian88ZwInQwrQhaK5WI3l2HrtUUgcURhLjvdmeGe8kkWvttUQBIb3A== X-Received: by 2002:a17:902:b947:: with SMTP id h7mr58521083pls.267.1560664418983; Sat, 15 Jun 2019 22:53:38 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id q198sm11341086pfq.155.2019.06.15.22.53.37 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 15 Jun 2019 22:53:38 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: <87pnod7ot4.fsf@gmail.com> Date: Sun, 16 Jun 2019 14:53:33 +0900 In-Reply-To: (Ricardo Wurmus's message of "Tue, 28 May 2019 16:53:17 +0200") Message-ID: <871rzu2i02.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Ricardo Wurmus writes: > And finally: Number 9! Yay! >> From 1290f9d1f0d594fdd4723d76b94116be25da9dd5 Mon Sep 17 00:00:00 2001 >> From: Maxim Cournoyer >> Date: Sat, 30 Mar 2019 20:27:35 -0400 >> Subject: [PATCH 9/9] import: pypi: Preserve package name case when forming >> pypi-uri. >> >> Fixes issue: #33046. > > Please change this to: > > Fixes . Done! >> * guix/build-system/python.scm (pypi-uri): Update the host URI to >> "files.pythonhosted.org". >> * guix/import/pypi.scm (make-pypi-sexp): Preserve the package name case when >> the source URL calls for it. > > Is the first change to use files.pythonhosted.org required to fix this? > Or is this unrelated? > > If it is required this looks fine to me. > > Thank you! The permanent redirection was found while fixing the issue; but it's better to have the fix separate. I've separated it into its own commit. Thank you! From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 16 02:05:42 2019 Received: (at 24450) by debbugs.gnu.org; 16 Jun 2019 06:05:42 +0000 Received: from localhost ([127.0.0.1]:39570 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcOIQ-0003hP-2k for submit@debbugs.gnu.org; Sun, 16 Jun 2019 02:05:42 -0400 Received: from mail-pf1-f170.google.com ([209.85.210.170]:44162) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcOIN-0003h9-B8 for 24450@debbugs.gnu.org; Sun, 16 Jun 2019 02:05:40 -0400 Received: by mail-pf1-f170.google.com with SMTP id t16so3853148pfe.11 for <24450@debbugs.gnu.org>; Sat, 15 Jun 2019 23:05:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=Vk0ZJ/OHcPTW1HyeeaeSlZlsjXxGqhG9wlqSvObCOCE=; b=KHzva0ByTq/PBDkKghSFTke1rMY2mprxrHwZ5mDfuktDRa0ih+cb2RNT+diA5px7bd k88kQge6sn+OgtQSo30GkGhAI007JXAR/7uCkjKnYI72AI3d9wB5J//tDgOGiDxHJ4JY CVzIe1cuMDQW/M3bO+hcNnxgaaYXVMh0QjG/dCRVVIhkHcJOEpy9ceKvZHJ7RtaMu0dx ecwjrqFadJ6TUgnfiiyzR4JhO8weSWFoIqhhaiSR6fvDigHfexx4/u94KlIIbaajBbqK BETZ2Q9h9Z8tq8CQHfoxGiOUI4IUrLhWlXM2sS8VMKGS2hhRcekB4iUR+h8C30JeSD9E lNeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=Vk0ZJ/OHcPTW1HyeeaeSlZlsjXxGqhG9wlqSvObCOCE=; b=P/NXhhXr8HSFsBTzKAUn/vDb0lNhtqd32bYR/PiAXD/K43lmjzuxUFfp0FWa/pNifS dI6Maorb0rUy8tSEuKx/rRzUPDlLEeYUVapHKBCF6N5+Kq+l4pMN6hESCU8Cq7Z0ghR4 bt+V3bJojNznyY5du3MmnZ50yZtfigisOc/FQRTRXHEuK/70yatlr9TF3cGn+pwtNo2Z lDZBymCfFSmz+APW6LovGUN1TvArrwF7KMsyVJ3DSSRVyTJdcrAxL88h+9fCXrYbpW55 IVBmL9p3WfSeYpNnVRDLrwAhZp3fULsrq2R7BT54LqAUhiG8DabrKrR5+BqvSazfscLH dZHg== X-Gm-Message-State: APjAAAXpoNi4vQB+ljtG99dLnqOlpeI8bO5rnKnzxomzP37+JXr7qWsr L735IR/DWeFopVFuUsV1Kfhk9eIPGSo= X-Google-Smtp-Source: APXvYqy9bDT6LDEAtMF2vGT2Sfi6FhL0K177Q5ZHeEAelWvCYvuVBH2n51rk0SdKYJh0ccVG6Glivw== X-Received: by 2002:a17:90a:dd42:: with SMTP id u2mr19753469pjv.118.1560665133208; Sat, 15 Jun 2019 23:05:33 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id m6sm8616954pjl.18.2019.06.15.23.05.31 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 15 Jun 2019 23:05:32 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: <87pnod7ot4.fsf@gmail.com> <87imtd6dtq.fsf@gmail.com> <87d0jldct8.fsf@mdc-berlin.de> Date: Sun, 16 Jun 2019 15:05:27 +0900 In-Reply-To: <87d0jldct8.fsf@mdc-berlin.de> (Ricardo Wurmus's message of "Mon, 10 Jun 2019 11:12:03 +0200") Message-ID: <87wohm12vs.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi again, Ricardo Wurmus writes: > Maxim Cournoyer writes: > >> While I agree that a regexp is a bigger hammer than basic string >> manipulation, I see some merit to it here: >> >> 1) We can be assured of conformance with upstream, again, per PEP-0508. >> 2) It is easier to extend; we might want to add parsing for the version >> spec in order to disregard dependencies specified for Python < 3, for >> example. >> >> The use of the PEP-0508 grammar to define the regexp is useful to detail >> in a more human-friendly language the components of the regexp. We >> could have otherwise used the more cryptic regexp for Python >> distribution names: >> >> --8<---------------cut here---------------start------------->8--- >> ^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$ >> --8<---------------cut here---------------end--------------->8--- >> >> So I guess that what I'm saying is that I prefer this approach to using >> string-index with invalid characters, for the reasons above. >> >> [0] https://www.python.org/dev/peps/pep-0508/ > > Okay, sounds good. Please make sure to note this in a comment, so that > I won=E2=80=99t be asking myself this same question in a year :) Done! Maxim From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 16 10:11:17 2019 Received: (at 24450) by debbugs.gnu.org; 16 Jun 2019 14:11:17 +0000 Received: from localhost ([127.0.0.1]:41208 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcVsK-0008P5-1k for submit@debbugs.gnu.org; Sun, 16 Jun 2019 10:11:17 -0400 Received: from mail-pl1-f182.google.com ([209.85.214.182]:42928) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcVsH-0008Op-Bm for 24450@debbugs.gnu.org; Sun, 16 Jun 2019 10:11:14 -0400 Received: by mail-pl1-f182.google.com with SMTP id ay6so769457plb.9 for <24450@debbugs.gnu.org>; Sun, 16 Jun 2019 07:11:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=su9QsJziuedXuj+nCCE3nSkSW786P9vOgQ41oGqoapM=; b=Z5KNuArDuviR51hZbeIrgrvQGs/0KHanzxhjGfRdiG0/Yu8sPITVkUOQjpGsHztnKm MhGdMKe90XQZP2C/E0kK85l2l8X96RAO52YnCMuiRwr3tf2yjcfxUMmQZ1zbXkOh7qck 7Kp2J+N4M2Ap0HTbcrTzm/SrxDR8ohNImfTYNGBKnuhdaY7x4MgaIK69YcrimXZ0SXXK CJPJYfOhoKVC6PTJM0psKLWyG75p5wqqPQCyomtQ+DIux4ypOXu46DbDJPHYSleDIxv2 +6ls0Vm/OxaG93wygZQCtjIIzvi2MdFQ8/dar/HEtsO51tbdhCkNiEGJrbA/24+FZvrd JUGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=su9QsJziuedXuj+nCCE3nSkSW786P9vOgQ41oGqoapM=; b=o/9c0XzDW1FlSxr5g7HEFwIMFH2Gy0qKw++96WMLBo3jCYXys/+mGmHpDLC4ewRrTI nAITb0J/30/L/m6cJwODFWOUsNoIYBt8mxC+DtH7H6uFYEe28sCrhTZrQvGun0Z89i/c J0ZFXuADvebH9T6VkCw9+AHtShLSHN8TGHvDja01V40t/yEKUbsqmXSoKpkBfLoHU5IW W9qTyJXfa9Kp83NB/Y5XpvG7KpwVJOHyGXlii4JOcSU92wisla6edfY3yRz6SPjWNU0H Eo+wXum6ZGncnCHFsQhpglfWw0ZOUwb95YV0NtnncAQzeIXgDYqYxP4t9kJUpEnBJoCS VBoQ== X-Gm-Message-State: APjAAAWFYsCP2BmPs2AW2WHbcqtW4knGlsu3jyWx1noI7OsCqIHP4foD ETekGvOC6WT3TcKpohUFlel2CQoQtC4= X-Google-Smtp-Source: APXvYqzDMsX9FUV/VkC9HYDe8flYHmAuzKOh0POWRMYOxwSITQp9tja3UELQ9zvAESAY4JGToNyZSQ== X-Received: by 2002:a17:902:e105:: with SMTP id cc5mr83815166plb.320.1560694266968; Sun, 16 Jun 2019 07:11:06 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id x17sm11345353pgk.72.2019.06.16.07.11.05 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sun, 16 Jun 2019 07:11:06 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: <87pnod7ot4.fsf@gmail.com> <87muiq5d7c.fsf@gmail.com> <87blz5dcap.fsf@mdc-berlin.de> Date: Sun, 16 Jun 2019 23:11:01 +0900 In-Reply-To: <87blz5dcap.fsf@mdc-berlin.de> (Ricardo Wurmus's message of "Mon, 10 Jun 2019 11:23:10 +0200") Message-ID: <87sgs91uyy.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello! Continued feedback about your much appreciated comments! :-) Ricardo Wurmus writes: > Maxim Cournoyer writes: > >>>> + ;; (extra) requirements. Non-optional requirements must ap= pear >>>> + ;; before any section is defined. >>>> + (if (or (eof-object? line) (section-header? line)) >>>> + (reverse result) >>>> + (cond >>>> + ((or (string-null? line) (comment? line)) >>>> + (loop result)) >>>> + (else >>>> + (loop (cons (clean-requirement line) >>>> + result)))))))))) >>>> + >>> >>> I think it would be better to use =E2=80=9Cmatch=E2=80=9D here instead = of nested =E2=80=9Clet=E2=80=9D, >>> =E2=80=9Cif=E2=80=9D and =E2=80=9Ccond=E2=80=9D. At least you can drop= the =E2=80=9Cif=E2=80=9D and just use cond. >>> >>> The loop let and the inner let can be merged. >> >> I'm not sure I understand; wouldn't merging the named let with the plain >> let mean adding an extra LINE argument to my LOOP procedure? I don't >> want that. > > Let=E2=80=99s forget about merging the nested =E2=80=9Clet=E2=80=9D, beca= use you would indeed > need to change a few more things. It=E2=80=99s fine to keep that as it i= s. But > (if =E2=80=A6 (cond =E2=80=A6)) is not pretty. At least it could be done= in one =E2=80=9Ccond=E2=80=9D: > > (cond > ((or (eof-object? line) (section-header? line)) > (reverse result)) > ((or (string-null? line) (comment? line)) > (loop result)) > (else > (loop (cons (clean-requirement line) > result)))) Agreed and fixed, thanks. >> Also, how could the above code be expressed using "match"? I'm using >> predicates which tests for (special) characters in a string; I don't see >> how the more primitive pattern language of "match" will enable me to do >> the same. > > =E2=80=9Cmatch=E2=80=9D has support for predicates, so you could do somet= hing like this: > > (match line > ((or (eof-object) (? section-header?)) > (reverse result)) > ((or '() (? comment?)) > (loop result)) > (_ (loop (cons (clean-requirement line) result)))) Oh, that's neat! I had no idea that predicates could be used with "match". '() would need to be replaced by "" to match the empty string. Another gotcha with "match", is that the "or" seems to evaluate every component, no matter if a early true condition was found; this resulted in the following error: --8<---------------cut here---------------start------------->8--- + (wrong-type-arg + "string-trim" + "Wrong type argument in position ~A (expecting ~A): ~S" + (1 "string" #) + (#)) result: FAIL --8<---------------cut here---------------end--------------->8--- Due to the "(or (eof-object) (? section-header?)" match clause evaluating the section-header? predicate despite the line being an EOF character. > This allows you to match =E2=80=9Ceof-object=E2=80=9D and '() directly. = Whenever I see > =E2=80=9Cstring-null?=E2=80=9D I think it might be better to =E2=80=9Cmat= ch=E2=80=9D on the empty list > directly. string-null? and an empty list are not the same, unless I'm missing somethi= ng. > But really, that=E2=80=99s up to you. I only feel strongly about avoidin= g =E2=80=9C(if > =E2=80=A6 (cond =E2=80=A6))=E2=80=9D. Due to the problem mentioned above, I stayed with "cond". Thanks! Maxim From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 16 10:30:12 2019 Received: (at 24450) by debbugs.gnu.org; 16 Jun 2019 14:30:12 +0000 Received: from localhost ([127.0.0.1]:41233 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcWAd-0000SX-W5 for submit@debbugs.gnu.org; Sun, 16 Jun 2019 10:30:12 -0400 Received: from mail-pg1-f174.google.com ([209.85.215.174]:37123) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcWAb-0000RS-NS for 24450@debbugs.gnu.org; Sun, 16 Jun 2019 10:30:10 -0400 Received: by mail-pg1-f174.google.com with SMTP id 20so4298891pgr.4 for <24450@debbugs.gnu.org>; Sun, 16 Jun 2019 07:30:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=b6TSodai/oaiDAQ15yVfxwC8QMg7no5hp9tcpwKRFRE=; b=TXuc7/V6PyeEMme+6bUDLuqDLXDXZWT9VJ2AeSAGeT+zqigcbU9pxubUA4Uz/gVYJF RjiyqpLFBuxgRBP7U/N1lcPDZiSfGjBzGJ9X4EB9e9mVNPKC93wqOhKja7xx96G2r5up 2o8bBHjS2e6POyCnyUqotqldITl/2lwTtLEIPV2SFkfn5NxZENFsVe3aKY5kwWTOW+/E vrV23paCyvmGJsfFJ2+E+3nQcfXwyzy2e4vnz6HGI/7p+hnP/yjD6YhDlu8LBNrZKr7n s14E9BE1fpLN4GZ5GY1cKrFpMJpX7t4FnGcnH+B+rxq5DoEQbxdHim4XnLynuBmCf6gW 6U0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=b6TSodai/oaiDAQ15yVfxwC8QMg7no5hp9tcpwKRFRE=; b=VBPGMjdaiRVvw3GoAGP+jAP22DSnMI9v55T/M3ESohFOBLJrfozeDKiO3g1CyNjNYL O5b7IdS7RgcM8WrFkblkbMtTnSenalajV55PD0ZjhWYDw6mbGZBf9Lu0dAPjB1n/7IAC c+fdZtSQ5z97S84LE141PsDp1RQSPomtm/eONBxxFIgsyCCBujub2eJxusBmttElPEAC 7J8r89j8ht5LmiH3G2+5Vc3rj6l1XWmEJk7GjV4wOnCpa9HMloNUdrDOyQfGfpAMcdkF XJrV/WrMF+Vh5RTmTh2Mb4S+Tp2vJuuqYVwPystVRjnN8s+bK5XyanRjESlWZwha8cXj yPAQ== X-Gm-Message-State: APjAAAVjnVs4gmBs7+7jYkwHA5Vj3kfSwcPMrJVvA0dYUQbZi9XLC/wU OF+zGta8nfB2dSFM42tBjvJTTuN/aDo= X-Google-Smtp-Source: APXvYqwMDoYg2xRLt4OPqH8d2uEimHhWKBhSZLfGAxopXgC4VBNjrg7duACBhhXEt8XQhomL1kx5fA== X-Received: by 2002:a65:42c9:: with SMTP id l9mr17371894pgp.32.1560695403518; Sun, 16 Jun 2019 07:30:03 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id x25sm9049301pfm.48.2019.06.16.07.30.01 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sun, 16 Jun 2019 07:30:02 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. References: <87imtb33tp.fsf@gmail.com> <87imtb9ujz.fsf@mdc-berlin.de> Date: Sun, 16 Jun 2019 23:29:58 +0900 In-Reply-To: <87imtb9ujz.fsf@mdc-berlin.de> (Ricardo Wurmus's message of "Wed, 12 Jun 2019 08:39:12 +0200") Message-ID: <87o92x1u3d.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hey Ricardo :-) Ricardo Wurmus writes: > Hi Maxim, > >>>> (call-with-input-file requires.txt >>>> (lambda (port) >>>> - (let loop ((result '())) >>>> + (let loop ((required-deps '()) >>>> + (test-deps '()) >>>> + (inside-test-section? #f) >>>> + (optional? #f)) >>>> (let ((line (read-line port))) >>>> - ;; Stop when a section is encountered, as sections contains= optional >>>> - ;; (extra) requirements. Non-optional requirements must ap= pear >>>> - ;; before any section is defined. >>>> - (if (or (eof-object? line) (section-header? line)) >>>> + (if (eof-object? line) >>>> ;; Duplicates can occur, since the same requirement can= be >>>> ;; listed multiple times with different conditional mar= kers, e.g. >>>> ;; pytest >=3D 3 ; python_version >=3D "3.3" >>>> ;; pytest < 3 ; python_version < "3.3" >>>> - (reverse (delete-duplicates result)) >>>> + (map (compose reverse delete-duplicates) >>>> + (list required-deps test-deps)) >>> >>> Looks like a list of lists to me. =E2=80=9Cdelete-duplicates=E2=80=9D = now won=E2=80=99t delete >>> a name that is in both =E2=80=9Crequired-deps=E2=80=9D as well as in = =E2=80=9Ctest-deps=E2=80=9D. Is >>> this acceptable? >> >> It is acceptable, as this corner case cannot exist given the current >> code (a requirement can exist in either required-deps or test-deps, but >> never in both). It also doesn't make sense that a run time requirement >> would also be listed as a test requirement, so that corner case is not >> likely to exist in the future either. > > I mentioned it because I believe I=E2=80=99ve seen this in the past where= the > importer would return some of the same inputs as both regular inputs and > test dependencies. OK! >>> Personally, I=E2=80=99m not a fan of using data structures for returning >>> multiple values, because we can simply return multiple values. >> >> I thought the Guile supported multiple values return value would be >> great here as well, but I've found that for this specific case here, a >> list of lists worked better, since the two lists contain requirements to >> be processed the same, which "map" can readily do (i.e. less ceremony is >> required). > > =E2=80=9Cmap=E2=80=9D can also operate on more than one list at a time: > > (call-with-values > (lambda () > (values (list 1 2 3) > (list 9 8 7))) > (lambda (a b) (map + a b))) > > =3D> (10 10 10) That's what I meant by "requires more ceremony". I can simply apply "map" to the return value of the function and get what I need, rather than having to use "values" in the callee, then "call-with-values" in the caller and establish a binding for each list. =20=20 > Of course, it would be simpler to just use a single list of tagged > items. Do you feel strongly about it? I don't; I'm open to try to use a tagged list if you feel this is worth it. Maxim From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 16 10:37:12 2019 Received: (at 24450) by debbugs.gnu.org; 16 Jun 2019 14:37:12 +0000 Received: from localhost ([127.0.0.1]:41242 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcWHJ-0000dB-VF for submit@debbugs.gnu.org; Sun, 16 Jun 2019 10:37:12 -0400 Received: from mail-pg1-f173.google.com ([209.85.215.173]:46817) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcWHF-0000cd-NZ for 24450@debbugs.gnu.org; Sun, 16 Jun 2019 10:37:04 -0400 Received: by mail-pg1-f173.google.com with SMTP id v9so4280488pgr.13 for <24450@debbugs.gnu.org>; Sun, 16 Jun 2019 07:37:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=aWQ8jSYOOMUk+mn96zVd65DpZhP3+795f+HGfCC5Uns=; b=ntG+2hx1fz+AukFECZEkXutzjEtSmbFCdb1y3OuGMqw354vOtKbU6jbW7UsmrtQAgw ba+oRQc6nYV4IHOpWbj3J/w/ojOwU0qXP0cFaOU+WHVbYYXChnj4O2jU8F3WGQDIFprl UBZkdyg4H75z3cwSqh48pvJX6ZeV8Q2ovdaS6U4HjNnaualn9N15flDVyv39xuHTjLfj spac3omgnKtFuZFvsaMQ2GoaTxQWgZsP98JlQSzpOuyXah1BpzdTq86mZVEH+4asvbl4 OoyD3+Oxa3+3SdwdZqEB5kdC1OVGHireayL3XWtQ7ZjllZN7F7Wzm2EDwriJqiMoJ1No fGsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=aWQ8jSYOOMUk+mn96zVd65DpZhP3+795f+HGfCC5Uns=; b=nHkOQHsG3Yf9U/7glcAHPnL/AW9I7d1uc3Hq0WatXfE8lA6rvzYoZxG+h4wnMJfz+y yHRpb+WtkIfDvmse4JiDQ2J4qC0iSDbgehkPbUuBIwQlAGu3BRWQRGCCw2QAmOQhwp/G gAgXzOnd2Psqu65MMITuSesnPlq4eHMxnTUz2YObplfiCg21KvsMc+OnaWaPZwsHAOEE kCDnOKwXvOE0gKSj232s84+x/hex/VyL1IbqXgGe5uruZbSn353zjjrp8tE3Ca0bnmJ1 bsLlVB9bjpgQ5C9iPK+5pOJARd4r6Xs81l3bnxbDMUwD8CU8wvBH0iaiJcwYpJplnJJ5 2RHA== X-Gm-Message-State: APjAAAVrVferdk7AUWogyzqLGrG057BI/fWOTUzI8pnX/REnSAvV9pvf vPa0fXSxrTt2OpKMvpEeQhVxvZP7mk4= X-Google-Smtp-Source: APXvYqzXQyA+4GsSyGV5x6aXB77xV+h/PR0N8toQJ1kUu+ljt9XgCAvVHG0WROau6i8Wd9J077dt5g== X-Received: by 2002:a63:fc15:: with SMTP id j21mr5700338pgi.217.1560695815586; Sun, 16 Jun 2019 07:36:55 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id t13sm8000400pjo.13.2019.06.16.07.36.53 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sun, 16 Jun 2019 07:36:54 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: bug#24450: [PATCHv3] Re: pypi importer outputs strange character series in optional dependency case. References: <87imtb33tp.fsf@gmail.com> <87imtb9ujz.fsf@mdc-berlin.de> <87o92x1u3d.fsf@gmail.com> Date: Sun, 16 Jun 2019 23:36:49 +0900 In-Reply-To: <87o92x1u3d.fsf@gmail.com> (Maxim Cournoyer's message of "Sun, 16 Jun 2019 23:29:58 +0900") Message-ID: <87pnndzjem.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain Here's the current patch set, version 3, with the modifications as discussed in the previous exchanges: --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-import-pypi-Do-not-consider-requirements.txt-files.patch Content-Transfer-Encoding: quoted-printable From=20215d0bde103e7e8df0d1d1df81965e973f38783a Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:00 -0400 Subject: [PATCH 01/12] import: pypi: Do not consider requirements.txt files. PyPI packages are mandated to have a setup.py file, which contains a listing of the required dependencies. The setuptools/distutils machinery embed metadata in the archives they produce, which contains this information. The= re is no need nor gain to collect the requirements from a "requirements.txt" file, as it is not the true record of dependencies for PyPI packages and may contain extraneous requirements or not exist at all. * guix/import/pypi.scm (guess-requirements): Update comment. [guess-requirements-from-source]: Do not attempt to parse the file requirements.txt. Streamline logic. * tests/pypi.scm (test-requires.txt): Rename from test-requirements, to hint at the file being tested. ("pypi->guix-package"): Adapt so that the fake package contains a requires.= txt file rather than a requirements.txt file. ("pypi->guix-package, wheels"): Likewise. =2D-- guix/import/pypi.scm | 35 +++++++++++++---------------------- tests/pypi.scm | 23 ++++++++++++----------- 2 files changed, 25 insertions(+), 33 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 3a20fc4b9b..8269aa61d7 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -206,35 +206,26 @@ cannot determine package dependencies")) (call-with-temporary-directory (lambda (dir) (let* ((pypi-name (string-take dirname (string-rindex dirname= #\-))) =2D (req-files (list (string-append dirname "/requiremen= ts.txt") =2D (string-append dirname "/" pypi-nam= e ".egg-info" =2D "/requires.txt"))) =2D (exit-codes (map (lambda (file-name) =2D (parameterize ((current-error-por= t (%make-void-port "rw+")) =2D (current-output-po= rt (%make-void-port "rw+"))) =2D (system* "tar" "xf" tarball "-C= " dir file-name))) =2D req-files))) =2D ;; Only one of these files needs to exist. =2D (if (any zero? exit-codes) =2D (match (find-files dir) =2D ((file . _) =2D (read-requirements file)) =2D (() =2D (warning (G_ "No requirements file found.\n")))) + (requires.txt (string-append dirname "/" pypi-name + ".egg-info" "/requires.tx= t")) + (exit-code (parameterize ((current-error-port (%make-v= oid-port "rw+")) + (current-output-port (%make-= void-port "rw+"))) + (system* "tar" "xf" tarball "-C" dir requ= ires.txt)))) + (if (zero? exit-code) + (read-requirements (string-append dir "/" requires.txt)) (begin =2D (warning (G_ "Failed to extract requirements files\= n")) + (warning + (G_ "Failed to extract file: ~a from source.~%") + requires.txt) '()))))) '()))) =20 =2D ;; First, try to compute the requirements using the wheel, since that = is the =2D ;; most reliable option. If a wheel is not provided for this package, = try =2D ;; getting them by reading either the "requirements.txt" file or the =2D ;; "requires.txt" from the egg-info directory from the source tarball.= Note =2D ;; that "requirements.txt" is not mandatory, so this is likely to fail. + ;; First, try to compute the requirements using the wheel, else, fallbac= k to + ;; reading the "requires.txt" from the egg-info directory from the source + ;; tarball. (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 =2D (define (compute-inputs source-url wheel-url tarball) "Given the SOURCE-URL of an already downloaded TARBALL, return a list of name/variable pairs describing the required inputs of this package. Also diff --git a/tests/pypi.scm b/tests/pypi.scm index 6daa44a6e7..a0271fffad 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -23,7 +23,7 @@ #:use-module (gcrypt hash) #:use-module (guix tests) #:use-module (guix build-system python) =2D #:use-module ((guix build utils) #:select (delete-file-recursively whi= ch)) + #:use-module ((guix build utils) #:select (delete-file-recursively which= mkdir-p)) #:use-module (srfi srfi-64) #:use-module (ice-9 match)) =20 @@ -55,11 +55,12 @@ (define test-source-hash "") =20 =2D(define test-requirements =2D"# A comment +(define test-requires.txt "\ +# A comment # A comment after a space bar =2Dbaz > 13.37") +baz > 13.37 +") =20 (define test-metadata "{ @@ -107,10 +108,10 @@ baz > 13.37") (match url ("https://example.com/foo-1.0.0.tar.gz" (begin =2D (mkdir "foo-1.0.0") =2D (with-output-to-file "foo-1.0.0/requirements.txt" + (mkdir-p "foo-1.0.0/foo.egg-info/") + (with-output-to-file "foo-1.0.0/foo.egg-info/requires.tx= t" (lambda () =2D (display test-requirements))) + (display test-requires.txt))) (system* "tar" "czvf" file-name "foo-1.0.0/") (delete-file-recursively "foo-1.0.0") (set! test-source-hash @@ -157,11 +158,11 @@ baz > 13.37") (lambda (url file-name) (match url ("https://example.com/foo-1.0.0.tar.gz" =2D (begin =2D (mkdir "foo-1.0.0") =2D (with-output-to-file "foo-1.0.0/requirements.txt" + (begin + (mkdir-p "foo-1.0.0/foo.egg-info/") + (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" (lambda () =2D (display test-requirements))) + (display test-requires.txt))) (system* "tar" "czvf" file-name "foo-1.0.0/") (delete-file-recursively "foo-1.0.0") (set! test-source-hash =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0001-import-pypi-Do-not-parse-optional-requirements-from-.patch Content-Transfer-Encoding: quoted-printable From=2022da422c3cea1f6d05a11123dd201c1ec54c9ff8 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:00 -0400 Subject: [PATCH] import: pypi: Do not parse optional requirements from sour= ce. * guix/import/pypi.scm: Export PARSE-REQUIRES.TXT. (clean-requirement): Move procedure to the top level. (guess-requirements): Move the READ-REQUIREMENTS procedure to the top level, and rename it to PARSE-REQUIRES.TXT. Move the CLEAN-REQUIREMENT procedure = to the top level. Move the COMMENT? functions inside the PARSE-REQUIRES.TXT procedure. (parse-requires.txt): Add a SECTION-HEADER? predicate, and use it to prevent parsing optional requirements. * tests/pypi.scm (test-requires-with-sections): New variable. ("parse-requires.txt, with sections"): New test. =2D-- guix/import/pypi.scm | 74 ++++++++++++++++++++++++++------------------ tests/pypi.scm | 14 +++++++++ 2 files changed, 58 insertions(+), 30 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 8269aa61d7..d9db876222 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -47,7 +47,8 @@ #:use-module (guix upstream) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix build-system python) =2D #:export (guix-package->pypi-name + #:export (parse-requires.txt + guix-package->pypi-name pypi-recursive-import pypi->guix-package %pypi-updater)) @@ -117,6 +118,47 @@ package definition." ((package-inputs ...) `((propagated-inputs (,'quasiquote ,package-inputs)))))) =20 +(define (clean-requirement s) + ;; Given a requirement LINE, as can be found in a setuptools requires.txt + ;; file, remove everything other than the actual name of the required + ;; package, and return it. + (cond + ((string-index s (char-set #\space #\> #\=3D #\<)) =3D> (cut string-tak= e s <>)) + (else s))) + +(define (parse-requires.txt requires.txt) + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of +requirement names." + ;; This is a very incomplete parser, whose job is to select the non-opti= onal + ;; dependencies and strip them out of any version information. + ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) + ;; library and the requirements grammar defined by PEP-0508 + ;; (https://www.python.org/dev/peps/pep-0508/). + + (define (comment? line) + ;; Return #t if the given LINE is a comment, #f otherwise. + (string-prefix? "#" (string-trim line))) + + (define (section-header? line) + ;; Return #t if the given LINE is a section header, #f otherwise. + (string-prefix? "[" (string-trim line))) + + (call-with-input-file requires.txt + (lambda (port) + (let loop ((result '())) + (let ((line (read-line port))) + ;; Stop when a section is encountered, as sections contain optio= nal + ;; (extra) requirements. Non-optional requirements must appear + ;; before any section is defined. + (if (or (eof-object? line) (section-header? line)) + (reverse result) + (cond + ((or (string-null? line) (comment? line)) + (loop result)) + (else + (loop (cons (clean-requirement line) + result)))))))))) + (define (guess-requirements source-url wheel-url tarball) "Given SOURCE-URL, WHEEL-URL and a TARBALL of the package, return a list of the required packages specified in the requirements.txt file. TARBALL = will @@ -139,34 +181,6 @@ be extracted in a temporary directory." cannot determine package dependencies")) #f))))) =20 =2D (define (clean-requirement s) =2D ;; Given a requirement LINE, as can be found in a Python requirement= s.txt =2D ;; file, remove everything other than the actual name of the required =2D ;; package, and return it. =2D (string-take s =2D (or (string-index s (lambda (chr) (member chr '(#\space #\> #\=3D = #\<)))) =2D (string-length s)))) =2D =2D (define (comment? line) =2D ;; Return #t if the given LINE is a comment, #f otherwise. =2D (eq? (string-ref (string-trim line) 0) #\#)) =2D =2D (define (read-requirements requirements-file) =2D ;; Given REQUIREMENTS-FILE, a Python requirements.txt file, return a= list =2D ;; of name/variable pairs describing the requirements. =2D (call-with-input-file requirements-file =2D (lambda (port) =2D (let loop ((result '())) =2D (let ((line (read-line port))) =2D (if (eof-object? line) =2D result =2D (cond =2D ((or (string-null? line) (comment? line)) =2D (loop result)) =2D (else =2D (loop (cons (clean-requirement line) =2D result)))))))))) =2D (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's ;; requirements. @@ -212,7 +226,7 @@ cannot determine package dependencies")) (current-output-port (%make-= void-port "rw+"))) (system* "tar" "xf" tarball "-C" dir requ= ires.txt)))) (if (zero? exit-code) =2D (read-requirements (string-append dir "/" requires.tx= t)) + (parse-requires.txt (string-append dir "/" requires.txt= )) (begin (warning (G_ "Failed to extract file: ~a from source.~%") diff --git a/tests/pypi.scm b/tests/pypi.scm index 6df69073dc..03455ba6be 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -62,6 +62,14 @@ bar baz > 13.37 ") =20 +(define test-requires-with-sections "\ +foo ~=3D 3 +bar !=3D 2 + +[test] +pytest (>=3D2.5.0) +") + (define test-metadata "{ \"run_requires\": [ @@ -101,6 +109,12 @@ baz > 13.37 (uri (list "https://bitheap.org/cram/cram-0.7.tar.gz" (pypi-uri "cram" "0.7")))))))) =20 +(test-equal "parse-requires.txt, with sections" + '("foo" "bar") + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-requires.txt test-requires-with-sections))) + (test-assert "pypi->guix-package" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0002-tests-pypi-Mute-the-output-of-tar.patch Content-Transfer-Encoding: quoted-printable From=2017419a5f9572b7b886544531324b7b2ec431555e Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Mon, 10 Jun 2019 12:20:08 +0900 Subject: [PATCH 02/12] tests: pypi: Mute the output of tar. The output of tar when creating archives for the purpose of tests is not useful, so we mute it. * tests/pypi.scm ("pypi->guix-package"): Mute the output of tar. ("pypi->guix-package, wheels"): Likewise. =2D-- tests/pypi.scm | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/pypi.scm b/tests/pypi.scm index a0271fffad..6df69073dc 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -112,7 +112,8 @@ baz > 13.37 (with-output-to-file "foo-1.0.0/foo.egg-info/requires.tx= t" (lambda () (display test-requires.txt))) =2D (system* "tar" "czvf" file-name "foo-1.0.0/") + (parameterize ((current-output-port (%make-void-port "rw= +"))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) (delete-file-recursively "foo-1.0.0") (set! test-source-hash (call-with-input-file file-name port-sha256)))) @@ -163,7 +164,8 @@ baz > 13.37 (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" (lambda () (display test-requires.txt))) =2D (system* "tar" "czvf" file-name "foo-1.0.0/") + (parameterize ((current-output-port (%make-void-port "rw+"= ))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) (delete-file-recursively "foo-1.0.0") (set! test-source-hash (call-with-input-file file-name port-sha256)))) =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0003-import-pypi-Do-not-parse-optional-requirements-from-.patch Content-Transfer-Encoding: quoted-printable From=2022da422c3cea1f6d05a11123dd201c1ec54c9ff8 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:00 -0400 Subject: [PATCH 03/12] import: pypi: Do not parse optional requirements from source. * guix/import/pypi.scm: Export PARSE-REQUIRES.TXT. (clean-requirement): Move procedure to the top level. (guess-requirements): Move the READ-REQUIREMENTS procedure to the top level, and rename it to PARSE-REQUIRES.TXT. Move the CLEAN-REQUIREMENT procedure = to the top level. Move the COMMENT? functions inside the PARSE-REQUIRES.TXT procedure. (parse-requires.txt): Add a SECTION-HEADER? predicate, and use it to prevent parsing optional requirements. * tests/pypi.scm (test-requires-with-sections): New variable. ("parse-requires.txt, with sections"): New test. =2D-- guix/import/pypi.scm | 74 ++++++++++++++++++++++++++------------------ tests/pypi.scm | 14 +++++++++ 2 files changed, 58 insertions(+), 30 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 8269aa61d7..d9db876222 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -47,7 +47,8 @@ #:use-module (guix upstream) #:use-module ((guix licenses) #:prefix license:) #:use-module (guix build-system python) =2D #:export (guix-package->pypi-name + #:export (parse-requires.txt + guix-package->pypi-name pypi-recursive-import pypi->guix-package %pypi-updater)) @@ -117,6 +118,47 @@ package definition." ((package-inputs ...) `((propagated-inputs (,'quasiquote ,package-inputs)))))) =20 +(define (clean-requirement s) + ;; Given a requirement LINE, as can be found in a setuptools requires.txt + ;; file, remove everything other than the actual name of the required + ;; package, and return it. + (cond + ((string-index s (char-set #\space #\> #\=3D #\<)) =3D> (cut string-tak= e s <>)) + (else s))) + +(define (parse-requires.txt requires.txt) + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of +requirement names." + ;; This is a very incomplete parser, whose job is to select the non-opti= onal + ;; dependencies and strip them out of any version information. + ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) + ;; library and the requirements grammar defined by PEP-0508 + ;; (https://www.python.org/dev/peps/pep-0508/). + + (define (comment? line) + ;; Return #t if the given LINE is a comment, #f otherwise. + (string-prefix? "#" (string-trim line))) + + (define (section-header? line) + ;; Return #t if the given LINE is a section header, #f otherwise. + (string-prefix? "[" (string-trim line))) + + (call-with-input-file requires.txt + (lambda (port) + (let loop ((result '())) + (let ((line (read-line port))) + ;; Stop when a section is encountered, as sections contain optio= nal + ;; (extra) requirements. Non-optional requirements must appear + ;; before any section is defined. + (if (or (eof-object? line) (section-header? line)) + (reverse result) + (cond + ((or (string-null? line) (comment? line)) + (loop result)) + (else + (loop (cons (clean-requirement line) + result)))))))))) + (define (guess-requirements source-url wheel-url tarball) "Given SOURCE-URL, WHEEL-URL and a TARBALL of the package, return a list of the required packages specified in the requirements.txt file. TARBALL = will @@ -139,34 +181,6 @@ be extracted in a temporary directory." cannot determine package dependencies")) #f))))) =20 =2D (define (clean-requirement s) =2D ;; Given a requirement LINE, as can be found in a Python requirement= s.txt =2D ;; file, remove everything other than the actual name of the required =2D ;; package, and return it. =2D (string-take s =2D (or (string-index s (lambda (chr) (member chr '(#\space #\> #\=3D = #\<)))) =2D (string-length s)))) =2D =2D (define (comment? line) =2D ;; Return #t if the given LINE is a comment, #f otherwise. =2D (eq? (string-ref (string-trim line) 0) #\#)) =2D =2D (define (read-requirements requirements-file) =2D ;; Given REQUIREMENTS-FILE, a Python requirements.txt file, return a= list =2D ;; of name/variable pairs describing the requirements. =2D (call-with-input-file requirements-file =2D (lambda (port) =2D (let loop ((result '())) =2D (let ((line (read-line port))) =2D (if (eof-object? line) =2D result =2D (cond =2D ((or (string-null? line) (comment? line)) =2D (loop result)) =2D (else =2D (loop (cons (clean-requirement line) =2D result)))))))))) =2D (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's ;; requirements. @@ -212,7 +226,7 @@ cannot determine package dependencies")) (current-output-port (%make-= void-port "rw+"))) (system* "tar" "xf" tarball "-C" dir requ= ires.txt)))) (if (zero? exit-code) =2D (read-requirements (string-append dir "/" requires.tx= t)) + (parse-requires.txt (string-append dir "/" requires.txt= )) (begin (warning (G_ "Failed to extract file: ~a from source.~%") diff --git a/tests/pypi.scm b/tests/pypi.scm index 6df69073dc..03455ba6be 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -62,6 +62,14 @@ bar baz > 13.37 ") =20 +(define test-requires-with-sections "\ +foo ~=3D 3 +bar !=3D 2 + +[test] +pytest (>=3D2.5.0) +") + (define test-metadata "{ \"run_requires\": [ @@ -101,6 +109,12 @@ baz > 13.37 (uri (list "https://bitheap.org/cram/cram-0.7.tar.gz" (pypi-uri "cram" "0.7")))))))) =20 +(test-equal "parse-requires.txt, with sections" + '("foo" "bar") + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-requires.txt test-requires-with-sections))) + (test-assert "pypi->guix-package" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0004-import-pypi-Improve-parsing-of-requirement-specifica.patch Content-Transfer-Encoding: quoted-printable From=20bbdfb0ff2cc0347df73bb21b1443d6ef1f138a43 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:01 -0400 Subject: [PATCH 04/12] import: pypi: Improve parsing of requirement specifications. The previous solution was fragile and could leave unwanted characters in a requirement name, such as '[' or ']'. Partially fixes . * guix/import/pypi.scm (use-modules): Export SPECIFICATION->REQUIREMENT-NAME (%requirement-name-regexp): New variable. (clean-requirement): Rename to... (specification->requirement-name): this, which now uses %requirement-name-regexp to select the requirement name from the requirement specification. (parse-requires.txt): Adapt. =2D-- guix/import/pypi.scm | 54 ++++++++++++++++++++++++++++++++------------ tests/pypi.scm | 12 ++++++++++ 2 files changed, 52 insertions(+), 14 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index d9db876222..6a881bda12 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -48,6 +48,7 @@ #:use-module ((guix licenses) #:prefix license:) #:use-module (guix build-system python) #:export (parse-requires.txt + specification->requirement-name guix-package->pypi-name pypi-recursive-import pypi->guix-package @@ -118,22 +119,47 @@ package definition." ((package-inputs ...) `((propagated-inputs (,'quasiquote ,package-inputs)))))) =20 =2D(define (clean-requirement s) =2D ;; Given a requirement LINE, as can be found in a setuptools requires.= txt =2D ;; file, remove everything other than the actual name of the required =2D ;; package, and return it. =2D (cond =2D ((string-index s (char-set #\space #\> #\=3D #\<)) =3D> (cut string-t= ake s <>)) =2D (else s))) +(define %requirement-name-regexp + ;; Regexp to match the requirement name in a requirement specification. + + ;; Some grammar, taken from PEP-0508 (see: + ;; https://www.python.org/dev/peps/pep-0508/). + + ;; Using this grammar makes the PEP-0508 regexp easier to understand for + ;; humans. The use of a regexp is preferred to more primitive string + ;; manipulations because we can more directly match what upstream uses + ;; (again, per PEP-0508). The regexp approach is also easier to extend, + ;; should we want to implement more completely the grammar of PEP-0508. + + ;; The unified rule can be expressed as: + ;; specification =3D wsp* ( url_req | name_req ) wsp* + + ;; where url_req is: + ;; url_req =3D name wsp* extras? wsp* urlspec wsp+ quoted_marker? + + ;; and where name_req is: + ;; name_req =3D name wsp* extras? wsp* versionspec? wsp* quoted_marker? + + ;; Thus, we need only matching NAME, which is expressed as: + ;; identifer_end =3D letterOrDigit | (('-' | '_' | '.' )* letterOrDigit) + ;; identifier =3D letterOrDigit identifier_end* + ;; name =3D identifier + (let* ((letter-or-digit "[A-Za-z0-9]") + (identifier-end (string-append "(" letter-or-digit "|" + "[-_.]*" letter-or-digit ")")) + (identifier (string-append "^" letter-or-digit identifier-end "*"= )) + (name identifier)) + (make-regexp name))) + +(define (specification->requirement-name spec) + "Given a specification SPEC, return the requirement name." + (match:substring + (or (regexp-exec %requirement-name-regexp spec) + (error (G_ "Could not extract requirement name in spec:") spec)))) =20 (define (parse-requires.txt requires.txt) "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of requirement names." =2D ;; This is a very incomplete parser, whose job is to select the non-op= tional =2D ;; dependencies and strip them out of any version information. =2D ;; Alternatively, we could implement a PEG parser with the (ice-9 peg) =2D ;; library and the requirements grammar defined by PEP-0508 =2D ;; (https://www.python.org/dev/peps/pep-0508/). =20 (define (comment? line) ;; Return #t if the given LINE is a comment, #f otherwise. @@ -156,7 +182,7 @@ requirement names." ((or (string-null? line) (comment? line)) (loop result)) (else =2D (loop (cons (clean-requirement line) + (loop (cons (specification->requirement-name line) result)))))))))) =20 (define (guess-requirements source-url wheel-url tarball) @@ -198,7 +224,7 @@ cannot determine package dependencies")) (hash-ref (list-ref run_requir= es 0) "requires") '()))) =2D (map clean-requirement requirements))))) + (map specification->requirement-name requirements))))) (lambda () (delete-file json-file) (rmdir dirname)))))) diff --git a/tests/pypi.scm b/tests/pypi.scm index 03455ba6be..c40be6c21d 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -55,6 +55,14 @@ (define test-source-hash "") =20 +(define test-specifications + '("Fizzy [foo, bar]" + "PickyThing<1.6,>1.9,!=3D1.9.6,<2.0a0,=3D=3D2.4c1" + "SomethingWithMarker[foo]>1.0;python_version<\"2.7\"" + "requests [security,tests] >=3D 2.8.1, =3D=3D 2.8.* ; python_version <= \"2.7\"" + "pip @ https://github.com/pypa/pip/archive/1.3.1.zip#\ +sha1=3Dda9234ee9982d4bbb3c72346a6de940a148ea686")) + (define test-requires.txt "\ # A comment # A comment after a space @@ -109,6 +117,10 @@ pytest (>=3D2.5.0) (uri (list "https://bitheap.org/cram/cram-0.7.tar.gz" (pypi-uri "cram" "0.7")))))))) =20 +(test-equal "specification->requirement-name" + '("Fizzy" "PickyThing" "SomethingWithMarker" "requests" "pip") + (map specification->requirement-name test-specifications)) + (test-equal "parse-requires.txt, with sections" '("foo" "bar") (mock ((ice-9 ports) call-with-input-file =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0005-import-pypi-Deduplicate-requirements.patch Content-Transfer-Encoding: quoted-printable From=20475d5e483e7bfdae8db271fc513172c4f05e7358 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:01 -0400 Subject: [PATCH 05/12] import: pypi: Deduplicate requirements. * guix/import/pypi.scm (parse-requires.txt): Remove potential duplicates. =2D-- guix/import/pypi.scm | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 6a881bda12..ad59a8b731 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -177,7 +177,11 @@ requirement names." ;; (extra) requirements. Non-optional requirements must appear ;; before any section is defined. (if (or (eof-object? line) (section-header? line)) =2D (reverse result) + ;; Duplicates can occur, since the same requirement can be + ;; listed multiple times with different conditional markers,= e.g. + ;; pytest >=3D 3 ; python_version >=3D "3.3" + ;; pytest < 3 ; python_version < "3.3" + (reverse (delete-duplicates result)) (cond ((or (string-null? line) (comment? line)) (loop result)) =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0006-import-pypi-Support-more-types-of-archives.patch Content-Transfer-Encoding: quoted-printable From=2049b2232b6a3421e3b7d3b59be5b965c5bb11392e Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:02 -0400 Subject: [PATCH 06/12] import: pypi: Support more types of archives. This change enables the PyPI importer to look for requirements in a source archive of a different type than "tar.gz" or "tar.bz2". Also, scan the sou= rce archive to find a requires.txt file. * guix/import/pypi.scm: (guess-requirements)[tarball-directory]: Remove pro= cedure. [guess-requirements-from-source]: Use COMRESSED-FILE? to determine if an archive type is supported, and some file extension logic that chooses either "tar" or "unzip" as the extractor. Search for the requires.txt file in the archive instead of using a static, expected location. (guess-requirements): Rename the TARBALL argument to ARCHIVE, to denote the archive format is no longer bound specifically to the Tar format. (compute-inputs): Likewise. * tests/pypi.scm ("pypi->guix-package, no wheel"): Mock the requires.txt at= a non-standard location. ("pypi->guix-package, no usable requirement file."): New test. =2D-- guix/import/pypi.scm | 77 +++++++++++++++++++------------------------- tests/pypi.scm | 52 ++++++++++++++++++++++++++++-- 2 files changed, 83 insertions(+), 46 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index ad59a8b731..a6106ab4ec 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -39,7 +39,8 @@ #:use-module ((guix build utils) #:select ((package-name->name+version . hyphen-package-name->name+version) =2D find-files)) + find-files + invoke)) #:use-module (guix import utils) #:use-module ((guix download) #:prefix download:) #:use-module (guix import json) @@ -189,28 +190,11 @@ requirement names." (loop (cons (specification->requirement-name line) result)))))))))) =20 =2D(define (guess-requirements source-url wheel-url tarball) =2D "Given SOURCE-URL, WHEEL-URL and a TARBALL of the package, return a li= st =2Dof the required packages specified in the requirements.txt file. TARBAL= L will +(define (guess-requirements source-url wheel-url archive) + "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a list +of the required packages specified in the requirements.txt file. ARCHIVE = will be extracted in a temporary directory." =20 =2D (define (tarball-directory url) =2D ;; Given the URL of the package's tarball, return the name of the di= rectory =2D ;; that will be created upon decompressing it. If the filetype is not =2D ;; supported, return #f. =2D ;; TODO: Support more archive formats. =2D (let ((basename (substring url (+ 1 (string-rindex url #\/))))) =2D (cond =2D ((string-suffix? ".tar.gz" basename) =2D (string-drop-right basename 7)) =2D ((string-suffix? ".tar.bz2" basename) =2D (string-drop-right basename 8)) =2D (else =2D (begin =2D (warning (G_ "Unsupported archive format: \ =2Dcannot determine package dependencies")) =2D #f))))) =2D (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's ;; requirements. @@ -239,29 +223,34 @@ cannot determine package dependencies")) (call-with-temporary-output-file (lambda (temp port) (if wheel-url =2D (and (url-fetch wheel-url temp) =2D (read-wheel-metadata temp)) =2D #f)))) + (and (url-fetch wheel-url temp) + (read-wheel-metadata temp)) + #f)))) =20 (define (guess-requirements-from-source) ;; Return the package's requirements by guessing them from the source. =2D (let ((dirname (tarball-directory source-url))) =2D (if (string? dirname) =2D (call-with-temporary-directory =2D (lambda (dir) =2D (let* ((pypi-name (string-take dirname (string-rindex dirna= me #\-))) =2D (requires.txt (string-append dirname "/" pypi-name =2D ".egg-info" "/requires.= txt")) =2D (exit-code (parameterize ((current-error-port (%make= -void-port "rw+")) =2D (current-output-port (%mak= e-void-port "rw+"))) =2D (system* "tar" "xf" tarball "-C" dir re= quires.txt)))) =2D (if (zero? exit-code) =2D (parse-requires.txt (string-append dir "/" requires.t= xt)) =2D (begin =2D (warning =2D (G_ "Failed to extract file: ~a from source.~%") =2D requires.txt) =2D '()))))) + (if (compressed-file? source-url) + (call-with-temporary-directory + (lambda (dir) + (parameterize ((current-error-port (%make-void-port "rw+")) + (current-output-port (%make-void-port "rw+"))) + (if (string=3D? "zip" (file-extension source-url)) + (invoke "unzip" archive "-d" dir) + (invoke "tar" "xf" archive "-C" dir))) + (let ((requires.txt-files + (find-files dir (lambda (abs-file-name _) + (string-match "\\.egg-info/requires.txt$" + abs-file-name))))) + (match requires.txt-files + (() + (warning (G_ "Cannot guess requirements from source archiv= e:\ + no requires.txt file found.~%")) + '()) + (else (parse-requires.txt (first requires.txt-files))))))) + (begin + (warning (G_ "Unsupported archive format; \ +cannot determine package dependencies from source archive: ~a~%") + (basename source-url)) '()))) =20 ;; First, try to compute the requirements using the wheel, else, fallbac= k to @@ -270,13 +259,13 @@ cannot determine package dependencies")) (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 =2D(define (compute-inputs source-url wheel-url tarball) =2D "Given the SOURCE-URL of an already downloaded TARBALL, return a list = of +(define (compute-inputs source-url wheel-url archive) + "Given the SOURCE-URL of an already downloaded ARCHIVE, return a list of name/variable pairs describing the required inputs of this package. Also return the unaltered list of upstream dependency names." (let ((dependencies (remove (cut string=3D? "argparse" <>) =2D (guess-requirements source-url wheel-url tarball)))) + (guess-requirements source-url wheel-url archive)))) (values (sort (map (lambda (input) (let ((guix-name (python->package-name input))) diff --git a/tests/pypi.scm b/tests/pypi.scm index c40be6c21d..b45d2c9d2f 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -20,6 +20,7 @@ (define-module (test-pypi) #:use-module (guix import pypi) #:use-module (guix base32) + #:use-module (guix memoization) #:use-module (gcrypt hash) #:use-module (guix tests) #:use-module (guix build-system python) @@ -134,8 +135,9 @@ pytest (>=3D2.5.0) (match url ("https://example.com/foo-1.0.0.tar.gz" (begin =2D (mkdir-p "foo-1.0.0/foo.egg-info/") =2D (with-output-to-file "foo-1.0.0/foo.egg-info/requires.= txt" + ;; Unusual requires.txt location should still be found. + (mkdir-p "foo-1.0.0/src/bizarre.egg-info") + (with-output-to-file "foo-1.0.0/src/bizarre.egg-info/req= uires.txt" (lambda () (display test-requires.txt))) (parameterize ((current-output-port (%make-void-port "rw= +"))) @@ -241,4 +243,50 @@ pytest (>=3D2.5.0) (x (pk 'fail x #f)))))) =20 +(test-assert "pypi->guix-package, no usable requirement file." + ;; Replace network resources with sample data. + (mock ((guix import utils) url-fetch + (lambda (url file-name) + (match url + ("https://example.com/foo-1.0.0.tar.gz" + (mkdir-p "foo-1.0.0/foo.egg-info/") + (parameterize ((current-output-port (%make-void-port "rw+"))) + (system* "tar" "czvf" file-name "foo-1.0.0/")) + (delete-file-recursively "foo-1.0.0") + (set! test-source-hash + (call-with-input-file file-name port-sha256))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #f) + (_ (error "Unexpected URL: " url))))) + (mock ((guix http-client) http-fetch + (lambda (url . rest) + (match url + ("https://pypi.org/pypi/foo/json" + (values (open-input-string test-json) + (string-length test-json))) + ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #= f) + (_ (error "Unexpected URL: " url))))) + ;; Not clearing the memoization cache here would mean return= ing the value + ;; computed in the previous test. + (invalidate-memoization! pypi->guix-package) + (match (pypi->guix-package "foo") + (('package + ('name "python-foo") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri ('pypi-uri "foo" 'version)) + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'python-build-system) + ('home-page "http://example.com") + ('synopsis "summary") + ('description "summary") + ('license 'license:lgpl2.0)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (pk 'fail x #f)))))) + (test-end "pypi") =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0007-import-pypi-Parse-wheel-METADATA-instead-of-metadata.patch Content-Transfer-Encoding: quoted-printable From=203a0082de1bed3eca6dedf71dec34efd0457bd58f Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 00:26:03 -0400 Subject: [PATCH 07/12] import: pypi: Parse wheel METADATA instead of metadata.json. With newer Wheel releases, there is no more metadata.json file; the METADATA file should be used instead (see: https://github.com/pypa/wheel/issues/195). This change updates our PyPI importer so that it uses the latter. * guix/import/pypi.scm (define-module): Remove unnecessary modules and expo= rt the PARSE-WHEEL-METADATA procedure. (parse-wheel-metadata): Add procedure. (guess-requirements): Use it. * tests/pypi.scm (test-metadata): Test it. =2D-- guix/import/pypi.scm | 90 +++++++++++++++++++++++++++----------------- tests/pypi.scm | 60 ++++++++++++++++++++++------- 2 files changed, 101 insertions(+), 49 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index a6106ab4ec..7cf1e92101 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -21,9 +21,7 @@ ;;; along with GNU Guix. If not, see . =20 (define-module (guix import pypi) =2D #:use-module (ice-9 binary-ports) #:use-module (ice-9 match) =2D #:use-module (ice-9 pretty-print) #:use-module (ice-9 regex) #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) @@ -31,9 +29,6 @@ #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) =2D #:use-module (rnrs bytevectors) =2D #:use-module (json) =2D #:use-module (web uri) #:use-module (guix ui) #:use-module (guix utils) #:use-module ((guix build utils) @@ -49,6 +44,7 @@ #:use-module ((guix licenses) #:prefix license:) #:use-module (guix build-system python) #:export (parse-requires.txt + parse-wheel-metadata specification->requirement-name guix-package->pypi-name pypi-recursive-import @@ -177,18 +173,49 @@ requirement names." ;; Stop when a section is encountered, as sections contain optio= nal ;; (extra) requirements. Non-optional requirements must appear ;; before any section is defined. =2D (if (or (eof-object? line) (section-header? line)) =2D ;; Duplicates can occur, since the same requirement can be =2D ;; listed multiple times with different conditional marker= s, e.g. =2D ;; pytest >=3D 3 ; python_version >=3D "3.3" =2D ;; pytest < 3 ; python_version < "3.3" =2D (reverse (delete-duplicates result)) =2D (cond =2D ((or (string-null? line) (comment? line)) =2D (loop result)) =2D (else =2D (loop (cons (specification->requirement-name line) =2D result)))))))))) + (cond + ((or (eof-object? line) (section-header? line)) + ;; Duplicates can occur, since the same requirement can be + ;; listed multiple times with different conditional markers, e= .g. + ;; pytest >=3D 3 ; python_version >=3D "3.3" + ;; pytest < 3 ; python_version < "3.3" + (reverse (delete-duplicates result))) + ((or (string-null? line) (comment? line)) + (loop result)) + (else + (loop (cons (specification->requirement-name line) + result))))))))) + +(define (parse-wheel-metadata metadata) + "Given METADATA, a Wheel metadata file, return a list of requirement nam= es." + ;; METADATA is a RFC-2822-like, header based file. + + (define (requires-dist-header? line) + ;; Return #t if the given LINE is a Requires-Dist header. + (string-match "^Requires-Dist: " line)) + + (define (requires-dist-value line) + (string-drop line (string-length "Requires-Dist: "))) + + (define (extra? line) + ;; Return #t if the given LINE is an "extra" requirement. + (string-match "extra =3D=3D '(.*)'" line)) + + (call-with-input-file metadata + (lambda (port) + (let loop ((requirements '())) + (let ((line (read-line port))) + ;; Stop at the first 'Provides-Extra' section: the non-optional + ;; requirements appear before the optional ones. + (cond + ((eof-object? line) + (reverse (delete-duplicates requirements))) + ((and (requires-dist-header? line) (not (extra? line))) + (loop (cons (specification->requirement-name + (requires-dist-value line)) + requirements))) + (else + (loop requirements)))))))) =20 (define (guess-requirements source-url wheel-url archive) "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a list @@ -197,25 +224,18 @@ be extracted in a temporary directory." =20 (define (read-wheel-metadata wheel-archive) ;; Given WHEEL-ARCHIVE, a ZIP Python wheel archive, return the package= 's =2D ;; requirements. + ;; requirements, or #f if the metadata file contained therein couldn't= be + ;; extracted. (let* ((dirname (wheel-url->extracted-directory wheel-url)) =2D (json-file (string-append dirname "/metadata.json"))) =2D (and (zero? (system* "unzip" "-q" wheel-archive json-file)) =2D (dynamic-wind =2D (const #t) =2D (lambda () =2D (call-with-input-file json-file =2D (lambda (port) =2D (let* ((metadata (json->scm port)) =2D (run_requires (hash-ref metadata "run_requires= ")) =2D (requirements (if run_requires =2D (hash-ref (list-ref run_requ= ires 0) =2D "requires") =2D '()))) =2D (map specification->requirement-name requirements))= ))) =2D (lambda () =2D (delete-file json-file) =2D (rmdir dirname)))))) + (metadata (string-append dirname "/METADATA"))) + (call-with-temporary-directory + (lambda (dir) + (if (zero? (system* "unzip" "-q" wheel-archive "-d" dir metadata)) + (parse-wheel-metadata (string-append dir "/" metadata)) + (begin + (warning + (G_ "Failed to extract file: ~a from wheel.~%") metadata) + #f)))))) =20 (define (guess-requirements-from-wheel) ;; Return the package's requirements using the wheel, or #f if an error diff --git a/tests/pypi.scm b/tests/pypi.scm index b45d2c9d2f..8b42c2f071 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -22,6 +22,7 @@ #:use-module (guix base32) #:use-module (guix memoization) #:use-module (gcrypt hash) + #:use-module (guix memoization) #:use-module (guix tests) #:use-module (guix build-system python) #:use-module ((guix build utils) #:select (delete-file-recursively which= mkdir-p)) @@ -79,17 +80,33 @@ bar !=3D 2 pytest (>=3D2.5.0) ") =20 =2D(define test-metadata =2D "{ =2D \"run_requires\": [ =2D { =2D \"requires\": [ =2D \"bar\", =2D \"baz (>13.37)\" =2D ] =2D } =2D ] =2D}") +(define test-metadata "\ +Classifier: Programming Language :: Python :: 3.7 +Requires-Dist: baz ~=3D 3 +Requires-Dist: bar !=3D 2 +Provides-Extra: test +pytest (>=3D2.5.0) +") + +(define test-metadata-with-extras " +Classifier: Programming Language :: Python :: 3.7 +Requires-Python: >=3D2.7, !=3D3.0.*, !=3D3.1.*, !=3D3.2.*, !=3D3.3.* +Requires-Dist: wrapt (<2,>=3D1) +Requires-Dist: bar + +Provides-Extra: dev +Requires-Dist: tox ; extra =3D=3D 'dev' +Requires-Dist: bumpversion (<1) ; extra =3D=3D 'dev' +") + +;;; Provides-Extra can appear before Requires-Dist. +(define test-metadata-with-extras-jedi "\ +Requires-Python: >=3D2.7, !=3D3.0.*, !=3D3.1.*, !=3D3.2.*, !=3D3.3.* +Provides-Extra: testing +Requires-Dist: parso (>=3D0.3.0) +Provides-Extra: testing +Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testing' +") =20 (test-begin "pypi") =20 @@ -128,6 +145,18 @@ pytest (>=3D2.5.0) call-with-input-string) (parse-requires.txt test-requires-with-sections))) =20 +(test-equal "parse-wheel-metadata, with extras" + '("wrapt" "bar") + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-wheel-metadata test-metadata-with-extras))) + +(test-equal "parse-wheel-metadata, with extras - Jedi" + '("parso") + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-wheel-metadata test-metadata-with-extras-jedi))) + (test-assert "pypi->guix-package" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch @@ -191,7 +220,7 @@ pytest (>=3D2.5.0) (mkdir-p "foo-1.0.0/foo.egg-info/") (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" (lambda () =2D (display test-requires.txt))) + (display "wrong data to make sure we're testing wheel= s "))) (parameterize ((current-output-port (%make-void-port "rw+"= ))) (system* "tar" "czvf" file-name "foo-1.0.0/")) (delete-file-recursively "foo-1.0.0") @@ -200,13 +229,13 @@ pytest (>=3D2.5.0) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" (begin (mkdir "foo-1.0.0.dist-info") =2D (with-output-to-file "foo-1.0.0.dist-info/metadata.json" + (with-output-to-file "foo-1.0.0.dist-info/METADATA" (lambda () (display test-metadata))) (let ((zip-file (string-append file-name ".zip"))) ;; zip always adds a "zip" extension to the file it cre= ates, ;; so we need to rename it. =2D (system* "zip" zip-file "foo-1.0.0.dist-info/metadata= .json") + (system* "zip" zip-file "foo-1.0.0.dist-info/METADATA") (rename-file zip-file file-name)) (delete-file-recursively "foo-1.0.0.dist-info"))) (_ (error "Unexpected URL: " url))))) @@ -218,6 +247,9 @@ pytest (>=3D2.5.0) (string-length test-json))) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" #= f) (_ (error "Unexpected URL: " url))))) + ;; Not clearing the memoization cache here would mean return= ing the value + ;; computed in the previous test. + (invalidate-memoization! pypi->guix-package) (match (pypi->guix-package "foo") (('package ('name "python-foo") =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0008-import-pypi-Fix-typo-in-docstring.patch Content-Transfer-Encoding: quoted-printable From=2004e119d50d8622bf5bf8f1a6c6c80cd51e3e1658 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Wed, 12 Jun 2019 11:34:23 +0900 Subject: [PATCH 08/12] import: pypi: Fix typo in docstring. * guix/import/pypi.scm (guess-requirements): Fix typo. =2D-- guix/import/pypi.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 7cf1e92101..d861dd960d 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -218,7 +218,7 @@ requirement names." (loop requirements)))))))) =20 (define (guess-requirements source-url wheel-url archive) =2D "Given SOURCE-URL, WHEEL-URL and a ARCHIVE of the package, return a li= st + "Given SOURCE-URL, WHEEL-URL and an ARCHIVE of the package, return a list of the required packages specified in the requirements.txt file. ARCHIVE = will be extracted in a temporary directory." =20 =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0009-import-pypi-Completely-mute-the-output-of-the-unzip-.patch Content-Transfer-Encoding: quoted-printable From=20141551b4dae1f7d9920495eddfd85d49e3a45262 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Wed, 12 Jun 2019 11:36:39 +0900 Subject: [PATCH 09/12] import: pypi: Completely mute the output of the "unz= ip" command. * guix/import/pypi.scm (guess-requirements): Completely mute the output of = the "unzip" command. =2D-- guix/import/pypi.scm | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index d861dd960d..23a1e69061 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -230,7 +230,10 @@ be extracted in a temporary directory." (metadata (string-append dirname "/METADATA"))) (call-with-temporary-directory (lambda (dir) =2D (if (zero? (system* "unzip" "-q" wheel-archive "-d" dir metadat= a)) + (if (zero? + (parameterize ((current-error-port (%make-void-port "rw+")) + (current-output-port (%make-void-port "rw+"))) + (system* "unzip" wheel-archive "-d" dir metadata))) (parse-wheel-metadata (string-append dir "/" metadata)) (begin (warning =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: attachment; filename=0010-import-pypi-Include-optional-test-inputs-as-native-i.patch Content-Transfer-Encoding: quoted-printable From=20b435bb15a1905fee13fcee5423a6d2dc2300fa9a Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 28 Mar 2019 23:12:26 -0400 Subject: [PATCH 10/12] import: pypi: Include optional test inputs as native-inputs. * guix/import/pypi.scm (maybe-inputs): Add INPUT-TYPE argument, and use it. (test-section?): New predicate. (parse-requires.txt): Collect the optional test inputs, and return them as = the second element of the returned list. (parse-wheel-metadata): Likewise. (guess-requirements): Adapt. (make-pypi-sexp): Likewise, and include the test inputs requirements as nat= ive inputs in the returned package expression. * tests/pypi.scm (test-requires.txt): Include a test section in the test-requires.txt data. (test-requires.txt-beaker): New variable. ("parse-requires.txt"): Adapt. ("parse-requires.txt - Beaker"): New test. ("parse-wheel-metadata, with extras"): Adapt. ("parse-wheel-metadata, with extras - Jedi"): Adapt. ("pypi->guix-package, no wheel"): Re-indent, and add the expected native-inputs. ("pypi->guix-package, wheels"): Likewise. =2D-- guix/import/pypi.scm | 177 ++++++++++++++++++++++++++++--------------- tests/pypi.scm | 79 ++++++++++++------- 2 files changed, 166 insertions(+), 90 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 23a1e69061..537431dd69 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -4,6 +4,7 @@ ;;; Copyright =C2=A9 2015, 2016, 2017 Ludovic Court=C3=A8s ;;; Copyright =C2=A9 2017 Mathieu Othacehe ;;; Copyright =C2=A9 2018 Ricardo Wurmus +;;; Copyright =C2=A9 2019 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -26,6 +27,7 @@ #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) @@ -107,14 +109,15 @@ package on PyPI." ((name version _ ...) (string-append name "-" version ".dist-info")))) =20 =2D(define (maybe-inputs package-inputs) +(define (maybe-inputs package-inputs input-type) "Given a list of PACKAGE-INPUTS, tries to generate the 'inputs' field of= a =2Dpackage definition." +package definition. INPUT-TYPE, a symbol, is used to populate the name of +the input field." (match package-inputs (() '()) ((package-inputs ...) =2D `((propagated-inputs (,'quasiquote ,package-inputs)))))) + `((,input-type (,'quasiquote ,package-inputs)))))) =20 (define %requirement-name-regexp ;; Regexp to match the requirement name in a requirement specification. @@ -154,9 +157,19 @@ package definition." (or (regexp-exec %requirement-name-regexp spec) (error (G_ "Could not extract requirement name in spec:") spec)))) =20 +(define (test-section? name) + "Return #t if the section name contains 'test' or 'dev'." + (any (cut string-contains-ci name <>) + '("test" "dev"))) + (define (parse-requires.txt requires.txt) =2D "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of =2Drequirement names." + "Given REQUIRES.TXT, a Setuptools requires.txt file, return a list of li= sts +of requirements. + +The first list contains the required dependencies while the second the +optional test dependencies. Note that currently, optional, non-test +dependencies are omitted since these can be difficult or expensive to +satisfy." =20 (define (comment? line) ;; Return #t if the given LINE is a comment, #f otherwise. @@ -168,26 +181,49 @@ requirement names." =20 (call-with-input-file requires.txt (lambda (port) =2D (let loop ((result '())) + (let loop ((required-deps '()) + (test-deps '()) + (inside-test-section? #f) + (optional? #f)) (let ((line (read-line port))) =2D ;; Stop when a section is encountered, as sections contain opt= ional =2D ;; (extra) requirements. Non-optional requirements must appear =2D ;; before any section is defined. (cond =2D ((or (eof-object? line) (section-header? line)) + ((eof-object? line) ;; Duplicates can occur, since the same requirement can be ;; listed multiple times with different conditional markers, e= .g. ;; pytest >=3D 3 ; python_version >=3D "3.3" ;; pytest < 3 ; python_version < "3.3" =2D (reverse (delete-duplicates result))) + (map (compose reverse delete-duplicates) + (list required-deps test-deps))) ((or (string-null? line) (comment? line)) =2D (loop result)) =2D (else + (loop required-deps test-deps inside-test-section? optional?)) + ((section-header? line) + ;; Encountering a section means that all the requirements + ;; listed below are optional. Since we want to pick only the + ;; test dependencies from the optional dependencies, we must + ;; track those separately. + (loop required-deps test-deps (test-section? line) #t)) + (inside-test-section? + (loop required-deps + (cons (specification->requirement-name line) + test-deps) + inside-test-section? optional?)) + ((not optional?) (loop (cons (specification->requirement-name line) =2D result))))))))) + required-deps) + test-deps inside-test-section? optional?)) + (optional? + ;; Skip optional items. + (loop required-deps test-deps inside-test-section? optional?)) + (else + (warning (G_ "parse-requires.txt reached an unexpected \ +condition on line ~a~%") line)))))))) =20 (define (parse-wheel-metadata metadata) =2D "Given METADATA, a Wheel metadata file, return a list of requirement n= ames." + "Given METADATA, a Wheel metadata file, return a list of lists of +requirements. + +Refer to the documentation of PARSE-REQUIRES.TXT for a description of the +returned value." ;; METADATA is a RFC-2822-like, header based file. =20 (define (requires-dist-header? line) @@ -201,21 +237,29 @@ requirement names." ;; Return #t if the given LINE is an "extra" requirement. (string-match "extra =3D=3D '(.*)'" line)) =20 + (define (test-requirement? line) + (and=3D> (match:substring (extra? line) 1) test-section?)) + (call-with-input-file metadata (lambda (port) =2D (let loop ((requirements '())) + (let loop ((required-deps '()) + (test-deps '())) (let ((line (read-line port))) =2D ;; Stop at the first 'Provides-Extra' section: the non-optional =2D ;; requirements appear before the optional ones. (cond ((eof-object? line) =2D (reverse (delete-duplicates requirements))) + (map (compose reverse delete-duplicates) + (list required-deps test-deps))) ((and (requires-dist-header? line) (not (extra? line))) (loop (cons (specification->requirement-name (requires-dist-value line)) =2D requirements))) + required-deps) + test-deps)) + ((and (requires-dist-header? line) (test-requirement? line)) + (loop required-deps + (cons (specification->requirement-name (requires-dist-va= lue line)) + test-deps))) (else =2D (loop requirements)))))))) + (loop required-deps test-deps)))))))) ;skip line =20 (define (guess-requirements source-url wheel-url archive) "Given SOURCE-URL, WHEEL-URL and an ARCHIVE of the package, return a list @@ -268,37 +312,46 @@ be extracted in a temporary directory." (() (warning (G_ "Cannot guess requirements from source archiv= e:\ no requires.txt file found.~%")) =2D '()) + (list '() '())) (else (parse-requires.txt (first requires.txt-files))))))) (begin (warning (G_ "Unsupported archive format; \ cannot determine package dependencies from source archive: ~a~%") (basename source-url)) =2D '()))) + (list '() '())))) =20 ;; First, try to compute the requirements using the wheel, else, fallbac= k to ;; reading the "requires.txt" from the egg-info directory from the source =2D ;; tarball. + ;; archive. (or (guess-requirements-from-wheel) (guess-requirements-from-source))) =20 (define (compute-inputs source-url wheel-url archive) =2D "Given the SOURCE-URL of an already downloaded ARCHIVE, return a list = of =2Dname/variable pairs describing the required inputs of this package. Also + "Given the SOURCE-URL and WHEEL-URL of an already downloaded ARCHIVE, re= turn +a pair of lists, each consisting of a list of name/variable pairs, for the +propagated inputs and the native inputs, respectively. Also return the unaltered list of upstream dependency names." =2D (let ((dependencies =2D (remove (cut string=3D? "argparse" <>) =2D (guess-requirements source-url wheel-url archive)))) =2D (values (sort =2D (map (lambda (input) =2D (let ((guix-name (python->package-name input))) =2D (list guix-name (list 'unquote (string->symbol gui= x-name))))) =2D dependencies) =2D (lambda args =2D (match args =2D (((a _ ...) (b _ ...)) =2D (string-ci) deps)) + + (define (requirement->package-name/sort deps) + (sort + (map (lambda (input) + (let ((guix-name (python->package-name input))) + (list guix-name (list 'unquote (string->symbol guix-name))))) + deps) + (lambda args + (match args + (((a _ ...) (b _ ...)) + (string-cipackage-name/sort strip-argparse)) + + (let ((dependencies (guess-requirements source-url wheel-url archive))) + (values (map process-requirements dependencies) + (concatenate dependencies)))) =20 (define (make-pypi-sexp name version source-url wheel-url home-page synops= is description license) @@ -307,29 +360,31 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION= , and LICENSE." (call-with-temporary-output-file (lambda (temp port) (and (url-fetch source-url temp) =2D (receive (input-package-names upstream-dependency-names) + (receive (guix-dependencies upstream-dependencies) (compute-inputs source-url wheel-url temp) =2D (values =2D `(package =2D (name ,(python->package-name name)) =2D (version ,version) =2D (source (origin =2D (method url-fetch) =2D =2D ;; Sometimes 'pypi-uri' doesn't quite work due= to mixed =2D ;; cases in NAME, for instance, as is the case= with =2D ;; "uwsgi". In that case, fall back to a full= URL. =2D (uri (pypi-uri ,(string-downcase name) version= )) =2D (sha256 =2D (base32 =2D ,(guix-hash-url temp))))) =2D (build-system python-build-system) =2D ,@(maybe-inputs input-package-names) =2D (home-page ,home-page) =2D (synopsis ,synopsis) =2D (description ,description) =2D (license ,(license->symbol license))) =2D upstream-dependency-names)))))) + (match guix-dependencies + ((required-inputs test-inputs) + (values + `(package + (name ,(python->package-name name)) + (version ,version) + (source (origin + (method url-fetch) + ;; Sometimes 'pypi-uri' doesn't quite work du= e to mixed + ;; cases in NAME, for instance, as is the cas= e with + ;; "uwsgi". In that case, fall back to a ful= l URL. + (uri (pypi-uri ,(string-downcase name) versio= n)) + (sha256 + (base32 + ,(guix-hash-url temp))))) + (build-system python-build-system) + ,@(maybe-inputs required-inputs 'propagated-inputs) + ,@(maybe-inputs test-inputs 'native-inputs) + (home-page ,home-page) + (synopsis ,synopsis) + (description ,description) + (license ,(license->symbol license))) + upstream-dependencies)))))))) =20 (define pypi->guix-package (memoize diff --git a/tests/pypi.scm b/tests/pypi.scm index 8b42c2f071..43d45f1dd8 100644 =2D-- a/tests/pypi.scm +++ b/tests/pypi.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2014 David Thompson ;;; Copyright =C2=A9 2016 Ricardo Wurmus +;;; Copyright =C2=A9 2019 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -68,11 +69,6 @@ sha1=3Dda9234ee9982d4bbb3c72346a6de940a148ea686")) (define test-requires.txt "\ # A comment # A comment after a space =2Dbar =2Dbaz > 13.37 =2D") =2D =2D(define test-requires-with-sections "\ foo ~=3D 3 bar !=3D 2 =20 @@ -80,12 +76,25 @@ bar !=3D 2 pytest (>=3D2.5.0) ") =20 +;; Beaker contains only optional dependencies. +(define test-requires.txt-beaker "\ +[crypto] +pycryptopp>=3D0.5.12 + +[cryptography] +cryptography + +[testsuite] +Mock +coverage +") + (define test-metadata "\ Classifier: Programming Language :: Python :: 3.7 Requires-Dist: baz ~=3D 3 Requires-Dist: bar !=3D 2 Provides-Extra: test =2Dpytest (>=3D2.5.0) +Requires-Dist: pytest (>=3D2.5.0) ; extra =3D=3D 'test' ") =20 (define test-metadata-with-extras " @@ -139,25 +148,31 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'test= ing' '("Fizzy" "PickyThing" "SomethingWithMarker" "requests" "pip") (map specification->requirement-name test-specifications)) =20 =2D(test-equal "parse-requires.txt, with sections" =2D '("foo" "bar") +(test-equal "parse-requires.txt" + (list '("foo" "bar") '("pytest")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) =2D (parse-requires.txt test-requires-with-sections))) + (parse-requires.txt test-requires.txt))) + +(test-equal "parse-requires.txt - Beaker" + (list '() '("Mock" "coverage")) + (mock ((ice-9 ports) call-with-input-file + call-with-input-string) + (parse-requires.txt test-requires.txt-beaker))) =20 (test-equal "parse-wheel-metadata, with extras" =2D '("wrapt" "bar") + (list '("wrapt" "bar") '("tox" "bumpversion")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) (parse-wheel-metadata test-metadata-with-extras))) =20 (test-equal "parse-wheel-metadata, with extras - Jedi" =2D '("parso") + (list '("parso") '("pytest")) (mock ((ice-9 ports) call-with-input-file call-with-input-string) (parse-wheel-metadata test-metadata-with-extras-jedi))) =20 =2D(test-assert "pypi->guix-package" +(test-assert "pypi->guix-package, no wheel" ;; Replace network resources with sample data. (mock ((guix import utils) url-fetch (lambda (url file-name) @@ -198,7 +213,10 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testi= ng' ('propagated-inputs ('quasiquote (("python-bar" ('unquote 'python-bar)) =2D ("python-baz" ('unquote 'python-baz))))) + ("python-foo" ('unquote 'python-foo))))) + ('native-inputs + ('quasiquote + (("python-pytest" ('unquote 'python-pytest))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary") @@ -219,25 +237,25 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'test= ing' (begin (mkdir-p "foo-1.0.0/foo.egg-info/") (with-output-to-file "foo-1.0.0/foo.egg-info/requires.txt" =2D (lambda () =2D (display "wrong data to make sure we're testing whe= els "))) + (lambda () + (display "wrong data to make sure we're testing wheels= "))) (parameterize ((current-output-port (%make-void-port "rw+"= ))) (system* "tar" "czvf" file-name "foo-1.0.0/")) =2D (delete-file-recursively "foo-1.0.0") =2D (set! test-source-hash =2D (call-with-input-file file-name port-sha256)))) + (delete-file-recursively "foo-1.0.0") + (set! test-source-hash + (call-with-input-file file-name port-sha256)))) ("https://example.com/foo-1.0.0-py2.py3-none-any.whl" =2D (begin =2D (mkdir "foo-1.0.0.dist-info") =2D (with-output-to-file "foo-1.0.0.dist-info/METADATA" =2D (lambda () =2D (display test-metadata))) =2D (let ((zip-file (string-append file-name ".zip"))) =2D ;; zip always adds a "zip" extension to the file it c= reates, =2D ;; so we need to rename it. =2D (system* "zip" zip-file "foo-1.0.0.dist-info/METADATA= ") =2D (rename-file zip-file file-name)) =2D (delete-file-recursively "foo-1.0.0.dist-info"))) + (begin + (mkdir "foo-1.0.0.dist-info") + (with-output-to-file "foo-1.0.0.dist-info/METADATA" + (lambda () + (display test-metadata))) + (let ((zip-file (string-append file-name ".zip"))) + ;; zip always adds a "zip" extension to the file it crea= tes, + ;; so we need to rename it. + (system* "zip" "-q" zip-file "foo-1.0.0.dist-info/METADA= TA") + (rename-file zip-file file-name)) + (delete-file-recursively "foo-1.0.0.dist-info"))) (_ (error "Unexpected URL: " url))))) (mock ((guix http-client) http-fetch (lambda (url . rest) @@ -265,6 +283,9 @@ Requires-Dist: pytest (>=3D3.1.0); extra =3D=3D 'testin= g' ('quasiquote (("python-bar" ('unquote 'python-bar)) ("python-baz" ('unquote 'python-baz))))) + ('native-inputs + ('quasiquote + (("python-pytest" ('unquote 'python-pytest))))) ('home-page "http://example.com") ('synopsis "summary") ('description "summary") =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0011-import-pypi-Update-the-host-URI.patch Content-Transfer-Encoding: quoted-printable From=20736e91e7c1d9d7e7c6f93342766fdcd4b78cbfa1 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Sun, 16 Jun 2019 14:52:25 +0900 Subject: [PATCH 11/12] import: pypi: Update the host URI. * guix/build-system/python.scm (pypi-uri): Update the host URI to "files.pythonhosted.org". =2D-- guix/build-system/python.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guix/build-system/python.scm b/guix/build-system/python.scm index b753940bad..e39c06528e 100644 =2D-- a/guix/build-system/python.scm +++ b/guix/build-system/python.scm @@ -50,7 +50,7 @@ "Return a URI string for the Python package hosted on the Python Package Index (PyPI) corresponding to NAME and VERSION. EXTENSION is the file name extension, such as '.tar.gz'." =2D (string-append "https://pypi.org/packages/source/" + (string-append "https://files.pythonhosted.org/packages/source/" (string-take name 1) "/" name "/" name "-" version extension)) =20 =2D-=20 2.21.0 --=-=-= Content-Type: text/x-patch Content-Disposition: attachment; filename=0012-import-pypi-Preserve-package-name-case-when-forming-.patch Content-Transfer-Encoding: quoted-printable From=205373f50088cd0d37e1dbc64e6ca505224a452d8e Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Sat, 30 Mar 2019 20:27:35 -0400 Subject: [PATCH 12/12] import: pypi: Preserve package name case when forming pypi-uri. Fixes . * guix/build-system/python.scm (pypi-uri): Update the host URI to "files.pythonhosted.org". * guix/import/pypi.scm (make-pypi-sexp): Preserve the package name case when the source URL calls for it. =2D-- guix/import/pypi.scm | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/guix/import/pypi.scm b/guix/import/pypi.scm index 537431dd69..ab7a024ee0 100644 =2D-- a/guix/import/pypi.scm +++ b/guix/import/pypi.scm @@ -368,15 +368,20 @@ VERSION, SOURCE-URL, HOME-PAGE, SYNOPSIS, DESCRIPTION= , and LICENSE." `(package (name ,(python->package-name name)) (version ,version) =2D (source (origin =2D (method url-fetch) =2D ;; Sometimes 'pypi-uri' doesn't quite work = due to mixed =2D ;; cases in NAME, for instance, as is the c= ase with =2D ;; "uwsgi". In that case, fall back to a f= ull URL. =2D (uri (pypi-uri ,(string-downcase name) vers= ion)) =2D (sha256 =2D (base32 =2D ,(guix-hash-url temp))))) + (source + (origin + (method url-fetch) + ;; PyPI URL are case sensitive, but sometimes a proj= ect + ;; named using mixed case has a URL using lower case= , so + ;; we must work around this inconsistency. For actu= al + ;; examples, compare the URLs of the "Deprecated" and + ;; "uWSGI" PyPI packages. + (uri ,(if (string-contains source-url name) + `(pypi-uri ,name version) + `(pypi-uri ,(string-downcase name) version= ))) + (sha256 + (base32 + ,(guix-hash-url temp))))) (build-system python-build-system) ,@(maybe-inputs required-inputs 'propagated-inputs) ,@(maybe-inputs test-inputs 'native-inputs) =2D-=20 2.21.0 --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEJ9WGpPiQCFQyn/CfEmDkZILmNWIFAl0GVAEACgkQEmDkZILm NWK5gQ/+KD7p1vIzRmPVotqZM9qXkBUjJlw+9LMwYwhCbdTM39+/inUh/vMO8j/9 3Qxtl0eVS35tpwY4Te3MrOhRYxjrhxhl7SuRzJr/UV/coA8Tm2C3SgKpnb2Vklp5 DBiwYcEuejdQMXOwLXZwdpUtx7zx8Ygu2fwlFait5jr4+psurhWw8Anb4wtW10Li ljBSOxiWBKkcUW5U9E9I/exelHTBWuFFcUU60ByQA9FLDu6/K8kFPIBCKOiKgDjK d/I1Kt5gSZ37qORbE/ITdFrodnp6+7Ty6NGwxLa18WQ6OO/bqdhDrjqt0jC47dWs mIQUrTclbwVSE/PCaDwBOQhcldKVPOFRFfVlKafq9xMa6uXnLgPOzbBIwnF0KgfH ylyCY13x/EZF7RcZE6hxj55LQY2koYCS0DmzU1ZBVxx9TETkGj+TXO9kfBA1oSkq FO26S/GXLF4yTUmRRrcwsd/y23TrV28RfrRfEQmTwZircbmz4sgeI1XmthKAcbOq k0joF4rfoU8FLpWyxOxJvzgk06FHUb4P9AQNyD+4zu1xWZ5f3IcSOfH8L4T7qfr6 zFhlQ46AG2xfdX9130M5c9bOVBm9iCf1smg6y+kn/yE1ZypiRVOlLQ8FR/TkNMEJ ysjyiBxdYcXLfR6KPAz+gjnoVyxqGg9F1mR1/h6T5u7G+sAsl5o= =zSir -----END PGP SIGNATURE----- --==-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 16 13:03:09 2019 Received: (at 24450) by debbugs.gnu.org; 16 Jun 2019 17:03:09 +0000 Received: from localhost ([127.0.0.1]:41342 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcYYf-0004Wz-3X for submit@debbugs.gnu.org; Sun, 16 Jun 2019 13:03:09 -0400 Received: from aibo.runbox.com ([91.220.196.211]:34148) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcYYc-0004Wq-Be for 24450@debbugs.gnu.org; Sun, 16 Jun 2019 13:03:07 -0400 Received: from [10.9.9.210] (helo=mailfront10.runbox.com) by mailtransmit02.runbox with esmtp (Exim 4.86_2) (envelope-from ) id 1hcYYa-00073P-Mu; Sun, 16 Jun 2019 19:03:04 +0200 Received: by mailfront10.runbox.com with esmtpsa (uid:892961 ) (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.82) id 1hcYY8-0002bF-Dr; Sun, 16 Jun 2019 19:02:36 +0200 Date: Sun, 16 Jun 2019 17:02:35 +0000 From: ng0 To: Maxim Cournoyer Subject: Re: bug#24450: pypi importer outputs strange character series in optional dependency case. Message-ID: <20190616170137.fdyj3bwevrfywr4r@uptimegirl> References: <87h99fipj1.fsf@we.make.ritual.n0.is> <87imw22tqf.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <87imw22tqf.fsf@gmail.com> X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: ng0 , 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Maxim, great to see this fixed. Weird it bumped into my view after almost 3 years. I trust that other people will have tested it, I'm all over the place and mostly doing NetBSD and GNUnet now. Sorry if I have written a reply before, the thread was marked as unread here. Maxim Cournoyer transcribed 3.5K bytes: > ng0 writes: > > > I think this should not happen with pypi import: > > > > (inputs > > `(("python-certifi==2016.2.28" > > ,python-certifi==2016.2.28) > > ("python-dateutil==2.5.3" > > ,python-dateutil==2.5.3) > > ("python-flask-babel==0.11.1" > > ,python-flask-babel==0.11.1) > > ("python-flask==0.11.1" ,python-flask==0.11.1) > > ("python-lxml==3.6.0" ,python-lxml==3.6.0) > > ("python-ndg-httpsclient==0.4.1" > > ,python-ndg-httpsclient==0.4.1) > > ("python-pyasn1-modules==0.0.8" > > ,python-pyasn1-modules==0.0.8) > > ("python-pyasn1==0.1.9" ,python-pyasn1==0.1.9) > > ("python-pygments==2.1.3" > > ,python-pygments==2.1.3) > > ("python-pyopenssl==0.15.1" > > ,python-pyopenssl==0.15.1) > > ("python-pyyaml==3.11" ,python-pyyaml==3.11) > > ("python-requests[socks]==2.10.0" > > ,#{python-requests\x5b;socks\x5d;==2.10.0}#) > > ("python-setuptools" ,python-setuptools))) > > > > > > I can understand the version numbers, I can also understand the optional > > socks building/module of the python-requests, but why does it read like > > Gobbledygook? Can't we improve the output here? > > > > For version numbers, this is not a format which happened recently which > > is exclusive for python build system right? This is just bad formated > > because of the pypi query. > > I will first try and not pin the application to these version numbers, > > maybe itjustworks™. > > > > > > To reproduce: "guix import pypi searx" > > This would now give (change to be sent for review soon): > > --8<---------------cut here---------------start------------->8--- > ./pre-inst-env guix import pypi searx > > Starting download of /tmp/guix-file.1wD8K4 > From https://files.pythonhosted.org/packages/75/3f/5941ad2d500ff7cf6f8da1022c78013dcd2207941d533586a8e7bfe699d3/searx-0.15.0.tar.gz... > …5.0.tar.gz 1.6MiB 729KiB/s 00:02 [##################] 100.0% > (package > (name "python-searx") > (version "0.15.0") > (source > (origin > (method url-fetch) > (uri (pypi-uri "searx" version)) > (sha256 > (base32 > "1gmww73q7wydkvlyz73wnr3sybpjn40wha7avnz9ak9m365zcjxf")))) > (build-system python-build-system) > (propagated-inputs > `(("python-certifi" ,python-certifi) > ("python-dateutil" ,python-dateutil) > ("python-flask" ,python-flask) > ("python-flask-babel" ,python-flask-babel) > ("python-idna" ,python-idna) > ("python-lxml" ,python-lxml) > ("python-pygments" ,python-pygments) > ("python-pyopenssl" ,python-pyopenssl) > ("python-pyyaml" ,python-pyyaml) > ("python-requests" ,python-requests))) > (native-inputs > `(("python-babel" ,python-babel) > ("python-cov-core" ,python-cov-core) > ("python-mock" ,python-mock) > ("python-nose2" ,python-nose2) > ("python-pep8" ,python-pep8) > ("python-plone.testing" ,python-plone.testing) > ("python-selenium" ,python-selenium) > ("python-splinter" ,python-splinter) > ("python-transifex-client" > ,python-transifex-client) > ("python-unittest2" ,python-unittest2) > ("python-zope.testrunner" > ,python-zope.testrunner))) > (home-page "https://github.com/asciimoo/searx") > (synopsis > "A privacy-respecting, hackable metasearch engine") > (description > "A privacy-respecting, hackable metasearch engine") > (license #f)) > --8<---------------cut here---------------end--------------->8--- > > > > From debbugs-submit-bounces@debbugs.gnu.org Sun Jun 16 21:41:28 2019 Received: (at 24450) by debbugs.gnu.org; 17 Jun 2019 01:41:28 +0000 Received: from localhost ([127.0.0.1]:41749 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcgeG-0006GI-8q for submit@debbugs.gnu.org; Sun, 16 Jun 2019 21:41:28 -0400 Received: from b2062.mx.srv.dfn.de ([194.95.234.172]:40229) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hcgeD-0006G6-7k for 24450@debbugs.gnu.org; Sun, 16 Jun 2019 21:41:26 -0400 Received: from localhost (localhost [127.0.0.1]) by b2062.mx.srv.dfn.de (Postfix) with ESMTP id 0D04B16005D; Mon, 17 Jun 2019 03:41:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=mdc-berlin.de; h= content-transfer-encoding:content-type:content-type:mime-version :message-id:date:date:in-reply-to:subject:subject:from:from :user-agent:references:received:received:received; s=mdc; t= 1560735678; x=1562550079; bh=rYw5RB+ky8e9iRvYYEKQzbX3Qprnf3mosXq XOU4DEgc=; b=RbEha0nX/jh6cmLEAUTdpLSjyhJZELIZj9435uGaC09935Kc165 bfrNdHRsYsOEgZAZrGnR+slX9Q6TPylZmEVut1Wcr/OJrmM3m+TmjpW1OWjsusfc pLas9E61cyosXlD+vgD/wiokR9ytIYwkJOR/hx5JXSck9MvlS7KHVyWY= Received: from b2062.mx.srv.dfn.de ([127.0.0.1]) by localhost (mgw4-tub.srv.dfn.de [127.0.0.1]) (amavisd-new, port 20134) with ESMTP id 86im6bWvFmHr; Mon, 17 Jun 2019 03:41:18 +0200 (CEST) Received: from SW-IT-P-CAS4.mdc-berlin.net (mgw10-4.mdc-berlin.de [141.80.113.59]) by b2062.mx.srv.dfn.de (Postfix) with ESMTPS; Mon, 17 Jun 2019 03:41:18 +0200 (CEST) Received: from localhost (84.173.77.188) by SW-IT-P-CAS4.mdc-berlin.net (141.80.113.59) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 17 Jun 2019 03:41:17 +0200 References: <87pnod7ot4.fsf@gmail.com> <87muiq5d7c.fsf@gmail.com> <87blz5dcap.fsf@mdc-berlin.de> <87sgs91uyy.fsf@gmail.com> User-agent: mu4e 1.2.0; emacs 26.2 From: Ricardo Wurmus To: Maxim Cournoyer Subject: Re: [PATCHv2] Re: pypi importer outputs strange character series in optional dependency case. In-Reply-To: <87sgs91uyy.fsf@gmail.com> X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Mon, 17 Jun 2019 03:41:16 +0200 Message-ID: <87k1dlouo3.fsf@mdc-berlin.de> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [84.173.77.188] X-TM-AS-Product-Ver: SMEX-12.5.0.1684-8.5.1010-24684.004 X-TM-AS-Result: No-2.940900-8.000000-10 X-TMASE-MatchedRID: O/y65JfDwwvOH9fyebH9T/HkpkyUphL9H7bpDOhZpjaRjx4hNpIk+CMn HClP1IKbWpmBc57aF+H5rnWvqSlt8+DocHyqS9HwAI0UpQvEYJn8SjXSxldPUwp+tuYb4NtQMkM 5PjR2vf20mkcodhQTcmLAc8HDb+/qSSOWVJeuO1CDGx/OQ1GV8khax4LkeV5t+gtHj7OwNO33FL eZXNZS4IzHo47z5Aa+JNCAsTF1JwFnBLaBbfNHM5ou+ffm85rV9RO0ppKBZpcl8qgGTx82BBrXK nAUyZduhyQllUUFsbKCw3tGTWERov+EpZ3dS9hVkX579PzRktSCO7HU4V6gix4khEYMlUzT2X7s U86hby5WXGvUUmKP2w== X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-TMASE-Result: 10--2.940900-8.000000 X-TMASE-Version: SMEX-12.5.0.1684-8.5.1010-24684.004 X-TM-SNTS-SMTP: 7CB6FB5FC6FFA35BF15DC540A22D2D6718FED99F20C6E3276C1693A1A6B914442000:9 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Maxim Cournoyer writes: >> This allows you to match =E2=80=9Ceof-object=E2=80=9D and '() directly. = Whenever I see >> =E2=80=9Cstring-null?=E2=80=9D I think it might be better to =E2=80=9Cma= tch=E2=80=9D on the empty list >> directly. > > string-null? and an empty list are not the same, unless I'm missing somet= hing. Yes, sorry, I meant =E2=80=9Cnull?=E2=80=9D. Using =E2=80=9Cstring-null?= =E2=80=9D is equivalent to matching the empty string, of course. >> But really, that=E2=80=99s up to you. I only feel strongly about avoidi= ng =E2=80=9C(if >> =E2=80=A6 (cond =E2=80=A6))=E2=80=9D. > > Due to the problem mentioned above, I stayed with "cond". Okay! Thanks. --=20 Ricardo From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 26 00:12:24 2019 Received: (at 24450) by debbugs.gnu.org; 26 Jun 2019 04:12:24 +0000 Received: from localhost ([127.0.0.1]:34071 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hfzIF-0006rD-VB for submit@debbugs.gnu.org; Wed, 26 Jun 2019 00:12:24 -0400 Received: from mail-pl1-f175.google.com ([209.85.214.175]:45506) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hfzIE-0006qy-OI for 24450@debbugs.gnu.org; Wed, 26 Jun 2019 00:12:23 -0400 Received: by mail-pl1-f175.google.com with SMTP id bi6so633255plb.12 for <24450@debbugs.gnu.org>; Tue, 25 Jun 2019 21:12:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=FpGyrHGindws9Eod7wJzfQeNc2OHCp6HXR0CRG6CYLA=; b=HuM+BFDbZhfnPRFaJfByIm8O4zbwi61aEqEcQeppP26ujSNjqJetk10Bfh520IHy6g nAP8pJ3hszT2LJJDKWz2hxm1U293AospI4VNTpwcQz6p+B29yoChOL3FdqVEOLQSPeou 1SI0QnZ8LuzHZWmQ2cMl1B1dFzr7KlBV35b8mUHTJtvdSMa9OrMfT8vhtrz6niNyXdD2 UZiuTVSKCKeziFNYbHXtZkLmznFXuhOeq9NsVeFzP3uQ6wisiomVdNgbx3rU2pq6P1O6 IaJuwyDhAazTzVk+CAA90h/P+r0kGF57O4yUIfo0gg39KHsFfj0VXzlhCWFSHzWqzlZM FJDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=FpGyrHGindws9Eod7wJzfQeNc2OHCp6HXR0CRG6CYLA=; b=QsDko0blxEk7jjhu795PA55NgyQeFLHqYwK7RFgkp0Ck67SIL62L2bFp4hhEtgseZW 8gI4x76I1SJZTgcj3G8OGkNWizexUObYNPghSTmDuVWFmAG4fyF6Ny6tnncMYbbU1zNa Iv3Oqou7LnLfJkjIJss4Ew84NFPVmpKzHfEUMmaPL2N/AaBnq8e7vJZImusNleHPcraF 1pgrhMaGn8p4P1QJfFFEdZKotcDEOS6/YEBhSxBvyOAR6Juswre88l4avCZ4FOb7ppJz LCavNI0yxOtWrrFxDJk0heCU4W4HWD8VHFhep0AcPco/FtIx8F40ACRESlau0rzMLFQ1 8F9g== X-Gm-Message-State: APjAAAWsas67xZmWQx4MFTu09+e+FcJBFl781R0Dk58uZNWpJfc+CyoW TsJhGepnD6UtqrgSAzD3FXAm5yGdagA= X-Google-Smtp-Source: APXvYqwhtA8L/c2i5cSxizn87VOfqTwAjn8exN1PgoJ9cqoZk4v1CE+nFsO3Ogwtf1tAJF1oltbC/g== X-Received: by 2002:a17:902:2ae6:: with SMTP id j93mr2832853plb.130.1561522336682; Tue, 25 Jun 2019 21:12:16 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id d19sm492162pjs.22.2019.06.25.21.12.15 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Tue, 25 Jun 2019 21:12:16 -0700 (PDT) From: Maxim Cournoyer To: ng0 Subject: Re: bug#24450: pypi importer outputs strange character series in optional dependency case. References: <87h99fipj1.fsf@we.make.ritual.n0.is> <87imw22tqf.fsf@gmail.com> <20190616170137.fdyj3bwevrfywr4r@uptimegirl> Date: Wed, 26 Jun 2019 13:12:11 +0900 In-Reply-To: <20190616170137.fdyj3bwevrfywr4r@uptimegirl> (ng0@we.make.ritual.n0.is's message of "Sun, 16 Jun 2019 17:02:35 +0000") Message-ID: <87woh9astg.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450 Cc: 24450@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hey ng0! ng0 writes: > Hi Maxim, > > great to see this fixed. Weird it bumped into my view after > almost 3 years. Yes! Bugs don't really expire, do they? :-) > I trust that other people will have tested > it, I'm all over the place and mostly doing NetBSD and GNUnet > now. I'm glad to know you're still involved with GNUnet :-) > Sorry if I have written a reply before, the thread was > marked as unread here. No problem! I'll wait a few more days to allow for commenting some more, then push. Cheers! Maxim From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 01 21:54:45 2019 Received: (at 24450-done) by debbugs.gnu.org; 2 Jul 2019 01:54:45 +0000 Received: from localhost ([127.0.0.1]:46368 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hi80L-0007EQ-1F for submit@debbugs.gnu.org; Mon, 01 Jul 2019 21:54:45 -0400 Received: from mail-pl1-f171.google.com ([209.85.214.171]:37571) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hi80H-0007E3-G6 for 24450-done@debbugs.gnu.org; Mon, 01 Jul 2019 21:54:44 -0400 Received: by mail-pl1-f171.google.com with SMTP id bh12so8247942plb.4 for <24450-done@debbugs.gnu.org>; Mon, 01 Jul 2019 18:54:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=uXnSRX3aEJB0PeiZLVyI8xzoUMGtEpvFKF78E0eSKUk=; b=RyO8rdIBnRrKkFunL0K32uHtSeJOnMyb/zWW2aAebQMeV+nms8SJMsxpsJ/GeSsCRi 164NaOD6pCbmBU3VKUC2b93Q951mSprp5vSOUfN8sX29xAx6N+pFm/orEtNTHMH8WD5k T1U+P+EA86ppYPXanmwDaQqxjMd9kU3ew/AxHWzLD0+rDIPzslcL4wUlewd2/NbmNJXH 3YqgDZqc1Fg7PwPtTool3XaFvynEsR/HYrJ5rVGIhnCUykJ76U+wsTmusl26CW2sEKwE 33s6x1cZmM207zIBSb2PtYqgVyuWWOIm4zimg13kxWIuhv4jeYDQHD7AXsqE2X2o4uuK MH7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=uXnSRX3aEJB0PeiZLVyI8xzoUMGtEpvFKF78E0eSKUk=; b=MPoqhS+qSVES9S+UDJ14A11qq4mI5K056ospcjvOPx/WS+aqyyME5sRUQOOTov2hcw 9TxHRxsuAEgQX4TCzKV6D905jTCEVVdAIoBjQWRCv7hU18NK/kYRwuHQ4+vF8YD9g+Ra pjzHHC/nfX9jhugw0lZ6AkcGIpnQzzYGhEk6QEXpRCAnggGyPz7PZQ0kmddjxjfvwvRK sUwQMnOumWGm7CqDthQbl8dgGPPW7FJOGtcXl6ogXZ6bNs5zStA7eWwjXBsN+W28p6vU biF7gmNMmIp09mWc8/1YNC73SrvoKNncPyxYDEJVgIBlHwhezCt/oj8Hfrt/iOVFRfds Hzgw== X-Gm-Message-State: APjAAAXnS1C0FmWPh3idDMbz9ZFw7GEPIj7UxuHI58OgIA9gjzW+aaAb SZGt2wUizxrFWiQKovuUKnsxr5X17zU= X-Google-Smtp-Source: APXvYqzwxEI8XPY1I/JdZ5vJCmxpbgD5mo14n5oj3v0U+3nxhYh3lxHY7pQ8CfEKz7foB7qKoq9Y4g== X-Received: by 2002:a17:902:6a2:: with SMTP id 31mr25134403plh.296.1562032475339; Mon, 01 Jul 2019 18:54:35 -0700 (PDT) Received: from kwak ([240f:c7:38e9:1:766b:a43b:9222:c603]) by smtp.gmail.com with ESMTPSA id u75sm13876431pgb.92.2019.07.01.18.54.33 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 01 Jul 2019 18:54:34 -0700 (PDT) From: Maxim Cournoyer To: Ricardo Wurmus Subject: Re: bug#24450: [PATCHv3] Re: pypi importer outputs strange character series in optional dependency case. References: <87imtb33tp.fsf@gmail.com> <87imtb9ujz.fsf@mdc-berlin.de> <87o92x1u3d.fsf@gmail.com> <87pnndzjem.fsf_-_@gmail.com> Date: Tue, 02 Jul 2019 10:54:30 +0900 In-Reply-To: <87pnndzjem.fsf_-_@gmail.com> (Maxim Cournoyer's message of "Sun, 16 Jun 2019 23:36:49 +0900") Message-ID: <87tvc5w695.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24450-done Cc: 24450-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Maxim Cournoyer writes: > Here's the current patch set, version 3, with the modifications > as discussed in the previous exchanges [...] I've now merged this patch series (version 3) into master, with top commit 4b60ab8c006964d026dee8cf5f1260eba0b2bb81. Closing. Thank you! Maxim From unknown Mon Aug 11 18:18:55 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Tue, 30 Jul 2019 11:24:05 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator