From unknown Wed Aug 20 03:37:39 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#49958] [PATCH] More flexibility in opam importer Resent-From: Alice BRENON Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 09 Aug 2021 15:04:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 49958 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 49958@debbugs.gnu.org X-Debbugs-Original-To: guix-patches@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.16285213978219 (code B ref -1); Mon, 09 Aug 2021 15:04:02 +0000 Received: (at submit) by debbugs.gnu.org; 9 Aug 2021 15:03:17 +0000 Received: from localhost ([127.0.0.1]:57477 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mD6o7-00028S-OY for submit@debbugs.gnu.org; Mon, 09 Aug 2021 11:03:17 -0400 Received: from lists.gnu.org ([209.51.188.17]:52684) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mD40t-0001HO-Fm for submit@debbugs.gnu.org; Mon, 09 Aug 2021 08:04:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49860) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mD40s-0006OB-No for guix-patches@gnu.org; Mon, 09 Aug 2021 08:04:14 -0400 Received: from lxc-smtp2.ens-lyon.fr ([140.77.167.81]:51116) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mD40p-0001ra-E5 for guix-patches@gnu.org; Mon, 09 Aug 2021 08:04:14 -0400 Received: from localhost (localhost [127.0.0.1]) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTP id 08232E28D6 for ; Mon, 9 Aug 2021 14:04:08 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.11.0 (20160426) (Debian) at ens-lyon.fr Received: from lxc-smtp2.ens-lyon.fr ([127.0.0.1]) by localhost (lxc-smtp2.ens-lyon.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id weeoDZ23OQLH for ; Mon, 9 Aug 2021 14:04:07 +0200 (CEST) Received: from localhost (unknown [78.194.167.103]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTPSA id E2994E0E67 for ; Mon, 9 Aug 2021 14:04:07 +0200 (CEST) Date: Mon, 9 Aug 2021 14:04:07 +0200 From: Alice BRENON Message-ID: <20210809140407.748fa019@ens-lyon.fr> Organization: ENS de Lyon X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.24; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/MP2=ddjP20LDcZckf5S4+Rd" Received-SPF: pass client-ip=140.77.167.81; envelope-from=alice.brenon@ens-lyon.fr; helo=lxc-smtp2.ens-lyon.fr X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.3 (/) X-Mailman-Approved-At: Mon, 09 Aug 2021 11:03:14 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --MP_/MP2=ddjP20LDcZckf5S4+Rd Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline Hello, I'd like to submit this patch for review, discussion and hopefully inclusion to guix' code. I recently tried to import grew[0], a NLP tool written in ocaml and distributed with opam but from a custom repository. The current importer prevented me to do so for several reasons: - the available repositories were hard-coded to be either opam's official repository or coq's - the repositories were expected to be distributed with git: while public git repositories do exist for coq and opam's official repository, they are not the source of truth for the opam tool one can use in an imperative setup like this: `opam repo add coq-released https://coq.inria.fr/opam/released` it entailed that assumptions were made about the freshness of the git repositories and the actual files served to opam, hence differences could theoretically be observed - it appears that the opam tool doesn't enforce as strict a structure on its repositories as its current documentation[1] suggests. Grew's own repository has all versions of each package directly under `/packages/` instead of in a separate subdirectory. While this deserves a clarification from opam's part, this patch hardens guix opam importer against such exotic layouts. - the unability to query several repositories at once rendered recursive imports inefficient, as some packages on a custom opam repository may still need dependencies from the official opam repository even if no guix package has been imported for it yet (this was the case with ocaml-ANSITerminal in my case) The current proposal attempts to solve these difficulties, and allowed me to actually import the guix declaration for grew and its dependencies. I'm still fixing the imported declaration and intend to submit a separate patch to add it to guix packages when it works. I added grew's custom opam repository to the list of known-repositories because it was my immediate target but this is of course not important and could be reverted in favour of coq and opam's official repository only. I used the emacs scripted auto-indenter to save a little time to my reviewers and had to discard many changes that weren't related to my changes to cut the noise. Maybe this file should be reindented in a separate commit following the approval or rejection of this patch proposal. Cheers, Alice [0]: https://grew.fr/ [1]: https://opam.ocaml.org/doc/Manual.html#Repositories --MP_/MP2=ddjP20LDcZckf5S4+Rd Content-Type: text/x-patch Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=0001-guix-opam.patch =46rom 3beac4d8caf3c6693237bc4f975029e58a20e0f4 Mon Sep 17 00:00:00 2001 From: Alice BRENON Date: Sat, 7 Aug 2021 19:50:10 +0200 Subject: [PATCH] guix: opam: - Add support for importing from several repositories - Add support for custom repositories, either from a URL or= a local path - Use actual opam repositories as defined by the opam tool as source * guix/scripts/import/opam.scm: pass all instances of --repo as a list to the importer * guix/import/opam.scm: - delay repo resolution (call to get-opam-repository from within opam-fetch instead of outside) - use the same repository source as CLI opam does (i.e. HTTP-served index.tar.gz instead of git repositories) - be more flexible on the repositories structure instead of expecting packages/PACKAGE-NAME/PACKAGE-NAME.VERSION/ --- guix/import/opam.scm | 145 +++++++++++++++++++++-------------- guix/scripts/import/opam.scm | 5 +- 2 files changed, 92 insertions(+), 58 deletions(-) diff --git a/guix/import/opam.scm b/guix/import/opam.scm index a35b01d277..c3fad7db65 100644 --- a/guix/import/opam.scm +++ b/guix/import/opam.scm @@ -2,6 +2,7 @@ ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2020 Martin Becze ;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -22,13 +23,16 @@ #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (ice-9 peg) + #:use-module (ice-9 popen) #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (ice-9 textual-ports) #:use-module (ice-9 vlist) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) + #:use-module (srfi srfi-26) #:use-module (web uri) + #:use-module ((guix build utils) #:select (dump-port find-files mkdir-p)) #:use-module (guix build-system) #:use-module (guix build-system ocaml) #:use-module (guix http-client) @@ -121,51 +125,78 @@ (define-peg-pattern condition-string all (and QUOTE (* STRCHR) QUOTE)) (define-peg-pattern condition-var all (+ (or (range #\a #\z) "-" ":"))) =20 -(define* (get-opam-repository #:optional repo) +(define (opam-cache-directory path) + (string-append (cache-directory #:ensure? #f) "/opam/" path)) + +(define known-repositories + '((opam . "https://opam.ocaml.org") + (coq . "https://coq.inria.fr/opam/released") + (grew . "http://opam.grew.fr"))) + +(define (repo-type repo) + (define (get-uri repo-root) + (let ((archive-file (string-append repo-root "/index.tar.gz"))) + (or (string->uri archive-file) (throw 'bad-uri archive-file)))) + (match (assoc-ref known-repositories (string->symbol repo)) + (#f (if (file-exists? repo) + `(local ,repo) + `(remote ,(get-uri repo)))) + (url `(remote ,(get-uri url))))) + +(define (update-repository-at output-folder input) + "Make sure the opam repository at OUTPUT-FOLDER is up-to-date with INPUT" + (let ((cached-date (if (file-exists? output-folder) + (stat:mtime (stat output-folder)) + (begin (mkdir-p output-folder) 0)))) + (begin + (and (> (stat:mtime (stat input)) cached-date) + (call-with-port + (open-pipe* OPEN_WRITE "tar" "xz" "-C" output-folder "-f" "-") + (cut dump-port input <>))) + output-folder))) + +(define* (get-opam-repository #:optional (repo "opam")) "Update or fetch the latest version of the opam repository and return the path to the repository." - (let ((url (cond - ((or (not repo) (equal? repo 'opam)) - "https://github.com/ocaml/opam-repository") - ((string-prefix? "coq-" (symbol->string repo)) - "https://github.com/coq/opam-coq-archive") - ((equal? repo 'coq) "https://github.com/coq/opam-coq-archiv= e") - (else (throw 'unknown-repository repo))))) - (receive (location commit _) - (update-cached-checkout url) - (cond - ((or (not repo) (equal? repo 'opam)) - location) - ((equal? repo 'coq) - (string-append location "/released")) - ((string-prefix? "coq-" (symbol->string repo)) - (string-append location "/" (substring (symbol->string repo) 4))) - (else location))))) + (match (repo-type repo) + (('local p) p) + (('remote source) (let ((cache (opam-cache-directory (uri-host source)= ))) + (call-with-port + (http-fetch/cached source) + (cut update-repository-at cache <>)))))) =20 ;; Prevent Guile 3 from inlining this procedure so we can mock it in tests. (set! get-opam-repository get-opam-repository) =20 -(define (latest-version versions) - "Find the most recent version from a list of versions." - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) versions)) +(define (get-version-and-file path) + "Analyse a candidate path and return an list containing information for = proper + version comparison as well as the source path for metadata." + (and-let* ((metadata-file (string-append path "/opam")) + (filename (basename path)) + (version (string-join (cdr (string-split filename #\.)) "."))) + (and (file-exists? metadata-file) + (eq? 'regular (stat:type (stat metadata-file))) + (if (string-prefix? "v" version) + `(V ,(substring version 1) ,metadata-file) + `(digits ,version ,metadata-file))))) + +(define (keep-max-version a b) + "Version comparison on the lists returned by the previous function takin= g the + janestreet re-versioning into account (v-prefixed come first)." + (match (cons a b) + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) + ((('V _ _) . _) a) + ((_ . ('V _ _)) b) + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a b)))) =20 (define (find-latest-version package repository) "Get the latest version of a package as described in the given repositor= y." - (let* ((dir (string-append repository "/packages/" package)) - (versions (scandir dir (lambda (name) (not (string-prefix? "." na= me)))))) - (if versions - (let ((versions (map - (lambda (dir) - (string-join (cdr (string-split dir #\.)) ".")) - versions))) - ;; Workaround for janestreet re-versionning - (let ((v-versions (filter (lambda (version) (string-prefix? "v" ve= rsion)) versions))) - (if (null? v-versions) - (latest-version versions) - (string-append "v" (latest-version (map (lambda (version) (sub= string version 1)) v-versions)))))) - (begin - (format #t (G_ "Package not found in opam repository: ~a~%") packa= ge) - #f)))) + (let ((packages (string-append repository "/packages")) + (filter (make-regexp (string-append "^" package "\\.")))) + (reduce keep-max-version #f + (filter-map + get-version-and-file + (find-files packages filter #:directories? #t))))) =20 (define (get-metadata opam-file) (with-input-from-file opam-file @@ -266,28 +297,30 @@ path to the repository." =20 (define (depends->native-inputs depends) (filter (lambda (name) (not (equal? "" name))) - (map dependency->native-input depends))) + (map dependency->native-input depends))) =20 (define (dependency-list->inputs lst) (map - (lambda (dependency) - (list dependency (list 'unquote (string->symbol dependency)))) - (ocaml-names->guix-names lst))) - -(define* (opam-fetch name #:optional (repository (get-opam-repository))) - (and-let* ((repository repository) - (version (find-latest-version name repository)) - (file (string-append repository "/packages/" name "/" name ".= " version "/opam"))) - `(("metadata" ,@(get-metadata file)) - ("version" . ,(if (string-prefix? "v" version) - (substring version 1) - version))))) - -(define* (opam->guix-package name #:key (repo 'opam) version) - "Import OPAM package NAME from REPOSITORY (a directory name) or, if -REPOSITORY is #f, from the official OPAM repository. Return a 'package' s= exp + (lambda (dependency) + (list dependency (list 'unquote (string->symbol dependency)))) + (ocaml-names->guix-names lst))) + +(define* (opam-fetch name #:optional (repositories-specs '("opam"))) + (or (fold (lambda (repository others) + (match (find-latest-version name repository) + ((_ version file) `(("metadata" ,@(get-metadata file)) + ("version" . ,version))) + (_ others))) + #f + (map get-opam-repository repositories-specs)) + (throw 'package-not-found repositories-specs))) + +(define* (opam->guix-package name #:key (repo '()) version) + "Import OPAM package NAME from REPOSITORIES (a list of names, URLs or lo= cal +paths, always including OPAM's official repository). Return a 'package' s= exp or #f on failure." - (and-let* ((opam-file (opam-fetch name (get-opam-repository repo))) + (and-let* ((with-opam (if (member "opam" repo) repo (cons "opam" repo))) + (opam-file (opam-fetch name with-opam)) (version (assoc-ref opam-file "version")) (opam-content (assoc-ref opam-file "metadata")) (url-dict (metadata-ref opam-content "url")) @@ -312,9 +345,7 @@ or #f on failure." (values `(package (name ,(ocaml-name->guix-name name)) - (version ,(if (string-prefix? "v" version) - (substring version 1) - version)) + (version ,version) (source (origin (method url-fetch) diff --git a/guix/scripts/import/opam.scm b/guix/scripts/import/opam.scm index 64164e7cc4..837a6ef40f 100644 --- a/guix/scripts/import/opam.scm +++ b/guix/scripts/import/opam.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -81,7 +82,9 @@ Import and convert the opam package for PACKAGE-NAME.\n")) #:build-options? #f)) =20 (let* ((opts (parse-options)) - (repo (and=3D> (assoc-ref opts 'repo) string->symbol)) + (repo (filter-map (match-lambda + (('repo . name) name) + (_ #f)) opts)) (args (filter-map (match-lambda (('argument . value) value) --=20 2.32.0 --MP_/MP2=ddjP20LDcZckf5S4+Rd-- From unknown Wed Aug 20 03:37:39 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#49958] [PATCH] More flexibility in opam importer Resent-From: Alice BRENON Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 09 Aug 2021 15:23:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49958 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 49958@debbugs.gnu.org Received: via spool by 49958-submit@debbugs.gnu.org id=B49958.162852257418581 (code B ref 49958); Mon, 09 Aug 2021 15:23:02 +0000 Received: (at 49958) by debbugs.gnu.org; 9 Aug 2021 15:22:54 +0000 Received: from localhost ([127.0.0.1]:57532 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mD777-0004pc-4S for submit@debbugs.gnu.org; Mon, 09 Aug 2021 11:22:54 -0400 Received: from lxc-smtp2.ens-lyon.fr ([140.77.167.81]:51792) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mD73x-0004kc-KF for 49958@debbugs.gnu.org; Mon, 09 Aug 2021 11:19:38 -0400 Received: from localhost (localhost [127.0.0.1]) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTP id 66B16E1F28 for <49958@debbugs.gnu.org>; Mon, 9 Aug 2021 17:19:36 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.11.0 (20160426) (Debian) at ens-lyon.fr Received: from lxc-smtp2.ens-lyon.fr ([127.0.0.1]) by localhost (lxc-smtp2.ens-lyon.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id X8LKfq-iJ4W6 for <49958@debbugs.gnu.org>; Mon, 9 Aug 2021 17:19:36 +0200 (CEST) Received: from localhost (unknown [78.194.167.103]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTPSA id 4D2D8E1A2D for <49958@debbugs.gnu.org>; Mon, 9 Aug 2021 17:19:36 +0200 (CEST) Date: Mon, 9 Aug 2021 17:19:35 +0200 From: Alice BRENON Message-ID: <20210809171935.05fac773@ens-lyon.fr> In-Reply-To: References: <20210809140407.748fa019@ens-lyon.fr> Organization: ENS de Lyon X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.24; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/cz_QVk7./OCdDAm5=mk01Dh" X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: First update of my patch thanks to the excellent remarks from roptat on IRC: the signature change for opam->guix-package (namely that the #:key parameter `repo` now expects a list instead of a single [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-Mailman-Approved-At: Mon, 09 Aug 2021 11:22:52 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) --MP_/cz_QVk7./OCdDAm5=mk01Dh Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline First update of my patch thanks to the excellent remarks from roptat on IRC: the signature change for opam->guix-package (namely that the #:key parameter `repo` now expects a list instead of a single element) broke the opam test, so here's a version passing a list containing `test-repo` instead of `test-repo` itself directly to fix the corresponding test. Alice Le Mon, 09 Aug 2021 15:04:02 +0000, help-debbugs@gnu.org (GNU bug Tracking System) a =C3=A9crit : > Thank you for filing a new bug report with debbugs.gnu.org. >=20 > This is an automatically generated reply to let you know your message > has been received. >=20 > Your message is being forwarded to the package maintainers and other > interested parties for their attention; they will reply in due course. >=20 > Your message has been sent to the package maintainer(s): > guix-patches@gnu.org >=20 > If you wish to submit further information on this problem, please > send it to 49958@debbugs.gnu.org. >=20 > Please do not send mail to help-debbugs@gnu.org unless you wish > to report a problem with the Bug-tracking system. >=20 --MP_/cz_QVk7./OCdDAm5=mk01Dh Content-Type: text/x-patch Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=0001-guix-opam.patch =46rom 8897e419e92fa9e6e5b91c554bf72722b2934953 Mon Sep 17 00:00:00 2001 From: Alice BRENON Date: Sat, 7 Aug 2021 19:50:10 +0200 Subject: [PATCH] guix: opam: - Add support for importing from several repositories - Add support for custom repositories, either from a URL or= a local path - Use actual opam repositories as defined by the opam tool as source * guix/scripts/import/opam.scm: pass all instances of --repo as a list to the importer * guix/import/opam.scm: - delay repo resolution (call to get-opam-repository from within opam-fetch instead of outside) - use the same repository source as CLI opam does (i.e. HTTP-served index.tar.gz instead of git repositories) - be more flexible on the repositories structure instead of expecting packages/PACKAGE-NAME/PACKAGE-NAME.VERSION/ * tests/opam.scm: update the opam->guix-package test since repo is now expected to be a list --- guix/import/opam.scm | 145 +++++++++++++++++++++-------------- guix/scripts/import/opam.scm | 5 +- tests/opam.scm | 2 +- 3 files changed, 93 insertions(+), 59 deletions(-) diff --git a/guix/import/opam.scm b/guix/import/opam.scm index a35b01d277..c3fad7db65 100644 --- a/guix/import/opam.scm +++ b/guix/import/opam.scm @@ -2,6 +2,7 @@ ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2020 Martin Becze ;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -22,13 +23,16 @@ #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (ice-9 peg) + #:use-module (ice-9 popen) #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (ice-9 textual-ports) #:use-module (ice-9 vlist) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) + #:use-module (srfi srfi-26) #:use-module (web uri) + #:use-module ((guix build utils) #:select (dump-port find-files mkdir-p)) #:use-module (guix build-system) #:use-module (guix build-system ocaml) #:use-module (guix http-client) @@ -121,51 +125,78 @@ (define-peg-pattern condition-string all (and QUOTE (* STRCHR) QUOTE)) (define-peg-pattern condition-var all (+ (or (range #\a #\z) "-" ":"))) =20 -(define* (get-opam-repository #:optional repo) +(define (opam-cache-directory path) + (string-append (cache-directory #:ensure? #f) "/opam/" path)) + +(define known-repositories + '((opam . "https://opam.ocaml.org") + (coq . "https://coq.inria.fr/opam/released") + (grew . "http://opam.grew.fr"))) + +(define (repo-type repo) + (define (get-uri repo-root) + (let ((archive-file (string-append repo-root "/index.tar.gz"))) + (or (string->uri archive-file) (throw 'bad-uri archive-file)))) + (match (assoc-ref known-repositories (string->symbol repo)) + (#f (if (file-exists? repo) + `(local ,repo) + `(remote ,(get-uri repo)))) + (url `(remote ,(get-uri url))))) + +(define (update-repository-at output-folder input) + "Make sure the opam repository at OUTPUT-FOLDER is up-to-date with INPUT" + (let ((cached-date (if (file-exists? output-folder) + (stat:mtime (stat output-folder)) + (begin (mkdir-p output-folder) 0)))) + (begin + (and (> (stat:mtime (stat input)) cached-date) + (call-with-port + (open-pipe* OPEN_WRITE "tar" "xz" "-C" output-folder "-f" "-") + (cut dump-port input <>))) + output-folder))) + +(define* (get-opam-repository #:optional (repo "opam")) "Update or fetch the latest version of the opam repository and return the path to the repository." - (let ((url (cond - ((or (not repo) (equal? repo 'opam)) - "https://github.com/ocaml/opam-repository") - ((string-prefix? "coq-" (symbol->string repo)) - "https://github.com/coq/opam-coq-archive") - ((equal? repo 'coq) "https://github.com/coq/opam-coq-archiv= e") - (else (throw 'unknown-repository repo))))) - (receive (location commit _) - (update-cached-checkout url) - (cond - ((or (not repo) (equal? repo 'opam)) - location) - ((equal? repo 'coq) - (string-append location "/released")) - ((string-prefix? "coq-" (symbol->string repo)) - (string-append location "/" (substring (symbol->string repo) 4))) - (else location))))) + (match (repo-type repo) + (('local p) p) + (('remote source) (let ((cache (opam-cache-directory (uri-host source)= ))) + (call-with-port + (http-fetch/cached source) + (cut update-repository-at cache <>)))))) =20 ;; Prevent Guile 3 from inlining this procedure so we can mock it in tests. (set! get-opam-repository get-opam-repository) =20 -(define (latest-version versions) - "Find the most recent version from a list of versions." - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) versions)) +(define (get-version-and-file path) + "Analyse a candidate path and return an list containing information for = proper + version comparison as well as the source path for metadata." + (and-let* ((metadata-file (string-append path "/opam")) + (filename (basename path)) + (version (string-join (cdr (string-split filename #\.)) "."))) + (and (file-exists? metadata-file) + (eq? 'regular (stat:type (stat metadata-file))) + (if (string-prefix? "v" version) + `(V ,(substring version 1) ,metadata-file) + `(digits ,version ,metadata-file))))) + +(define (keep-max-version a b) + "Version comparison on the lists returned by the previous function takin= g the + janestreet re-versioning into account (v-prefixed come first)." + (match (cons a b) + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) + ((('V _ _) . _) a) + ((_ . ('V _ _)) b) + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a b)))) =20 (define (find-latest-version package repository) "Get the latest version of a package as described in the given repositor= y." - (let* ((dir (string-append repository "/packages/" package)) - (versions (scandir dir (lambda (name) (not (string-prefix? "." na= me)))))) - (if versions - (let ((versions (map - (lambda (dir) - (string-join (cdr (string-split dir #\.)) ".")) - versions))) - ;; Workaround for janestreet re-versionning - (let ((v-versions (filter (lambda (version) (string-prefix? "v" ve= rsion)) versions))) - (if (null? v-versions) - (latest-version versions) - (string-append "v" (latest-version (map (lambda (version) (sub= string version 1)) v-versions)))))) - (begin - (format #t (G_ "Package not found in opam repository: ~a~%") packa= ge) - #f)))) + (let ((packages (string-append repository "/packages")) + (filter (make-regexp (string-append "^" package "\\.")))) + (reduce keep-max-version #f + (filter-map + get-version-and-file + (find-files packages filter #:directories? #t))))) =20 (define (get-metadata opam-file) (with-input-from-file opam-file @@ -266,28 +297,30 @@ path to the repository." =20 (define (depends->native-inputs depends) (filter (lambda (name) (not (equal? "" name))) - (map dependency->native-input depends))) + (map dependency->native-input depends))) =20 (define (dependency-list->inputs lst) (map - (lambda (dependency) - (list dependency (list 'unquote (string->symbol dependency)))) - (ocaml-names->guix-names lst))) - -(define* (opam-fetch name #:optional (repository (get-opam-repository))) - (and-let* ((repository repository) - (version (find-latest-version name repository)) - (file (string-append repository "/packages/" name "/" name ".= " version "/opam"))) - `(("metadata" ,@(get-metadata file)) - ("version" . ,(if (string-prefix? "v" version) - (substring version 1) - version))))) - -(define* (opam->guix-package name #:key (repo 'opam) version) - "Import OPAM package NAME from REPOSITORY (a directory name) or, if -REPOSITORY is #f, from the official OPAM repository. Return a 'package' s= exp + (lambda (dependency) + (list dependency (list 'unquote (string->symbol dependency)))) + (ocaml-names->guix-names lst))) + +(define* (opam-fetch name #:optional (repositories-specs '("opam"))) + (or (fold (lambda (repository others) + (match (find-latest-version name repository) + ((_ version file) `(("metadata" ,@(get-metadata file)) + ("version" . ,version))) + (_ others))) + #f + (map get-opam-repository repositories-specs)) + (throw 'package-not-found repositories-specs))) + +(define* (opam->guix-package name #:key (repo '()) version) + "Import OPAM package NAME from REPOSITORIES (a list of names, URLs or lo= cal +paths, always including OPAM's official repository). Return a 'package' s= exp or #f on failure." - (and-let* ((opam-file (opam-fetch name (get-opam-repository repo))) + (and-let* ((with-opam (if (member "opam" repo) repo (cons "opam" repo))) + (opam-file (opam-fetch name with-opam)) (version (assoc-ref opam-file "version")) (opam-content (assoc-ref opam-file "metadata")) (url-dict (metadata-ref opam-content "url")) @@ -312,9 +345,7 @@ or #f on failure." (values `(package (name ,(ocaml-name->guix-name name)) - (version ,(if (string-prefix? "v" version) - (substring version 1) - version)) + (version ,version) (source (origin (method url-fetch) diff --git a/guix/scripts/import/opam.scm b/guix/scripts/import/opam.scm index 64164e7cc4..837a6ef40f 100644 --- a/guix/scripts/import/opam.scm +++ b/guix/scripts/import/opam.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -81,7 +82,9 @@ Import and convert the opam package for PACKAGE-NAME.\n")) #:build-options? #f)) =20 (let* ((opts (parse-options)) - (repo (and=3D> (assoc-ref opts 'repo) string->symbol)) + (repo (filter-map (match-lambda + (('repo . name) name) + (_ #f)) opts)) (args (filter-map (match-lambda (('argument . value) value) diff --git a/tests/opam.scm b/tests/opam.scm index f1e3b70cb0..dcfbd50e5a 100644 --- a/tests/opam.scm +++ b/tests/opam.scm @@ -90,7 +90,7 @@ url { (with-output-to-file (string-append my-package "/opam") (lambda _ (format #t "~a" test-opam-file)))) - (match (opam->guix-package "foo" #:repo test-repo) + (match (opam->guix-package "foo" #:repo `(,test-repo)) (('package ('name "ocaml-foo") ('version "1.0.0") --=20 2.32.0 --MP_/cz_QVk7./OCdDAm5=mk01Dh-- From unknown Wed Aug 20 03:37:39 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#49958] [PATCH] More flexibility in opam importer Resent-From: Alice BRENON Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 10 Aug 2021 12:05:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49958 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 49958@debbugs.gnu.org Received: via spool by 49958-submit@debbugs.gnu.org id=B49958.162859706716922 (code B ref 49958); Tue, 10 Aug 2021 12:05:02 +0000 Received: (at 49958) by debbugs.gnu.org; 10 Aug 2021 12:04:27 +0000 Received: from localhost ([127.0.0.1]:58609 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mDQUV-0004Ok-V3 for submit@debbugs.gnu.org; Tue, 10 Aug 2021 08:04:27 -0400 Received: from lxc-smtp2.ens-lyon.fr ([140.77.167.81]:56616) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mDQUR-0004Oa-SY for 49958@debbugs.gnu.org; Tue, 10 Aug 2021 08:04:18 -0400 Received: from localhost (localhost [127.0.0.1]) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTP id 85144E1922 for <49958@debbugs.gnu.org>; Tue, 10 Aug 2021 14:04:14 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.11.0 (20160426) (Debian) at ens-lyon.fr Received: from lxc-smtp2.ens-lyon.fr ([127.0.0.1]) by localhost (lxc-smtp2.ens-lyon.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6d3e9IXngV2P for <49958@debbugs.gnu.org>; Tue, 10 Aug 2021 14:04:14 +0200 (CEST) Received: from localhost (unknown [78.194.167.103]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTPSA id 68CDBE131F for <49958@debbugs.gnu.org>; Tue, 10 Aug 2021 14:04:14 +0200 (CEST) Date: Tue, 10 Aug 2021 14:04:13 +0200 From: Alice BRENON Message-ID: <20210810140413.2f7d2f1b@ens-lyon.fr> In-Reply-To: <20210809171935.05fac773@ens-lyon.fr> References: <20210809140407.748fa019@ens-lyon.fr> <20210809171935.05fac773@ens-lyon.fr> Organization: ENS de Lyon X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.24; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/_3eM992o5H9LkZokTQ7DVuJ" X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Following further discussion on IRC, here is a version which - removes an unneeded mocked value in the test - updates the documentation to explain the new behaviour of the --repo option for this importer - improves the general quality of code and commit message Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) --MP_/_3eM992o5H9LkZokTQ7DVuJ Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline Following further discussion on IRC, here is a version which - removes an unneeded mocked value in the test - updates the documentation to explain the new behaviour of the --repo option for this importer - improves the general quality of code and commit message --MP_/_3eM992o5H9LkZokTQ7DVuJ Content-Type: text/x-patch Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=0001-guix-opam-More-flexibility-in-the-importer.patch =46rom 3a14538aaa8f74274c2b0a924f9c4542daa4af75 Mon Sep 17 00:00:00 2001 From: Alice BRENON Date: Sat, 7 Aug 2021 19:50:10 +0200 Subject: [PATCH] guix: opam: More flexibility in the importer * guix/scripts/import/opam.scm: pass all instances of --repo as a list to the importer * guix/import/opam.scm (opam-fetch): stop expecting "expanded" repositories and call get-opam-repository instead to keep values "symbolic" as long as possible and factorize. (get-opam-repository): use the same repository source as CLI opam does (i.e. HTTP-served index.tar.gz instead of git repositories) (find-latest-version): be more flexible on the repositories structure instead of expecting packages/PACKAGE-NAME/PACKAGE-NAME.VERSION/ * tests/opam.scm: update the call to opam->guix-package since repo is now expected to be a list and remove the mocked get-opam-repository deprecated by the support for local folders by the actual implementation * doc/guix.texi: document the new semantics and valid arguments for the --repo option --- doc/guix.texi | 20 ++++- guix/import/opam.scm | 155 +++++++++++++++++++++-------------- guix/scripts/import/opam.scm | 7 +- tests/opam.scm | 68 ++++++++------- 4 files changed, 151 insertions(+), 99 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 4eb5324b51..e3fb37c3ce 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -94,6 +94,7 @@ Copyright @copyright{} 2021 Xinglu Chen@* Copyright @copyright{} 2021 Raghav Gururajan@* Copyright @copyright{} 2021 Domagoj Stolfa@* Copyright @copyright{} 2021 Hui Lu@* +Copyright @copyright{} 2021 Alice Brenon@* =20 Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or @@ -11612,7 +11613,24 @@ Traverse the dependency graph of the given upstrea= m package recursively and generate package expressions for all those packages that are not yet in Guix. @item --repo -Select the given repository (a repository name). Possible values include: + +Add the given repository to the list that will be searched. This can be us= ed +several times and valid arguments for this option may be + +@itemize +@item the name of a known repositories (one of @code{opam}, @code{coq} or + @code{coq-released} (equivalent), @code{coq-core-dev}, + @code{coq-extra-dev} or @code{grew}) +@item the URL of a repository as expected by the @code{opam repository add} + command (for instance, the URL equivalent of the above @code{opam} n= ame + would be @uref{https://opam.ocaml.org}) +@item the path to a local copy of a repository (a directory containing a s= ub- + directory @file{packages/}) +@end itemize + +The list searched always includes @code{opam} so calling this importer wit= h the +option @code{--repo=3Dopam}, although valid, is useless. + @itemize @item @code{opam}, the default opam repository, @item @code{coq} or @code{coq-released}, the stable repository for coq pac= kages, diff --git a/guix/import/opam.scm b/guix/import/opam.scm index a35b01d277..6d88ccab76 100644 --- a/guix/import/opam.scm +++ b/guix/import/opam.scm @@ -2,6 +2,7 @@ ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2020 Martin Becze ;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -22,21 +23,24 @@ #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (ice-9 peg) + #:use-module ((ice-9 popen) #:select (open-pipe*)) #:use-module (ice-9 receive) - #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (ice-9 textual-ports) #:use-module (ice-9 vlist) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) - #:use-module (web uri) + #:use-module ((srfi srfi-26) #:select (cut)) + #:use-module ((web uri) #:select (string->uri uri->string)) + #:use-module ((guix build utils) #:select (dump-port find-files mkdir-p)) #:use-module (guix build-system) #:use-module (guix build-system ocaml) #:use-module (guix http-client) - #:use-module (guix git) #:use-module (guix ui) #:use-module (guix packages) #:use-module (guix upstream) - #:use-module (guix utils) + #:use-module ((guix utils) #:select (cache-directory + version>? + call-with-temporary-output-file)) #:use-module (guix import utils) #:use-module ((guix licenses) #:prefix license:) #:export (opam->guix-package @@ -121,51 +125,80 @@ (define-peg-pattern condition-string all (and QUOTE (* STRCHR) QUOTE)) (define-peg-pattern condition-var all (+ (or (range #\a #\z) "-" ":"))) =20 -(define* (get-opam-repository #:optional repo) +(define (opam-cache-directory path) + (string-append (cache-directory #:ensure? #f) "/opam/" path)) + +(define known-repositories + '((opam . "https://opam.ocaml.org") + (coq . "https://coq.inria.fr/opam/released") + (coq-released . "https://coq.inria.fr/opam/released") + (coq-core-dev . "https://coq.inria.fr/opam/core-dev") + (coq-extra-dev . "https://coq.inria.fr/opam/extra-dev") + (grew . "http://opam.grew.fr"))) + +(define (repo-type repo) + (define (get-uri repo-root) + (let ((archive-file (string-append repo-root "/index.tar.gz"))) + (or (string->uri archive-file) + (warning (G_ "'~a' is not a valid URI~%") archive-file)))) + (match (assoc-ref known-repositories (string->symbol repo)) + (#f (if (file-exists? repo) + `(local ,repo) + `(remote ,(get-uri repo)))) + (url `(remote ,(get-uri url))))) + +(define (update-repository input) + "Make sure the cache for opam repository INPUT is up-to-date" + (let* ((output (opam-cache-directory (basename (port-filename input)))) + (cached-date (if (file-exists? output) + (stat:mtime (stat output)) + (begin (mkdir-p output) 0)))) + (when (> (stat:mtime (stat input)) cached-date) + (call-with-port + (open-pipe* OPEN_WRITE "tar" "xz" "-C" output "-f" "-") + (cut dump-port input <>))) + output)) + +(define* (get-opam-repository #:optional (repo "opam")) "Update or fetch the latest version of the opam repository and return the path to the repository." - (let ((url (cond - ((or (not repo) (equal? repo 'opam)) - "https://github.com/ocaml/opam-repository") - ((string-prefix? "coq-" (symbol->string repo)) - "https://github.com/coq/opam-coq-archive") - ((equal? repo 'coq) "https://github.com/coq/opam-coq-archiv= e") - (else (throw 'unknown-repository repo))))) - (receive (location commit _) - (update-cached-checkout url) - (cond - ((or (not repo) (equal? repo 'opam)) - location) - ((equal? repo 'coq) - (string-append location "/released")) - ((string-prefix? "coq-" (symbol->string repo)) - (string-append location "/" (substring (symbol->string repo) 4))) - (else location))))) + (match (repo-type repo) + (('local p) p) + (('remote #t) #f) + (('remote r) (call-with-port (http-fetch/cached r) update-repository))= )) =20 ;; Prevent Guile 3 from inlining this procedure so we can mock it in tests. (set! get-opam-repository get-opam-repository) =20 -(define (latest-version versions) - "Find the most recent version from a list of versions." - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) versions)) +(define (get-version-and-file path) + "Analyse a candidate path and return an list containing information for = proper + version comparison as well as the source path for metadata." + (and-let* ((metadata-file (string-append path "/opam")) + (filename (basename path)) + (version (string-join (cdr (string-split filename #\.)) "."))) + (and (file-exists? metadata-file) + (eq? 'regular (stat:type (stat metadata-file))) + (if (string-prefix? "v" version) + `(V ,(substring version 1) ,metadata-file) + `(digits ,version ,metadata-file))))) + +(define (keep-max-version a b) + "Version comparison on the lists returned by the previous function takin= g the + janestreet re-versioning into account (v-prefixed come first)." + (match (cons a b) + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) + ((('V _ _) . _) a) + ((_ . ('V _ _)) b) + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a b)))) =20 (define (find-latest-version package repository) "Get the latest version of a package as described in the given repositor= y." - (let* ((dir (string-append repository "/packages/" package)) - (versions (scandir dir (lambda (name) (not (string-prefix? "." na= me)))))) - (if versions - (let ((versions (map - (lambda (dir) - (string-join (cdr (string-split dir #\.)) ".")) - versions))) - ;; Workaround for janestreet re-versionning - (let ((v-versions (filter (lambda (version) (string-prefix? "v" ve= rsion)) versions))) - (if (null? v-versions) - (latest-version versions) - (string-append "v" (latest-version (map (lambda (version) (sub= string version 1)) v-versions)))))) - (begin - (format #t (G_ "Package not found in opam repository: ~a~%") packa= ge) - #f)))) + (let ((packages (string-append repository "/packages")) + (filter (make-regexp (string-append "^" package "\\.")))) + (reduce keep-max-version #f + (filter-map + get-version-and-file + (find-files packages filter #:directories? #t))))) =20 (define (get-metadata opam-file) (with-input-from-file opam-file @@ -266,28 +299,30 @@ path to the repository." =20 (define (depends->native-inputs depends) (filter (lambda (name) (not (equal? "" name))) - (map dependency->native-input depends))) + (map dependency->native-input depends))) =20 (define (dependency-list->inputs lst) (map - (lambda (dependency) - (list dependency (list 'unquote (string->symbol dependency)))) - (ocaml-names->guix-names lst))) - -(define* (opam-fetch name #:optional (repository (get-opam-repository))) - (and-let* ((repository repository) - (version (find-latest-version name repository)) - (file (string-append repository "/packages/" name "/" name ".= " version "/opam"))) - `(("metadata" ,@(get-metadata file)) - ("version" . ,(if (string-prefix? "v" version) - (substring version 1) - version))))) - -(define* (opam->guix-package name #:key (repo 'opam) version) - "Import OPAM package NAME from REPOSITORY (a directory name) or, if -REPOSITORY is #f, from the official OPAM repository. Return a 'package' s= exp + (lambda (dependency) + (list dependency (list 'unquote (string->symbol dependency)))) + (ocaml-names->guix-names lst))) + +(define* (opam-fetch name #:optional (repositories-specs '("opam"))) + (or (fold (lambda (repository others) + (match (find-latest-version name repository) + ((_ version file) `(("metadata" ,@(get-metadata file)) + ("version" . ,version))) + (_ others))) + #f + (filter-map get-opam-repository repositories-specs)) + (leave (G_ "Package '~a' not found~%") name))) + +(define* (opam->guix-package name #:key (repo '()) version) + "Import OPAM package NAME from REPOSITORIES (a list of names, URLs or lo= cal +paths, always including OPAM's official repository). Return a 'package' s= exp or #f on failure." - (and-let* ((opam-file (opam-fetch name (get-opam-repository repo))) + (and-let* ((with-opam (if (member "opam" repo) repo (cons "opam" repo))) + (opam-file (opam-fetch name with-opam)) (version (assoc-ref opam-file "version")) (opam-content (assoc-ref opam-file "metadata")) (url-dict (metadata-ref opam-content "url")) @@ -312,9 +347,7 @@ or #f on failure." (values `(package (name ,(ocaml-name->guix-name name)) - (version ,(if (string-prefix? "v" version) - (substring version 1) - version)) + (version ,version) (source (origin (method url-fetch) diff --git a/guix/scripts/import/opam.scm b/guix/scripts/import/opam.scm index 64164e7cc4..cb0c4b5d19 100644 --- a/guix/scripts/import/opam.scm +++ b/guix/scripts/import/opam.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -46,7 +47,7 @@ Import and convert the opam package for PACKAGE-NAME.\n")) (display (G_ " -r, --recursive import packages recursively")) (display (G_ " - --repo import packages from this opam repository")) + --repo import packages from this opam repository (name, = URL or local path)")) (display (G_ " -V, --version display version information and exit")) (newline) @@ -81,7 +82,9 @@ Import and convert the opam package for PACKAGE-NAME.\n")) #:build-options? #f)) =20 (let* ((opts (parse-options)) - (repo (and=3D> (assoc-ref opts 'repo) string->symbol)) + (repo (filter-map (match-lambda + (('repo . name) name) + (_ #f)) opts)) (args (filter-map (match-lambda (('argument . value) value) diff --git a/tests/opam.scm b/tests/opam.scm index f1e3b70cb0..1536b74339 100644 --- a/tests/opam.scm +++ b/tests/opam.scm @@ -82,41 +82,39 @@ url { (set! test-source-hash (call-with-input-file file-name port-sha256)))) (_ (error "Unexpected URL: " url))))) - (mock ((guix import opam) get-opam-repository - (const test-repo)) - (let ((my-package (string-append test-repo - "/packages/foo/foo.1.0.0"))) - (mkdir-p my-package) - (with-output-to-file (string-append my-package "/opam") - (lambda _ - (format #t "~a" test-opam-file)))) - (match (opam->guix-package "foo" #:repo test-repo) - (('package - ('name "ocaml-foo") - ('version "1.0.0") - ('source ('origin - ('method 'url-fetch) - ('uri "https://example.org/foo-1.0.0.tar.gz") - ('sha256 - ('base32 - (? string? hash))))) - ('build-system 'ocaml-build-system) - ('propagated-inputs - ('quasiquote - (("ocaml-zarith" ('unquote 'ocaml-zarith))))) - ('native-inputs - ('quasiquote - (("ocaml-alcotest" ('unquote 'ocaml-alcotest)) - ("ocamlbuild" ('unquote 'ocamlbuild))))) - ('home-page "https://example.org/") - ('synopsis "Some example package") - ('description "This package is just an example.") - ('license 'license:bsd-3)) - (string=3D? (bytevector->nix-base32-string - test-source-hash) - hash)) - (x - (pk 'fail x #f)))))) + (let ((my-package (string-append test-repo + "/packages/foo/foo.1.0.0"))) + (mkdir-p my-package) + (with-output-to-file (string-append my-package "/opam") + (lambda _ + (format #t "~a" test-opam-file)))) + (match (opam->guix-package "foo" #:repo (list test-repo)) + (('package + ('name "ocaml-foo") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri "https://example.org/foo-1.0.0.tar.gz") + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'ocaml-build-system) + ('propagated-inputs + ('quasiquote + (("ocaml-zarith" ('unquote 'ocaml-zarith))))) + ('native-inputs + ('quasiquote + (("ocaml-alcotest" ('unquote 'ocaml-alcotest)) + ("ocamlbuild" ('unquote 'ocamlbuild))))) + ('home-page "https://example.org/") + ('synopsis "Some example package") + ('description "This package is just an example.") + ('license 'license:bsd-3)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (pk 'fail x #f))))) =20 ;; Test the opam file parser ;; We fold over some test cases. Each case is a pair of the string to pars= e and the --=20 2.32.0 --MP_/_3eM992o5H9LkZokTQ7DVuJ-- From unknown Wed Aug 20 03:37:39 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#49958] [PATCH] More flexibility in opam importer Resent-From: Alice BRENON Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 10 Aug 2021 16:49:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49958 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 49958@debbugs.gnu.org Received: via spool by 49958-submit@debbugs.gnu.org id=B49958.16286141188775 (code B ref 49958); Tue, 10 Aug 2021 16:49:01 +0000 Received: (at 49958) by debbugs.gnu.org; 10 Aug 2021 16:48:38 +0000 Received: from localhost ([127.0.0.1]:60756 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mDUvX-0002HN-7N for submit@debbugs.gnu.org; Tue, 10 Aug 2021 12:48:38 -0400 Received: from lxc-smtp2.ens-lyon.fr ([140.77.167.81]:54068) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mDUvU-0002HC-GR for 49958@debbugs.gnu.org; Tue, 10 Aug 2021 12:48:29 -0400 Received: from localhost (localhost [127.0.0.1]) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTP id 14500E2B98 for <49958@debbugs.gnu.org>; Tue, 10 Aug 2021 18:48:27 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.11.0 (20160426) (Debian) at ens-lyon.fr Received: from lxc-smtp2.ens-lyon.fr ([127.0.0.1]) by localhost (lxc-smtp2.ens-lyon.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QQtoutRkzxDa for <49958@debbugs.gnu.org>; Tue, 10 Aug 2021 18:48:27 +0200 (CEST) Received: from localhost (unknown [78.194.167.103]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTPSA id ECD07E1728 for <49958@debbugs.gnu.org>; Tue, 10 Aug 2021 18:48:26 +0200 (CEST) Date: Tue, 10 Aug 2021 18:48:26 +0200 From: Alice BRENON Message-ID: <20210810184826.17d14aab@ens-lyon.fr> In-Reply-To: <20210810140413.2f7d2f1b@ens-lyon.fr> References: <20210809140407.748fa019@ens-lyon.fr> <20210809171935.05fac773@ens-lyon.fr> <20210810140413.2f7d2f1b@ens-lyon.fr> Organization: ENS de Lyon X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.24; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/83hnYGjSLbcyLCazTJrKRll" X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: - rephrase the documentation. - remove unnecessary #:ensure #f key in call to (cache-directory). - clarify what happens when string->uri fails and yields a warning by passing a 'bad-repo symbol and co [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) --MP_/83hnYGjSLbcyLCazTJrKRll Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline - rephrase the documentation. - remove unnecessary #:ensure #f key in call to (cache-directory). - clarify what happens when string->uri fails and yields a warning by passing a 'bad-repo symbol and commenting at its elimination site. - make a separate function of get-uri now that it's become as large as its caller repo-type. - mention the possibility to call --repo several times in help message. - add points at the end of sentences in commit message and in this e-mail. --MP_/83hnYGjSLbcyLCazTJrKRll Content-Type: text/x-patch Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=0001-guix-opam-More-flexibility-in-the-importer.patch =46rom 49e8236f81462501e89aa40d4e1b77bcc3fbb0ad Mon Sep 17 00:00:00 2001 From: Alice BRENON Date: Sat, 7 Aug 2021 19:50:10 +0200 Subject: [PATCH] guix: opam: More flexibility in the importer. * guix/scripts/import/opam.scm: pass all instances of --repo as a list to the importer. * guix/import/opam.scm (opam-fetch): stop expecting "expanded" repositories and call get-opam-repository instead to keep values "symbolic" as long as possible and factorize. (get-opam-repository): use the same repository source as CLI opam does (i.e. HTTP-served index.tar.gz instead of git repositories). (find-latest-version): be more flexible on the repositories structure instead of expecting packages/PACKAGE-NAME/PACKAGE-NAME.VERSION/. * tests/opam.scm: update the call to opam->guix-package since repo is now expected to be a list and remove the mocked get-opam-repository deprecated by the support for local folders by the actual implementation. * doc/guix.texi: document the new semantics and valid arguments for the --repo option. --- doc/guix.texi | 25 ++++-- guix/import/opam.scm | 158 +++++++++++++++++++++-------------- guix/scripts/import/opam.scm | 8 +- tests/opam.scm | 68 ++++++++------- 4 files changed, 155 insertions(+), 104 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 4eb5324b51..4a911e4c0f 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -94,6 +94,7 @@ Copyright @copyright{} 2021 Xinglu Chen@* Copyright @copyright{} 2021 Raghav Gururajan@* Copyright @copyright{} 2021 Domagoj Stolfa@* Copyright @copyright{} 2021 Hui Lu@* +Copyright @copyright{} 2021 Alice Brenon@* =20 Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or @@ -11612,14 +11613,26 @@ Traverse the dependency graph of the given upstre= am package recursively and generate package expressions for all those packages that are not yet in Guix. @item --repo -Select the given repository (a repository name). Possible values include: + +By default, packages are searched in the official OPAM repository. This +option, which can be used more than once, lets you add other +repositories where to look for packages. It accepts as valid arguments: + @itemize -@item @code{opam}, the default opam repository, -@item @code{coq} or @code{coq-released}, the stable repository for coq pac= kages, -@item @code{coq-core-dev}, the repository that contains development versio= ns of coq, -@item @code{coq-extra-dev}, the repository that contains development versi= ons - of coq packages. +@item the name of a known repository - can be one of @code{opam}, + @code{coq} (equivalent to @code{coq-released}), + @code{coq-core-dev}, @code{coq-extra-dev} or @code{grew}. +@item the URL of a repository as expected by the @code{opam repository + add} command (for instance, the URL equivalent of the above + @code{opam} name would be @uref{https://opam.ocaml.org}). +@item the path to a local copy of a repository (a directory containing a + @file{packages/} sub-directory). @end itemize + +Please note that repositories added with this option do not replace the +default @code{opam} repository, so calling this importer with the option +@code{--repo=3Dopam} is redundant. + @end table =20 @item go diff --git a/guix/import/opam.scm b/guix/import/opam.scm index a35b01d277..0e6cae72c4 100644 --- a/guix/import/opam.scm +++ b/guix/import/opam.scm @@ -2,6 +2,7 @@ ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2020 Martin Becze ;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -22,21 +23,24 @@ #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (ice-9 peg) + #:use-module ((ice-9 popen) #:select (open-pipe*)) #:use-module (ice-9 receive) - #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (ice-9 textual-ports) #:use-module (ice-9 vlist) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) - #:use-module (web uri) + #:use-module ((srfi srfi-26) #:select (cut)) + #:use-module ((web uri) #:select (string->uri uri->string)) + #:use-module ((guix build utils) #:select (dump-port find-files mkdir-p)) #:use-module (guix build-system) #:use-module (guix build-system ocaml) #:use-module (guix http-client) - #:use-module (guix git) #:use-module (guix ui) #:use-module (guix packages) #:use-module (guix upstream) - #:use-module (guix utils) + #:use-module ((guix utils) #:select (cache-directory + version>? + call-with-temporary-output-file)) #:use-module (guix import utils) #:use-module ((guix licenses) #:prefix license:) #:export (opam->guix-package @@ -121,51 +125,83 @@ (define-peg-pattern condition-string all (and QUOTE (* STRCHR) QUOTE)) (define-peg-pattern condition-var all (+ (or (range #\a #\z) "-" ":"))) =20 -(define* (get-opam-repository #:optional repo) +(define (opam-cache-directory path) + (string-append (cache-directory) "/opam/" path)) + +(define known-repositories + '((opam . "https://opam.ocaml.org") + (coq . "https://coq.inria.fr/opam/released") + (coq-released . "https://coq.inria.fr/opam/released") + (coq-core-dev . "https://coq.inria.fr/opam/core-dev") + (coq-extra-dev . "https://coq.inria.fr/opam/extra-dev") + (grew . "http://opam.grew.fr"))) + +(define (get-uri repo-root) + (let ((archive-file (string-append repo-root "/index.tar.gz"))) + (or (string->uri archive-file) + (begin + (warning (G_ "'~a' is not a valid URI~%") archive-file) + 'bad-repo)))) + +(define (repo-type repo) + (match (assoc-ref known-repositories (string->symbol repo)) + (#f (if (file-exists? repo) + `(local ,repo) + `(remote ,(get-uri repo)))) + (url `(remote ,(get-uri url))))) + +(define (update-repository input) + "Make sure the cache for opam repository INPUT is up-to-date" + (let* ((output (opam-cache-directory (basename (port-filename input)))) + (cached-date (if (file-exists? output) + (stat:mtime (stat output)) + (begin (mkdir-p output) 0)))) + (when (> (stat:mtime (stat input)) cached-date) + (call-with-port + (open-pipe* OPEN_WRITE "tar" "xz" "-C" output "-f" "-") + (cut dump-port input <>))) + output)) + +(define* (get-opam-repository #:optional (repo "opam")) "Update or fetch the latest version of the opam repository and return the path to the repository." - (let ((url (cond - ((or (not repo) (equal? repo 'opam)) - "https://github.com/ocaml/opam-repository") - ((string-prefix? "coq-" (symbol->string repo)) - "https://github.com/coq/opam-coq-archive") - ((equal? repo 'coq) "https://github.com/coq/opam-coq-archiv= e") - (else (throw 'unknown-repository repo))))) - (receive (location commit _) - (update-cached-checkout url) - (cond - ((or (not repo) (equal? repo 'opam)) - location) - ((equal? repo 'coq) - (string-append location "/released")) - ((string-prefix? "coq-" (symbol->string repo)) - (string-append location "/" (substring (symbol->string repo) 4))) - (else location))))) + (match (repo-type repo) + (('local p) p) + (('remote 'bad-repo) #f) ; to weed it out during filter-map in opam-fe= tch + (('remote r) (call-with-port (http-fetch/cached r) update-repository))= )) =20 ;; Prevent Guile 3 from inlining this procedure so we can mock it in tests. (set! get-opam-repository get-opam-repository) =20 -(define (latest-version versions) - "Find the most recent version from a list of versions." - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) versions)) +(define (get-version-and-file path) + "Analyse a candidate path and return an list containing information for = proper + version comparison as well as the source path for metadata." + (and-let* ((metadata-file (string-append path "/opam")) + (filename (basename path)) + (version (string-join (cdr (string-split filename #\.)) "."))) + (and (file-exists? metadata-file) + (eq? 'regular (stat:type (stat metadata-file))) + (if (string-prefix? "v" version) + `(V ,(substring version 1) ,metadata-file) + `(digits ,version ,metadata-file))))) + +(define (keep-max-version a b) + "Version comparison on the lists returned by the previous function takin= g the + janestreet re-versioning into account (v-prefixed come first)." + (match (cons a b) + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) + ((('V _ _) . _) a) + ((_ . ('V _ _)) b) + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a b)))) =20 (define (find-latest-version package repository) "Get the latest version of a package as described in the given repositor= y." - (let* ((dir (string-append repository "/packages/" package)) - (versions (scandir dir (lambda (name) (not (string-prefix? "." na= me)))))) - (if versions - (let ((versions (map - (lambda (dir) - (string-join (cdr (string-split dir #\.)) ".")) - versions))) - ;; Workaround for janestreet re-versionning - (let ((v-versions (filter (lambda (version) (string-prefix? "v" ve= rsion)) versions))) - (if (null? v-versions) - (latest-version versions) - (string-append "v" (latest-version (map (lambda (version) (sub= string version 1)) v-versions)))))) - (begin - (format #t (G_ "Package not found in opam repository: ~a~%") packa= ge) - #f)))) + (let ((packages (string-append repository "/packages")) + (filter (make-regexp (string-append "^" package "\\.")))) + (reduce keep-max-version #f + (filter-map + get-version-and-file + (find-files packages filter #:directories? #t))))) =20 (define (get-metadata opam-file) (with-input-from-file opam-file @@ -266,28 +302,30 @@ path to the repository." =20 (define (depends->native-inputs depends) (filter (lambda (name) (not (equal? "" name))) - (map dependency->native-input depends))) + (map dependency->native-input depends))) =20 (define (dependency-list->inputs lst) (map - (lambda (dependency) - (list dependency (list 'unquote (string->symbol dependency)))) - (ocaml-names->guix-names lst))) - -(define* (opam-fetch name #:optional (repository (get-opam-repository))) - (and-let* ((repository repository) - (version (find-latest-version name repository)) - (file (string-append repository "/packages/" name "/" name ".= " version "/opam"))) - `(("metadata" ,@(get-metadata file)) - ("version" . ,(if (string-prefix? "v" version) - (substring version 1) - version))))) - -(define* (opam->guix-package name #:key (repo 'opam) version) - "Import OPAM package NAME from REPOSITORY (a directory name) or, if -REPOSITORY is #f, from the official OPAM repository. Return a 'package' s= exp + (lambda (dependency) + (list dependency (list 'unquote (string->symbol dependency)))) + (ocaml-names->guix-names lst))) + +(define* (opam-fetch name #:optional (repositories-specs '("opam"))) + (or (fold (lambda (repository others) + (match (find-latest-version name repository) + ((_ version file) `(("metadata" ,@(get-metadata file)) + ("version" . ,version))) + (_ others))) + #f + (filter-map get-opam-repository repositories-specs)) + (leave (G_ "Package '~a' not found~%") name))) + +(define* (opam->guix-package name #:key (repo '()) version) + "Import OPAM package NAME from REPOSITORIES (a list of names, URLs or lo= cal +paths, always including OPAM's official repository). Return a 'package' s= exp or #f on failure." - (and-let* ((opam-file (opam-fetch name (get-opam-repository repo))) + (and-let* ((with-opam (if (member "opam" repo) repo (cons "opam" repo))) + (opam-file (opam-fetch name with-opam)) (version (assoc-ref opam-file "version")) (opam-content (assoc-ref opam-file "metadata")) (url-dict (metadata-ref opam-content "url")) @@ -312,9 +350,7 @@ or #f on failure." (values `(package (name ,(ocaml-name->guix-name name)) - (version ,(if (string-prefix? "v" version) - (substring version 1) - version)) + (version ,version) (source (origin (method url-fetch) diff --git a/guix/scripts/import/opam.scm b/guix/scripts/import/opam.scm index 64164e7cc4..834ac34cb0 100644 --- a/guix/scripts/import/opam.scm +++ b/guix/scripts/import/opam.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -46,7 +47,8 @@ Import and convert the opam package for PACKAGE-NAME.\n")) (display (G_ " -r, --recursive import packages recursively")) (display (G_ " - --repo import packages from this opam repository")) + --repo import packages from this opam repository (name, = URL or local path) + can be used more than once")) (display (G_ " -V, --version display version information and exit")) (newline) @@ -81,7 +83,9 @@ Import and convert the opam package for PACKAGE-NAME.\n")) #:build-options? #f)) =20 (let* ((opts (parse-options)) - (repo (and=3D> (assoc-ref opts 'repo) string->symbol)) + (repo (filter-map (match-lambda + (('repo . name) name) + (_ #f)) opts)) (args (filter-map (match-lambda (('argument . value) value) diff --git a/tests/opam.scm b/tests/opam.scm index f1e3b70cb0..1536b74339 100644 --- a/tests/opam.scm +++ b/tests/opam.scm @@ -82,41 +82,39 @@ url { (set! test-source-hash (call-with-input-file file-name port-sha256)))) (_ (error "Unexpected URL: " url))))) - (mock ((guix import opam) get-opam-repository - (const test-repo)) - (let ((my-package (string-append test-repo - "/packages/foo/foo.1.0.0"))) - (mkdir-p my-package) - (with-output-to-file (string-append my-package "/opam") - (lambda _ - (format #t "~a" test-opam-file)))) - (match (opam->guix-package "foo" #:repo test-repo) - (('package - ('name "ocaml-foo") - ('version "1.0.0") - ('source ('origin - ('method 'url-fetch) - ('uri "https://example.org/foo-1.0.0.tar.gz") - ('sha256 - ('base32 - (? string? hash))))) - ('build-system 'ocaml-build-system) - ('propagated-inputs - ('quasiquote - (("ocaml-zarith" ('unquote 'ocaml-zarith))))) - ('native-inputs - ('quasiquote - (("ocaml-alcotest" ('unquote 'ocaml-alcotest)) - ("ocamlbuild" ('unquote 'ocamlbuild))))) - ('home-page "https://example.org/") - ('synopsis "Some example package") - ('description "This package is just an example.") - ('license 'license:bsd-3)) - (string=3D? (bytevector->nix-base32-string - test-source-hash) - hash)) - (x - (pk 'fail x #f)))))) + (let ((my-package (string-append test-repo + "/packages/foo/foo.1.0.0"))) + (mkdir-p my-package) + (with-output-to-file (string-append my-package "/opam") + (lambda _ + (format #t "~a" test-opam-file)))) + (match (opam->guix-package "foo" #:repo (list test-repo)) + (('package + ('name "ocaml-foo") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri "https://example.org/foo-1.0.0.tar.gz") + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'ocaml-build-system) + ('propagated-inputs + ('quasiquote + (("ocaml-zarith" ('unquote 'ocaml-zarith))))) + ('native-inputs + ('quasiquote + (("ocaml-alcotest" ('unquote 'ocaml-alcotest)) + ("ocamlbuild" ('unquote 'ocamlbuild))))) + ('home-page "https://example.org/") + ('synopsis "Some example package") + ('description "This package is just an example.") + ('license 'license:bsd-3)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (pk 'fail x #f))))) =20 ;; Test the opam file parser ;; We fold over some test cases. Each case is a pair of the string to pars= e and the --=20 2.32.0 --MP_/83hnYGjSLbcyLCazTJrKRll-- From unknown Wed Aug 20 03:37:39 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#49958] [PATCH] More flexibility in opam importer Resent-From: Xinglu Chen Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 13 Aug 2021 07:39:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49958 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Alice BRENON , 49958@debbugs.gnu.org Received: via spool by 49958-submit@debbugs.gnu.org id=B49958.16288402848847 (code B ref 49958); Fri, 13 Aug 2021 07:39:01 +0000 Received: (at 49958) by debbugs.gnu.org; 13 Aug 2021 07:38:04 +0000 Received: from localhost ([127.0.0.1]:40458 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mERlO-0002IG-RZ for submit@debbugs.gnu.org; Fri, 13 Aug 2021 03:38:04 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:59548 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mERlK-0002Hy-2f for 49958@debbugs.gnu.org; Fri, 13 Aug 2021 03:37:57 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1628840264; bh=HQlzVtAUqSZvrM/g7PTOCxhUWRwPrqPTB0Gn61Y/eOM=; h=From:To:Subject:In-Reply-To:References:Date; b=SboMHInNfIc2IWUCBAl/1VkI4C9SCN/8JQaeig+nnNHaO3IzdDZCKluobOnipo8RP cXWpnc386Cf1oeIKzVfoqXPJTKFINqUXXp3wICQhptDuB97yBzdmno0T8UNsODGCPh CThIFv1PbgEhj2GmtgrXU1UaCiMzL3AXD+73yw0g= In-Reply-To: <20210810184826.17d14aab@ens-lyon.fr> References: <20210809140407.748fa019@ens-lyon.fr> <20210809171935.05fac773@ens-lyon.fr> <20210810140413.2f7d2f1b@ens-lyon.fr> <20210810184826.17d14aab@ens-lyon.fr> Date: Fri, 13 Aug 2021 09:37:43 +0200 Message-ID: <87pmuhhk1k.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Tue, Aug 10 2021, Alice BRENON wrote: > - rephrase the documentation. > - remove unnecessary #:ensure #f key in call to (cache-directory). > - clarify what happens when string->uri fails and yields a warning by > passing a 'bad-repo symbo [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Tue, Aug 10 2021, Alice BRENON wrote: > - rephrase the documentation. > - remove unnecessary #:ensure #f key in call to (cache-directory). > - clarify what happens when string->uri fails and yields a warning by > passing a 'bad-repo symbo [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Tue, Aug 10 2021, Alice BRENON wrote: > - rephrase the documentation. > - remove unnecessary #:ensure #f key in call to (cache-directory). > - clarify what happens when string->uri fails and yields a warning by > passing a 'bad-repo symbol and commenting at its elimination site. > - make a separate function of get-uri now that it's become as large as > its caller repo-type. > - mention the possibility to call --repo several times in help message. > - add points at the end of sentences in commit message and in this > e-mail. > From 49e8236f81462501e89aa40d4e1b77bcc3fbb0ad Mon Sep 17 00:00:00 2001 > From: Alice BRENON > Date: Sat, 7 Aug 2021 19:50:10 +0200 > Subject: [PATCH] guix: opam: More flexibility in the importer. > > * guix/scripts/import/opam.scm: pass all instances of --repo as a list > to the importer. > * guix/import/opam.scm (opam-fetch): stop expecting "expanded" > repositories and call get-opam-repository instead to keep values > "symbolic" as long as possible and factorize. > (get-opam-repository): use the same repository source as CLI opam does > (i.e. HTTP-served index.tar.gz instead of git repositories). > (find-latest-version): be more flexible on the repositories structure > instead of expecting packages/PACKAGE-NAME/PACKAGE-NAME.VERSION/. > * tests/opam.scm: update the call to opam->guix-package since repo is > now expected to be a list and remove the mocked get-opam-repository > deprecated by the support for local folders by the actual > implementation. > * doc/guix.texi: document the new semantics and valid arguments for the > --repo option. > --- > doc/guix.texi | 25 ++++-- > guix/import/opam.scm | 158 +++++++++++++++++++++-------------- > guix/scripts/import/opam.scm | 8 +- > tests/opam.scm | 68 ++++++++------- > 4 files changed, 155 insertions(+), 104 deletions(-) > > diff --git a/doc/guix.texi b/doc/guix.texi > index 4eb5324b51..4a911e4c0f 100644 > --- a/doc/guix.texi > +++ b/doc/guix.texi > @@ -94,6 +94,7 @@ Copyright @copyright{} 2021 Xinglu Chen@* > Copyright @copyright{} 2021 Raghav Gururajan@* > Copyright @copyright{} 2021 Domagoj Stolfa@* > Copyright @copyright{} 2021 Hui Lu@* > +Copyright @copyright{} 2021 Alice Brenon@* >=20=20 > Permission is granted to copy, distribute and/or modify this document > under the terms of the GNU Free Documentation License, Version 1.3 or > @@ -11612,14 +11613,26 @@ Traverse the dependency graph of the given upst= ream package recursively > and generate package expressions for all those packages that are not yet > in Guix. > @item --repo > -Select the given repository (a repository name). Possible values includ= e: > + > +By default, packages are searched in the official OPAM repository. This > +option, which can be used more than once, lets you add other > +repositories where to look for packages. =E2=80=9Clets you add other repositories where to look for package=E2=80=9D= sounds a bit weird, maybe lets you add other repositories which will be used to lookup packages. ? > @itemize > -@item @code{opam}, the default opam repository, > -@item @code{coq} or @code{coq-released}, the stable repository for coq p= ackages, > -@item @code{coq-core-dev}, the repository that contains development vers= ions of coq, > -@item @code{coq-extra-dev}, the repository that contains development ver= sions > - of coq packages. > +@item the name of a known repository - can be one of @code{opam}, > + @code{coq} (equivalent to @code{coq-released}), > + @code{coq-core-dev}, @code{coq-extra-dev} or @code{grew}. > +@item the URL of a repository as expected by the @code{opam repository > + add} command (for instance, the URL equivalent of the above > + @code{opam} name would be @uref{https://opam.ocaml.org}). > +@item the path to a local copy of a repository (a directory containing a > + @file{packages/} sub-directory). > @end itemize > + > +Please note that repositories added with this option do not replace the > +default @code{opam} repository, so calling this importer with the option > +@code{--repo=3Dopam} is redundant. What happens if I specify an additional repository, and a package exists in that repository _and_ the default opam repository? From which repository will the package be imported from? > diff --git a/guix/import/opam.scm b/guix/import/opam.scm > index a35b01d277..0e6cae72c4 100644 > --- a/guix/import/opam.scm > +++ b/guix/import/opam.scm > @@ -2,6 +2,7 @@ > ;;; Copyright =C2=A9 2018 Julien Lepiller > ;;; Copyright =C2=A9 2020 Martin Becze > ;;; Copyright =C2=A9 2021 Xinglu Chen > +;;; Copyright =C2=A9 2021 Alice Brenon > ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -22,21 +23,24 @@ > #:use-module (ice-9 ftw) > #:use-module (ice-9 match) > #:use-module (ice-9 peg) > + #:use-module ((ice-9 popen) #:select (open-pipe*)) > #:use-module (ice-9 receive) > - #:use-module ((ice-9 rdelim) #:select (read-line)) > #:use-module (ice-9 textual-ports) > #:use-module (ice-9 vlist) > #:use-module (srfi srfi-1) > #:use-module (srfi srfi-2) > - #:use-module (web uri) > + #:use-module ((srfi srfi-26) #:select (cut)) > + #:use-module ((web uri) #:select (string->uri uri->string)) > + #:use-module ((guix build utils) #:select (dump-port find-files mkdir-= p)) > #:use-module (guix build-system) > #:use-module (guix build-system ocaml) > #:use-module (guix http-client) > - #:use-module (guix git) > #:use-module (guix ui) > #:use-module (guix packages) > #:use-module (guix upstream) > - #:use-module (guix utils) > + #:use-module ((guix utils) #:select (cache-directory > + version>? > + call-with-temporary-output-file)) > #:use-module (guix import utils) > #:use-module ((guix licenses) #:prefix license:) > #:export (opam->guix-package > @@ -121,51 +125,83 @@ > (define-peg-pattern condition-string all (and QUOTE (* STRCHR) QUOTE)) > (define-peg-pattern condition-var all (+ (or (range #\a #\z) "-" ":"))) >=20=20 > -(define* (get-opam-repository #:optional repo) > +(define (opam-cache-directory path) > + (string-append (cache-directory) "/opam/" path)) > + > +(define known-repositories > + '((opam . "https://opam.ocaml.org") > + (coq . "https://coq.inria.fr/opam/released") > + (coq-released . "https://coq.inria.fr/opam/released") > + (coq-core-dev . "https://coq.inria.fr/opam/core-dev") > + (coq-extra-dev . "https://coq.inria.fr/opam/extra-dev") > + (grew . "http://opam.grew.fr"))) > + > +(define (get-uri repo-root) > + (let ((archive-file (string-append repo-root "/index.tar.gz"))) > + (or (string->uri archive-file) > + (begin > + (warning (G_ "'~a' is not a valid URI~%") archive-file) > + 'bad-repo)))) > + > +(define (repo-type repo) > + (match (assoc-ref known-repositories (string->symbol repo)) > + (#f (if (file-exists? repo) > + `(local ,repo) > + `(remote ,(get-uri repo)))) > + (url `(remote ,(get-uri url))))) > + > +(define (update-repository input) > + "Make sure the cache for opam repository INPUT is up-to-date" > + (let* ((output (opam-cache-directory (basename (port-filename input)))) > + (cached-date (if (file-exists? output) > + (stat:mtime (stat output)) > + (begin (mkdir-p output) 0)))) > + (when (> (stat:mtime (stat input)) cached-date) > + (call-with-port > + (open-pipe* OPEN_WRITE "tar" "xz" "-C" output "-f" "-") > + (cut dump-port input <>))) > + output)) > + > +(define* (get-opam-repository #:optional (repo "opam")) > "Update or fetch the latest version of the opam repository and return = the > path to the repository." > - (let ((url (cond > - ((or (not repo) (equal? repo 'opam)) > - "https://github.com/ocaml/opam-repository") > - ((string-prefix? "coq-" (symbol->string repo)) > - "https://github.com/coq/opam-coq-archive") > - ((equal? repo 'coq) "https://github.com/coq/opam-coq-arch= ive") > - (else (throw 'unknown-repository repo))))) > - (receive (location commit _) > - (update-cached-checkout url) > - (cond > - ((or (not repo) (equal? repo 'opam)) > - location) > - ((equal? repo 'coq) > - (string-append location "/released")) > - ((string-prefix? "coq-" (symbol->string repo)) > - (string-append location "/" (substring (symbol->string repo) 4)= )) > - (else location))))) > + (match (repo-type repo) > + (('local p) p) > + (('remote 'bad-repo) #f) ; to weed it out during filter-map in opam-= fetch > + (('remote r) (call-with-port (http-fetch/cached r) update-repository= )))) >=20=20 > ;; Prevent Guile 3 from inlining this procedure so we can mock it in tes= ts. > (set! get-opam-repository get-opam-repository) >=20=20 > -(define (latest-version versions) > - "Find the most recent version from a list of versions." > - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) versions)) > +(define (get-version-and-file path) > + "Analyse a candidate path and return an list containing information fo= r proper > + version comparison as well as the source path for metadata." > + (and-let* ((metadata-file (string-append path "/opam")) > + (filename (basename path)) > + (version (string-join (cdr (string-split filename #\.)) "."= ))) > + (and (file-exists? metadata-file) > + (eq? 'regular (stat:type (stat metadata-file))) > + (if (string-prefix? "v" version) > + `(V ,(substring version 1) ,metadata-file) > + `(digits ,version ,metadata-file))))) What happens if some other prefix is used, e.g., =E2=80=9Crelease-=E2=80=9D= or =E2=80=9CV-=E2=80=9D? Also, why not just return the version number and the metadata file; we don=E2=80=99t really care about the prefix do we? > +(define (keep-max-version a b) > + "Version comparison on the lists returned by the previous function tak= ing the > + janestreet re-versioning into account (v-prefixed come first)." > + (match (cons a b) > + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) > + ((('V _ _) . _) a) > + ((_ . ('V _ _)) b) > + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a b)))) >=20=20 > (define (find-latest-version package repository) > "Get the latest version of a package as described in the given reposit= ory." > - (let* ((dir (string-append repository "/packages/" package)) > - (versions (scandir dir (lambda (name) (not (string-prefix? "." = name)))))) > - (if versions > - (let ((versions (map > - (lambda (dir) > - (string-join (cdr (string-split dir #\.)) ".")) > - versions))) > - ;; Workaround for janestreet re-versionning > - (let ((v-versions (filter (lambda (version) (string-prefix? "v" = version)) versions))) > - (if (null? v-versions) > - (latest-version versions) > - (string-append "v" (latest-version (map (lambda (version) (s= ubstring version 1)) v-versions)))))) > - (begin > - (format #t (G_ "Package not found in opam repository: ~a~%") pac= kage) > - #f)))) > + (let ((packages (string-append repository "/packages")) > + (filter (make-regexp (string-append "^" package "\\.")))) > + (reduce keep-max-version #f > + (filter-map > + get-version-and-file > + (find-files packages filter #:directories? #t))))) >=20=20 > (define (get-metadata opam-file) > (with-input-from-file opam-file > @@ -266,28 +302,30 @@ path to the repository." >=20=20 > (define (depends->native-inputs depends) > (filter (lambda (name) (not (equal? "" name))) > - (map dependency->native-input depends))) > + (map dependency->native-input depends))) >=20=20 > (define (dependency-list->inputs lst) > (map > - (lambda (dependency) > - (list dependency (list 'unquote (string->symbol dependency)))) > - (ocaml-names->guix-names lst))) > - > -(define* (opam-fetch name #:optional (repository (get-opam-repository))) > - (and-let* ((repository repository) > - (version (find-latest-version name repository)) > - (file (string-append repository "/packages/" name "/" name = "." version "/opam"))) > - `(("metadata" ,@(get-metadata file)) > - ("version" . ,(if (string-prefix? "v" version) > - (substring version 1) > - version))))) > - > -(define* (opam->guix-package name #:key (repo 'opam) version) > - "Import OPAM package NAME from REPOSITORY (a directory name) or, if > -REPOSITORY is #f, from the official OPAM repository. Return a 'package'= sexp > + (lambda (dependency) > + (list dependency (list 'unquote (string->symbol dependency)))) > + (ocaml-names->guix-names lst))) > + > +(define* (opam-fetch name #:optional (repositories-specs '("opam"))) > + (or (fold (lambda (repository others) > + (match (find-latest-version name repository) > + ((_ version file) `(("metadata" ,@(get-metadata file)) > + ("version" . ,version))) > + (_ others))) > + #f > + (filter-map get-opam-repository repositories-specs)) > + (leave (G_ "Package '~a' not found~%") name))) Nit: I wouldn=E2=80=99t capitalize =E2=80=9Cpackage=E2=80=9D, otherwise the= error message looks like this guix import: error: Package 'equations' not found Otherwise, LGTM! Great to see more work on the importers! :-) --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmEWIUcVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5sloP/j9GlNL6cVOqlrmzWPOSblnPHGlz Ds5mRFshWiBa52F8BlmRW4p3XnDoahHqewPVMOQef8xxOYzKSKhNo+qUmqfO22Mj z1qFD4gX6c3O9YMuRXfcDynvTaesFLkU0ddu+OaP634fYk62uP922p3JVk8cV8VZ /IgIjermbTNMtY7AfEMBwr37To4Hdq7J8lyq1VQdKxXKbmyqDGBoQC0LELNfmwlp oKjWmUxGC31kxuzZU0iBq84eEcE0kkuXc+kThXZZ6coIoVShMrFB6HoxLM+PiQev br39GgJzlhW2ypFDbYVzGpQdcuiVPr5zPIZuaBFXm40NYMWpLZw+Xhuv9kKHChb6 puRFQlywrxBre6gfx4rUstRWgOjZPUUdeRM+yhn2Kt5pSQat+9bpR8TnoMeMX0hD BkO+hxBqKc072g9WSZ3jgAh40WJiMxv6H34BO8RgTtJ72hS81l7PIw5Mxl+JpCRa iQAPgOrJ9rLE3h59WIueGJ129n5sGc3JxFMwR4Vp0h6OnaSM3uClWDZmlsmhUP34 o0KsI7KU0ErkL3q4gZESK8wrHvT88inCmaW+r4cS5+qk4TSpFe3MgSFtEEM4m3Z0 dHkhusWz87r3/A5P238Qku6MAsKwgutx8PmNzcuyuyvSl8tjDM4GeF9kT+Rmi6eF 5sqW8OLvBFGlzFqF =34CG -----END PGP SIGNATURE----- --=-=-=-- From unknown Wed Aug 20 03:37:39 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#49958] [PATCH] More flexibility in opam importer Resent-From: Alice BRENON Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 13 Aug 2021 11:12:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49958 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Xinglu Chen Cc: 49958@debbugs.gnu.org Received: via spool by 49958-submit@debbugs.gnu.org id=B49958.16288530826750 (code B ref 49958); Fri, 13 Aug 2021 11:12:02 +0000 Received: (at 49958) by debbugs.gnu.org; 13 Aug 2021 11:11:22 +0000 Received: from localhost ([127.0.0.1]:40593 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mEV5n-0001kf-FX for submit@debbugs.gnu.org; Fri, 13 Aug 2021 07:11:22 -0400 Received: from lxc-smtp2.ens-lyon.fr ([140.77.167.81]:39012) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mEV5j-0001kV-Vn for 49958@debbugs.gnu.org; Fri, 13 Aug 2021 07:11:13 -0400 Received: from localhost (localhost [127.0.0.1]) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTP id 91280E2DBD; Fri, 13 Aug 2021 13:11:10 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.11.0 (20160426) (Debian) at ens-lyon.fr Received: from lxc-smtp2.ens-lyon.fr ([127.0.0.1]) by localhost (lxc-smtp2.ens-lyon.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6D4NyVWOfuI3; Fri, 13 Aug 2021 13:11:10 +0200 (CEST) Received: from localhost (unknown [78.194.167.103]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTPSA id 5001EE2CFA; Fri, 13 Aug 2021 13:11:10 +0200 (CEST) Date: Fri, 13 Aug 2021 13:11:09 +0200 From: Alice BRENON Message-ID: <20210813131109.14c25204@ens-lyon.fr> In-Reply-To: <87pmuhhk1k.fsf@yoctocell.xyz> References: <20210809140407.748fa019@ens-lyon.fr> <20210809171935.05fac773@ens-lyon.fr> <20210810140413.2f7d2f1b@ens-lyon.fr> <20210810184826.17d14aab@ens-lyon.fr> <87pmuhhk1k.fsf@yoctocell.xyz> Organization: ENS de Lyon X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.24; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/CNH00mJos/vwxi9.UARMLah" X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Le Fri, 13 Aug 2021 09:37:43 +0200, Xinglu Chen a =?UTF-8?Q?=C3=A9crit?= : > On Tue, Aug 10 2021, Alice BRENON wrote: > > =?UTF-8?Q?[=E2=80=A6]?= > > include: + > > +By default, packages are searched in the official OPAM repository. > > This +option, which can be used more than once, lets you [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) --MP_/CNH00mJos/vwxi9.UARMLah Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Le Fri, 13 Aug 2021 09:37:43 +0200, Xinglu Chen a =C3=A9crit : > On Tue, Aug 10 2021, Alice BRENON wrote: >=20 > [=E2=80=A6] > > include: + > > +By default, packages are searched in the official OPAM repository. > > This +option, which can be used more than once, lets you add other > > +repositories where to look for packages. =20 >=20 > =E2=80=9Clets you add other repositories where to look for package=E2=80= =9D sounds a > bit weird, maybe >=20 > lets you add other repositories which will be used to lookup > packages. >=20 > ? Ok, as discussed on IRC, trying "lets you add other repositories which will be searched for packages". > What happens if I specify an additional repository, and a package > exists in that repository _and_ the default opam repository? From > which repository will the package be imported from? That is the beauty of it: the repositories are assumed to be passed by order of preference, defaulting to the official opam repositories only if packages haven't been found anywhere else. Writing this makes me realize that indeed, starting with --repo=3Dopam isn't entirely redundant: it could be used to prevent an otherwise interesting repo from overriding stuff if opam already provides it (let's assume some "super-opam" with a couple additional packages, and custom versions of existing opam packages). Calling `--repo=3Dsuper-opam` would use the super-opam versions as soon as a package exists in super-opam, while `--repo=3Dopam --repo=3Dsuper-opam` would take the super-opam versions only when none exist in opam. A much simpler use-case would be to locally override only some packages in a repo, and pass --repo=3Doverriden-repo --repo=3Dnormal-repo. This behaviour relies on the implementation of opam-fetch and how folds work in guile. Since in the importer script options are stacked as they are retrieved from the CLI arguments, and repositories are then just filter-maped from that list, they end up in a list by reverse order of preference. In opam->guix-package, 'opam gets push on the top if it's not already there somewhere. So what we get as input of opam-fetch is a list of repositories-specs by reverse order of preference. Now fold applies the accumulator to each item in order, so, last elements has the final say, i.e. the last elements which yield results in find-latests are preferred over the earlier elements of the list. This works for the same reason why `(lambda (l) (fold cons '() l)` will reverse its input list. It's slightly inefficient because it means all repositories are searched, in reverse order of preference, but I haven't figured how to get a lazy fold in guile. Granted, I could have written the recursion explicitly to get that. Will fix if performance matters. Also, versions are not compared between repositories, as soon as a repo provides one version of a given package, the latest of all the versions this one provides is used in the output, no matter the contents of other repositories. This is useful to allow "downgrades" by masking parts of repository which have too recent versions. So, thanks for your remark, the documentation deserved a clearer explanation of it. > [=E2=80=A6] > > -(define (latest-version versions) > > - "Find the most recent version from a list of versions." > > - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) > > versions)) +(define (get-version-and-file path) > > + "Analyse a candidate path and return an list containing > > information for proper > > + version comparison as well as the source path for metadata." > > + (and-let* ((metadata-file (string-append path "/opam")) > > + (filename (basename path)) > > + (version (string-join (cdr (string-split filename > > #\.)) "."))) > > + (and (file-exists? metadata-file) > > + (eq? 'regular (stat:type (stat metadata-file))) > > + (if (string-prefix? "v" version) > > + `(V ,(substring version 1) ,metadata-file) > > + `(digits ,version ,metadata-file))))) =20 >=20 > What happens if some other prefix is used, e.g., =E2=80=9Crelease-=E2=80= =9D or =E2=80=9CV-=E2=80=9D? >=20 It would get marked as a 'digit. In a previous draft before I started sending this series of patches, it was called 'R, standing for "regular", then I thought it was not very meaningful, and, since the versions were to my knowledge supposed to be either v[0-9]+(\.[0-9]+)* or [0-9]+(\.[0-9]+)*, I thought I could call that default case "digits" to clearly indicate what I was trying to refer to. I could change it to 'other if it matters too much, but the important thing here is that we distinguish between v-prefixed (the so-called "janestreet re-versionning" mentioned inside the implementation of find-latest on current d87d6d6 master) and other versions because =E2=AC=87=EF=B8=8F > Also, why not just return the version number and the metadata file; we > don=E2=80=99t really care about the prefix do we? >=20 yes we do ! the former latest-version finder handled strings, and dropped this prefix or put it back on the fly, but the logic implemented was: if there are v-prefixed versions, find the greatest of them, without the initial "v", if there aren't, just find the greatest of all versions. This implies that v-prefixed versions are considered more important and automatically greater than non-prefixed versions, no matter what the numbers, which is why this information must be kept. I'm just playing ADTs in guile here, "parsing" the version string only once to retain a symbolic representation of it that will at first glance allow to identify the type of version used and access the relevant digits for comparison. The comparison is used right after: > > +(define (keep-max-version a b) > > + "Version comparison on the lists returned by the previous > > function taking the > > + janestreet re-versioning into account (v-prefixed come first)." > > + (match (cons a b) > > + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) > > + ((('V _ _) . _) a) > > + ((_ . ('V _ _)) b) > > + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a > > b))))=20 and used in the reduce in find-latest-version. So keeping this 'V is what will help janestreet-re-versionned packages "skip the line" by being automatically greater than any non v-prefixed package (thus, v0.0.1 is greater than 13.2, which is the current behaviour). > > (define (find-latest-version package repository) > > "Get the latest version of a package as described in the given > > repository." > > - (let* ((dir (string-append repository "/packages/" package)) > > - (versions (scandir dir (lambda (name) (not > > (string-prefix? "." name)))))) > > - (if versions > > - (let ((versions (map > > - (lambda (dir) > > - (string-join (cdr (string-split dir > > #\.)) ".")) > > - versions))) > > - ;; Workaround for janestreet re-versionning > > - (let ((v-versions (filter (lambda (version) > > (string-prefix? "v" version)) versions))) > > - (if (null? v-versions) > > - (latest-version versions) > > - (string-append "v" (latest-version (map (lambda > > (version) (substring version 1)) v-versions)))))) > > - (begin > > - (format #t (G_ "Package not found in opam repository: > > ~a~%") package) > > - #f)))) > > + (let ((packages (string-append repository "/packages")) > > + (filter (make-regexp (string-append "^" package "\\.")))) > > + (reduce keep-max-version #f > > + (filter-map > > + get-version-and-file > > + (find-files packages filter #:directories? #t))))) > > =20 > [=E2=80=A6] > > + (filter-map get-opam-repository repositories-specs)) > > + (leave (G_ "Package '~a' not found~%") name))) =20 >=20 > Nit: I wouldn=E2=80=99t capitalize =E2=80=9Cpackage=E2=80=9D, otherwise t= he error message > looks like this >=20 > guix import: error: Package 'equations' not found a very neat tip, thank you ! : ) --MP_/CNH00mJos/vwxi9.UARMLah Content-Type: text/x-patch Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=0001-guix-opam-More-flexibility-in-the-importer.patch =46rom cde8b2a5d88d89bfea31c86d3ae94d37c1d3c83f Mon Sep 17 00:00:00 2001 From: Alice BRENON Date: Sat, 7 Aug 2021 19:50:10 +0200 Subject: [PATCH] guix: opam: More flexibility in the importer. * guix/scripts/import/opam.scm: pass all instances of --repo as a list to the importer. * guix/import/opam.scm (opam-fetch): stop expecting "expanded" repositories and call get-opam-repository instead to keep values "symbolic" as long as possible and factorize. (get-opam-repository): use the same repository source as CLI opam does (i.e. HTTP-served index.tar.gz instead of git repositories). (find-latest-version): be more flexible on the repositories structure instead of expecting packages/PACKAGE-NAME/PACKAGE-NAME.VERSION/. * tests/opam.scm: update the call to opam->guix-package since repo is now expected to be a list and remove the mocked get-opam-repository deprecated by the support for local folders by the actual implementation. * doc/guix.texi: document the new semantics and valid arguments for the --repo option. --- doc/guix.texi | 30 +++++-- guix/import/opam.scm | 158 +++++++++++++++++++++-------------- guix/scripts/import/opam.scm | 8 +- tests/opam.scm | 68 ++++++++------- 4 files changed, 160 insertions(+), 104 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 78c1c09858..2d36561186 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -94,6 +94,7 @@ Copyright @copyright{} 2021 Xinglu Chen@* Copyright @copyright{} 2021 Raghav Gururajan@* Copyright @copyright{} 2021 Domagoj Stolfa@* Copyright @copyright{} 2021 Hui Lu@* +Copyright @copyright{} 2021 Alice Brenon@* =20 Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or @@ -11612,14 +11613,31 @@ Traverse the dependency graph of the given upstre= am package recursively and generate package expressions for all those packages that are not yet in Guix. @item --repo -Select the given repository (a repository name). Possible values include: + +By default, packages are searched in the official OPAM repository. This +option, which can be used more than once, lets you add other +repositories which will be searched for packages. It accepts as valid +arguments: + @itemize -@item @code{opam}, the default opam repository, -@item @code{coq} or @code{coq-released}, the stable repository for coq pac= kages, -@item @code{coq-core-dev}, the repository that contains development versio= ns of coq, -@item @code{coq-extra-dev}, the repository that contains development versi= ons - of coq packages. +@item the name of a known repository - can be one of @code{opam}, + @code{coq} (equivalent to @code{coq-released}), + @code{coq-core-dev}, @code{coq-extra-dev} or @code{grew}. +@item the URL of a repository as expected by the @code{opam repository + add} command (for instance, the URL equivalent of the above + @code{opam} name would be @uref{https://opam.ocaml.org}). +@item the path to a local copy of a repository (a directory containing a + @file{packages/} sub-directory). @end itemize + +Repositories must be passed to this option by order of preference and do +not replace the default @code{opam} which is always failed-back to. + +Also, please note that versions are not compared accross repositories. +The first repository (from left to right) that has at least one version +of a given package will prevail over any others and the version imported +will be the latest one found @emph{in this repository only}. + @end table =20 @item go diff --git a/guix/import/opam.scm b/guix/import/opam.scm index a35b01d277..fe13d29f03 100644 --- a/guix/import/opam.scm +++ b/guix/import/opam.scm @@ -2,6 +2,7 @@ ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2020 Martin Becze ;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -22,21 +23,24 @@ #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (ice-9 peg) + #:use-module ((ice-9 popen) #:select (open-pipe*)) #:use-module (ice-9 receive) - #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (ice-9 textual-ports) #:use-module (ice-9 vlist) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) - #:use-module (web uri) + #:use-module ((srfi srfi-26) #:select (cut)) + #:use-module ((web uri) #:select (string->uri uri->string)) + #:use-module ((guix build utils) #:select (dump-port find-files mkdir-p)) #:use-module (guix build-system) #:use-module (guix build-system ocaml) #:use-module (guix http-client) - #:use-module (guix git) #:use-module (guix ui) #:use-module (guix packages) #:use-module (guix upstream) - #:use-module (guix utils) + #:use-module ((guix utils) #:select (cache-directory + version>? + call-with-temporary-output-file)) #:use-module (guix import utils) #:use-module ((guix licenses) #:prefix license:) #:export (opam->guix-package @@ -121,51 +125,83 @@ (define-peg-pattern condition-string all (and QUOTE (* STRCHR) QUOTE)) (define-peg-pattern condition-var all (+ (or (range #\a #\z) "-" ":"))) =20 -(define* (get-opam-repository #:optional repo) +(define (opam-cache-directory path) + (string-append (cache-directory) "/opam/" path)) + +(define known-repositories + '((opam . "https://opam.ocaml.org") + (coq . "https://coq.inria.fr/opam/released") + (coq-released . "https://coq.inria.fr/opam/released") + (coq-core-dev . "https://coq.inria.fr/opam/core-dev") + (coq-extra-dev . "https://coq.inria.fr/opam/extra-dev") + (grew . "http://opam.grew.fr"))) + +(define (get-uri repo-root) + (let ((archive-file (string-append repo-root "/index.tar.gz"))) + (or (string->uri archive-file) + (begin + (warning (G_ "'~a' is not a valid URI~%") archive-file) + 'bad-repo)))) + +(define (repo-type repo) + (match (assoc-ref known-repositories (string->symbol repo)) + (#f (if (file-exists? repo) + `(local ,repo) + `(remote ,(get-uri repo)))) + (url `(remote ,(get-uri url))))) + +(define (update-repository input) + "Make sure the cache for opam repository INPUT is up-to-date" + (let* ((output (opam-cache-directory (basename (port-filename input)))) + (cached-date (if (file-exists? output) + (stat:mtime (stat output)) + (begin (mkdir-p output) 0)))) + (when (> (stat:mtime (stat input)) cached-date) + (call-with-port + (open-pipe* OPEN_WRITE "tar" "xz" "-C" output "-f" "-") + (cut dump-port input <>))) + output)) + +(define* (get-opam-repository #:optional (repo "opam")) "Update or fetch the latest version of the opam repository and return the path to the repository." - (let ((url (cond - ((or (not repo) (equal? repo 'opam)) - "https://github.com/ocaml/opam-repository") - ((string-prefix? "coq-" (symbol->string repo)) - "https://github.com/coq/opam-coq-archive") - ((equal? repo 'coq) "https://github.com/coq/opam-coq-archiv= e") - (else (throw 'unknown-repository repo))))) - (receive (location commit _) - (update-cached-checkout url) - (cond - ((or (not repo) (equal? repo 'opam)) - location) - ((equal? repo 'coq) - (string-append location "/released")) - ((string-prefix? "coq-" (symbol->string repo)) - (string-append location "/" (substring (symbol->string repo) 4))) - (else location))))) + (match (repo-type repo) + (('local p) p) + (('remote 'bad-repo) #f) ; to weed it out during filter-map in opam-fe= tch + (('remote r) (call-with-port (http-fetch/cached r) update-repository))= )) =20 ;; Prevent Guile 3 from inlining this procedure so we can mock it in tests. (set! get-opam-repository get-opam-repository) =20 -(define (latest-version versions) - "Find the most recent version from a list of versions." - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) versions)) +(define (get-version-and-file path) + "Analyse a candidate path and return an list containing information for = proper + version comparison as well as the source path for metadata." + (and-let* ((metadata-file (string-append path "/opam")) + (filename (basename path)) + (version (string-join (cdr (string-split filename #\.)) "."))) + (and (file-exists? metadata-file) + (eq? 'regular (stat:type (stat metadata-file))) + (if (string-prefix? "v" version) + `(V ,(substring version 1) ,metadata-file) + `(digits ,version ,metadata-file))))) + +(define (keep-max-version a b) + "Version comparison on the lists returned by the previous function takin= g the + janestreet re-versioning into account (v-prefixed come first)." + (match (cons a b) + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) + ((('V _ _) . _) a) + ((_ . ('V _ _)) b) + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a b)))) =20 (define (find-latest-version package repository) "Get the latest version of a package as described in the given repositor= y." - (let* ((dir (string-append repository "/packages/" package)) - (versions (scandir dir (lambda (name) (not (string-prefix? "." na= me)))))) - (if versions - (let ((versions (map - (lambda (dir) - (string-join (cdr (string-split dir #\.)) ".")) - versions))) - ;; Workaround for janestreet re-versionning - (let ((v-versions (filter (lambda (version) (string-prefix? "v" ve= rsion)) versions))) - (if (null? v-versions) - (latest-version versions) - (string-append "v" (latest-version (map (lambda (version) (sub= string version 1)) v-versions)))))) - (begin - (format #t (G_ "Package not found in opam repository: ~a~%") packa= ge) - #f)))) + (let ((packages (string-append repository "/packages")) + (filter (make-regexp (string-append "^" package "\\.")))) + (reduce keep-max-version #f + (filter-map + get-version-and-file + (find-files packages filter #:directories? #t))))) =20 (define (get-metadata opam-file) (with-input-from-file opam-file @@ -266,28 +302,30 @@ path to the repository." =20 (define (depends->native-inputs depends) (filter (lambda (name) (not (equal? "" name))) - (map dependency->native-input depends))) + (map dependency->native-input depends))) =20 (define (dependency-list->inputs lst) (map - (lambda (dependency) - (list dependency (list 'unquote (string->symbol dependency)))) - (ocaml-names->guix-names lst))) - -(define* (opam-fetch name #:optional (repository (get-opam-repository))) - (and-let* ((repository repository) - (version (find-latest-version name repository)) - (file (string-append repository "/packages/" name "/" name ".= " version "/opam"))) - `(("metadata" ,@(get-metadata file)) - ("version" . ,(if (string-prefix? "v" version) - (substring version 1) - version))))) - -(define* (opam->guix-package name #:key (repo 'opam) version) - "Import OPAM package NAME from REPOSITORY (a directory name) or, if -REPOSITORY is #f, from the official OPAM repository. Return a 'package' s= exp + (lambda (dependency) + (list dependency (list 'unquote (string->symbol dependency)))) + (ocaml-names->guix-names lst))) + +(define* (opam-fetch name #:optional (repositories-specs '("opam"))) + (or (fold (lambda (repository others) + (match (find-latest-version name repository) + ((_ version file) `(("metadata" ,@(get-metadata file)) + ("version" . ,version))) + (_ others))) + #f + (filter-map get-opam-repository repositories-specs)) + (leave (G_ "package '~a' not found~%") name))) + +(define* (opam->guix-package name #:key (repo '()) version) + "Import OPAM package NAME from REPOSITORIES (a list of names, URLs or lo= cal +paths, always including OPAM's official repository). Return a 'package' s= exp or #f on failure." - (and-let* ((opam-file (opam-fetch name (get-opam-repository repo))) + (and-let* ((with-opam (if (member "opam" repo) repo (cons "opam" repo))) + (opam-file (opam-fetch name with-opam)) (version (assoc-ref opam-file "version")) (opam-content (assoc-ref opam-file "metadata")) (url-dict (metadata-ref opam-content "url")) @@ -312,9 +350,7 @@ or #f on failure." (values `(package (name ,(ocaml-name->guix-name name)) - (version ,(if (string-prefix? "v" version) - (substring version 1) - version)) + (version ,version) (source (origin (method url-fetch) diff --git a/guix/scripts/import/opam.scm b/guix/scripts/import/opam.scm index 64164e7cc4..834ac34cb0 100644 --- a/guix/scripts/import/opam.scm +++ b/guix/scripts/import/opam.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -46,7 +47,8 @@ Import and convert the opam package for PACKAGE-NAME.\n")) (display (G_ " -r, --recursive import packages recursively")) (display (G_ " - --repo import packages from this opam repository")) + --repo import packages from this opam repository (name, = URL or local path) + can be used more than once")) (display (G_ " -V, --version display version information and exit")) (newline) @@ -81,7 +83,9 @@ Import and convert the opam package for PACKAGE-NAME.\n")) #:build-options? #f)) =20 (let* ((opts (parse-options)) - (repo (and=3D> (assoc-ref opts 'repo) string->symbol)) + (repo (filter-map (match-lambda + (('repo . name) name) + (_ #f)) opts)) (args (filter-map (match-lambda (('argument . value) value) diff --git a/tests/opam.scm b/tests/opam.scm index f1e3b70cb0..1536b74339 100644 --- a/tests/opam.scm +++ b/tests/opam.scm @@ -82,41 +82,39 @@ url { (set! test-source-hash (call-with-input-file file-name port-sha256)))) (_ (error "Unexpected URL: " url))))) - (mock ((guix import opam) get-opam-repository - (const test-repo)) - (let ((my-package (string-append test-repo - "/packages/foo/foo.1.0.0"))) - (mkdir-p my-package) - (with-output-to-file (string-append my-package "/opam") - (lambda _ - (format #t "~a" test-opam-file)))) - (match (opam->guix-package "foo" #:repo test-repo) - (('package - ('name "ocaml-foo") - ('version "1.0.0") - ('source ('origin - ('method 'url-fetch) - ('uri "https://example.org/foo-1.0.0.tar.gz") - ('sha256 - ('base32 - (? string? hash))))) - ('build-system 'ocaml-build-system) - ('propagated-inputs - ('quasiquote - (("ocaml-zarith" ('unquote 'ocaml-zarith))))) - ('native-inputs - ('quasiquote - (("ocaml-alcotest" ('unquote 'ocaml-alcotest)) - ("ocamlbuild" ('unquote 'ocamlbuild))))) - ('home-page "https://example.org/") - ('synopsis "Some example package") - ('description "This package is just an example.") - ('license 'license:bsd-3)) - (string=3D? (bytevector->nix-base32-string - test-source-hash) - hash)) - (x - (pk 'fail x #f)))))) + (let ((my-package (string-append test-repo + "/packages/foo/foo.1.0.0"))) + (mkdir-p my-package) + (with-output-to-file (string-append my-package "/opam") + (lambda _ + (format #t "~a" test-opam-file)))) + (match (opam->guix-package "foo" #:repo (list test-repo)) + (('package + ('name "ocaml-foo") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri "https://example.org/foo-1.0.0.tar.gz") + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'ocaml-build-system) + ('propagated-inputs + ('quasiquote + (("ocaml-zarith" ('unquote 'ocaml-zarith))))) + ('native-inputs + ('quasiquote + (("ocaml-alcotest" ('unquote 'ocaml-alcotest)) + ("ocamlbuild" ('unquote 'ocamlbuild))))) + ('home-page "https://example.org/") + ('synopsis "Some example package") + ('description "This package is just an example.") + ('license 'license:bsd-3)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (pk 'fail x #f))))) =20 ;; Test the opam file parser ;; We fold over some test cases. Each case is a pair of the string to pars= e and the --=20 2.32.0 --MP_/CNH00mJos/vwxi9.UARMLah-- From unknown Wed Aug 20 03:37:39 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#49958] [PATCH] More flexibility in opam importer Resent-From: Xinglu Chen Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 13 Aug 2021 13:14:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49958 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Alice BRENON Cc: 49958@debbugs.gnu.org Received: via spool by 49958-submit@debbugs.gnu.org id=B49958.162886042520737 (code B ref 49958); Fri, 13 Aug 2021 13:14:01 +0000 Received: (at 49958) by debbugs.gnu.org; 13 Aug 2021 13:13:45 +0000 Received: from localhost ([127.0.0.1]:40862 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mEX0G-0005OL-I4 for submit@debbugs.gnu.org; Fri, 13 Aug 2021 09:13:44 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:39160 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mEX0A-0005O2-Uf for 49958@debbugs.gnu.org; Fri, 13 Aug 2021 09:13:39 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1628860404; bh=yTUN3qsUeH8+oU2eD8kwglWY5vrWlvqft0BjC3fIMdU=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=WHiXyfPr/VF/VzFQCKlU74t0AhlZ3U+GBZiMnHIvL/3PWe4MKOqZ4hyS+i4xMhc3N YYLE3fE9qFsepgU8n+skTRTRT3abcS0Q1YYB+v4P3gwZyDs2ELSlZYzwYx0uJUrrfB 0vzr5V1a62LlCMRVdZiWf7+XJXPvswdUArDCegrA= In-Reply-To: <20210813131109.14c25204@ens-lyon.fr> References: <20210809140407.748fa019@ens-lyon.fr> <20210809171935.05fac773@ens-lyon.fr> <20210810140413.2f7d2f1b@ens-lyon.fr> <20210810184826.17d14aab@ens-lyon.fr> <87pmuhhk1k.fsf@yoctocell.xyz> <20210813131109.14c25204@ens-lyon.fr> Date: Fri, 13 Aug 2021 15:13:14 +0200 Message-ID: <87h7fth4id.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Fri, Aug 13 2021, Alice BRENON wrote: > Le Fri, 13 Aug 2021 09:37:43 +0200, > Xinglu Chen a =?UTF-8?Q?=C3=A9crit?= : > >> On Tue, Aug 10 2021, Alice BRENON wrote: >> >> =?UTF-8?Q?[=E2=80=A6]?= >> > include: + >> > +By default, packages are searched [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Fri, Aug 13 2021, Alice BRENON wrote: > Le Fri, 13 Aug 2021 09:37:43 +0200, > Xinglu Chen a =?UTF-8?Q?=C3=A9crit?= : > >> On Tue, Aug 10 2021, Alice BRENON wrote: >> >> =?UTF-8?Q?[=E2=80=A6]?= >> > include: + >> > +By default, packages are searched [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Fri, Aug 13 2021, Alice BRENON wrote: > Le Fri, 13 Aug 2021 09:37:43 +0200, > Xinglu Chen a =C3=A9crit : > >> On Tue, Aug 10 2021, Alice BRENON wrote: >>=20 >> [=E2=80=A6] >> > include: + >> > +By default, packages are searched in the official OPAM repository. >> > This +option, which can be used more than once, lets you add other >> > +repositories where to look for packages.=20=20 >>=20 >> =E2=80=9Clets you add other repositories where to look for package=E2=80= =9D sounds a >> bit weird, maybe >>=20 >> lets you add other repositories which will be used to lookup >> packages. >>=20 >> ? > > Ok, as discussed on IRC, trying "lets you add other repositories which > will be searched for packages". > > >> What happens if I specify an additional repository, and a package >> exists in that repository _and_ the default opam repository? From >> which repository will the package be imported from? > > That is the beauty of it: the repositories are assumed to be passed by > order of preference, defaulting to the official opam repositories only > if packages haven't been found anywhere else. Writing this makes me > realize that indeed, starting with --repo=3Dopam isn't entirely > redundant: it could be used to prevent an otherwise interesting repo > from overriding stuff if opam already provides it (let's assume some > "super-opam" with a couple additional packages, and custom versions of > existing opam packages). > > Calling `--repo=3Dsuper-opam` would use the super-opam versions as soon > as a package exists in super-opam, while `--repo=3Dopam > --repo=3Dsuper-opam` would take the super-opam versions only when none > exist in opam. > > A much simpler use-case would be to locally override only some > packages in a repo, and pass --repo=3Doverriden-repo --repo=3Dnormal-repo. > > This behaviour relies on the implementation of opam-fetch and how folds > work in guile. > > Since in the importer script options are stacked as they are retrieved > from the CLI arguments, and repositories are then just filter-maped from > that list, they end up in a list by reverse order of preference. In > opam->guix-package, 'opam gets push on the top if it's not already > there somewhere. So what we get as input of opam-fetch is a list of > repositories-specs by reverse order of preference. Now fold applies the > accumulator to each item in order, so, last elements has the final say, > i.e. the last elements which yield results in find-latests are > preferred over the earlier elements of the list. This works for the > same reason why `(lambda (l) (fold cons '() l)` will reverse its input > list. It's slightly inefficient because it means all repositories are > searched, in reverse order of preference, but I haven't figured how to > get a lazy fold in guile. Granted, I could have written the recursion > explicitly to get that. Will fix if performance matters. > > Also, versions are not compared between repositories, as soon as a repo > provides one version of a given package, the latest of all the versions > this one provides is used in the output, no matter the contents of > other repositories. This is useful to allow "downgrades" by masking > parts of repository which have too recent versions. > > So, thanks for your remark, the documentation deserved a clearer > explanation of it. Thanks for the explanation! And great that you also documented this to avoid ambiguity. >> > -(define (latest-version versions) >> > - "Find the most recent version from a list of versions." >> > - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) >> > versions)) +(define (get-version-and-file path) >> > + "Analyse a candidate path and return an list containing >> > information for proper >> > + version comparison as well as the source path for metadata." >> > + (and-let* ((metadata-file (string-append path "/opam")) >> > + (filename (basename path)) >> > + (version (string-join (cdr (string-split filename >> > #\.)) "."))) >> > + (and (file-exists? metadata-file) >> > + (eq? 'regular (stat:type (stat metadata-file))) >> > + (if (string-prefix? "v" version) >> > + `(V ,(substring version 1) ,metadata-file) >> > + `(digits ,version ,metadata-file)))))=20=20 >>=20 >> What happens if some other prefix is used, e.g., =E2=80=9Crelease-=E2=80= =9D or =E2=80=9CV-=E2=80=9D? >>=20 > > It would get marked as a 'digit. In a previous draft before I started > sending this series of patches, it was called 'R, standing for > "regular", then I thought it was not very meaningful, and, since the > versions were to my knowledge supposed to be either v[0-9]+(\.[0-9]+)* > or [0-9]+(\.[0-9]+)*, I thought I could call that default case "digits" > to clearly indicate what I was trying to refer to. I could change it to > 'other if it matters too much, but the important thing here is that we > distinguish between v-prefixed (the so-called "janestreet > re-versionning" mentioned inside the implementation of find-latest on > current d87d6d6 master) and other versions because =E2=AC=87=EF=B8=8F Oh, OK, I wasn=E2=80=99t aware of this =E2=80=9Cjanestreet re-versionning= =E2=80=9D thing, so only janestreet package have the =E2=80=9Cv=E2=80=9D prefix, right? That e= xplains why versions prefixed with =E2=80=9Cv=E2=80=9D are always greater than those no= t prefixed with anything (in =E2=80=98keep-max-version=E2=80=99). >> Also, why not just return the version number and the metadata file; we >> don=E2=80=99t really care about the prefix do we? >>=20 > > yes we do ! the former latest-version finder handled strings, and > dropped this prefix or put it back on the fly, but the logic > implemented was: if there are v-prefixed versions, find the greatest of > them, without the initial "v", if there aren't, just find the greatest > of all versions. This implies that v-prefixed versions are considered > more important and automatically greater than non-prefixed versions, no > matter what the numbers, which is why this information must be kept. Ah, understood. > I'm just playing ADTs in guile here, "parsing" the version string only > once to retain a symbolic representation of it that will at first > glance allow to identify the type of version used and access the > relevant digits for comparison. The comparison is used right after: > >> > +(define (keep-max-version a b) >> > + "Version comparison on the lists returned by the previous >> > function taking the >> > + janestreet re-versioning into account (v-prefixed come first)." >> > + (match (cons a b) >> > + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) >> > + ((('V _ _) . _) a) >> > + ((_ . ('V _ _)) b) >> > + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a >> > b))))=20 > > and used in the reduce in find-latest-version. So keeping this 'V is > what will help janestreet-re-versionned packages "skip the line" by > being automatically greater than any non v-prefixed package (thus, > v0.0.1 is greater than 13.2, which is the current behaviour). > >> > (define (find-latest-version package repository) >> > "Get the latest version of a package as described in the given >> > repository." >> > - (let* ((dir (string-append repository "/packages/" package)) >> > - (versions (scandir dir (lambda (name) (not >> > (string-prefix? "." name)))))) >> > - (if versions >> > - (let ((versions (map >> > - (lambda (dir) >> > - (string-join (cdr (string-split dir >> > #\.)) ".")) >> > - versions))) >> > - ;; Workaround for janestreet re-versionning >> > - (let ((v-versions (filter (lambda (version) >> > (string-prefix? "v" version)) versions))) >> > - (if (null? v-versions) >> > - (latest-version versions) >> > - (string-append "v" (latest-version (map (lambda >> > (version) (substring version 1)) v-versions)))))) >> > - (begin >> > - (format #t (G_ "Package not found in opam repository: >> > ~a~%") package) >> > - #f)))) >> > + (let ((packages (string-append repository "/packages")) >> > + (filter (make-regexp (string-append "^" package "\\.")))) >> > + (reduce keep-max-version #f >> > + (filter-map >> > + get-version-and-file >> > + (find-files packages filter #:directories? #t))))) >> >=20=20 >> [=E2=80=A6] >> > + (filter-map get-opam-repository repositories-specs)) >> > + (leave (G_ "Package '~a' not found~%") name)))=20=20 >>=20 >> Nit: I wouldn=E2=80=99t capitalize =E2=80=9Cpackage=E2=80=9D, otherwise = the error message >> looks like this >>=20 >> guix import: error: Package 'equations' not found > > a very neat tip, thank you ! : ) You are welcome, and thank you for working on this! > From cde8b2a5d88d89bfea31c86d3ae94d37c1d3c83f Mon Sep 17 00:00:00 2001 > From: Alice BRENON > Date: Sat, 7 Aug 2021 19:50:10 +0200 > Subject: [PATCH] guix: opam: More flexibility in the importer. > > * guix/scripts/import/opam.scm: pass all instances of --repo as a list > to the importer. Nit: The word after the =E2=80=9C:=E2=80=9D is usually capitalized, so =E2= =80=9CPass=E2=80=9D instead of =E2=80=9Cpass=E2=80=9D in this case. Sorry for not noticing this earlier; = the person committing the patch can probably fixup the commit message, so no need to send a reroll just for this small fix. :-) > * guix/import/opam.scm (opam-fetch): stop expecting "expanded" > repositories and call get-opam-repository instead to keep values > "symbolic" as long as possible and factorize. > (get-opam-repository): use the same repository source as CLI opam does > (i.e. HTTP-served index.tar.gz instead of git repositories). > (find-latest-version): be more flexible on the repositories structure > instead of expecting packages/PACKAGE-NAME/PACKAGE-NAME.VERSION/. > * tests/opam.scm: update the call to opam->guix-package since repo is > now expected to be a list and remove the mocked get-opam-repository > deprecated by the support for local folders by the actual > implementation. > * doc/guix.texi: document the new semantics and valid arguments for the > --repo option. > --- > doc/guix.texi | 30 +++++-- > guix/import/opam.scm | 158 +++++++++++++++++++++-------------- > guix/scripts/import/opam.scm | 8 +- > tests/opam.scm | 68 ++++++++------- > 4 files changed, 160 insertions(+), 104 deletions(-) > > diff --git a/doc/guix.texi b/doc/guix.texi > index 78c1c09858..2d36561186 100644 > --- a/doc/guix.texi > +++ b/doc/guix.texi > @@ -94,6 +94,7 @@ Copyright @copyright{} 2021 Xinglu Chen@* > Copyright @copyright{} 2021 Raghav Gururajan@* > Copyright @copyright{} 2021 Domagoj Stolfa@* > Copyright @copyright{} 2021 Hui Lu@* > +Copyright @copyright{} 2021 Alice Brenon@* >=20=20 > Permission is granted to copy, distribute and/or modify this document > under the terms of the GNU Free Documentation License, Version 1.3 or > @@ -11612,14 +11613,31 @@ Traverse the dependency graph of the given upst= ream package recursively > and generate package expressions for all those packages that are not yet > in Guix. > @item --repo > -Select the given repository (a repository name). Possible values includ= e: > + > +By default, packages are searched in the official OPAM repository. This > +option, which can be used more than once, lets you add other > +repositories which will be searched for packages. It accepts as valid > +arguments: > + > @itemize > -@item @code{opam}, the default opam repository, > -@item @code{coq} or @code{coq-released}, the stable repository for coq p= ackages, > -@item @code{coq-core-dev}, the repository that contains development vers= ions of coq, > -@item @code{coq-extra-dev}, the repository that contains development ver= sions > - of coq packages. > +@item the name of a known repository - can be one of @code{opam}, > + @code{coq} (equivalent to @code{coq-released}), > + @code{coq-core-dev}, @code{coq-extra-dev} or @code{grew}. > +@item the URL of a repository as expected by the @code{opam repository > + add} command (for instance, the URL equivalent of the above > + @code{opam} name would be @uref{https://opam.ocaml.org}). > +@item the path to a local copy of a repository (a directory containing a > + @file{packages/} sub-directory). > @end itemize > + > +Repositories must be passed to this option by order of preference and do > +not replace the default @code{opam} which is always failed-back to. I suggest Repositories should be passed to this option by the order of preference. The additional repositories will not replace the default @code{opam} repository, which is always kept as a fallback. WDYT? > +Also, please note that versions are not compared accross repositories. > +The first repository (from left to right) that has at least one version > +of a given package will prevail over any others and the version imported ^ Missing comma. The rest looks good! :-) --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmEWb+oVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5+lAP/3SaIU3lETZzT0Uud0i+ai/tqJ8q 6hZqXaa1HY652u9i/a9IYxUDdMRKX62b1WPrOLmSYw+QOhPZ7ob81iNlPTqxFnXT wXQYSr7zth/sMccZtoT3Zjkv79PWXkS0V4d8S/0dYvwfdI2tClL/qhoig0v2rSGL tYVL8qJRA4ZTcZhV3X4ICB5NnbublCZu5TlD0T5CxdqPU73ssOwU9bA1t3zQTuz4 TDjlIEEdDjdv2SCrHDHbnZpgnue2qaH2PMls+jz3gwZk4WNTHXGQiI3ikYRVMIhG XJjdIqG97e1lNpPGutz7szg3vz66yOlvOFiqzdK3H+GVsWHfmLV0ryWKcaYN8J2a TfglQJ5pu+Xcf92Iujeqm3vUdRvVCTZzPNYnISxewtG7jMBu/wsE5ly0Zncf0Jea DL8pLSok7Uq3CcRGxZ8JDxwPcCYCy6Tw9CHvLdaMddQtbXnlq1C85jswfXY6S1dY bW3tZdOCHNQnUU+JegVpK9zntOsZaPQwv6B/3FMkhS23o/NGMorz8LIDwQExPIji fMTDImb1Syff2Keky6BmLOQZjMQvBgwQigRf3dUGjjwhQY1ade+/gVHn7vc2wJOm mPD1lxu6o4qTcKvmpLwZn/evEXlnOxGb9A9R6vpkO13mp4uCVUdUQDI0vm+hgVR6 2SZ93oNIrGwzMNNI =i9iN -----END PGP SIGNATURE----- --=-=-=-- From unknown Wed Aug 20 03:37:39 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#49958] [PATCH] More flexibility in opam importer Resent-From: Alice BRENON Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 13 Aug 2021 13:48:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 49958 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Xinglu Chen Cc: 49958@debbugs.gnu.org Received: via spool by 49958-submit@debbugs.gnu.org id=B49958.16288624527837 (code B ref 49958); Fri, 13 Aug 2021 13:48:02 +0000 Received: (at 49958) by debbugs.gnu.org; 13 Aug 2021 13:47:32 +0000 Received: from localhost ([127.0.0.1]:40939 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mEXWv-0001xp-Qs for submit@debbugs.gnu.org; Fri, 13 Aug 2021 09:47:32 -0400 Received: from lxc-smtp2.ens-lyon.fr ([140.77.167.81]:51910) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mEXWp-0001t3-S3 for 49958@debbugs.gnu.org; Fri, 13 Aug 2021 09:47:24 -0400 Received: from localhost (localhost [127.0.0.1]) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTP id 73F4AE2A62; Fri, 13 Aug 2021 15:47:18 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.11.0 (20160426) (Debian) at ens-lyon.fr Received: from lxc-smtp2.ens-lyon.fr ([127.0.0.1]) by localhost (lxc-smtp2.ens-lyon.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AQPzSkqODeOw; Fri, 13 Aug 2021 15:47:18 +0200 (CEST) Received: from localhost (unknown [78.194.167.103]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTPSA id 3F45FE2A23; Fri, 13 Aug 2021 15:47:18 +0200 (CEST) Date: Fri, 13 Aug 2021 15:47:17 +0200 From: Alice BRENON Message-ID: <20210813154717.33ad6816@ens-lyon.fr> In-Reply-To: <87h7fth4id.fsf@yoctocell.xyz> References: <20210809140407.748fa019@ens-lyon.fr> <20210809171935.05fac773@ens-lyon.fr> <20210810140413.2f7d2f1b@ens-lyon.fr> <20210810184826.17d14aab@ens-lyon.fr> <87pmuhhk1k.fsf@yoctocell.xyz> <20210813131109.14c25204@ens-lyon.fr> <87h7fth4id.fsf@yoctocell.xyz> Organization: ENS de Lyon X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.24; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/fSB.2bEAthp3yMbWns2W_5J" X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Re-phrased the new documentation explaining how versions are selected from the repositories in case several contain the same package. Sincerely hope it's the last one : ) From daadaa73c41799279d732bbb3aae4d3091d88e2b Mon Sep 17 00:00:00 2001 From: Alice BRENON Date: Sat, 7 Aug 2021 19:50:10 +0200 Subject: [PATCH] guix: opam: More flexibility [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) --MP_/fSB.2bEAthp3yMbWns2W_5J Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline Re-phrased the new documentation explaining how versions are selected from the repositories in case several contain the same package. Sincerely hope it's the last one : ) --MP_/fSB.2bEAthp3yMbWns2W_5J Content-Type: text/x-patch Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=0001-guix-opam-More-flexibility-in-the-importer.patch =46rom daadaa73c41799279d732bbb3aae4d3091d88e2b Mon Sep 17 00:00:00 2001 From: Alice BRENON Date: Sat, 7 Aug 2021 19:50:10 +0200 Subject: [PATCH] guix: opam: More flexibility in the importer. * guix/scripts/import/opam.scm: Pass all instances of --repo as a list to the importer. * guix/import/opam.scm (opam-fetch): Stop expecting "expanded" repositories and call get-opam-repository instead to keep values "symbolic" as long as possible and factorize. (get-opam-repository): Use the same repository source as CLI opam does (i.e. HTTP-served index.tar.gz instead of git repositories). (find-latest-version): Be more flexible on the repositories structure instead of expecting packages/PACKAGE-NAME/PACKAGE-NAME.VERSION/. * tests/opam.scm: Update the call to opam->guix-package since repo is now expected to be a list and remove the mocked get-opam-repository deprecated by the support for local folders by the actual implementation. * doc/guix.texi: Document the new semantics and valid arguments for the --repo option. --- doc/guix.texi | 31 +++++-- guix/import/opam.scm | 158 +++++++++++++++++++++-------------- guix/scripts/import/opam.scm | 8 +- tests/opam.scm | 68 ++++++++------- 4 files changed, 161 insertions(+), 104 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 78c1c09858..ce6a163c87 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -94,6 +94,7 @@ Copyright @copyright{} 2021 Xinglu Chen@* Copyright @copyright{} 2021 Raghav Gururajan@* Copyright @copyright{} 2021 Domagoj Stolfa@* Copyright @copyright{} 2021 Hui Lu@* +Copyright @copyright{} 2021 Alice Brenon@* =20 Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or @@ -11612,14 +11613,32 @@ Traverse the dependency graph of the given upstre= am package recursively and generate package expressions for all those packages that are not yet in Guix. @item --repo -Select the given repository (a repository name). Possible values include: + +By default, packages are searched in the official OPAM repository. This +option, which can be used more than once, lets you add other +repositories which will be searched for packages. It accepts as valid +arguments: + @itemize -@item @code{opam}, the default opam repository, -@item @code{coq} or @code{coq-released}, the stable repository for coq pac= kages, -@item @code{coq-core-dev}, the repository that contains development versio= ns of coq, -@item @code{coq-extra-dev}, the repository that contains development versi= ons - of coq packages. +@item the name of a known repository - can be one of @code{opam}, + @code{coq} (equivalent to @code{coq-released}), + @code{coq-core-dev}, @code{coq-extra-dev} or @code{grew}. +@item the URL of a repository as expected by the @code{opam repository + add} command (for instance, the URL equivalent of the above + @code{opam} name would be @uref{https://opam.ocaml.org}). +@item the path to a local copy of a repository (a directory containing a + @file{packages/} sub-directory). @end itemize + +Repositories are assumed to be passed to this option by order of +preference. The additional repositories will not replace the default +@code{opam} repository, which is always kept as a fallback. + +Also, please note that versions are not compared accross repositories. +The first repository (from left to right) that has at least one version +of a given package will prevail over any others, and the version +imported will be the latest one found @emph{in this repository only}. + @end table =20 @item go diff --git a/guix/import/opam.scm b/guix/import/opam.scm index a35b01d277..fe13d29f03 100644 --- a/guix/import/opam.scm +++ b/guix/import/opam.scm @@ -2,6 +2,7 @@ ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2020 Martin Becze ;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -22,21 +23,24 @@ #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (ice-9 peg) + #:use-module ((ice-9 popen) #:select (open-pipe*)) #:use-module (ice-9 receive) - #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (ice-9 textual-ports) #:use-module (ice-9 vlist) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) - #:use-module (web uri) + #:use-module ((srfi srfi-26) #:select (cut)) + #:use-module ((web uri) #:select (string->uri uri->string)) + #:use-module ((guix build utils) #:select (dump-port find-files mkdir-p)) #:use-module (guix build-system) #:use-module (guix build-system ocaml) #:use-module (guix http-client) - #:use-module (guix git) #:use-module (guix ui) #:use-module (guix packages) #:use-module (guix upstream) - #:use-module (guix utils) + #:use-module ((guix utils) #:select (cache-directory + version>? + call-with-temporary-output-file)) #:use-module (guix import utils) #:use-module ((guix licenses) #:prefix license:) #:export (opam->guix-package @@ -121,51 +125,83 @@ (define-peg-pattern condition-string all (and QUOTE (* STRCHR) QUOTE)) (define-peg-pattern condition-var all (+ (or (range #\a #\z) "-" ":"))) =20 -(define* (get-opam-repository #:optional repo) +(define (opam-cache-directory path) + (string-append (cache-directory) "/opam/" path)) + +(define known-repositories + '((opam . "https://opam.ocaml.org") + (coq . "https://coq.inria.fr/opam/released") + (coq-released . "https://coq.inria.fr/opam/released") + (coq-core-dev . "https://coq.inria.fr/opam/core-dev") + (coq-extra-dev . "https://coq.inria.fr/opam/extra-dev") + (grew . "http://opam.grew.fr"))) + +(define (get-uri repo-root) + (let ((archive-file (string-append repo-root "/index.tar.gz"))) + (or (string->uri archive-file) + (begin + (warning (G_ "'~a' is not a valid URI~%") archive-file) + 'bad-repo)))) + +(define (repo-type repo) + (match (assoc-ref known-repositories (string->symbol repo)) + (#f (if (file-exists? repo) + `(local ,repo) + `(remote ,(get-uri repo)))) + (url `(remote ,(get-uri url))))) + +(define (update-repository input) + "Make sure the cache for opam repository INPUT is up-to-date" + (let* ((output (opam-cache-directory (basename (port-filename input)))) + (cached-date (if (file-exists? output) + (stat:mtime (stat output)) + (begin (mkdir-p output) 0)))) + (when (> (stat:mtime (stat input)) cached-date) + (call-with-port + (open-pipe* OPEN_WRITE "tar" "xz" "-C" output "-f" "-") + (cut dump-port input <>))) + output)) + +(define* (get-opam-repository #:optional (repo "opam")) "Update or fetch the latest version of the opam repository and return the path to the repository." - (let ((url (cond - ((or (not repo) (equal? repo 'opam)) - "https://github.com/ocaml/opam-repository") - ((string-prefix? "coq-" (symbol->string repo)) - "https://github.com/coq/opam-coq-archive") - ((equal? repo 'coq) "https://github.com/coq/opam-coq-archiv= e") - (else (throw 'unknown-repository repo))))) - (receive (location commit _) - (update-cached-checkout url) - (cond - ((or (not repo) (equal? repo 'opam)) - location) - ((equal? repo 'coq) - (string-append location "/released")) - ((string-prefix? "coq-" (symbol->string repo)) - (string-append location "/" (substring (symbol->string repo) 4))) - (else location))))) + (match (repo-type repo) + (('local p) p) + (('remote 'bad-repo) #f) ; to weed it out during filter-map in opam-fe= tch + (('remote r) (call-with-port (http-fetch/cached r) update-repository))= )) =20 ;; Prevent Guile 3 from inlining this procedure so we can mock it in tests. (set! get-opam-repository get-opam-repository) =20 -(define (latest-version versions) - "Find the most recent version from a list of versions." - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) versions)) +(define (get-version-and-file path) + "Analyse a candidate path and return an list containing information for = proper + version comparison as well as the source path for metadata." + (and-let* ((metadata-file (string-append path "/opam")) + (filename (basename path)) + (version (string-join (cdr (string-split filename #\.)) "."))) + (and (file-exists? metadata-file) + (eq? 'regular (stat:type (stat metadata-file))) + (if (string-prefix? "v" version) + `(V ,(substring version 1) ,metadata-file) + `(digits ,version ,metadata-file))))) + +(define (keep-max-version a b) + "Version comparison on the lists returned by the previous function takin= g the + janestreet re-versioning into account (v-prefixed come first)." + (match (cons a b) + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) + ((('V _ _) . _) a) + ((_ . ('V _ _)) b) + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a b)))) =20 (define (find-latest-version package repository) "Get the latest version of a package as described in the given repositor= y." - (let* ((dir (string-append repository "/packages/" package)) - (versions (scandir dir (lambda (name) (not (string-prefix? "." na= me)))))) - (if versions - (let ((versions (map - (lambda (dir) - (string-join (cdr (string-split dir #\.)) ".")) - versions))) - ;; Workaround for janestreet re-versionning - (let ((v-versions (filter (lambda (version) (string-prefix? "v" ve= rsion)) versions))) - (if (null? v-versions) - (latest-version versions) - (string-append "v" (latest-version (map (lambda (version) (sub= string version 1)) v-versions)))))) - (begin - (format #t (G_ "Package not found in opam repository: ~a~%") packa= ge) - #f)))) + (let ((packages (string-append repository "/packages")) + (filter (make-regexp (string-append "^" package "\\.")))) + (reduce keep-max-version #f + (filter-map + get-version-and-file + (find-files packages filter #:directories? #t))))) =20 (define (get-metadata opam-file) (with-input-from-file opam-file @@ -266,28 +302,30 @@ path to the repository." =20 (define (depends->native-inputs depends) (filter (lambda (name) (not (equal? "" name))) - (map dependency->native-input depends))) + (map dependency->native-input depends))) =20 (define (dependency-list->inputs lst) (map - (lambda (dependency) - (list dependency (list 'unquote (string->symbol dependency)))) - (ocaml-names->guix-names lst))) - -(define* (opam-fetch name #:optional (repository (get-opam-repository))) - (and-let* ((repository repository) - (version (find-latest-version name repository)) - (file (string-append repository "/packages/" name "/" name ".= " version "/opam"))) - `(("metadata" ,@(get-metadata file)) - ("version" . ,(if (string-prefix? "v" version) - (substring version 1) - version))))) - -(define* (opam->guix-package name #:key (repo 'opam) version) - "Import OPAM package NAME from REPOSITORY (a directory name) or, if -REPOSITORY is #f, from the official OPAM repository. Return a 'package' s= exp + (lambda (dependency) + (list dependency (list 'unquote (string->symbol dependency)))) + (ocaml-names->guix-names lst))) + +(define* (opam-fetch name #:optional (repositories-specs '("opam"))) + (or (fold (lambda (repository others) + (match (find-latest-version name repository) + ((_ version file) `(("metadata" ,@(get-metadata file)) + ("version" . ,version))) + (_ others))) + #f + (filter-map get-opam-repository repositories-specs)) + (leave (G_ "package '~a' not found~%") name))) + +(define* (opam->guix-package name #:key (repo '()) version) + "Import OPAM package NAME from REPOSITORIES (a list of names, URLs or lo= cal +paths, always including OPAM's official repository). Return a 'package' s= exp or #f on failure." - (and-let* ((opam-file (opam-fetch name (get-opam-repository repo))) + (and-let* ((with-opam (if (member "opam" repo) repo (cons "opam" repo))) + (opam-file (opam-fetch name with-opam)) (version (assoc-ref opam-file "version")) (opam-content (assoc-ref opam-file "metadata")) (url-dict (metadata-ref opam-content "url")) @@ -312,9 +350,7 @@ or #f on failure." (values `(package (name ,(ocaml-name->guix-name name)) - (version ,(if (string-prefix? "v" version) - (substring version 1) - version)) + (version ,version) (source (origin (method url-fetch) diff --git a/guix/scripts/import/opam.scm b/guix/scripts/import/opam.scm index 64164e7cc4..834ac34cb0 100644 --- a/guix/scripts/import/opam.scm +++ b/guix/scripts/import/opam.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -46,7 +47,8 @@ Import and convert the opam package for PACKAGE-NAME.\n")) (display (G_ " -r, --recursive import packages recursively")) (display (G_ " - --repo import packages from this opam repository")) + --repo import packages from this opam repository (name, = URL or local path) + can be used more than once")) (display (G_ " -V, --version display version information and exit")) (newline) @@ -81,7 +83,9 @@ Import and convert the opam package for PACKAGE-NAME.\n")) #:build-options? #f)) =20 (let* ((opts (parse-options)) - (repo (and=3D> (assoc-ref opts 'repo) string->symbol)) + (repo (filter-map (match-lambda + (('repo . name) name) + (_ #f)) opts)) (args (filter-map (match-lambda (('argument . value) value) diff --git a/tests/opam.scm b/tests/opam.scm index f1e3b70cb0..1536b74339 100644 --- a/tests/opam.scm +++ b/tests/opam.scm @@ -82,41 +82,39 @@ url { (set! test-source-hash (call-with-input-file file-name port-sha256)))) (_ (error "Unexpected URL: " url))))) - (mock ((guix import opam) get-opam-repository - (const test-repo)) - (let ((my-package (string-append test-repo - "/packages/foo/foo.1.0.0"))) - (mkdir-p my-package) - (with-output-to-file (string-append my-package "/opam") - (lambda _ - (format #t "~a" test-opam-file)))) - (match (opam->guix-package "foo" #:repo test-repo) - (('package - ('name "ocaml-foo") - ('version "1.0.0") - ('source ('origin - ('method 'url-fetch) - ('uri "https://example.org/foo-1.0.0.tar.gz") - ('sha256 - ('base32 - (? string? hash))))) - ('build-system 'ocaml-build-system) - ('propagated-inputs - ('quasiquote - (("ocaml-zarith" ('unquote 'ocaml-zarith))))) - ('native-inputs - ('quasiquote - (("ocaml-alcotest" ('unquote 'ocaml-alcotest)) - ("ocamlbuild" ('unquote 'ocamlbuild))))) - ('home-page "https://example.org/") - ('synopsis "Some example package") - ('description "This package is just an example.") - ('license 'license:bsd-3)) - (string=3D? (bytevector->nix-base32-string - test-source-hash) - hash)) - (x - (pk 'fail x #f)))))) + (let ((my-package (string-append test-repo + "/packages/foo/foo.1.0.0"))) + (mkdir-p my-package) + (with-output-to-file (string-append my-package "/opam") + (lambda _ + (format #t "~a" test-opam-file)))) + (match (opam->guix-package "foo" #:repo (list test-repo)) + (('package + ('name "ocaml-foo") + ('version "1.0.0") + ('source ('origin + ('method 'url-fetch) + ('uri "https://example.org/foo-1.0.0.tar.gz") + ('sha256 + ('base32 + (? string? hash))))) + ('build-system 'ocaml-build-system) + ('propagated-inputs + ('quasiquote + (("ocaml-zarith" ('unquote 'ocaml-zarith))))) + ('native-inputs + ('quasiquote + (("ocaml-alcotest" ('unquote 'ocaml-alcotest)) + ("ocamlbuild" ('unquote 'ocamlbuild))))) + ('home-page "https://example.org/") + ('synopsis "Some example package") + ('description "This package is just an example.") + ('license 'license:bsd-3)) + (string=3D? (bytevector->nix-base32-string + test-source-hash) + hash)) + (x + (pk 'fail x #f))))) =20 ;; Test the opam file parser ;; We fold over some test cases. Each case is a pair of the string to pars= e and the --=20 2.32.0 --MP_/fSB.2bEAthp3yMbWns2W_5J-- From unknown Wed Aug 20 03:37:39 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Alice BRENON Subject: bug#49958: closed (Re: [bug#49958] [PATCH] More flexibility in opam importer) Message-ID: References: <20210821000748.59cce44e@tachikoma.lepiller.eu> <20210809140407.748fa019@ens-lyon.fr> X-Gnu-PR-Message: they-closed 49958 X-Gnu-PR-Package: guix-patches X-Gnu-PR-Keywords: patch Reply-To: 49958@debbugs.gnu.org Date: Fri, 20 Aug 2021 22:09:03 +0000 Content-Type: multipart/mixed; boundary="----------=_1629497343-9201-1" This is a multi-part message in MIME format... ------------=_1629497343-9201-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #49958: [PATCH] More flexibility in opam importer which was filed against the guix-patches package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 49958@debbugs.gnu.org. --=20 49958: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D49958 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1629497343-9201-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 49958-done) by debbugs.gnu.org; 20 Aug 2021 22:08:03 +0000 Received: from localhost ([127.0.0.1]:35636 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mHCgF-0002Mk-DB for submit@debbugs.gnu.org; Fri, 20 Aug 2021 18:08:03 -0400 Received: from lepiller.eu ([89.234.186.109]:55546) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mHCg9-0002MG-Tv for 49958-done@debbugs.gnu.org; Fri, 20 Aug 2021 18:08:02 -0400 Received: from lepiller.eu (localhost [127.0.0.1]) by lepiller.eu (OpenSMTPD) with ESMTP id ff259e35; Fri, 20 Aug 2021 22:07:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=lepiller.eu; h=date:from :to:cc:subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=dkim; bh=WipX77ypthEe vET9KEQB04sCYDe2mQy6h3FLOiqDGSE=; b=RK+m0DF/sjmKh9kVCpHJJ3UE3eNO heLJjnidJrPMwq1UM4lt97BswWfpSU9/YM4i7EsWbfvKsIwxHuY5rvdtF2V7TooU k83xyd8fT/uPGOXyo8alaXebXfbYt8Mu528kSWFD8ZtEd5pkRexOGKxT2NjFyxDj aq6fUXwqMlXqrTCKgXMaMuqbal8I1E6loEqNwlLdbV5xU5gNq8BTGhNLpumzV3Tc dif2ZV0uGQbWtgtYtootNJ7XpB+PQA41f6Gm1gxelLJXaYLwCULd0t7ESBHcMu27 37eOaqon4nzOiSkVJVf+2Mu2gd0oqvJSFzTbP/ZxQcxQMex62QU5BcxGdg== Received: by lepiller.eu (OpenSMTPD) with ESMTPSA id a7f3ada3 (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Fri, 20 Aug 2021 22:07:55 +0000 (UTC) Date: Sat, 21 Aug 2021 00:07:48 +0200 From: Julien Lepiller To: Alice BRENON Subject: Re: [bug#49958] [PATCH] More flexibility in opam importer Message-ID: <20210821000748.59cce44e@tachikoma.lepiller.eu> In-Reply-To: <20210813154717.33ad6816@ens-lyon.fr> References: <20210809140407.748fa019@ens-lyon.fr> <20210809171935.05fac773@ens-lyon.fr> <20210810140413.2f7d2f1b@ens-lyon.fr> <20210810184826.17d14aab@ens-lyon.fr> <87pmuhhk1k.fsf@yoctocell.xyz> <20210813131109.14c25204@ens-lyon.fr> <87h7fth4id.fsf@yoctocell.xyz> <20210813154717.33ad6816@ens-lyon.fr> X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.32; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 49958-done Cc: 49958-done@debbugs.gnu.org, Xinglu Chen X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Le Fri, 13 Aug 2021 15:47:17 +0200, Alice BRENON a =C3=A9crit : > Re-phrased the new documentation explaining how versions are selected > from the repositories in case several contain the same package. >=20 > Sincerely hope it's the last one : ) Pushed to master as fc29c80b9635ff490bcc768c774442043cb1e231, thanks! ------------=_1629497343-9201-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 9 Aug 2021 15:03:17 +0000 Received: from localhost ([127.0.0.1]:57477 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mD6o7-00028S-OY for submit@debbugs.gnu.org; Mon, 09 Aug 2021 11:03:17 -0400 Received: from lists.gnu.org ([209.51.188.17]:52684) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mD40t-0001HO-Fm for submit@debbugs.gnu.org; Mon, 09 Aug 2021 08:04:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:49860) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mD40s-0006OB-No for guix-patches@gnu.org; Mon, 09 Aug 2021 08:04:14 -0400 Received: from lxc-smtp2.ens-lyon.fr ([140.77.167.81]:51116) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mD40p-0001ra-E5 for guix-patches@gnu.org; Mon, 09 Aug 2021 08:04:14 -0400 Received: from localhost (localhost [127.0.0.1]) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTP id 08232E28D6 for ; Mon, 9 Aug 2021 14:04:08 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.11.0 (20160426) (Debian) at ens-lyon.fr Received: from lxc-smtp2.ens-lyon.fr ([127.0.0.1]) by localhost (lxc-smtp2.ens-lyon.fr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id weeoDZ23OQLH for ; Mon, 9 Aug 2021 14:04:07 +0200 (CEST) Received: from localhost (unknown [78.194.167.103]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by lxc-smtp2.ens-lyon.fr (Postfix) with ESMTPSA id E2994E0E67 for ; Mon, 9 Aug 2021 14:04:07 +0200 (CEST) Date: Mon, 9 Aug 2021 14:04:07 +0200 From: Alice BRENON To: guix-patches@gnu.org Subject: [PATCH] More flexibility in opam importer Message-ID: <20210809140407.748fa019@ens-lyon.fr> Organization: ENS de Lyon X-Mailer: Claws Mail 4.0.0 (GTK+ 3.24.24; x86_64-pc-linux-gnu) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="MP_/MP2=ddjP20LDcZckf5S4+Rd" Received-SPF: pass client-ip=140.77.167.81; envelope-from=alice.brenon@ens-lyon.fr; helo=lxc-smtp2.ens-lyon.fr X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: 0.3 (/) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Mon, 09 Aug 2021 11:03:14 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --MP_/MP2=ddjP20LDcZckf5S4+Rd Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline Hello, I'd like to submit this patch for review, discussion and hopefully inclusion to guix' code. I recently tried to import grew[0], a NLP tool written in ocaml and distributed with opam but from a custom repository. The current importer prevented me to do so for several reasons: - the available repositories were hard-coded to be either opam's official repository or coq's - the repositories were expected to be distributed with git: while public git repositories do exist for coq and opam's official repository, they are not the source of truth for the opam tool one can use in an imperative setup like this: `opam repo add coq-released https://coq.inria.fr/opam/released` it entailed that assumptions were made about the freshness of the git repositories and the actual files served to opam, hence differences could theoretically be observed - it appears that the opam tool doesn't enforce as strict a structure on its repositories as its current documentation[1] suggests. Grew's own repository has all versions of each package directly under `/packages/` instead of in a separate subdirectory. While this deserves a clarification from opam's part, this patch hardens guix opam importer against such exotic layouts. - the unability to query several repositories at once rendered recursive imports inefficient, as some packages on a custom opam repository may still need dependencies from the official opam repository even if no guix package has been imported for it yet (this was the case with ocaml-ANSITerminal in my case) The current proposal attempts to solve these difficulties, and allowed me to actually import the guix declaration for grew and its dependencies. I'm still fixing the imported declaration and intend to submit a separate patch to add it to guix packages when it works. I added grew's custom opam repository to the list of known-repositories because it was my immediate target but this is of course not important and could be reverted in favour of coq and opam's official repository only. I used the emacs scripted auto-indenter to save a little time to my reviewers and had to discard many changes that weren't related to my changes to cut the noise. Maybe this file should be reindented in a separate commit following the approval or rejection of this patch proposal. Cheers, Alice [0]: https://grew.fr/ [1]: https://opam.ocaml.org/doc/Manual.html#Repositories --MP_/MP2=ddjP20LDcZckf5S4+Rd Content-Type: text/x-patch Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename=0001-guix-opam.patch =46rom 3beac4d8caf3c6693237bc4f975029e58a20e0f4 Mon Sep 17 00:00:00 2001 From: Alice BRENON Date: Sat, 7 Aug 2021 19:50:10 +0200 Subject: [PATCH] guix: opam: - Add support for importing from several repositories - Add support for custom repositories, either from a URL or= a local path - Use actual opam repositories as defined by the opam tool as source * guix/scripts/import/opam.scm: pass all instances of --repo as a list to the importer * guix/import/opam.scm: - delay repo resolution (call to get-opam-repository from within opam-fetch instead of outside) - use the same repository source as CLI opam does (i.e. HTTP-served index.tar.gz instead of git repositories) - be more flexible on the repositories structure instead of expecting packages/PACKAGE-NAME/PACKAGE-NAME.VERSION/ --- guix/import/opam.scm | 145 +++++++++++++++++++++-------------- guix/scripts/import/opam.scm | 5 +- 2 files changed, 92 insertions(+), 58 deletions(-) diff --git a/guix/import/opam.scm b/guix/import/opam.scm index a35b01d277..c3fad7db65 100644 --- a/guix/import/opam.scm +++ b/guix/import/opam.scm @@ -2,6 +2,7 @@ ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2020 Martin Becze ;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -22,13 +23,16 @@ #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (ice-9 peg) + #:use-module (ice-9 popen) #:use-module (ice-9 receive) #:use-module ((ice-9 rdelim) #:select (read-line)) #:use-module (ice-9 textual-ports) #:use-module (ice-9 vlist) #:use-module (srfi srfi-1) #:use-module (srfi srfi-2) + #:use-module (srfi srfi-26) #:use-module (web uri) + #:use-module ((guix build utils) #:select (dump-port find-files mkdir-p)) #:use-module (guix build-system) #:use-module (guix build-system ocaml) #:use-module (guix http-client) @@ -121,51 +125,78 @@ (define-peg-pattern condition-string all (and QUOTE (* STRCHR) QUOTE)) (define-peg-pattern condition-var all (+ (or (range #\a #\z) "-" ":"))) =20 -(define* (get-opam-repository #:optional repo) +(define (opam-cache-directory path) + (string-append (cache-directory #:ensure? #f) "/opam/" path)) + +(define known-repositories + '((opam . "https://opam.ocaml.org") + (coq . "https://coq.inria.fr/opam/released") + (grew . "http://opam.grew.fr"))) + +(define (repo-type repo) + (define (get-uri repo-root) + (let ((archive-file (string-append repo-root "/index.tar.gz"))) + (or (string->uri archive-file) (throw 'bad-uri archive-file)))) + (match (assoc-ref known-repositories (string->symbol repo)) + (#f (if (file-exists? repo) + `(local ,repo) + `(remote ,(get-uri repo)))) + (url `(remote ,(get-uri url))))) + +(define (update-repository-at output-folder input) + "Make sure the opam repository at OUTPUT-FOLDER is up-to-date with INPUT" + (let ((cached-date (if (file-exists? output-folder) + (stat:mtime (stat output-folder)) + (begin (mkdir-p output-folder) 0)))) + (begin + (and (> (stat:mtime (stat input)) cached-date) + (call-with-port + (open-pipe* OPEN_WRITE "tar" "xz" "-C" output-folder "-f" "-") + (cut dump-port input <>))) + output-folder))) + +(define* (get-opam-repository #:optional (repo "opam")) "Update or fetch the latest version of the opam repository and return the path to the repository." - (let ((url (cond - ((or (not repo) (equal? repo 'opam)) - "https://github.com/ocaml/opam-repository") - ((string-prefix? "coq-" (symbol->string repo)) - "https://github.com/coq/opam-coq-archive") - ((equal? repo 'coq) "https://github.com/coq/opam-coq-archiv= e") - (else (throw 'unknown-repository repo))))) - (receive (location commit _) - (update-cached-checkout url) - (cond - ((or (not repo) (equal? repo 'opam)) - location) - ((equal? repo 'coq) - (string-append location "/released")) - ((string-prefix? "coq-" (symbol->string repo)) - (string-append location "/" (substring (symbol->string repo) 4))) - (else location))))) + (match (repo-type repo) + (('local p) p) + (('remote source) (let ((cache (opam-cache-directory (uri-host source)= ))) + (call-with-port + (http-fetch/cached source) + (cut update-repository-at cache <>)))))) =20 ;; Prevent Guile 3 from inlining this procedure so we can mock it in tests. (set! get-opam-repository get-opam-repository) =20 -(define (latest-version versions) - "Find the most recent version from a list of versions." - (fold (lambda (a b) (if (version>? a b) a b)) (car versions) versions)) +(define (get-version-and-file path) + "Analyse a candidate path and return an list containing information for = proper + version comparison as well as the source path for metadata." + (and-let* ((metadata-file (string-append path "/opam")) + (filename (basename path)) + (version (string-join (cdr (string-split filename #\.)) "."))) + (and (file-exists? metadata-file) + (eq? 'regular (stat:type (stat metadata-file))) + (if (string-prefix? "v" version) + `(V ,(substring version 1) ,metadata-file) + `(digits ,version ,metadata-file))))) + +(define (keep-max-version a b) + "Version comparison on the lists returned by the previous function takin= g the + janestreet re-versioning into account (v-prefixed come first)." + (match (cons a b) + ((('V va _) . ('V vb _)) (if (version>? va vb) a b)) + ((('V _ _) . _) a) + ((_ . ('V _ _)) b) + ((('digits va _) . ('digits vb _)) (if (version>? va vb) a b)))) =20 (define (find-latest-version package repository) "Get the latest version of a package as described in the given repositor= y." - (let* ((dir (string-append repository "/packages/" package)) - (versions (scandir dir (lambda (name) (not (string-prefix? "." na= me)))))) - (if versions - (let ((versions (map - (lambda (dir) - (string-join (cdr (string-split dir #\.)) ".")) - versions))) - ;; Workaround for janestreet re-versionning - (let ((v-versions (filter (lambda (version) (string-prefix? "v" ve= rsion)) versions))) - (if (null? v-versions) - (latest-version versions) - (string-append "v" (latest-version (map (lambda (version) (sub= string version 1)) v-versions)))))) - (begin - (format #t (G_ "Package not found in opam repository: ~a~%") packa= ge) - #f)))) + (let ((packages (string-append repository "/packages")) + (filter (make-regexp (string-append "^" package "\\.")))) + (reduce keep-max-version #f + (filter-map + get-version-and-file + (find-files packages filter #:directories? #t))))) =20 (define (get-metadata opam-file) (with-input-from-file opam-file @@ -266,28 +297,30 @@ path to the repository." =20 (define (depends->native-inputs depends) (filter (lambda (name) (not (equal? "" name))) - (map dependency->native-input depends))) + (map dependency->native-input depends))) =20 (define (dependency-list->inputs lst) (map - (lambda (dependency) - (list dependency (list 'unquote (string->symbol dependency)))) - (ocaml-names->guix-names lst))) - -(define* (opam-fetch name #:optional (repository (get-opam-repository))) - (and-let* ((repository repository) - (version (find-latest-version name repository)) - (file (string-append repository "/packages/" name "/" name ".= " version "/opam"))) - `(("metadata" ,@(get-metadata file)) - ("version" . ,(if (string-prefix? "v" version) - (substring version 1) - version))))) - -(define* (opam->guix-package name #:key (repo 'opam) version) - "Import OPAM package NAME from REPOSITORY (a directory name) or, if -REPOSITORY is #f, from the official OPAM repository. Return a 'package' s= exp + (lambda (dependency) + (list dependency (list 'unquote (string->symbol dependency)))) + (ocaml-names->guix-names lst))) + +(define* (opam-fetch name #:optional (repositories-specs '("opam"))) + (or (fold (lambda (repository others) + (match (find-latest-version name repository) + ((_ version file) `(("metadata" ,@(get-metadata file)) + ("version" . ,version))) + (_ others))) + #f + (map get-opam-repository repositories-specs)) + (throw 'package-not-found repositories-specs))) + +(define* (opam->guix-package name #:key (repo '()) version) + "Import OPAM package NAME from REPOSITORIES (a list of names, URLs or lo= cal +paths, always including OPAM's official repository). Return a 'package' s= exp or #f on failure." - (and-let* ((opam-file (opam-fetch name (get-opam-repository repo))) + (and-let* ((with-opam (if (member "opam" repo) repo (cons "opam" repo))) + (opam-file (opam-fetch name with-opam)) (version (assoc-ref opam-file "version")) (opam-content (assoc-ref opam-file "metadata")) (url-dict (metadata-ref opam-content "url")) @@ -312,9 +345,7 @@ or #f on failure." (values `(package (name ,(ocaml-name->guix-name name)) - (version ,(if (string-prefix? "v" version) - (substring version 1) - version)) + (version ,version) (source (origin (method url-fetch) diff --git a/guix/scripts/import/opam.scm b/guix/scripts/import/opam.scm index 64164e7cc4..837a6ef40f 100644 --- a/guix/scripts/import/opam.scm +++ b/guix/scripts/import/opam.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2018 Julien Lepiller ;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; Copyright =C2=A9 2021 Alice Brenon ;;; ;;; This file is part of GNU Guix. ;;; @@ -81,7 +82,9 @@ Import and convert the opam package for PACKAGE-NAME.\n")) #:build-options? #f)) =20 (let* ((opts (parse-options)) - (repo (and=3D> (assoc-ref opts 'repo) string->symbol)) + (repo (filter-map (match-lambda + (('repo . name) name) + (_ #f)) opts)) (args (filter-map (match-lambda (('argument . value) value) --=20 2.32.0 --MP_/MP2=ddjP20LDcZckf5S4+Rd-- ------------=_1629497343-9201-1--