From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 03 11:51:23 2021 Received: (at submit) by debbugs.gnu.org; 3 Sep 2021 15:51:24 +0000 Received: from localhost ([127.0.0.1]:45788 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMBTP-0007xA-Ar for submit@debbugs.gnu.org; Fri, 03 Sep 2021 11:51:23 -0400 Received: from lists.gnu.org ([209.51.188.17]:53982) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMBTN-0007x3-TK for submit@debbugs.gnu.org; Fri, 03 Sep 2021 11:51:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56522) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mMBTM-0006Q2-VL for guix-patches@gnu.org; Fri, 03 Sep 2021 11:51:21 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:58080 helo=mail.yoctocell.xyz) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mMBTJ-0004iq-L0 for guix-patches@gnu.org; Fri, 03 Sep 2021 11:51:20 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1630684266; bh=cdtrGN9Y8W0hGMyXnMzGssKIL6kIknZ9jKKMhgeCheg=; h=From:To:Subject:Date; b=FwV1KD+rP9LxCUJ7+HnDaOh1f8ePrHqNIdQtltOERgyZRm2C2o9vH9D84sO4ClQ6H J4NWCChfgbAgLHQCFKrBBYNQ1AjtlVgSMIUeUioc93ezzwQ77P5S7U5AtaX0QvpviM QBEz3uekTyR2cezX71GwZRBGeVDi2K7ubxuOgayw= To: guix-patches@gnu.org Subject: [PATCH] import: Add 'generic-git' updater. Message-Id: Date: Fri, 03 Sep 2021 17:50:56 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=87.96.130.155; envelope-from=public@yoctocell.xyz; helo=mail.yoctocell.xyz X-Spam_score_int: 30 X-Spam_score: 3.0 X-Spam_bar: +++ X-Spam_report: (3.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FROM_SUSPICIOUS_NTLD=0.5, FROM_SUSPICIOUS_NTLD_FP=1.598, PDS_OTHER_BAD_TLD=1.999, RDNS_DYNAMIC=0.982, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-Spam-Score: 1.7 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: * guix/import/git.scm: New file. * doc/guix.texi (Invoking guix refresh): Document it. * Makefile.am (MODULES): Register it. --- This patch adds a new ‘generic-git’ updater which can check for new [...] Content analysis details: (1.7 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 1.0 SPF_SOFTFAIL SPF: sender does not match SPF record (softfail) -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [209.51.188.17 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [209.51.188.17 listed in list.dnswl.org] 0.5 FROM_SUSPICIOUS_NTLD_FP From abused NTLD 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.2 (/) * guix/import/git.scm: New file. * doc/guix.texi (Invoking guix refresh): Document it. * Makefile.am (MODULES): Register it. --- This patch adds a new =E2=80=98generic-git=E2=80=99 updater which can check= for new tags for package hosted on Git repos. However, it cannot download Git repos and update the package definitions, i.e. =E2=80=98guix refresh -u=E2=80=99.= There is a pending patch that would add this feature though[1]. =E2=80=98guix refresh -L=E2=80=99 now reports Available updaters: [=E2=80=A6] 94.5% of the packages are covered by these updaters. We are getting close to 100% :-) See it in action! --8<---------------cut here---------------start------------->8--- $ ./pre-inst-env guix refresh harmonist scdoc gmnisrv gnu/packages/web.scm:7931:4: warning: no tags were found for package `gmnis= rv' gnu/packages/web.scm:7931:4: warning: 'generic-git' updater failed to deter= mine available releases for gmnisrv gnu/packages/man.scm:339:12: scdoc would be upgraded from 1.10.1 to 1.11.1 gnu/packages/games.scm:9433:2: warning: failed to fetch Git repository for = package `harmonist' gnu/packages/games.scm:9433:2: warning: 'generic-git' updater failed to det= ermine available releases for harmonist --8<---------------cut here---------------end--------------->8--- =20=20 [1]: Makefile.am | 1 + doc/guix.texi | 27 ++++++ guix/import/git.scm | 223 ++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 251 insertions(+) create mode 100644 guix/import/git.scm diff --git a/Makefile.am b/Makefile.am index 3c79760734..c4d3a456b1 100644 --- a/Makefile.am +++ b/Makefile.am @@ -254,6 +254,7 @@ MODULES =3D \ guix/import/egg.scm \ guix/import/elpa.scm \ guix/import/gem.scm \ + guix/import/git.scm \ guix/import/github.scm \ guix/import/gnome.scm \ guix/import/gnu.scm \ diff --git a/doc/guix.texi b/doc/guix.texi index 36a0c7f5ec..26afb1607a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -11920,6 +11920,33 @@ the updater for @uref{https://launchpad.net, Launc= hpad} packages. @item generic-html a generic updater that crawls the HTML page where the source tarball of the package is hosted, when applicable. +@item generic-git +a generic updater for packages hosted on Git repositories. It tries to +be smart about parsing Git tag names, but if it is not able to parse the +tag name and compare tags correctly, users can define the following +properties for a package. + +@itemize +@item @code{tag-prefix}: a regular expression for matching a prefix of +the tag name. + +@item @code{tag-suffix}: a regular expression for matching a suffix of +the tag name. + +@item @code{tag-version-delimiter}: a string used as the delimiter in +the tag name for separating the numbers of the version. +@end itemize + +@lisp +(package + (name "foo") + ;; ... + (properties + '((tag-prefix . "^release0-") + (tag-suffix . "[a-z]?$") + (tag-version-delimiter . ":")))) +@end lisp=20=20=20=20=20=20 + @end table =20 For instance, the following command only checks for updates of Emacs diff --git a/guix/import/git.scm b/guix/import/git.scm new file mode 100644 index 0000000000..9a654c1972 --- /dev/null +++ b/guix/import/git.scm @@ -0,0 +1,223 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2021 Xinglu Chen +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix import git) + #:use-module (git) + #:use-module (guix build utils) + #:use-module (guix diagnostics) + #:use-module (guix git) + #:use-module (guix git-download) + #:use-module (guix i18n) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 regex) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) + #:use-module (srfi srfi-28) + #:use-module (srfi srfi-34) + #:use-module (srfi srfi-35) + #:use-module (srfi srfi-71) + #:export (%generic-git-updater)) + +;;; Commentary: +;;; +;;; This module provides a generic package updater for packages hosted on = Git +;;; repositories. +;;; +;;; It tries to be smart about tag names, but if it is not automatically a= ble +;;; to parse the tag names correctly, users can set the `tag-prefix', +;;; `tag-suffix' and `tag-version-delimiter' properties of the package to = make +;;; the updater parse the Git tag name correctly. +;;; +;;; Code: + +;;; Errors & warnings + +(define-condition-type &git-tag-error &error + git-tag-error? + (kind git-tag-error-kind)) + +(define (git-tag-error kind) + (raise (condition (&message (message (format "bad `~a' property"))) + (&git-tag-error + (kind kind))))) + +(define (git-tag-warning package c) + (warning (package-location package) + (G_ "~a for package `~a'~%") + (condition-message c) + (package-name package))) + +(define-condition-type &git-no-tags-error &error + git-no-tags-error?) + +(define (git-no-tags-error) + (raise (condition (&message (message "no tags were found")) + (&git-no-tags-error)))) + +(define (git-no-tags-warning package c) + (warning (package-location package) + (G_ "~a for package `~a'~%") + (condition-message c) + (package-name package))) + +(define (git-fetch-warning package) + (warning (package-location package) + (G_ "failed to fetch Git repository for package `~a'~%") + (package-name package))) + + +;;; Helper functions + +(define (string-split* str delim) + "Like `string-split', but DELIM is a string instead of a +char-set." + (filter (lambda (str) (not (equal? str ""))) + (string-split str (string->char-set delim)))) + +(define* (get-version package tag #:key prefix suffix delim) + (define delim* (if delim delim ".")) + (define prefix-regexp "^[^0-9]*") + (define suffix-regexp (string-append "[^0-9" (regexp-quote delim*) "]*$"= )) + (define delim-regexp (string-append "^[0-9]+" (regexp-quote delim*) "[0-= 9]+")) + + (define no-prefix + (let ((match (string-match (or prefix prefix-regexp) tag))) + (if match + (regexp-substitute #f match 'post) + (git-tag-error 'tag-prefix)))) + + (define no-suffix + (let ((match (string-match (or suffix suffix-regexp) no-prefix))) + (if match + (regexp-substitute #f match 'pre) + (git-tag-error 'tag-suffix)))) + + (define no-delims + (if (string-match delim-regexp no-suffix) + (string-split* no-suffix delim*) + (git-tag-error 'tag-version-delimiter))) + + (string-join no-delims ".")) + +(define (sort-tags tags) + "Sort TAGS, a list if Git tags, such that the latest tag is the last ele= ment." + (sort tags (lambda (a b) + (eq? (version-compare a b) '<)))) + + +;;; Updater + +(define (get-remote url git-uri) + "Given a URL and GIT-URI, a record, return the ``origin'= ' remote." + (let* ((checkout (update-cached-checkout url + #:recursive? + (git-reference-recursive? git-u= ri))) + (repository (repository-open checkout))) + (remote-lookup repository "origin"))) + +(define (get-latest-tag remote) + "Given a Git REMOTE, return that latest tag available." + (remote-connect remote) + + (define tags + (sort-tags + (map (lambda (tag) + (string-drop tag (string-length "refs/tags/"))) + (filter (lambda (ref) + ;; Every tag has two refs: + ;; + ;; * refs/tags/1.2.3^{} + ;; * refs/tags/1.2.3 + ;; + ;; remove the one with the trailing ^{} + (and (not (string-suffix? "^{}" ref)) + (string-prefix? "refs/tags/" ref))) + (map (lambda (remote-head) + (remote-head-name remote-head)) + (remote-ls remote)))))) + + (remote-disconnect remote) + + (if (null? tags) + (git-no-tags-error) + (last tags))) + +(define (latest-git-tag-version package tag-prefix tag-suffix + tag-version-delimiter) + "Given a PACKAGE, the TAG-PREFIX, TAG-SUFFIX, and TAG-VERSION-DELIMITER +properties of PACKAGE, returns the latest version of PACKAGE." + (guard (c ((eq? (exception-kind c) 'git-error) + (git-fetch-warning package) + #f) + ((git-tag-error? c) + (git-tag-warning package c) + #f) + ((git-no-tags-error? c) + (git-no-tags-warning package c) + #f)) + (let* ((source (package-source package)) + (git-uri (origin-uri source)) + (url (git-reference-url (origin-uri source))) + (remote (get-remote url git-uri)) + (latest-tag (get-latest-tag remote))) + (get-version package + latest-tag + #:prefix tag-prefix + #:suffix tag-suffix + #:delim tag-version-delimiter)))) + +(define (git-package? package) + "Whether the origin of PACKAGE is a Git repostiory." + (match (package-source package) + ((? origin? origin) + (and (eq? (origin-method origin) git-fetch) + (git-reference? (origin-uri origin)))) + (_ #f))) + +(define (latest-git-release package) + "Return the latest release of PACKAGE." + (let* ((name (package-name package)) + (properties (package-properties package)) + (tag-prefix (assq-ref properties 'tag-prefix)) + (tag-suffix (assq-ref properties 'tag-suffix)) + (tag-version-delimiter (assq-ref properties 'tag-version-delimite= r)) + (old-version (package-version package)) + (url (git-reference-url (origin-uri (package-source package)))) + (new-version (latest-git-tag-version package + tag-prefix + tag-suffix + tag-version-delimiter))) + + (if new-version + (upstream-source + (package name) + (version new-version) + (urls (list url))) + ;; No new release or no tags available. + #f))) + +(define %generic-git-updater + (upstream-updater + (name 'generic-git) + (description "Updater for packages hosted on Git repositories") + (pred git-package?) + (latest latest-git-release))) base-commit: 9540323458de87b0b8aa421e449a4fe27af7c393 --=20 2.33.0 From debbugs-submit-bounces@debbugs.gnu.org Sat Sep 04 20:19:50 2021 Received: (at 50359) by debbugs.gnu.org; 5 Sep 2021 00:19:50 +0000 Received: from localhost ([127.0.0.1]:48868 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMfsz-0008N7-MW for submit@debbugs.gnu.org; Sat, 04 Sep 2021 20:19:50 -0400 Received: from out2.migadu.com ([188.165.223.204]:31959) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMfsw-0008Mx-8W for 50359@debbugs.gnu.org; Sat, 04 Sep 2021 20:19:48 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1630801184; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: references:references; bh=MKFB+NpBec8yWeoNs7VQUyviuqZy0NVyZz//X2MEXDk=; b=N7dMN8FCGRaqZ4MVYnaa0ZkhEz6dwSP6219YhbpdiasHZj7T2amr2GrNyTHWnL6nvYVECP o2kMA4ywPhE8kgITkLTq7R0ZKTqQjWS/bj3SvMoPk08s/icXgtpPyG1VHg4VFN0LlDiRxW 0sGXZAvfPDH35SoIFg1PQIsooxcs+eg= From: Sarah Morgensen To: Xinglu Chen Subject: Re: [bug#50359] [PATCH] import: Add 'generic-git' updater. References: Date: Sat, 04 Sep 2021 17:19:40 -0700 Message-ID: <86k0jvkh5v.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: iskarian@mgsn.dev X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hello, Thanks for the patch! Glad to see this idea becoming more polished. Xinglu Chen writes: Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [188.165.223.204 listed in wl.mailspike.net] -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hello, Thanks for the patch! Glad to see this idea becoming more polished. Xinglu Chen writes: > * guix/import/git.scm: New file. > * doc/guix.texi (Invoking guix refresh): Document it. > * Makefile.am (MODULES): Register it. > --- > This patch adds a new =E2=80=98generic-git=E2=80=99 updater which can che= ck for new tags > for package hosted on Git repos. However, it cannot download Git repos > and update the package definitions, i.e. =E2=80=98guix refresh -u=E2=80= =99. There is a > pending patch that would add this feature though[1]. > > =E2=80=98guix refresh -L=E2=80=99 now reports > > Available updaters: > [=E2=80=A6] > 94.5% of the packages are covered by these updaters. > > We are getting close to 100% :-) Wow, that is close! > > See it in action! > > $ ./pre-inst-env guix refresh harmonist scdoc gmnisrv > gnu/packages/web.scm:7931:4: warning: no tags were found for package `gmn= isrv' > gnu/packages/web.scm:7931:4: warning: 'generic-git' updater failed to det= ermine available releases for gmnisrv > gnu/packages/man.scm:339:12: scdoc would be upgraded from 1.10.1 to 1.11.1 > gnu/packages/games.scm:9433:2: warning: failed to fetch Git repository fo= r package `harmonist' > gnu/packages/games.scm:9433:2: warning: 'generic-git' updater failed to d= etermine available releases for harmonist FWIW, harmonist and a few other packages fail to work because they use an old git protocol which is not supported by libgit2. [...] > + > +@itemize > +@item @code{tag-prefix}: a regular expression for matching a prefix of > +the tag name. > + > +@item @code{tag-suffix}: a regular expression for matching a suffix of > +the tag name. > + > +@item @code{tag-version-delimiter}: a string used as the delimiter in > +the tag name for separating the numbers of the version. > +@end itemize > + > +@lisp > +(package > + (name "foo") > + ;; ... > + (properties > + '((tag-prefix . "^release0-") > + (tag-suffix . "[a-z]?$") > + (tag-version-delimiter . ":")))) > +@end lisp ^ extra whitespace I do like the selection of (prefix, suffix, delimiter), though I think there are only one or two packages which use a different delimiter. [...] > +;;; Errors & warnings > + > +(define-condition-type &git-tag-error &error > + git-tag-error? > + (kind git-tag-error-kind)) > + > +(define (git-tag-error kind) > + (raise (condition (&message (message (format "bad `~a' property"))) > + (&git-tag-error > + (kind kind))))) When I trigger this error, I get: --8<---------------cut here---------------start------------->8--- In ice-9/exceptions.scm: 406:15 6 (latest-git-release _) In ice-9/boot-9.scm: 1752:10 5 (with-exception-handler _ _ #:unwind? _ # _) In guix/import/git.scm: 59:39 4 (get-version _ _ #:prefix _ #:suffix _ #:delim _) In unknown file: 3 (simple-format #f "bad `~a' property") In ice-9/boot-9.scm: 1685:16 2 (raise-exception _ #:continuable? _) 1683:16 1 (raise-exception _ #:continuable? _) 1685:16 0 (raise-exception _ #:continuable? _) ice-9/boot-9.scm:1685:16: In procedure raise-exception: In procedure simple-format: FORMAT: Missing argument for ~a --8<---------------cut here---------------end--------------->8--- > + > +(define (git-tag-warning package c) > + (warning (package-location package) > + (G_ "~a for package `~a'~%") > + (condition-message c) > + (package-name package))) > + > +(define-condition-type &git-no-tags-error &error > + git-no-tags-error?) > + > +(define (git-no-tags-error) > + (raise (condition (&message (message "no tags were found")) > + (&git-no-tags-error)))) > + > +(define (git-no-tags-warning package c) > + (warning (package-location package) > + (G_ "~a for package `~a'~%") > + (condition-message c) > + (package-name package))) > + > +(define (git-fetch-warning package) > + (warning (package-location package) > + (G_ "failed to fetch Git repository for package `~a'~%") > + (package-name package))) > + > + > +;;; Helper functions > + > +(define (string-split* str delim) > + "Like `string-split', but DELIM is a string instead of a > +char-set." > + (filter (lambda (str) (not (equal? str ""))) > + (string-split str (string->char-set delim)))) (string-split* "1:2.3" ":.") -> ("1" "2" "3") (string-split* "1a2b3" "ab") -> ("1" "2" "3") Is this what you intended? The documentation above makes it sound like the whole string serves as the delimiter. > + > +(define* (get-version package tag #:key prefix suffix delim) PACKAGE is not used by this procedure. > + (define delim* (if delim delim ".")) > + (define prefix-regexp "^[^0-9]*") > + (define suffix-regexp (string-append "[^0-9" (regexp-quote delim*) "]*= $")) With a delimiter of '.', this would say the suffix of '1.2.3.prerelease' is 'prerelease', not '.prerelease'. Is this correct? (I would be tempted to just remove delim* from this.) > + (define delim-regexp (string-append "^[0-9]+" (regexp-quote delim*) "[= 0-9]+")) This fails to account for versions which use non-numerics, such as (all taken from the package-version field of packages using git-fetch and which use this version as the tag): 1.0.0-beta.0 0.0.9.4f 4.4-git.1 5.2.0-alpha 0.2.0-alpha-199-g3e7a475 20200701.154658.b0d6223 12-068oasis4 4.0.0.dev8 0.32-14-gcdfe14e 2.8-fix-2 There are about 50-60 packages like this. I'm not sure how much effort should be spent including them, and for some of them I'm not sure what our ideal behavior *is*. Even if we could reliably detect them, should "alpha" or "dev" packages be returned by the updater? Upon investigation, there is a deeper problem: version-compare thinks "5.2.0" is a lower version than "5.2.0-alpha", and that "4.0.0" is lower than "4.0.0.dev8". scheme@(guile-user)> (version-compare "5.1.9" "5.2.0") $5 =3D < scheme@(guile-user)> (version-compare "5.2.0" "5.2.0-alpha") $6 =3D < scheme@(guile-user)> (version-compare "4.0.0" "4.0.0.dev8") $7 =3D < > + > + (define no-prefix > + (let ((match (string-match (or prefix prefix-regexp) tag))) > + (if match > + (regexp-substitute #f match 'post) > + (git-tag-error 'tag-prefix)))) > + > + (define no-suffix > + (let ((match (string-match (or suffix suffix-regexp) no-prefix))) > + (if match > + (regexp-substitute #f match 'pre) > + (git-tag-error 'tag-suffix)))) > + > + (define no-delims > + (if (string-match delim-regexp no-suffix) > + (string-split* no-suffix delim*) > + (git-tag-error 'tag-version-delimiter))) This throws an error if the version doesn't have any delimiter. Actually, it throws an error in a lot of other cases too, often saying the 'tag-version-delimiter is wrong when it's something else. Consider the tags from the "openjpeg" package, sorted by 'sort-tags': arelease opj0-97 start v2.1.1 v2.1.2 v2.2.0 v2.3.0 v2.3.1 v2.4.0 version.1.0 version.1.1 version.1.2 version.1.3 version.1.4 version.1.5 version.1.5.1 version.1.5.2 version.2.0 version.2.0.1 version.2.1 wg1n6848 At first, 'get-version' throws an error because "wg1n6848" doesn't have a delimiter. But even disregarding that, it would return "version.2.1" -> "2.1" as the latest version. Probably we should process all tags with 'get-version' (simply skipping any that don't parse) and use that to sort the tags. If none parse with 'get-version' we could use the "no tags" error or have a separate error for "there were tags but we couldn't process them". And this lets us just do something like (untested): (define* (get-version tag #:key prefix suffix delim) (define delim-rx (regexp-quote (or delim "."))) (define prefix-rx (or prefix "[^[:digit:]]*")) (define suffix-rx (or suffix ".*")) (define version-char-rx (string-append "[^" delim-rx "[:punct:]]")) (define tag-rx (string-append "^" prefix "(" version-char-rx "+(" delim-rx version-char-rx ")*)" suffix-rx "$")) (and=3D> (string-match tag-rx tag) (cut match-substring <> 1))) Though at this point, 'tag-rx' should probably be constructed and compiled outside the loop. > + > + (string-join no-delims ".")) > + > +(define (sort-tags tags) > + "Sort TAGS, a list if Git tags, such that the latest tag is the last e= lement." > + (sort tags (lambda (a b) > + (eq? (version-compare a b) '<)))) > + > + > +;;; Updater > + > +(define (get-remote url git-uri) > + "Given a URL and GIT-URI, a record, return the ``origi= n'' remote." > + (let* ((checkout (update-cached-checkout url > + #:recursive? > + (git-reference-recursive? git= -uri))) > + (repository (repository-open checkout))) > + (remote-lookup repository "origin"))) We surely don't want 'update-cached-checkout' since that fetches the whole repo history! I've attached a patch below (based on top of this one) which brings the total time-per-package to under 1s. I moved it to (guix git) to make use of 'with-libgit2' which ensures we use system certificates. Apologies for such a long reply. I hope it was helpful :) -- Sarah --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0001-git-Add-ls-remote-refs.patch Content-Description: git: Add 'ls-remote-refs'. >From 0b0973034711e15b52702c0aec0c653dfd41928c Mon Sep 17 00:00:00 2001 Message-Id: <0b0973034711e15b52702c0aec0c653dfd41928c.1630800771.git.iskarian@mgsn.dev> From: Sarah Morgensen Date: Fri, 3 Sep 2021 22:40:02 -0700 Subject: [PATCH] git: Add 'ls-remote-refs'. --- guix/git.scm | 33 +++++++++++++++++++++++++++++++ guix/import/git.scm | 47 ++++++++++----------------------------------- 2 files changed, 43 insertions(+), 37 deletions(-) diff --git a/guix/git.scm b/guix/git.scm index 9c6f326c36..b784fd6d20 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -56,6 +56,8 @@ commit-difference commit-relation + ls-remote-refs + git-checkout git-checkout? git-checkout-url @@ -556,6 +558,37 @@ objects: 'ancestor (meaning that OLD is an ancestor of NEW), 'descendant, or (if (set-contains? oldest new) 'descendant 'unrelated)))))) + +;; +;;; Remote operations. +;;; + +(define* (ls-remote-refs url #:key tags?) + "Return the list of references advertised at Git repository URL. If TAGS? +is true, limit to only refs/tags." + (define (ref? ref) + ;; Like `git ls-remote --refs', only show actual references. + (and (string-prefix? "refs/" ref) + (not (string-suffix? "^{}" ref)))) + + (define (tag? ref) + (string-prefix? "refs/tags/" ref)) + + (define (include? ref) + (and ref? + (or (not tags?) (tag? ref)))) + + (with-libgit2 + (with-temporary-directory + (lambda (cache-directory) + (let* ((repository (repository-init cache-directory)) + ;; Create an in-memory remote so we don't touch disk. + (remote (remote-create-anonymous repository url))) + (remote-connect remote) + (remote-disconnect remote) + (repository-close! repository) + + (filter include? (map remote-head-name (remote-ls remote)))))))) ;;; diff --git a/guix/import/git.scm b/guix/import/git.scm index 9a654c1972..097a2f70bc 100644 --- a/guix/import/git.scm +++ b/guix/import/git.scm @@ -17,7 +17,6 @@ ;;; along with GNU Guix. If not, see . (define-module (guix import git) - #:use-module (git) #:use-module (guix build utils) #:use-module (guix diagnostics) #:use-module (guix git) @@ -126,40 +125,15 @@ char-set." ;;; Updater -(define (get-remote url git-uri) - "Given a URL and GIT-URI, a record, return the ``origin'' remote." - (let* ((checkout (update-cached-checkout url - #:recursive? - (git-reference-recursive? git-uri))) - (repository (repository-open checkout))) - (remote-lookup repository "origin"))) - -(define (get-latest-tag remote) - "Given a Git REMOTE, return that latest tag available." - (remote-connect remote) - - (define tags - (sort-tags - (map (lambda (tag) - (string-drop tag (string-length "refs/tags/"))) - (filter (lambda (ref) - ;; Every tag has two refs: - ;; - ;; * refs/tags/1.2.3^{} - ;; * refs/tags/1.2.3 - ;; - ;; remove the one with the trailing ^{} - (and (not (string-suffix? "^{}" ref)) - (string-prefix? "refs/tags/" ref))) - (map (lambda (remote-head) - (remote-head-name remote-head)) - (remote-ls remote)))))) - - (remote-disconnect remote) - - (if (null? tags) - (git-no-tags-error) - (last tags))) +(define (get-latest-tag url) + "Return the latest tag available from the Git repository at URL." + (let ((tags (map (cut string-drop <> (string-length "refs/tags/")) + (ls-remote-refs url #:tags? #t)))) + + (if (null? tags) + (git-no-tags-error) + (last (sort-tags tags))))) + (define (latest-git-tag-version package tag-prefix tag-suffix tag-version-delimiter) @@ -177,8 +151,7 @@ properties of PACKAGE, returns the latest version of PACKAGE." (let* ((source (package-source package)) (git-uri (origin-uri source)) (url (git-reference-url (origin-uri source))) - (remote (get-remote url git-uri)) - (latest-tag (get-latest-tag remote))) + (latest-tag (get-latest-tag url))) (get-version package latest-tag #:prefix tag-prefix base-commit: 522a3bf99cbc21a9093f63280b9508cd69b94ff0 prerequisite-patch-id: c60e771d96884a78a014e145723562a619c1a0e0 -- 2.32.0 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sat Sep 04 21:03:24 2021 Received: (at 50359) by debbugs.gnu.org; 5 Sep 2021 01:03:25 +0000 Received: from localhost ([127.0.0.1]:48883 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMgZA-0000vs-PL for submit@debbugs.gnu.org; Sat, 04 Sep 2021 21:03:24 -0400 Received: from out0.migadu.com ([94.23.1.103]:15773) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMgZ8-0000vi-K4 for 50359@debbugs.gnu.org; Sat, 04 Sep 2021 21:03:23 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1630803800; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=OrhwAzr3CgSgU7rh4QKxREoUhoPYi+iq4m9BIe0do3c=; b=hza3sadN31Zz00TLXUwkLSPglhxSDEHcFm7pAfs1P83bhKq+dnktAeoZyaectVAkGLCiUb arcl8uWSsuCuK944DvPOW7MmSlLIyKL6zxL911NocTj5e70+l9/ayq5yBg7QCPn09PGSYd XX/Z2hejPFiEL+493BWbebsqbvtXavE= From: Sarah Morgensen To: Xinglu Chen Subject: Re: [bug#50359] [PATCH] import: Add 'generic-git' updater. References: <86k0jvkh5v.fsf@mgsn.dev> Date: Sat, 04 Sep 2021 18:03:18 -0700 In-Reply-To: <86k0jvkh5v.fsf@mgsn.dev> (Sarah Morgensen's message of "Sat, 04 Sep 2021 17:19:40 -0700 (40 minutes, 11 seconds ago)") Message-ID: <86czpnkf55.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: text/plain X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: iskarian@mgsn.dev X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Apologies, in my patch, 'with-temporary-directory' should be 'call-with-temporary-directory'... Sarah Morgensen writes: > +(define* (ls-remote-refs url #:key tags?) > + "Return the list of references advertised at Git repository URL. If TAGS? > +is true, limit to only refs/tags." > + (define (ref? ref) > + ;; Like `git ls-remote --refs', only show actual references. > + (and (string-prefix? "refs/" ref) > + (not (string-suffix? "^{}" ref)))) > + > + (define (tag? ref) > + (string-prefix? "refs/tags/" ref)) > + > + (define (include? ref) > + (and ref? > + (or (not tags?) (tag? ref)))) > + > + (with-libgit2 > + (with-temporary-directory ...right here. -- Sarah From debbugs-submit-bounces@debbugs.gnu.org Sun Sep 05 06:36:38 2021 Received: (at 50359) by debbugs.gnu.org; 5 Sep 2021 10:36:39 +0000 Received: from localhost ([127.0.0.1]:49368 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMpVu-0003H3-BC for submit@debbugs.gnu.org; Sun, 05 Sep 2021 06:36:38 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:34286 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMpVs-0003Gj-Lc for 50359@debbugs.gnu.org; Sun, 05 Sep 2021 06:36:37 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1630838188; bh=eXYEC//qZ2lxCKHOpD/Aij4T6gIZE8/Mcyd/QwTDA68=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=uT2UZp/QQBJ+dxk7Tj9GLEfHRF0msM/uZuZkEYo+fpvPRkSJ169xnebdhf9b529qP 5+UXfYRRSEcGFNtPYYH1R9NkTqK5x9b8ecuiRqgB6MFA51O9VkZjv9/KRXRHzIovr4 rpdXDdtdrDsuGO56O5fH9sv8J8k/GNWr4a0Bv1II= To: Sarah Morgensen Subject: Re: [bug#50359] [PATCH] import: Add 'generic-git' updater. In-Reply-To: <86k0jvkh5v.fsf@mgsn.dev> References: <86k0jvkh5v.fsf@mgsn.dev> Date: Sun, 05 Sep 2021 12:36:24 +0200 Message-ID: <87h7ez48d3.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Sat, Sep 04 2021, Sarah Morgensen wrote: > Hello, > > Thanks for the patch! Glad to see this idea becoming more polished. > > Xinglu Chen writes: > >> * guix/import/git.scm: New file. >> * doc/guix.texi (Invoking guix [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Sat, Sep 04 2021, Sarah Morgensen wrote: > Hello, > > Thanks for the patch! Glad to see this idea becoming more polished. > > Xinglu Chen writes: > >> * guix/import/git.scm: New file. >> * doc/guix.texi (Invoking guix [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Sat, Sep 04 2021, Sarah Morgensen wrote: > Hello, > > Thanks for the patch! Glad to see this idea becoming more polished. > > Xinglu Chen writes: > >> * guix/import/git.scm: New file. >> * doc/guix.texi (Invoking guix refresh): Document it. >> * Makefile.am (MODULES): Register it. >> --- >> This patch adds a new =E2=80=98generic-git=E2=80=99 updater which can ch= eck for new tags >> for package hosted on Git repos. However, it cannot download Git repos >> and update the package definitions, i.e. =E2=80=98guix refresh -u=E2=80= =99. There is a >> pending patch that would add this feature though[1]. >> >> =E2=80=98guix refresh -L=E2=80=99 now reports >> >> Available updaters: >> [=E2=80=A6] >> 94.5% of the packages are covered by these updaters. >> >> We are getting close to 100% :-) > > Wow, that is close! > >> >> See it in action! >> >> $ ./pre-inst-env guix refresh harmonist scdoc gmnisrv >> gnu/packages/web.scm:7931:4: warning: no tags were found for package `gm= nisrv' >> gnu/packages/web.scm:7931:4: warning: 'generic-git' updater failed to de= termine available releases for gmnisrv >> gnu/packages/man.scm:339:12: scdoc would be upgraded from 1.10.1 to 1.11= .1 >> gnu/packages/games.scm:9433:2: warning: failed to fetch Git repository f= or package `harmonist' >> gnu/packages/games.scm:9433:2: warning: 'generic-git' updater failed to = determine available releases for harmonist > > FWIW, harmonist and a few other packages fail to work because they use > an old git protocol which is not supported by libgit2. > > [...] >> + >> +@itemize >> +@item @code{tag-prefix}: a regular expression for matching a prefix of >> +the tag name. >> + >> +@item @code{tag-suffix}: a regular expression for matching a suffix of >> +the tag name. >> + >> +@item @code{tag-version-delimiter}: a string used as the delimiter in >> +the tag name for separating the numbers of the version. >> +@end itemize >> + >> +@lisp >> +(package >> + (name "foo") >> + ;; ... >> + (properties >> + '((tag-prefix . "^release0-") >> + (tag-suffix . "[a-z]?$") >> + (tag-version-delimiter . ":")))) >> +@end lisp > ^ extra whitespace > > I do like the selection of (prefix, suffix, delimiter), though I think > there are only one or two packages which use a different delimiter. > > [...] >> +;;; Errors & warnings >> + >> +(define-condition-type &git-tag-error &error >> + git-tag-error? >> + (kind git-tag-error-kind)) >> + >> +(define (git-tag-error kind) >> + (raise (condition (&message (message (format "bad `~a' property"))) >> + (&git-tag-error >> + (kind kind))))) > > When I trigger this error, I get: > --8<---------------cut here---------------start------------->8--- > In ice-9/exceptions.scm: > 406:15 6 (latest-git-release _) > In ice-9/boot-9.scm: > 1752:10 5 (with-exception-handler _ _ #:unwind? _ # _) > In guix/import/git.scm: > 59:39 4 (get-version _ _ #:prefix _ #:suffix _ #:delim _) > In unknown file: > 3 (simple-format #f "bad `~a' property") > In ice-9/boot-9.scm: > 1685:16 2 (raise-exception _ #:continuable? _) > 1683:16 1 (raise-exception _ #:continuable? _) > 1685:16 0 (raise-exception _ #:continuable? _) > > ice-9/boot-9.scm:1685:16: In procedure raise-exception: > In procedure simple-format: FORMAT: Missing argument for ~a > --8<---------------cut here---------------end--------------->8--- Oops, it should be (format "bad `~a' property" kind) =20=20 >> + >> +(define (git-tag-warning package c) >> + (warning (package-location package) >> + (G_ "~a for package `~a'~%") >> + (condition-message c) >> + (package-name package))) >> + >> +(define-condition-type &git-no-tags-error &error >> + git-no-tags-error?) >> + >> +(define (git-no-tags-error) >> + (raise (condition (&message (message "no tags were found")) >> + (&git-no-tags-error)))) >> + >> +(define (git-no-tags-warning package c) >> + (warning (package-location package) >> + (G_ "~a for package `~a'~%") >> + (condition-message c) >> + (package-name package))) >> + >> +(define (git-fetch-warning package) >> + (warning (package-location package) >> + (G_ "failed to fetch Git repository for package `~a'~%") >> + (package-name package))) >> + >> + >> +;;; Helper functions >> + >> +(define (string-split* str delim) >> + "Like `string-split', but DELIM is a string instead of a >> +char-set." >> + (filter (lambda (str) (not (equal? str ""))) >> + (string-split str (string->char-set delim)))) > > (string-split* "1:2.3" ":.") -> ("1" "2" "3") > (string-split* "1a2b3" "ab") -> ("1" "2" "3") > > Is this what you intended? The documentation above makes it sound like > the whole string serves as the delimiter. It=E2=80=99s not what I wanted, indeed. I will try to fix it. >> + >> +(define* (get-version package tag #:key prefix suffix delim) > > PACKAGE is not used by this procedure. Good catch, it was some leftover I forgot to remove. >> + (define delim* (if delim delim ".")) >> + (define prefix-regexp "^[^0-9]*") >> + (define suffix-regexp (string-append "[^0-9" (regexp-quote delim*) "]= *$")) > > With a delimiter of '.', this would say the suffix of '1.2.3.prerelease' > is 'prerelease', not '.prerelease'. Is this correct? (I would be > tempted to just remove delim* from this.) Good point, I think removing =E2=80=98delim*=E2=80=99 would be a good idea. >> + (define delim-regexp (string-append "^[0-9]+" (regexp-quote delim*) "= [0-9]+")) > > This fails to account for versions which use non-numerics, such as (all > taken from the package-version field of packages using git-fetch and > which use this version as the tag): > > 1.0.0-beta.0 > 0.0.9.4f > 4.4-git.1 > 5.2.0-alpha > 0.2.0-alpha-199-g3e7a475 > 20200701.154658.b0d6223 > 12-068oasis4 > 4.0.0.dev8 > 0.32-14-gcdfe14e > 2.8-fix-2 > > There are about 50-60 packages like this. > > I'm not sure how much effort should be spent including them, and for > some of them I'm not sure what our ideal behavior *is*. Even if we > could reliably detect them, should "alpha" or "dev" packages be returned > by the updater? I don=E2=80=99t think we usually include alpha or rc releases, so updater probably shouldn=E2=80=99t return them either. Not sure how we would try to detect alpha/beta/rc releases, though, besides running something like (string-match? "(alpha|beta|rc|dev)" TAG) On each tag. > Upon investigation, there is a deeper problem: version-compare thinks > "5.2.0" is a lower version than "5.2.0-alpha", and that "4.0.0" is lower > than "4.0.0.dev8". > > scheme@(guile-user)> (version-compare "5.1.9" "5.2.0") > $5 =3D < > scheme@(guile-user)> (version-compare "5.2.0" "5.2.0-alpha") > $6 =3D < > scheme@(guile-user)> (version-compare "4.0.0" "4.0.0.dev8") > $7 =3D < Maybe we should filter the tags before comparing them; that should get rid of these pre-release tags. >> + >> + (define no-prefix >> + (let ((match (string-match (or prefix prefix-regexp) tag))) >> + (if match >> + (regexp-substitute #f match 'post) >> + (git-tag-error 'tag-prefix)))) >> + >> + (define no-suffix >> + (let ((match (string-match (or suffix suffix-regexp) no-prefix))) >> + (if match >> + (regexp-substitute #f match 'pre) >> + (git-tag-error 'tag-suffix)))) >> + >> + (define no-delims >> + (if (string-match delim-regexp no-suffix) >> + (string-split* no-suffix delim*) >> + (git-tag-error 'tag-version-delimiter))) > > This throws an error if the version doesn't have any delimiter. Setting the =E2=80=98tag-version-delimiter=E2=80=99 prefix to an empty stri= ng would solve this, right? Or, maybe we should just get rid of the delimiter thing since only a few packages use a different delimiter. > Actually, it throws an error in a lot of other cases too, often saying > the 'tag-version-delimiter is wrong when it's something else. Consider > the tags from the "openjpeg" package, sorted by 'sort-tags': > > arelease > opj0-97 > start > v2.1.1 > v2.1.2 > v2.2.0 > v2.3.0 > v2.3.1 > v2.4.0 > version.1.0 > version.1.1 > version.1.2 > version.1.3 > version.1.4 > version.1.5 > version.1.5.1 > version.1.5.2 > version.2.0 > version.2.0.1 > version.2.1 > wg1n6848 > > At first, 'get-version' throws an error because "wg1n6848" doesn't have > a delimiter. But even disregarding that, it would return "version.2.1" > -> "2.1" as the latest version. > > Probably we should process all tags with 'get-version' (simply skipping > any that don't parse) and use that to sort the tags. If none parse with > 'get-version' we could use the "no tags" error or have a separate error > for "there were tags but we couldn't process them". Ah, yes, that would be a good idea. > And this lets us just do something like (untested): > > (define* (get-version tag #:key prefix suffix delim) > (define delim-rx (regexp-quote (or delim "."))) > (define prefix-rx (or prefix "[^[:digit:]]*")) > (define suffix-rx (or suffix ".*")) > (define version-char-rx > (string-append "[^" delim-rx "[:punct:]]")) > > (define tag-rx > (string-append "^" prefix "(" version-char-rx "+(" > delim-rx version-char-rx ")*)" suffix-rx "$")) > > (and=3D> (string-match tag-rx tag) > (cut match-substring <> 1))) > > Though at this point, 'tag-rx' should probably be constructed and > compiled outside the loop. > >> + >> + (string-join no-delims ".")) >> + >> +(define (sort-tags tags) >> + "Sort TAGS, a list if Git tags, such that the latest tag is the last = element." >> + (sort tags (lambda (a b) >> + (eq? (version-compare a b) '<)))) >> + >> + >> +;;; Updater >> + >> +(define (get-remote url git-uri) >> + "Given a URL and GIT-URI, a record, return the ``orig= in'' remote." >> + (let* ((checkout (update-cached-checkout url >> + #:recursive? >> + (git-reference-recursive? gi= t-uri))) >> + (repository (repository-open checkout))) >> + (remote-lookup repository "origin"))) > > We surely don't want 'update-cached-checkout' since that fetches the > whole repo history! I've attached a patch below (based on top of this > one) which brings the total time-per-package to under 1s. I moved it to > (guix git) to make use of 'with-libgit2' which ensures we use system > certificates. > > Apologies for such a long reply. I hope it was helpful :) No worries, it definitely helped a lot, thank you! --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmE0nagVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5vCAQAI0BtYCXoZojo9O8Y4O128gOfCvo ETQvtPRNMILtZ8mkv+6aY0gnB1foGIEUDWb7pdZs3IBY6Z7H9yAM/pIaWnlfldum 6x/3m6ESO1GVyEFsSZxZ4pPa1xeQ3+qIRrKqIPN8fpmyttxzEDR54djq8yBARlCR RhSjEf+r65nsUZqlOBRfWCDp/Gx7oreb6+RiFo7ZYeBsLESlcLt8SmUbJSXzw0b5 gvigA232i3povwdbYdfGiX3np9K6xhaXeXcGO5pPike5jlJlpLoSYmBP0aqhHOqb TgqUvmCoxHcnMc8ytEVR02CdSix3RBhP8z7+fbyb1emTGVDiLCGfHCUn8L1jg2pF 6zsxKEGbLshNHkkolSIlm1YuHELiPqo8s9aNCtPFYyfcpn5iNjjx/P6B0Ywb1Gm6 uFiZ4f4PT3+slS07WWwZz+AbJnts+NFuhP26TOSjDym/WzM2XNOYAxlmcjoEDriz +H8RFOmt6p8nnE2oBxLZkRq1+6jQBJHrCJc6+ZrJm3N3ZlFa0qdbRMwEeSkHrj7Y nJgp+ZdIBqm2zbvw/xyquzytysnCuu2mBlQBPuWlRzoa9HYTa4FW6YEa7lXX8FDf 75tgRA+A2o/Cjv19zn6XpJjtIYW1Rdg7NhJqWmqUoPpMMf9BqNw5vcXztXSM2/f6 A05zw5m5IumjEdiy =ZAEz -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Sep 05 09:11:32 2021 Received: (at 50359) by debbugs.gnu.org; 5 Sep 2021 13:11:32 +0000 Received: from localhost ([127.0.0.1]:49429 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMrvm-0000hQ-VS for submit@debbugs.gnu.org; Sun, 05 Sep 2021 09:11:32 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:37500 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mMrvk-0000h8-4q for 50359@debbugs.gnu.org; Sun, 05 Sep 2021 09:11:30 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1630847480; bh=symRydbRMAn9xIMs0ijSQ604qu2+5ZzJH8i12xOxy80=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=OExsn8Dfdto3EQ8A6pj+ET1SXwFNisN2wE0oHoxKzKMS3bzhC2e4Llu6C6J7+S7aY HIIQUuQsIF0r1RhfdGVP99bwJBdDUo7TCRHSRCHD69qaNrXkhGY9vpvi8UREGTU5f3 3Q3ogLDRRVn8stXY5FTbeBVckGsQ6PE7trg9My3k= To: Sarah Morgensen Subject: Re: [bug#50359] [PATCH] import: Add 'generic-git' updater. In-Reply-To: <86k0jvkh5v.fsf@mgsn.dev> References: <86k0jvkh5v.fsf@mgsn.dev> Date: Sun, 05 Sep 2021 15:11:18 +0200 Message-ID: <87eea3416x.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Some more comments after some testing On Sat, Sep 04 2021, Sarah Morgensen wrote: > Hello, > > Thanks for the patch! Glad to see this idea becoming more polished. > > Xinglu Chen writes: > >> * guix/import/git.scm: New file. >> * doc/guix.texi (Invoking guix [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Some more comments after some testing On Sat, Sep 04 2021, Sarah Morgensen wrote: > Hello, > > Thanks for the patch! Glad to see this idea becoming more polished. > > Xinglu Chen writes: > >> * guix/import/git.scm: New file. >> * doc/guix.texi (Invoking guix [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Some more comments after some testing On Sat, Sep 04 2021, Sarah Morgensen wrote: > Hello, > > Thanks for the patch! Glad to see this idea becoming more polished. > > Xinglu Chen writes: > >> * guix/import/git.scm: New file. >> * doc/guix.texi (Invoking guix refresh): Document it. >> * Makefile.am (MODULES): Register it. >> --- >> This patch adds a new =E2=80=98generic-git=E2=80=99 updater which can ch= eck for new tags >> for package hosted on Git repos. However, it cannot download Git repos >> and update the package definitions, i.e. =E2=80=98guix refresh -u=E2=80= =99. There is a >> pending patch that would add this feature though[1]. >> >> =E2=80=98guix refresh -L=E2=80=99 now reports >> >> Available updaters: >> [=E2=80=A6] >> 94.5% of the packages are covered by these updaters. >> >> We are getting close to 100% :-) > > Wow, that is close! > >> >> See it in action! >> >> $ ./pre-inst-env guix refresh harmonist scdoc gmnisrv >> gnu/packages/web.scm:7931:4: warning: no tags were found for package `gm= nisrv' >> gnu/packages/web.scm:7931:4: warning: 'generic-git' updater failed to de= termine available releases for gmnisrv >> gnu/packages/man.scm:339:12: scdoc would be upgraded from 1.10.1 to 1.11= .1 >> gnu/packages/games.scm:9433:2: warning: failed to fetch Git repository f= or package `harmonist' >> gnu/packages/games.scm:9433:2: warning: 'generic-git' updater failed to = determine available releases for harmonist > > FWIW, harmonist and a few other packages fail to work because they use > an old git protocol which is not supported by libgit2. > > [...] >> + >> +@itemize >> +@item @code{tag-prefix}: a regular expression for matching a prefix of >> +the tag name. >> + >> +@item @code{tag-suffix}: a regular expression for matching a suffix of >> +the tag name. >> + >> +@item @code{tag-version-delimiter}: a string used as the delimiter in >> +the tag name for separating the numbers of the version. >> +@end itemize >> + >> +@lisp >> +(package >> + (name "foo") >> + ;; ... >> + (properties >> + '((tag-prefix . "^release0-") >> + (tag-suffix . "[a-z]?$") >> + (tag-version-delimiter . ":")))) >> +@end lisp > ^ extra whitespace > > I do like the selection of (prefix, suffix, delimiter), though I think > there are only one or two packages which use a different delimiter. > > [...] >> +;;; Errors & warnings >> + >> +(define-condition-type &git-tag-error &error >> + git-tag-error? >> + (kind git-tag-error-kind)) >> + >> +(define (git-tag-error kind) >> + (raise (condition (&message (message (format "bad `~a' property"))) >> + (&git-tag-error >> + (kind kind))))) > > When I trigger this error, I get: > --8<---------------cut here---------------start------------->8--- > In ice-9/exceptions.scm: > 406:15 6 (latest-git-release _) > In ice-9/boot-9.scm: > 1752:10 5 (with-exception-handler _ _ #:unwind? _ # _) > In guix/import/git.scm: > 59:39 4 (get-version _ _ #:prefix _ #:suffix _ #:delim _) > In unknown file: > 3 (simple-format #f "bad `~a' property") > In ice-9/boot-9.scm: > 1685:16 2 (raise-exception _ #:continuable? _) > 1683:16 1 (raise-exception _ #:continuable? _) > 1685:16 0 (raise-exception _ #:continuable? _) > > ice-9/boot-9.scm:1685:16: In procedure raise-exception: > In procedure simple-format: FORMAT: Missing argument for ~a > --8<---------------cut here---------------end--------------->8--- > >> + >> +(define (git-tag-warning package c) >> + (warning (package-location package) >> + (G_ "~a for package `~a'~%") >> + (condition-message c) >> + (package-name package))) >> + >> +(define-condition-type &git-no-tags-error &error >> + git-no-tags-error?) >> + >> +(define (git-no-tags-error) >> + (raise (condition (&message (message "no tags were found")) >> + (&git-no-tags-error)))) >> + >> +(define (git-no-tags-warning package c) >> + (warning (package-location package) >> + (G_ "~a for package `~a'~%") >> + (condition-message c) >> + (package-name package))) >> + >> +(define (git-fetch-warning package) >> + (warning (package-location package) >> + (G_ "failed to fetch Git repository for package `~a'~%") >> + (package-name package))) >> + >> + >> +;;; Helper functions >> + >> +(define (string-split* str delim) >> + "Like `string-split', but DELIM is a string instead of a >> +char-set." >> + (filter (lambda (str) (not (equal? str ""))) >> + (string-split str (string->char-set delim)))) > > (string-split* "1:2.3" ":.") -> ("1" "2" "3") > (string-split* "1a2b3" "ab") -> ("1" "2" "3") > > Is this what you intended? The documentation above makes it sound like > the whole string serves as the delimiter. > >> + >> +(define* (get-version package tag #:key prefix suffix delim) > > PACKAGE is not used by this procedure. > >> + (define delim* (if delim delim ".")) >> + (define prefix-regexp "^[^0-9]*") >> + (define suffix-regexp (string-append "[^0-9" (regexp-quote delim*) "]= *$")) > > With a delimiter of '.', this would say the suffix of '1.2.3.prerelease' > is 'prerelease', not '.prerelease'. Is this correct? (I would be > tempted to just remove delim* from this.) > >> + (define delim-regexp (string-append "^[0-9]+" (regexp-quote delim*) "= [0-9]+")) > > This fails to account for versions which use non-numerics, such as (all > taken from the package-version field of packages using git-fetch and > which use this version as the tag): > > 1.0.0-beta.0 > 0.0.9.4f > 4.4-git.1 > 5.2.0-alpha > 0.2.0-alpha-199-g3e7a475 > 20200701.154658.b0d6223 > 12-068oasis4 > 4.0.0.dev8 > 0.32-14-gcdfe14e > 2.8-fix-2 > > There are about 50-60 packages like this. > > I'm not sure how much effort should be spent including them, and for > some of them I'm not sure what our ideal behavior *is*. Even if we > could reliably detect them, should "alpha" or "dev" packages be returned > by the updater? > > Upon investigation, there is a deeper problem: version-compare thinks > "5.2.0" is a lower version than "5.2.0-alpha", and that "4.0.0" is lower > than "4.0.0.dev8". > > scheme@(guile-user)> (version-compare "5.1.9" "5.2.0") > $5 =3D < > scheme@(guile-user)> (version-compare "5.2.0" "5.2.0-alpha") > $6 =3D < > scheme@(guile-user)> (version-compare "4.0.0" "4.0.0.dev8") > $7 =3D < > >> + >> + (define no-prefix >> + (let ((match (string-match (or prefix prefix-regexp) tag))) >> + (if match >> + (regexp-substitute #f match 'post) >> + (git-tag-error 'tag-prefix)))) >> + >> + (define no-suffix >> + (let ((match (string-match (or suffix suffix-regexp) no-prefix))) >> + (if match >> + (regexp-substitute #f match 'pre) >> + (git-tag-error 'tag-suffix)))) >> + >> + (define no-delims >> + (if (string-match delim-regexp no-suffix) >> + (string-split* no-suffix delim*) >> + (git-tag-error 'tag-version-delimiter))) > > This throws an error if the version doesn't have any delimiter. > > Actually, it throws an error in a lot of other cases too, often saying > the 'tag-version-delimiter is wrong when it's something else. Consider > the tags from the "openjpeg" package, sorted by 'sort-tags': > > arelease > opj0-97 > start > v2.1.1 > v2.1.2 > v2.2.0 > v2.3.0 > v2.3.1 > v2.4.0 > version.1.0 > version.1.1 > version.1.2 > version.1.3 > version.1.4 > version.1.5 > version.1.5.1 > version.1.5.2 > version.2.0 > version.2.0.1 > version.2.1 > wg1n6848 > > At first, 'get-version' throws an error because "wg1n6848" doesn't have > a delimiter. But even disregarding that, it would return "version.2.1" > -> "2.1" as the latest version. > > Probably we should process all tags with 'get-version' (simply skipping > any that don't parse) and use that to sort the tags. If none parse with > 'get-version' we could use the "no tags" error or have a separate error > for "there were tags but we couldn't process them". > > And this lets us just do something like (untested): > > (define* (get-version tag #:key prefix suffix delim) > (define delim-rx (regexp-quote (or delim "."))) > (define prefix-rx (or prefix "[^[:digit:]]*")) > (define suffix-rx (or suffix ".*")) > (define version-char-rx > (string-append "[^" delim-rx "[:punct:]]")) > > (define tag-rx > (string-append "^" prefix "(" version-char-rx "+(" > delim-rx version-char-rx ")*)" suffix-rx "$")) This wouldn=E2=80=99t match anything if the version is just a plain number, e.g., 1 or 09. > > (and=3D> (string-match tag-rx tag) > (cut match-substring <> 1))) With this, something like =E2=80=9C1.4.0rc1-450-g2725ef99d=E2=80=9D will re= sult in =E2=80=9C1.4.0=E2=80=9D being returned, which is incorrect. Changing (cut match:substring <> 1) to just =E2=80=98match:substring=E2=80=99 would solve= the issue, but then pre-release tags, which we usually don=E2=80=99t want, would also= get matched. Not sure what the best option would be in this case. > Though at this point, 'tag-rx' should probably be constructed and > compiled outside the loop. > >> + >> + (string-join no-delims ".")) >> + >> +(define (sort-tags tags) >> + "Sort TAGS, a list if Git tags, such that the latest tag is the last = element." >> + (sort tags (lambda (a b) >> + (eq? (version-compare a b) '<)))) >> + >> + >> +;;; Updater >> + >> +(define (get-remote url git-uri) >> + "Given a URL and GIT-URI, a record, return the ``orig= in'' remote." >> + (let* ((checkout (update-cached-checkout url >> + #:recursive? >> + (git-reference-recursive? gi= t-uri))) >> + (repository (repository-open checkout))) >> + (remote-lookup repository "origin"))) > > We surely don't want 'update-cached-checkout' since that fetches the > whole repo history! I've attached a patch below (based on top of this > one) which brings the total time-per-package to under 1s. I moved it to > (guix git) to make use of 'with-libgit2' which ensures we use system > certificates. > > Apologies for such a long reply. I hope it was helpful :) > > -- > Sarah > > From 0b0973034711e15b52702c0aec0c653dfd41928c Mon Sep 17 00:00:00 2001 > Message-Id: <0b0973034711e15b52702c0aec0c653dfd41928c.1630800771.git.iska= rian@mgsn.dev> > From: Sarah Morgensen > Date: Fri, 3 Sep 2021 22:40:02 -0700 > Subject: [PATCH] git: Add 'ls-remote-refs'. > > --- > guix/git.scm | 33 +++++++++++++++++++++++++++++++ > guix/import/git.scm | 47 ++++++++++----------------------------------- > 2 files changed, 43 insertions(+), 37 deletions(-) > > diff --git a/guix/git.scm b/guix/git.scm > index 9c6f326c36..b784fd6d20 100644 > --- a/guix/git.scm > +++ b/guix/git.scm > @@ -56,6 +56,8 @@ > commit-difference > commit-relation >=20=20 > + ls-remote-refs > + > git-checkout > git-checkout? > git-checkout-url > @@ -556,6 +558,37 @@ objects: 'ancestor (meaning that OLD is an ancestor = of NEW), 'descendant, or > (if (set-contains? oldest new) > 'descendant > 'unrelated)))))) > + > +;; > +;;; Remote operations. > +;;; > + > +(define* (ls-remote-refs url #:key tags?) > + "Return the list of references advertised at Git repository URL. If T= AGS? > +is true, limit to only refs/tags." > + (define (ref? ref) > + ;; Like `git ls-remote --refs', only show actual references. > + (and (string-prefix? "refs/" ref) > + (not (string-suffix? "^{}" ref)))) > + > + (define (tag? ref) > + (string-prefix? "refs/tags/" ref)) > + > + (define (include? ref) > + (and ref? > + (or (not tags?) (tag? ref)))) > + > + (with-libgit2 > + (with-temporary-directory > + (lambda (cache-directory) > + (let* ((repository (repository-init cache-directory)) > + ;; Create an in-memory remote so we don't touch disk. > + (remote (remote-create-anonymous repository url))) > + (remote-connect remote) > + (remote-disconnect remote) > + (repository-close! repository) > + > + (filter include? (map remote-head-name (remote-ls remote)))))))) > For some reason it seems to include refs that do and don=E2=80=99t end with =E2=80=9C^{}=E2=80=9D =2D-8<---------------cut here---------------start------------->8--- scheme@(guile-user)> (ls-remote-refs "https://github.com/clementine-player/= Clementine" #:tags? #t) $6 =3D ("refs/tags/0.1" "refs/tags/0.1^{}" "refs/tags/0.2" "refs/tags/0.2^{= }" "refs/tags/0.3" "refs/tags/0.3^{}" "refs/tags/0.3.1" "refs/tags/0.3.1^{}= " "refs/tags/0.3.2" "refs/tags/0.3.2^{}" "refs/tags/0.3rc1" "refs/tags/0.3r= c1^{}" "refs/tags/0.4" "refs/tags/0.4^{}" "refs/tags/0.4.1" "refs/tags/0.4.= 1^{}" "refs/tags/0.4.2" "refs/tags/0.4.2^{}" "refs/tags/0.4rc1" "refs/tags/= 0.4rc1^{}" "refs/tags/0.5" "refs/tags/0.5^{}" "refs/tags/0.5.1" "refs/tags/= 0.5.1^{}" "refs/tags/0.5.2" "refs/tags/0.5.2^{}" "refs/tags/0.5.3" "refs/ta= gs/0.5.3^{}" "refs/tags/0.5rc1" "refs/tags/0.5rc1^{}" "refs/tags/0.6" "refs= /tags/0.6^{}" "refs/tags/0.6rc1" "refs/tags/0.6rc1^{}" "refs/tags/0.7" "ref= s/tags/0.7^{}" "refs/tags/0.7.1" "refs/tags/0.7.1^{}" "refs/tags/0.7.2" "re= fs/tags/0.7.2^{}" "refs/tags/0.7.3" "refs/tags/0.7.3^{}" "refs/tags/0.7rc1"= "refs/tags/0.7rc1^{}" "refs/tags/1.0" "refs/tags/1.0^{}" "refs/tags/1.0.1"= "refs/tags/1.0.1^{}" "refs/tags/1.0rc1" "refs/tags/1.0rc1^{}" "refs/tags/1= .1" "refs/tags/1.1^{}" "refs/tags/1.1.1" "refs/tags/1.1.1^{}" "refs/tags/1.= 2" "refs/tags/1.2^{}" "refs/tags/1.2.1" "refs/tags/1.2.1^{}" "refs/tags/1.2= .2" "refs/tags/1.2.2^{}" "refs/tags/1.2.3" "refs/tags/1.2.3^{}" "refs/tags/= 1.3" "refs/tags/1.3.1" "refs/tags/1.3rc1" "refs/tags/1.4.0rc1" "refs/tags/1= .4.0rc1^{}" "refs/tags/1.4.0rc1-153-g06ba55549" "refs/tags/1.4.0rc1-156-gca= 6f42fae" "refs/tags/1.4.0rc1-157-g176b1d6c7" "refs/tags/1.4.0rc1-163-gef302= 1dff" "refs/tags/1.4.0rc1-167-gb0c92ae78" "refs/tags/1.4.0rc1-168-g6285c11b= c" "refs/tags/1.4.0rc1-169-g934fd336d" "refs/tags/1.4.0rc1-170-g509c65ced" = "refs/tags/1.4.0rc1-171-g0ecb77335" "refs/tags/1.4.0rc1-172-gb007e54b3" "re= fs/tags/1.4.0rc1-174-gcb64d9705" "refs/tags/1.4.0rc1-176-g7e7d271b3" "refs/= tags/1.4.0rc1-177-g096203ac8" "refs/tags/1.4.0rc1-188-g83fc376b0" "refs/tag= s/1.4.0rc1-189-g58569d9d0" "refs/tags/1.4.0rc1-194-gbaea2d488" "refs/tags/1= .4.0rc1-198-g6a5cb0712" "refs/tags/1.4.0rc1-200-g18497dcb6" "refs/tags/1.4.= 0rc1-201-gf46241e75" "refs/tags/1.4.0rc1-202-g833f8256c" "refs/tags/1.4.0rc= 1-203-gbc1674700" "refs/tags/1.4.0rc1-204-g912589439" "refs/tags/1.4.0rc1-2= 06-g8f56fbb83" "refs/tags/1.4.0rc1-207-g879dfa3d7" "refs/tags/1.4.0rc1-211-= g949c20abd" "refs/tags/1.4.0rc1-230-gc934fef63" "refs/tags/1.4.0rc1-231-g60= a46d193" "refs/tags/1.4.0rc1-234-g0271f43cc" "refs/tags/1.4.0rc1-235-g92b16= 0d2a" "refs/tags/1.4.0rc1-236-g13ee11f81" "refs/tags/1.4.0rc1-237-g54f200d9= b" "refs/tags/1.4.0rc1-239-gfa067bf5c" "refs/tags/1.4.0rc1-241-ge7c5c76ea" = "refs/tags/1.4.0rc1-242-gcf1067e74" "refs/tags/1.4.0rc1-243-g5612c9cb5" "re= fs/tags/1.4.0rc1-244-g84099f249" "refs/tags/1.4.0rc1-245-g0555cf5a3" "refs/= tags/1.4.0rc1-246-gf90babefa" "refs/tags/1.4.0rc1-247-g1a73918f9" "refs/tag= s/1.4.0rc1-248-ged0078b8d" "refs/tags/1.4.0rc1-250-ga63a37a7a" "refs/tags/1= .4.0rc1-251-g6f5fe724b" "refs/tags/1.4.0rc1-252-gc8d56776a" "refs/tags/1.4.= 0rc1-253-g00f9597d3" "refs/tags/1.4.0rc1-254-gbf3d3db23" "refs/tags/1.4.0rc= 1-257-g236cfa7ad" "refs/tags/1.4.0rc1-258-g84fc00d55" "refs/tags/1.4.0rc1-2= 61-g48ae27b4a" "refs/tags/1.4.0rc1-262-g536f34526" "refs/tags/1.4.0rc1-263-= g4c9241db1" "refs/tags/1.4.0rc1-264-g22537a450" "refs/tags/1.4.0rc1-265-g22= cfade4a" "refs/tags/1.4.0rc1-268-gc299c198d" "refs/tags/1.4.0rc1-269-gcf8d2= 004b" "refs/tags/1.4.0rc1-270-g6900197a8" "refs/tags/1.4.0rc1-271-g56ed6d4f= 7" "refs/tags/1.4.0rc1-272-gedb8c3b4e" "refs/tags/1.4.0rc1-274-g1ef5ec259" = "refs/tags/1.4.0rc1-275-g0d25a1b39" "refs/tags/1.4.0rc1-276-g8c25c443c" "re= fs/tags/1.4.0rc1-279-g76a24a0a2" "refs/tags/1.4.0rc1-280-gcf279e6f4" "refs/= tags/1.4.0rc1-282-gad882cc99" "refs/tags/1.4.0rc1-283-g0fcb1df20" "refs/tag= s/1.4.0rc1-284-g7d28e8700" "refs/tags/1.4.0rc1-285-gebf9ebf08" "refs/tags/1= .4.0rc1-289-g834b1d451" "refs/tags/1.4.0rc1-290-g3bfaf3ff3" "refs/tags/1.4.= 0rc1-291-gc75fa0077" "refs/tags/1.4.0rc1-292-gdd9ed2334" "refs/tags/1.4.0rc= 1-293-g1f7607b1d" "refs/tags/1.4.0rc1-294-g987fe047c" "refs/tags/1.4.0rc1-2= 95-gee72793b5" "refs/tags/1.4.0rc1-296-g68d375c43" "refs/tags/1.4.0rc1-310-= gd131c66f0" "refs/tags/1.4.0rc1-315-g16843da41" "refs/tags/1.4.0rc1-318-g44= af6f9d5" "refs/tags/1.4.0rc1-319-gd3e327022" "refs/tags/1.4.0rc1-320-g3a4d7= f3a3" "refs/tags/1.4.0rc1-321-g2d280734a" "refs/tags/1.4.0rc1-322-g6821f6d7= b" "refs/tags/1.4.0rc1-323-g29aad2ae3" "refs/tags/1.4.0rc1-324-g06855ea6c" = "refs/tags/1.4.0rc1-325-g598f84007" "refs/tags/1.4.0rc1-326-gd0bf92f06" "re= fs/tags/1.4.0rc1-327-g7b3a0f397" "refs/tags/1.4.0rc1-328-ge9b62fa34" "refs/= tags/1.4.0rc1-329-gf7bece3b8" "refs/tags/1.4.0rc1-332-g62d2f0de9" "refs/tag= s/1.4.0rc1-340-g2172732b1" "refs/tags/1.4.0rc1-341-g54f7637ad" "refs/tags/1= .4.0rc1-342-g2bac3626c" "refs/tags/1.4.0rc1-343-gb49afcc5b" "refs/tags/1.4.= 0rc1-344-gad354276b" "refs/tags/1.4.0rc1-345-g9e8d4434a" "refs/tags/1.4.0rc= 1-346-g4e3e9c8d1" "refs/tags/1.4.0rc1-347-gfc4cb6fc7" "refs/tags/1.4.0rc1-3= 48-gcac606186" "refs/tags/1.4.0rc1-349-g16d09ace0" "refs/tags/1.4.0rc1-350-= geefb96bdc" "refs/tags/1.4.0rc1-351-g1daf43f91" "refs/tags/1.4.0rc1-352-gaa= ee0b701" "refs/tags/1.4.0rc1-353-gae4948ce3" "refs/tags/1.4.0rc1-354-gd970b= 7400" "refs/tags/1.4.0rc1-355-gc856a6617" "refs/tags/1.4.0rc1-356-gd417aed2= 9" "refs/tags/1.4.0rc1-357-geec7641ef" "refs/tags/1.4.0rc1-358-gc536dc88e" = "refs/tags/1.4.0rc1-360-gb2044a5be" "refs/tags/1.4.0rc1-361-gf17e29f41" "re= fs/tags/1.4.0rc1-362-g7b3e2dfd8" "refs/tags/1.4.0rc1-363-gf60c42224" "refs/= tags/1.4.0rc1-364-gc4d22d441" "refs/tags/1.4.0rc1-365-g41b1ba8ff" "refs/tag= s/1.4.0rc1-366-g20f49c445" "refs/tags/1.4.0rc1-368-g1a0b288a8" "refs/tags/1= .4.0rc1-369-gf5c904b26" "refs/tags/1.4.0rc1-370-gcca48b1eb" "refs/tags/1.4.= 0rc1-371-gdf262c5c7" "refs/tags/1.4.0rc1-372-g01f072764" "refs/tags/1.4.0rc= 1-373-gba8fc09a6" "refs/tags/1.4.0rc1-374-g91bad31f6" "refs/tags/1.4.0rc1-3= 77-gccba649f6" "refs/tags/1.4.0rc1-378-ga3a51ae11" "refs/tags/1.4.0rc1-379-= gcfcd0a956" "refs/tags/1.4.0rc1-380-gd7966c828" "refs/tags/1.4.0rc1-384-g41= 513527c" "refs/tags/1.4.0rc1-386-gbbb6a773f" "refs/tags/1.4.0rc1-387-g627dd= c398" "refs/tags/1.4.0rc1-388-g6a6ef729e" "refs/tags/1.4.0rc1-389-g51c600a5= 3" "refs/tags/1.4.0rc1-390-gaf4810a58" "refs/tags/1.4.0rc1-391-g863a66824" = "refs/tags/1.4.0rc1-392-g9f8093a22" "refs/tags/1.4.0rc1-393-gc999fc70e" "re= fs/tags/1.4.0rc1-394-g870969ef4" "refs/tags/1.4.0rc1-395-gfaab7fa6c" "refs/= tags/1.4.0rc1-396-g549544517" "refs/tags/1.4.0rc1-397-g616ccc6fd" "refs/tag= s/1.4.0rc1-398-g0393d865c" "refs/tags/1.4.0rc1-399-ga012e7e27" "refs/tags/1= .4.0rc1-400-g87cd3d2ab" "refs/tags/1.4.0rc1-401-gdc2c1e111" "refs/tags/1.4.= 0rc1-402-g63a73a4a5" "refs/tags/1.4.0rc1-403-g2b99d32be" "refs/tags/1.4.0rc= 1-406-g409c6b89d" "refs/tags/1.4.0rc1-407-g3efa68f07" "refs/tags/1.4.0rc1-4= 08-g8f863bc96" "refs/tags/1.4.0rc1-409-g8201c1035" "refs/tags/1.4.0rc1-410-= g479f1d4de" "refs/tags/1.4.0rc1-413-g25d3fca07" "refs/tags/1.4.0rc1-414-g8c= 774e388" "refs/tags/1.4.0rc1-416-g7b9430982" "refs/tags/1.4.0rc1-417-gf7796= 52aa" "refs/tags/1.4.0rc1-418-gb3aed042e" "refs/tags/1.4.0rc1-420-g596cd9b0= a" "refs/tags/1.4.0rc1-421-ge1e559732" "refs/tags/1.4.0rc1-422-gace5234e6" = "refs/tags/1.4.0rc1-423-g2dd424a19" "refs/tags/1.4.0rc1-425-g4f5bf1cc6" "re= fs/tags/1.4.0rc1-426-g72e2e62eb" "refs/tags/1.4.0rc1-427-gcf842a8c5" "refs/= tags/1.4.0rc1-428-g81a3c0f83" "refs/tags/1.4.0rc1-429-gf1678fd33" "refs/tag= s/1.4.0rc1-430-g7854aefdd" "refs/tags/1.4.0rc1-431-ga9e193234" "refs/tags/1= .4.0rc1-432-g447e91a68" "refs/tags/1.4.0rc1-433-g76c87146d" "refs/tags/1.4.= 0rc1-434-ga7a32b08b" "refs/tags/1.4.0rc1-436-g8c2ab8fa0" "refs/tags/1.4.0rc= 1-438-gcb88954a3" "refs/tags/1.4.0rc1-439-g79ca9147e" "refs/tags/1.4.0rc1-4= 40-g7ba322b10" "refs/tags/1.4.0rc1-441-gb9a844263" "refs/tags/1.4.0rc1-442-= g78d4c4f3f" "refs/tags/1.4.0rc1-444-g8d11e9ffa" "refs/tags/1.4.0rc1-446-g18= eef830a" "refs/tags/1.4.0rc1-447-g8db8b1e78" "refs/tags/1.4.0rc1-448-g816fd= 88d4" "refs/tags/1.4.0rc1-449-g50ee78613" "refs/tags/1.4.0rc1-450-g2725ef99= d" "refs/tags/1.4.0rc1-451-g66ea25bca" "refs/tags/1.4.0rc1-453-g281da0532" = "refs/tags/1.4.0rc1-454-g57a6fe4f2" "refs/tags/1.4.0rc1-456-geb7a9bfa4" "re= fs/tags/1.4.0rc1-457-g8f3772b59" "refs/tags/1.4.0rc1-459-ge84f87f62" "refs/= tags/1.4.0rc1-461-gf7b6708e4" "refs/tags/1.4.0rc1-462-gfffc50c79" "refs/tag= s/1.4.0rc1-463-gf7ed4a309" "refs/tags/1.4.0rc1-464-gcde0343a6" "refs/tags/1= .4.0rc1-465-gb69dd2d90" "refs/tags/1.4.0rc1-466-gd9a48b90b" "refs/tags/1.4.= 0rc1-467-gd93bd9ca2" "refs/tags/1.4.0rc1-468-g1a3828e2c" "refs/tags/1.4.0rc= 1-469-gb40d9ed44" "refs/tags/1.4.0rc1-471-gb989a674a" "refs/tags/1.4.0rc1-4= 72-g4e8a12f37" "refs/tags/1.4.0rc1-473-gbce55d0ef" "refs/tags/1.4.0rc1-477-= g576731767" "refs/tags/1.4.0rc1-480-g05f513ab6" "refs/tags/1.4.0rc1-481-g2b= 988ed7b" "refs/tags/1.4.0rc1-482-g0c099ab6f" "refs/tags/1.4.0rc1-483-gc7f5c= 0f40" "refs/tags/1.4.0rc1-484-g2d8a56b7c" "refs/tags/1.4.0rc1-486-gf92690c1= 4" "refs/tags/1.4.0rc1-487-g15474ada3" "refs/tags/1.4.0rc1-488-g7bb0c59f2" = "refs/tags/1.4.0rc1-489-g6314c8cb2" "refs/tags/1.4.0rc1-491-g651eee13e" "re= fs/tags/1.4.0rc1-494-gdfb953a78" "refs/tags/1.4.0rc1-495-g10bf5dc17" "refs/= tags/1.4.0rc1-496-gcef1d7e74" "refs/tags/1.4.0rc1-497-g3bd15aea0" "refs/tag= s/1.4.0rc1-498-g681e7bea5" "refs/tags/1.4.0rc1-502-gaf75ebbd6" "refs/tags/1= .4.0rc1-509-g89e9b20df" "refs/tags/1.4.0rc1-510-g3f34b332c" "refs/tags/1.4.= 0rc1-512-g8b2f7f08a" "refs/tags/1.4.0rc1-514-g05e450c3c" "refs/tags/1.4.0rc= 1-515-g1154c0f54" "refs/tags/1.4.0rc1-516-g8b566b2a7" "refs/tags/1.4.0rc1-5= 18-g3244cf083" "refs/tags/1.4.0rc1-519-gd1e9ee9f9" "refs/tags/1.4.0rc1-520-= gc394d7d2d" "refs/tags/1.4.0rc1-521-gb68b12010" "refs/tags/1.4.0rc1-522-gfd= b3f7ac3" "refs/tags/1.4.0rc1-525-gc12294c5e" "refs/tags/1.4.0rc1-526-g88189= 8f84" "refs/tags/1.4.0rc1-527-g438e8ca61" "refs/tags/1.4.0rc1-528-g86d782cb= 6" "refs/tags/1.4.0rc1-531-g641279072" "refs/tags/1.4.0rc1-533-gf4e70face" = "refs/tags/1.4.0rc1-534-gd13410c91" "refs/tags/1.4.0rc1-536-g4edf77082" "re= fs/tags/1.4.0rc1-537-gada6752ea" "refs/tags/1.4.0rc1-538-g15fdad3d5" "refs/= tags/1.4.0rc1-540-g4f86e0b2b" "refs/tags/1.4.0rc1-541-ge077df22d" "refs/tag= s/1.4.0rc1-542-g8a7120e1e" "refs/tags/1.4.0rc1-544-g3b8519fda" "refs/tags/1= .4.0rc1-545-g2d6bb4abd" "refs/tags/1.4.0rc1-548-g354f6a23e" "refs/tags/1.4.= 0rc1-549-ge8875faf8" "refs/tags/1.4.0rc1-550-g72c1f91c0" "refs/tags/1.4.0rc= 1-551-g144bdc249" "refs/tags/1.4.0rc1-552-gdb55c541b" "refs/tags/1.4.0rc1-5= 53-ga86558f9a" "refs/tags/1.4.0rc1-554-g2d34588b8" "refs/tags/1.4.0rc1-555-= g32944a15d" "refs/tags/1.4.0rc1-556-g3440f90a6" "refs/tags/1.4.0rc1-557-g00= 9642d12" "refs/tags/1.4.0rc1-558-g47f7b307f" "refs/tags/1.4.0rc1-559-ge7364= 263b" "refs/tags/1.4.0rc1-560-g7303f72ee" "refs/tags/1.4.0rc1-562-g99ee1394= a" "refs/tags/1.4.0rc1-563-g163ebe71d" "refs/tags/1.4.0rc1-564-g429d8ee0f" = "refs/tags/1.4.0rc1-565-g6b21079fd" "refs/tags/1.4.0rc1-566-gf04657e7e" "re= fs/tags/1.4.0rc1-567-g280a514eb" "refs/tags/1.4.0rc1-568-gc51d2f954" "refs/= tags/1.4.0rc1-569-gf17b79a10" "refs/tags/1.4.0rc1-570-g73c0af197" "refs/tag= s/1.4.0rc1-571-g5f75bde39" "refs/tags/1.4.0rc1-572-g59f6d95b8" "refs/tags/1= .4.0rc1-573-g8258c78c0" "refs/tags/1.4.0rc1-574-gb2ed9499f" "refs/tags/1.4.= 0rc1-575-g94f4f65a6" "refs/tags/1.4.0rc1-576-g7e48b78c1" "refs/tags/1.4.0rc= 1-577-gfc83e4127" "refs/tags/1.4.0rc1-578-gd59ed1e70" "refs/tags/1.4.0rc1-5= 79-g8fddc816a" "refs/tags/1.4.0rc1-585-g8c1bdc1a4" "refs/tags/1.4.0rc1-586-= g20647e8a9" "refs/tags/1.4.0rc1-587-g708385c71" "refs/tags/1.4.0rc1-588-g9a= 337a9ef" "refs/tags/1.4.0rc1-589-gf48888a43" "refs/tags/1.4.0rc1-591-g579d8= 6904" "refs/tags/1.4.0rc1-593-g783213f9c" "refs/tags/1.4.0rc1-594-gf5d3079d= b" "refs/tags/1.4.0rc1-596-g590bcf1c7" "refs/tags/1.4.0rc1-597-g83157100c" = "refs/tags/1.4.0rc1-598-gd16d9ba28" "refs/tags/1.4.0rc1-600-g3f614464e" "re= fs/tags/1.4.0rc1-601-ga7468dcd4" "refs/tags/1.4.0rc1-602-g89155ace7" "refs/= tags/1.4.0rc1-603-g75de59703" "refs/tags/1.4.0rc1-604-g1309c76be" "refs/tag= s/1.4.0rc1-613-ge756f2d68" "refs/tags/1.4.0rc1-614-g89831f8dc" "refs/tags/1= .4.0rc1-617-g776bd3b02" "refs/tags/1.4.0rc1-618-gf071075e8" "refs/tags/1.4.= 0rc1-619-gd71eba97f" "refs/tags/1.4.0rc1-620-g684c9d232" "refs/tags/1.4.0rc= 1-621-g2132e99fb" "refs/tags/1.4.0rc1-622-gf7369d2c4" "refs/tags/1.4.0rc1-6= 23-gf67475375" "refs/tags/1.4.0rc1-624-g72cfdf25a" "refs/tags/1.4.0rc1-626-= g058fe6f4b" "refs/tags/1.4.0rc1-627-g0dbefa306" "refs/tags/1.4.0rc1-628-gb0= 9ab3ff3" "refs/tags/1.4.0rc1-629-g612767c87" "refs/tags/1.4.0rc1-631-g4e4fc= cc07" "refs/tags/1.4.0rc1-633-g3a00403ad" "refs/tags/1.4.0rc1-634-g4aa4f4fc= e" "refs/tags/1.4.0rc1-635-g418a36693" "refs/tags/1.4.0rc1-636-g2bf8f1388" = "refs/tags/1.4.0rc1-637-gffdaeba09" "refs/tags/1.4.0rc1-638-gc3c77aef1" "re= fs/tags/1.4.0rc1-639-g11bd0db03" "refs/tags/1.4.0rc1-657-g57b5911f1" "refs/= tags/1.4.0rc1-658-g6240fd3d0" "refs/tags/1.4.0rc1-659-g54be35f52" "refs/tag= s/1.4.0rc1-660-ge46503d0c" "refs/tags/1.4.0rc1-661-g62cb889a3" "refs/tags/1= .4.0rc1-662-g5ab81fd8b" "refs/tags/1.4.0rc1-663-gf9854e564" "refs/tags/1.4.= 0rc1-664-g1db1e3231" "refs/tags/1.4.0rc1-665-g67a947f11" "refs/tags/1.4.0rc= 1-666-g4a83f8c81" "refs/tags/1.4.0rc1-668-gf35a640ce" "refs/tags/1.4.0rc1-6= 69-g67aa15418" "refs/tags/1.4.0rc1-670-g8c660e278" "refs/tags/1.4.0rc1-671-= g25b537cf2" "refs/tags/1.4.0rc1-672-ga5fd484a6" "refs/tags/1.4.0rc1-673-gdb= 8de64ab" "refs/tags/1.4.0rc1-674-g7cb5f5c80" "refs/tags/1.4.0rc1-675-ga5e84= bbe9" "refs/tags/1.4.0rc1-676-g6b2918ee9" "refs/tags/1.4.0rc1-677-g4acfdae7= 4" "refs/tags/1.4.0rc1-678-g2902a8786" "refs/tags/1.4.0rc1-679-gb3b769f0e" = "refs/tags/1.4.0rc1-680-g4d3474840" "refs/tags/1.4.0rc1-681-g598e660ae" "re= fs/tags/1.4.0rc1-682-g0c1b6a2a4" "refs/tags/1.4.0rc1-683-g320a1b81c" "refs/= tags/1.4.0rc1-684-g1d1d3b157" "refs/tags/1.4.0rc1-685-gf379ad84d" "refs/tag= s/1.4.0rc1-686-gdaa2f25e3" "refs/tags/1.4.0rc1-687-g1e39ce29a" "refs/tags/1= .4.0rc1-688-g98dd3e48a" "refs/tags/1.4.0rc1-689-g6982b4781" "refs/tags/1.4.= 0rc1-690-gc0c903767" "refs/tags/1.4.0rc1-691-gdbe15e5e9" "refs/tags/1.4.0rc= 1-692-g224c475b5" "refs/tags/1.4.0rc1-693-gac3a0d33f" "refs/tags/1.4.0rc1-6= 94-g102317e5c" "refs/tags/1.4.0rc1-695-ge2d6759d5" "refs/tags/1.4.0rc1-696-= gbf424ce98" "refs/tags/1.4.0rc1-697-gcddc08e14" "refs/tags/1.4.0rc1-698-gb5= 5e54388" "refs/tags/1.4.0rc1-699-g327d5fdac" "refs/tags/1.4.0rc1-700-g03e13= c69e" "refs/tags/1.4.0rc1-701-g8682d4de4" "refs/tags/1.4.0rc1-702-g922afe50= 6" "refs/tags/1.4.0rc1-708-gc8c110efa" "refs/tags/1.4.0rc1-709-g628ff6582" = "refs/tags/1.4.0rc1-710-g7eb62b626" "refs/tags/1.4.0rc1-711-g3b7d5880f" "re= fs/tags/1.4.0rc1-712-g769d8bbe6" "refs/tags/1.4.0rc1-713-gc58335c6c" "refs/= tags/1.4.0rc1-715-ge556a59ae" "refs/tags/1.4.0rc1-716-g2cca75d93") scheme@(guile-user)> =2D-8<---------------cut here---------------end--------------->8--- --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmE0wfcVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5v8QP/1CC/R9sAsolZ2UOQ6I8R4RXOTqx 2eGD9CyQGvD+xxliaP+cWzom/ti8vjGf8EEQfSIZ4F5db3z1VdbfU8XmMg1lwhXx /pdH5tjCjASrWioMKMZh7ZI4VMjZmSTCtvkGqsIiYdZ1xP11bJyIqDCxzib1Ue61 irN2Yh9UEQzyrW5a+TQrUkEWAY35ayNusiJHA0ANml6k7SyM6Y3CSTCfr0zk9EK+ UEiGuTpdUnezR/1czsxl/CvCepVmS9Xm6Ys1BO8v1DzXAtKGzxne4n5QuTUEL2Lo 5yFkIEtiFo1ccu6kbh59KsmOsljVtBZOuWNsEnM5d2ybKVyD350n2XoEOztv+VUQ vZjpqbuDXt17MovXQGmNU6mS0r176X9KlXPN227uR7rZc9CyiTfcAvttrTMJ8pFG FNJwhbuAzX271hrOHkLgAbj03Q6B1J5kyTMnfsD0jct4NDypsKZnw5CGltRyGCSZ d9R52wyeiNIJv4nwxrfe3a4Oi1tIkJI81nqOM674Z1v0lGqUIk0yC63d/A+whngs 0cOs/qCiZDF+xxFF/3ptXKSsJyKSBu2cLZbp7aUhOj1Oqj40zE2Itf2qz6wgpXc0 wovuyEPjLb6SF3z9Fz1YwPusgm99zWyrM4/XMMZ6UWMQAd3jkCL8CBYm7pavbFsr QnYIhfSYyoo7stod =mfA0 -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Sep 05 23:14:56 2021 Received: (at 50359) by debbugs.gnu.org; 6 Sep 2021 03:14:56 +0000 Received: from localhost ([127.0.0.1]:51288 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mN560-0000W4-IS for submit@debbugs.gnu.org; Sun, 05 Sep 2021 23:14:56 -0400 Received: from out2.migadu.com ([188.165.223.204]:13490) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mN55x-0000Vt-3z for 50359@debbugs.gnu.org; Sun, 05 Sep 2021 23:14:54 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1630898091; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xUrn6lZE3xVKauxh73rs8SYlgQuSTAK1Bn2VQe/8pXk=; b=LvTReMGbQawATevtl9is1eatz2cFz9S72jIAZrnTo53GLiKakkU00gpLyCCgfiF4ZzztXB V3tj8yyF1RJAOgbsflLR1wmTAlP+VnIqwreOMyWAM4vW/4BuVDdIwcUxvIm2DT7cJpw1xx GuIJNX8nVJ7ReXZiYL5EO3R8kjVd72o= From: Sarah Morgensen To: Xinglu Chen Subject: Re: [bug#50359] [PATCH] import: Add 'generic-git' updater. References: <86k0jvkh5v.fsf@mgsn.dev> <87eea3416x.fsf@yoctocell.xyz> Date: Sun, 05 Sep 2021 20:14:49 -0700 In-Reply-To: <87eea3416x.fsf@yoctocell.xyz> (Xinglu Chen's message of "Sun, 05 Sep 2021 15:11:18 +0200 (5 hours, 24 minutes, 26 seconds ago)") Message-ID: <861r62jsye.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: iskarian@mgsn.dev X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hello, Xinglu Chen writes: >> And this lets us just do something like (untested): >> >> (define* (get-version tag #:key prefix suffix delim) >> (define delim-rx (regexp-quote (or delim "."))) >> (define prefix-rx (or prefix "[^ [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [188.165.223.204 listed in wl.mailspike.net] -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) Hello, Xinglu Chen writes: >> And this lets us just do something like (untested): >> >> (define* (get-version tag #:key prefix suffix delim) >> (define delim-rx (regexp-quote (or delim "."))) >> (define prefix-rx (or prefix "[^[:digit:]]*")) >> (define suffix-rx (or suffix ".*")) >> (define version-char-rx >> (string-append "[^" delim-rx "[:punct:]]")) >> >> (define tag-rx >> (string-append "^" prefix "(" version-char-rx "+(" >> delim-rx version-char-rx ")*)" suffix-rx "$")) > > This wouldn=E2=80=99t match anything if the version is just a plain numbe= r, > e.g., 1 or 09. It does, but I had many errors in the definition. Again, apologies. I shouldn't send emails that late, haha. This method should read: --8<---------------cut here---------------start------------->8--- (define* (get-version tag #:key prefix suffix delim) (define delim-rx (regexp-quote (or delim "."))) (define prefix-rx (or prefix "[^[:digit:]]*")) (define suffix-rx (or suffix ".*")) (define version-char-rx (string-append "[^" delim-rx "[:punct:]]")) (define tag-rx (string-append "^" prefix-rx "(" version-char-rx "+(" delim-rx version-char-rx "+)*)" suffix-rx "$")) (and=3D> (string-match tag-rx tag) (cut match:substring <> 1))) --8<---------------cut here---------------end--------------->8--- > > With this, something like =E2=80=9C1.4.0rc1-450-g2725ef99d=E2=80=9D will = result in > =E2=80=9C1.4.0=E2=80=9D being returned, which is incorrect. Changing (cut > match:substring <> 1) to just =E2=80=98match:substring=E2=80=99 would sol= ve the issue, > but then pre-release tags, which we usually don=E2=80=99t want, would al= so get > matched. Not sure what the best option would be in this case. > With the fixed method above: scheme@(emacs-guix)> (get-version "8") $16 =3D "8" scheme@(emacs-guix)> (get-version "1.4.0rc1-450-g2725ef99d") $17 =3D "1.4.0rc1" But, we still get: scheme@(emacs-guix)> (get-version "1.4.0-rc1") $18 =3D "1.4.0" which leads us to what you talked about in your other message. [...] >> +(define* (ls-remote-refs url #:key tags?) >> + "Return the list of references advertised at Git repository URL. If = TAGS? >> +is true, limit to only refs/tags." >> + (define (ref? ref) >> + ;; Like `git ls-remote --refs', only show actual references. >> + (and (string-prefix? "refs/" ref) >> + (not (string-suffix? "^{}" ref)))) >> + >> + (define (tag? ref) >> + (string-prefix? "refs/tags/" ref)) >> + >> + (define (include? ref) >> + (and ref? This should be: (and (ref? ref) >> + (or (not tags?) (tag? ref)))) >> + >> + (with-libgit2 >> + (with-temporary-directory >> + (lambda (cache-directory) >> + (let* ((repository (repository-init cache-directory)) >> + ;; Create an in-memory remote so we don't touch disk. >> + (remote (remote-create-anonymous repository url))) >> + (remote-connect remote) >> + (remote-disconnect remote) >> + (repository-close! repository) >> + >> + (filter include? (map remote-head-name (remote-ls remote)))))))) >> > > For some reason it seems to include refs that do and don=E2=80=99t end wi= th > =E2=80=9C^{}=E2=80=9D Sorry, another typo I missed. See above. -- Sarah From debbugs-submit-bounces@debbugs.gnu.org Mon Sep 06 01:40:15 2021 Received: (at 50359) by debbugs.gnu.org; 6 Sep 2021 05:40:15 +0000 Received: from localhost ([127.0.0.1]:51401 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mN7Mc-0006U2-Hv for submit@debbugs.gnu.org; Mon, 06 Sep 2021 01:40:15 -0400 Received: from out0.migadu.com ([94.23.1.103]:32941) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mN7MZ-0006To-NK for 50359@debbugs.gnu.org; Mon, 06 Sep 2021 01:40:13 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1630906809; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Vc21lp4iSw+Pfq9KNBd4IwNHCdIxnnMcXqV5hIbaYqw=; b=AR5kqWPz/Bu+HBa80vmM5kB+w3L4XO6NyGAGNSAH//wg8932gj+hrvp8527DMR99ksy/sT gw1kw9bYwqcnKTFko1INrvO3ZqVYv3jm4MfqAlYDdzcz9ug3Lflc9YkP8tVNFtxPKsWuSo 8DOgHXc9++Cw4jhGuZaJ5BCNPm4U7eQ= From: Sarah Morgensen To: Xinglu Chen Subject: Re: [bug#50359] [PATCH] import: Add 'generic-git' updater. References: <86k0jvkh5v.fsf@mgsn.dev> <87h7ez48d3.fsf@yoctocell.xyz> Date: Sun, 05 Sep 2021 22:40:07 -0700 In-Reply-To: <87h7ez48d3.fsf@yoctocell.xyz> (Xinglu Chen's message of "Sun, 05 Sep 2021 12:36:24 +0200 (16 hours, 38 minutes, 30 seconds ago)") Message-ID: <86y28ai7ns.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: iskarian@mgsn.dev X-Spam-Score: 1.3 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi, Xinglu Chen writes: >> There are about 50-60 packages like this. >> >> I'm not sure how much effort should be spent including them, and for >> some of them I'm not sure what our ideal behavior *is*. Even if we >> could r [...] Content analysis details: (1.3 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/, low trust [94.23.1.103 listed in list.dnswl.org] -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [94.23.1.103 listed in wl.mailspike.net] -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.3 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, Xinglu Chen writes: >> There are about 50-60 packages like this. >> >> I'm not sure how much effort should be spent including them, and for >> some of them I'm not sure what our ideal behavior *is*. Even if we >> could reliably detect them, should "alpha" or "dev" packages be returned >> by the updater? > > I don=E2=80=99t think we usually include alpha or rc releases, so updater > probably shouldn=E2=80=99t return them either. Not sure how we would try= to > detect alpha/beta/rc releases, though, besides running something like > > (string-match? "(alpha|beta|rc|dev)" TAG) > > On each tag. That heuristic is pretty good. (It might miss a few, but I'd rather accidentally include some alpha/beta/rc releases than risk excluding real ones.) We could then safely sort tags with just the prefix removed -- this takes c= are of "1.1f" coming after "1.1", and so on. Actually, it looks like there's only a few packages with a suffix; instead, we can probably just only use a suffix if they provide 'tag-suffix. This is all of them (and the "dev" ones probably shouldn't be suffixes, just part of the version): (commit (string-append "v" version "-stable")))) (commit (string-append version "-stable")))) (commit (string-append version "-Leia")))) (commit (string-append "haddock-" version "-release")))) (commit (string-append "v" version "-8.13")))) (commit (string-append "v" version "-oss")))) (commit (string-append "v" version "-stable")))) (commit (string-append "ddskk-" version "_" code-name)))) (commit (string-append version "-freebsdport")))) (commit (string-append version "-dev")))) (commit (string-append version "-release-20210531143054")))) (commit (string-append version "-release-20210412001032")))) (commit (string-append "v" version "-debian")))) (commit (string-append version "dev")))) (commit (string-append version "_Linux")) (commit (string-append version "R")))) (commit (string-append "jdk-" version "-ga")))) (commit (string-append "jdk-" version "-ga")))) (commit (string-append "jdk-" version "-ga")))) (commit (string-append "jdk-" version "-ga")))) (commit (string-append version "-opt")))) (commit (string-append "1.1-" version "-RELEASE")))) Additionally, these are all the weird version strings I could find that are actually used as the commit: --=-=-= Content-Type: text/plain Content-Disposition: inline; filename=versions.txt Content-Description: versions 1.2.2.rc2 12-068oasis4 1.2-2 0.F-2 0.2.0-alpha 5.1.0-b2 1.5-11 1.9.14-20210407 0.9.3b 2.8-fix-2 2020-05-19 10-11.0.0 1.0.12-2 0.9.3+16.04.20160218-0ubuntu1 2.12.c 1.1+11 5.1+4.06.0 4.2-411 1.4.0rc1-450-g2725ef99d 2.0.0-alpha14 60.2.3-2 1.21c 1.0.0rc4 2.1.0b1 1.02r6 0.32-14-gcdfe14e 3.0.0a3 2.00a2.3 1.0beta.18 2.1b 2.7.8a 2.7.3a 1.1.alpha19 1.0.2-rc4 0.5.3+git20200502 1.0.3-rc3 4.0.0.dev8 3.0.0beta1-24-g024cc9fa2 0.16-2-ge145396 2.0b6 2.0M10 1.0.7+0 1.16.0+5 0.4.0+1 2.2.10+0 4.3.1+2 2.13.93+0 2.10.4+0 1.0.5+5 0.21.0+0 3.3.4+0 2.68.1+0 0.10.1+1 6.9.10-12+3 2.0.1+2 3.100.0+1 0.14.0+2 0.1.6+2 3.3.0+0 1.8.7+0 1.3.0+2 1.42.0+0 1.16.1+0 2.35.0+0 1.6.37+5 4.1.0+1 2.36.0+0 1.3.6+4 2.10.1+0 2.26.0+0 1.3.4+0 1.1.1+2 1.3.1+1 8.44.0+0 0.40.1+0 5.15.2+0 1.17.0+3 1.18.0+3 2020.7.14+0 3.0.0+1 0.9.1+3 2.9.12+0 0.1.0+2 1.6.9+2 1.0.9+3 1.13.0+2 1.2.0+3 1.1.3+3 1.3.4+2 5.0.3+3 1.7.10+3 1.1.4+3 1.1.0+3 1.5.2+3 0.9.10+3 0.4.0+1 0.4.0+1 0.4.0+1 0.3.9+1 0.4.1+1 1.4.2+3 2.27.0+3 1.4.0+2 1.1.34+0 1.2.12+1 1.5.0+0 2021-06-07 1.2.2-5-g20dc8ed 2.1-20201229 3.0-rc1 1.32.0-0 20200701.154658.b0d6223 3.9-0 R63-10032.B 0.58.2.a 0.0.0+git20200527 3.028R 3.001R 2.57b 1.0.0-20201130134442-10cb98267c6c 0.0.0-20161123171359-e6a2ba005892 0.0.0-20210615171337-6886f2dfbf5b 2020-11-10 5.2.0-alpha 2021-01-01 2021-02-28 2020-11-01 4.4-git.1 1.217-2 3.3.06-1 0.2.0-alpha-199-g3e7a475 1.0.0-beta.0 1.20190621-4 1.9.0-147-g61edec1ef 0.0.9.4f --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Yep, it looks like the above would work for the majority of these. That's probably Good Enough^tm. >>> + (define no-delims >>> + (if (string-match delim-regexp no-suffix) >>> + (string-split* no-suffix delim*) >>> + (git-tag-error 'tag-version-delimiter))) >> >> This throws an error if the version doesn't have any delimiter. > > Setting the =E2=80=98tag-version-delimiter=E2=80=99 prefix to an empty st= ring would > solve this, right? Or, maybe we should just get rid of the delimiter > thing since only a few packages use a different delimiter. IMO, just get rid of the delimiter. If we wanted to be *that* flexible, we could make it so they provide a tag->version proc instead of (prefix, suffix, delimiter). -- Sarah --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon Sep 06 08:20:53 2021 Received: (at 50359) by debbugs.gnu.org; 6 Sep 2021 12:20:53 +0000 Received: from localhost ([127.0.0.1]:52280 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNDcK-0007kr-Kz for submit@debbugs.gnu.org; Mon, 06 Sep 2021 08:20:53 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:37942 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNDcI-0007ka-Ag for 50359@debbugs.gnu.org; Mon, 06 Sep 2021 08:20:51 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1630930837; bh=rTeAfqqkrZ8uCrLD42TmYnmw6Zfc0mWDeJW9zRplpEc=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=MYYVUOoe+DrZ96dkt00vEYMFEyxSINEtCD0m6tE9lhiRsm8VlNqbqiwK2gv9ooEVL OTFRKDlomoQ8fgVPqa4Aa/k+tTnu2tQFbO4GngHrLonWRvNWg20F1hGOi7bGdIBzS8 hTGdJsdFudoC2F/mSQqDu+WCMDT9WzKcjQD9cP/g= To: Sarah Morgensen Subject: Re: [bug#50359] [PATCH] import: Add 'generic-git' updater. In-Reply-To: <86y28ai7ns.fsf@mgsn.dev> References: <86k0jvkh5v.fsf@mgsn.dev> <87h7ez48d3.fsf@yoctocell.xyz> <86y28ai7ns.fsf@mgsn.dev> Date: Mon, 06 Sep 2021 14:20:34 +0200 Message-ID: <87y28928vh.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Sun, Sep 05 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >>> There are about 50-60 packages like this. >>> >>> I'm not sure how much effort should be spent including them, and for >>> some of them I'm n [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Sun, Sep 05 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >>> There are about 50-60 packages like this. >>> >>> I'm not sure how much effort should be spent including them, and for >>> some of them I'm n [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Sun, Sep 05 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >>> There are about 50-60 packages like this. >>> >>> I'm not sure how much effort should be spent including them, and for >>> some of them I'm not sure what our ideal behavior *is*. Even if we >>> could reliably detect them, should "alpha" or "dev" packages be returned >>> by the updater? >> >> I don=E2=80=99t think we usually include alpha or rc releases, so updater >> probably shouldn=E2=80=99t return them either. Not sure how we would tr= y to >> detect alpha/beta/rc releases, though, besides running something like >> >> (string-match? "(alpha|beta|rc|dev)" TAG) >> >> On each tag. > > That heuristic is pretty good. (It might miss a few, but I'd rather > accidentally include some alpha/beta/rc releases than risk excluding > real ones.) > > We could then safely sort tags with just the prefix removed -- this takes= care > of "1.1f" coming after "1.1", and so on. > > Actually, it looks like there's only a few packages with a suffix; > instead, we can probably just only use a suffix if they provide > 'tag-suffix. This is all of them (and the "dev" ones probably shouldn't > be suffixes, just part of the version): Yeah, I think that sounds like a reasonable thing to do. > (commit (string-append "v" version "-stable")))) > (commit (string-append version "-stable")))) > (commit (string-append version "-Leia")))) > (commit (string-append "haddock-" version "-release")))) > (commit (string-append "v" version "-8.13")))) > (commit (string-append "v" version "-oss")))) > (commit (string-append "v" version "-stable")))) > (commit (string-append "ddskk-" version "_" code-name)))) > (commit (string-append version "-freebsdport")))) > (commit (string-append version "-dev")))) > (commit (string-append version "-release-20210531143054")))) > (commit (string-append version "-release-20210412001032")))) > (commit (string-append "v" version "-debian")))) > (commit (string-append version "dev")))) > (commit (string-append version "_Linux")) > (commit (string-append version "R")))) > (commit (string-append "jdk-" version "-ga")))) > (commit (string-append "jdk-" version "-ga")))) > (commit (string-append "jdk-" version "-ga")))) > (commit (string-append "jdk-" version "-ga")))) > (commit (string-append version "-opt")))) > (commit (string-append "1.1-" version "-RELEASE")))) > > Additionally, these are all the weird version strings I could find that > are actually used as the commit: > 1.2.2.rc2 > 12-068oasis4 > 1.2-2 > 0.F-2 > 0.2.0-alpha > 5.1.0-b2 > 1.5-11 > 1.9.14-20210407 > 0.9.3b > 2.8-fix-2 > 2020-05-19 > 10-11.0.0 > 1.0.12-2 > 0.9.3+16.04.20160218-0ubuntu1 > 2.12.c > 1.1+11 > 5.1+4.06.0 > 4.2-411 > 1.4.0rc1-450-g2725ef99d > 2.0.0-alpha14 > 60.2.3-2 > 1.21c > 1.0.0rc4 > 2.1.0b1 > 1.02r6 > 0.32-14-gcdfe14e > 3.0.0a3 > 2.00a2.3 > 1.0beta.18 > 2.1b > 2.7.8a > 2.7.3a > 1.1.alpha19 > 1.0.2-rc4 > 0.5.3+git20200502 > 1.0.3-rc3 > 4.0.0.dev8 > 3.0.0beta1-24-g024cc9fa2 > 0.16-2-ge145396 > 2.0b6 > 2.0M10 > 1.0.7+0 > 1.16.0+5 > 0.4.0+1 > 2.2.10+0 > 4.3.1+2 > 2.13.93+0 > 2.10.4+0 > 1.0.5+5 > 0.21.0+0 > 3.3.4+0 > 2.68.1+0 > 0.10.1+1 > 6.9.10-12+3 > 2.0.1+2 > 3.100.0+1 > 0.14.0+2 > 0.1.6+2 > 3.3.0+0 > 1.8.7+0 > 1.3.0+2 > 1.42.0+0 > 1.16.1+0 > 2.35.0+0 > 1.6.37+5 > 4.1.0+1 > 2.36.0+0 > 1.3.6+4 > 2.10.1+0 > 2.26.0+0 > 1.3.4+0 > 1.1.1+2 > 1.3.1+1 > 8.44.0+0 > 0.40.1+0 > 5.15.2+0 > 1.17.0+3 > 1.18.0+3 > 2020.7.14+0 > 3.0.0+1 > 0.9.1+3 > 2.9.12+0 > 0.1.0+2 > 1.6.9+2 > 1.0.9+3 > 1.13.0+2 > 1.2.0+3 > 1.1.3+3 > 1.3.4+2 > 5.0.3+3 > 1.7.10+3 > 1.1.4+3 > 1.1.0+3 > 1.5.2+3 > 0.9.10+3 > 0.4.0+1 > 0.4.0+1 > 0.4.0+1 > 0.3.9+1 > 0.4.1+1 > 1.4.2+3 > 2.27.0+3 > 1.4.0+2 > 1.1.34+0 > 1.2.12+1 > 1.5.0+0 > 2021-06-07 > 1.2.2-5-g20dc8ed > 2.1-20201229 > 3.0-rc1 > 1.32.0-0 > 20200701.154658.b0d6223 > 3.9-0 > R63-10032.B > 0.58.2.a > 0.0.0+git20200527 > 3.028R > 3.001R > 2.57b > 1.0.0-20201130134442-10cb98267c6c > 0.0.0-20161123171359-e6a2ba005892 > 0.0.0-20210615171337-6886f2dfbf5b > 2020-11-10 > 5.2.0-alpha > 2021-01-01 > 2021-02-28 > 2020-11-01 > 4.4-git.1 > 1.217-2 > 3.3.06-1 > 0.2.0-alpha-199-g3e7a475 > 1.0.0-beta.0 > 1.20190621-4 > 1.9.0-147-g61edec1ef > 0.0.9.4f > > Yep, it looks like the above would work for the majority of these. > That's probably Good Enough^tm. Any trick you used to find all of there weird version numbers? :-) >>>> + (define no-delims >>>> + (if (string-match delim-regexp no-suffix) >>>> + (string-split* no-suffix delim*) >>>> + (git-tag-error 'tag-version-delimiter))) >>> >>> This throws an error if the version doesn't have any delimiter. >> >> Setting the =E2=80=98tag-version-delimiter=E2=80=99 prefix to an empty s= tring would >> solve this, right? Or, maybe we should just get rid of the delimiter >> thing since only a few packages use a different delimiter. > > IMO, just get rid of the delimiter. If we wanted to be *that* flexible, > we could make it so they provide a tag->version proc instead of (prefix, > suffix, delimiter). a =E2=80=98tag->version=E2=80=99 procedure would probably make things a bit= too complicated for the people writing package definitions. For example, having a delimiter would make it easy to match a tag like =E2=80=9C2021-01-01-release=E2=80=9D Delimiter is =E2=80=9C.=E2=80=9D (sorry if this hurts your eyes ;-)) scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^\\.[:punct:= ]]+(\\.[^\\.[:punct:]]+)*).*$" "2021-01-01-release") 1) $28 =3D "2021" Delimiter is =E2=80=9C-=E2=80=9D scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^-[:punct:]]= +(-[^-[:punct:]]+)*).*$" "2021-01-01-release") 1) $29 =3D "2021-01-01-release" And then, setting the suffix to =E2=80=9C-release=E2=80=9D would match just= the version part. On Sun, Sep 05 2021, Sarah Morgensen wrote (again): >>> And this lets us just do something like (untested): >>> >>> (define* (get-version tag #:key prefix suffix delim) >>> (define delim-rx (regexp-quote (or delim "."))) >>> (define prefix-rx (or prefix "[^[:digit:]]*")) >>> (define suffix-rx (or suffix ".*")) >>> (define version-char-rx >>> (string-append "[^" delim-rx "[:punct:]]")) >>> >>> (define tag-rx >>> (string-append "^" prefix "(" version-char-rx "+(" >>> delim-rx version-char-rx ")*)" suffix-rx "$")) >> >> This wouldn=E2=80=99t match anything if the version is just a plain numb= er, >> e.g., 1 or 09. > > It does, Oh, I missed the extra pair of parens, sorry about that. > but I had many errors in the definition. Again, apologies. I > shouldn't send emails that late, haha. This method should read: > > --8<---------------cut here---------------start------------->8--- > (define* (get-version tag #:key prefix suffix delim) > (define delim-rx (regexp-quote (or delim "."))) > (define prefix-rx (or prefix "[^[:digit:]]*")) > (define suffix-rx (or suffix ".*")) > (define version-char-rx > (string-append "[^" delim-rx "[:punct:]]")) > > (define tag-rx > (string-append "^" prefix-rx "(" version-char-rx "+(" > delim-rx version-char-rx "+)*)" suffix-rx "$")) > (and=3D> (string-match tag-rx tag) > (cut match:substring <> 1))) > --8<---------------cut here---------------end--------------->8--- > >> >> With this, something like =E2=80=9C1.4.0rc1-450-g2725ef99d=E2=80=9D will= result in >> =E2=80=9C1.4.0=E2=80=9D being returned, which is incorrect. Changing (c= ut >> match:substring <> 1) to just =E2=80=98match:substring=E2=80=99 would so= lve the issue, >> but then pre-release tags, which we usually don=E2=80=99t want, would a= lso get >> matched. Not sure what the best option would be in this case. >> > > With the fixed method above: > > scheme@(emacs-guix)> (get-version "8") > $16 =3D "8" > scheme@(emacs-guix)> (get-version "1.4.0rc1-450-g2725ef99d") > $17 =3D "1.4.0rc1" > > But, we still get: > > scheme@(emacs-guix)> (get-version "1.4.0-rc1") > $18 =3D "1.4.0" > > which leads us to what you talked about in your other message. Hmm, maybe we could check that if (string-append "1.4.0" suffix) is a suffix on of the tag, or is this too much overhead? This would only work if we set the suffix to the empty string by default. Since (string-match "1.4.0$" "1.4.0-rc1") returns #f, this tag would get filtered out. But then a suffix would have to be specified to match "1.4.0rc1-450-g2725ef99d". Not sure what the best option is here. >>> +(define* (ls-remote-refs url #:key tags?) >>> + "Return the list of references advertised at Git repository URL. If= TAGS? >>> +is true, limit to only refs/tags." >>> + (define (ref? ref) >>> + ;; Like `git ls-remote --refs', only show actual references. >>> + (and (string-prefix? "refs/" ref) >>> + (not (string-suffix? "^{}" ref)))) >>> + >>> + (define (tag? ref) >>> + (string-prefix? "refs/tags/" ref)) >>> + >>> + (define (include? ref) >>> + (and ref? > > This should be: > (and (ref? ref) Ah, problem solved! :-) >>> + (or (not tags?) (tag? ref)))) >>> + >>> + (with-libgit2 >>> + (with-temporary-directory >>> + (lambda (cache-directory) >>> + (let* ((repository (repository-init cache-directory)) >>> + ;; Create an in-memory remote so we don't touch disk. >>> + (remote (remote-create-anonymous repository url))) >>> + (remote-connect remote) >>> + (remote-disconnect remote) >>> + (repository-close! repository) >>> + >>> + (filter include? (map remote-head-name (remote-ls remote))))))= )) >>> >> >> For some reason it seems to include refs that do and don=E2=80=99t end w= ith >> =E2=80=9C^{}=E2=80=9D > > Sorry, another typo I missed. See above. No worries, thanks for taking the time to review this! --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmE2B5IVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5BgIQAIZHtTauloSjYVC1T6cs8epK0r8B gUw3Dk3ZdBbjd9TfpfjsUnaGN2o6KwBKTnpj5atLPiM2Vo5an8BLmSOTlJQ9+O8p DBiKYY0kKdknJYhboSIKj/ejCRIkdLQy6Wj6ucWmYmxipqk+muBO0c5qoMsIYHFp iLUH0rrT7R0PMkBOSGrGgLi5Y4RRBlM9O4QqMp4RZ5Vw+brn6o3Z/+ja9/wlPSl9 md9W0g3D8d7eTmu/ATW71GDaI3UndGOUeCW4X3uTV0SqPUkyFuLKFFsqiQMxZHe1 iQ+uu1vj6shpxa+tXvpzqT3CdPDRzU8UyCtPNCUc8TaoLLj+SRlsdGv/xe1DSRYa lZhX1kxkUdJxpAjKuyU4i9N2b0Wftbmsfg4PATQJgP/XbY9RiSp6pXNY0uKNIhLF CBEoC5cq3MDjmLMswaipdlcVG4a5gxzhGI37aMBpE/of9hCDlY+rxth02m4nr4Rf 6SMeXcwYlcLNTU8FRw90d6aE+8NJxH4XR5VOW/CvZw6aArzYE3lwUvyYmFD78xfu 9LfehCH0DR3q98GCabyswVxpMrNdv8Z+wVNlpF+kDnCDzcD9vIhjKU6RTkzjp1UN zc5hQv4VjEqTs2hVSjf04X9L0s6gO2TRjpFJtquPq+GPqnozLJBuzIt+v0MRj/AH DF2EcH0wmD9yKpdI =l3fr -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon Sep 06 21:01:00 2021 Received: (at 50359) by debbugs.gnu.org; 7 Sep 2021 01:01:01 +0000 Received: from localhost ([127.0.0.1]:54822 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNPTw-0004XY-33 for submit@debbugs.gnu.org; Mon, 06 Sep 2021 21:01:00 -0400 Received: from out2.migadu.com ([188.165.223.204]:62202) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNPTs-0004XL-RF for 50359@debbugs.gnu.org; Mon, 06 Sep 2021 21:00:58 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1630976455; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=g9fElk3LvhkfJh78D55tZ1OXS453jmAETo9WcoQbHpk=; b=ktJw6RDJpOrKXdxxSvG+IYg6UFk76jZUmg2MEy1uc6S/6qqaZb/GRz0q6iA8IVlL/QyqsS nRDsxV5rBwjfHyFKordYG21MVSKF4TS56QrlSToI2xfLcKbTcZuffIIcyX5mwcjnMIzyiW xtjygFP2c0riO009Hvzg6Cj4hNHkglo= From: Sarah Morgensen To: Xinglu Chen Subject: Re: [bug#50359] [PATCH] import: Add 'generic-git' updater. References: <86k0jvkh5v.fsf@mgsn.dev> <87h7ez48d3.fsf@yoctocell.xyz> <86y28ai7ns.fsf@mgsn.dev> <87y28928vh.fsf@yoctocell.xyz> Date: Mon, 06 Sep 2021 18:00:52 -0700 In-Reply-To: <87y28928vh.fsf@yoctocell.xyz> (Xinglu Chen's message of "Mon, 06 Sep 2021 14:20:34 +0200 (5 hours, 48 minutes ago)") Message-ID: <86pmtli4hn.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: iskarian@mgsn.dev X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi, Xinglu Chen writes: > Any trick you used to find all of there weird version numbers? :-) Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [188.165.223.204 listed in wl.mailspike.net] X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, Xinglu Chen writes: > Any trick you used to find all of there weird version numbers? :-) This monstrosity: rg -U -B4 --pcre2 '(?!\(let.*(\n.*){0,1})\(version "([^\n"]*[^0-9\.][^"\n= ]*)".*(\n.*){0,10}commit.*version' gnu/packages and to show just the versions: rg -Uor '$2' --pcre2 --no-filename --no-line-number=20 >> IMO, just get rid of the delimiter. If we wanted to be *that* flexible, >> we could make it so they provide a tag->version proc instead of (prefix, >> suffix, delimiter). > > a =E2=80=98tag->version=E2=80=99 procedure would probably make things a b= it too > complicated for the people writing package definitions. For example, > having a delimiter would make it easy to match a tag like > =E2=80=9C2021-01-01-release=E2=80=9D > > Delimiter is =E2=80=9C.=E2=80=9D (sorry if this hurts your eyes ;-)) > > scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^\\.[:punc= t:]]+(\\.[^\\.[:punct:]]+)*).*$" "2021-01-01-release") 1) > $28 =3D "2021" > > Delimiter is =E2=80=9C-=E2=80=9D > > scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^-[:punct:= ]]+(-[^-[:punct:]]+)*).*$" "2021-01-01-release") 1) > $29 =3D "2021-01-01-release" > > And then, setting the suffix to =E2=80=9C-release=E2=80=9D would match ju= st the version > part. Right. I missed that. In that vein, should we keep the dashes in "2021-01-01" or convert them to periods? What about when a tag has underscores? What if a repo has tags in both formats? Then "3.0.1" would be considered older than "2011-01-01". Maybe we should just add an extra bit to detect a date format and only consider it when there's no "proper versions"? Aaaand I fell down a rabbit hole after that :) I've attached a patch with what I've done. It still has lots of issues--it requires the tag to contain at least one version delimiter, it requires the first character of the version to be a number... it might not even be better than before I touched it, and even so the added complexity might not be worth it. But if you'd like to take it for a spin, I've attached it (it applies straight on master). -- Sarah --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline; filename=0001-import-Add-generic-git-updater.patch Content-Transfer-Encoding: quoted-printable Content-Description: the rabbit hole >From 08bd59a7fa1aa9735a1794672ce8d1f683d3d6db Mon Sep 17 00:00:00 2001 Message-Id: <08bd59a7fa1aa9735a1794672ce8d1f683d3d6db.1630975873.git.iskari= an@mgsn.dev> From: Xinglu Chen Date: Fri, 3 Sep 2021 17:50:56 +0200 Subject: [PATCH] import: Add 'generic-git' updater. * guix/import/git.scm: New file. * doc/guix.texi (Invoking guix refresh): Document it. * Makefile.am (MODULES): Register it. --- Makefile.am | 1 + doc/guix.texi | 27 +++++++ guix/git.scm | 33 ++++++++ guix/import/git.scm | 191 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 252 insertions(+) create mode 100644 guix/import/git.scm diff --git a/Makefile.am b/Makefile.am index 3c79760734..c4d3a456b1 100644 --- a/Makefile.am +++ b/Makefile.am @@ -254,6 +254,7 @@ MODULES =3D \ guix/import/egg.scm \ guix/import/elpa.scm \ guix/import/gem.scm \ + guix/import/git.scm \ guix/import/github.scm \ guix/import/gnome.scm \ guix/import/gnu.scm \ diff --git a/doc/guix.texi b/doc/guix.texi index 36a0c7f5ec..26afb1607a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -11920,6 +11920,33 @@ the updater for @uref{https://launchpad.net, Launc= hpad} packages. @item generic-html a generic updater that crawls the HTML page where the source tarball of the package is hosted, when applicable. +@item generic-git +a generic updater for packages hosted on Git repositories. It tries to +be smart about parsing Git tag names, but if it is not able to parse the +tag name and compare tags correctly, users can define the following +properties for a package. + +@itemize +@item @code{tag-prefix}: a regular expression for matching a prefix of +the tag name. + +@item @code{tag-suffix}: a regular expression for matching a suffix of +the tag name. + +@item @code{tag-version-delimiter}: a string used as the delimiter in +the tag name for separating the numbers of the version. +@end itemize + +@lisp +(package + (name "foo") + ;; ... + (properties + '((tag-prefix . "^release0-") + (tag-suffix . "[a-z]?$") + (tag-version-delimiter . ":")))) +@end lisp=20=20=20=20=20=20 + @end table =20 For instance, the following command only checks for updates of Emacs diff --git a/guix/git.scm b/guix/git.scm index 9c6f326c36..c5d0d2da8e 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -56,6 +56,8 @@ commit-difference commit-relation =20 + ls-remote-refs + git-checkout git-checkout? git-checkout-url @@ -556,6 +558,37 @@ objects: 'ancestor (meaning that OLD is an ancestor of= NEW), 'descendant, or (if (set-contains? oldest new) 'descendant 'unrelated)))))) + +;; +;;; Remote operations. +;;; + +(define* (ls-remote-refs url #:key tags?) + "Return the list of references advertised at Git repository URL. If TAG= S? +is true, limit to only refs/tags." + (define (ref? ref) + ;; Like `git ls-remote --refs', only show actual references. + (and (string-prefix? "refs/" ref) + (not (string-suffix? "^{}" ref)))) + + (define (tag? ref) + (string-prefix? "refs/tags/" ref)) + + (define (include? ref) + (and (ref? ref) + (or (not tags?) (tag? ref)))) + + (with-libgit2 + (call-with-temporary-directory + (lambda (cache-directory) + (let* ((repository (repository-init cache-directory)) + ;; Create an in-memory remote so we don't touch disk. + (remote (remote-create-anonymous repository url))) + (remote-connect remote) + (remote-disconnect remote) + (repository-close! repository) + + (filter include? (map remote-head-name (remote-ls remote)))))))) =20 ;;; diff --git a/guix/import/git.scm b/guix/import/git.scm new file mode 100644 index 0000000000..8568981af2 --- /dev/null +++ b/guix/import/git.scm @@ -0,0 +1,191 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix import git) + #:use-module (guix build utils) + #:use-module (guix diagnostics) + #:use-module (guix git) + #:use-module (guix git-download) + #:use-module (guix i18n) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module (ice-9 format) + #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 regex) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) + #:use-module (srfi srfi-34) + #:use-module (srfi srfi-35) + #:use-module (srfi srfi-71) + #:export (%generic-git-updater)) + +;;; Commentary: +;;; +;;; This module provides a generic package updater for packages hosted on = Git +;;; repositories. +;;; +;;; It tries to be smart about tag names, but if it is not automatically a= ble +;;; to parse the tag names correctly, users can set the `tag-prefix', +;;; `tag-suffix' and `tag-version-delimiter' properties of the package to = make +;;; the updater parse the Git tag name correctly. +;;; +;;; Code: + +;;; Errors & warnings + +(define-condition-type &git-no-valid-tags-error &error + git-no-valid-tags-error?) + +(define (git-no-valid-tags-error) + (raise (condition (&message (message "no valid tags found")) + (&git-no-valid-tags-error)))) + +(define-condition-type &git-no-tags-error &error + git-no-tags-error?) + +(define (git-no-tags-error) + (raise (condition (&message (message "no tags were found")) + (&git-no-tags-error)))) + + +;;; Updater + +(define* (get-version-mapping tags #:key prefix suffix delim) + (define (guess-delim) + (let ((total (length tags)) + (dots (reduce + 0 (map (cut string-count <> #\.) tags))) + (dashes (reduce + 0 (map (cut string-count <> #\-) tags))) + (underscores (reduce + 0 (map (cut string-count <> #\_) tags)))) + ;; (format #t "total: ~d, dots: ~d, dashes ~d, underscores ~d~%" + ;; total dots dashes underscores) + (cond + ((> dots (* total 0.35)) ".") + ((> dashes (* total 0.8)) "-") + ((> underscores (* total 0.8)) "_") + (else ".")))) + + (define delim-rx (regexp-quote (or delim (guess-delim)))) + (define suffix-rx (or suffix "")) + + (define prefix-rx + (make-regexp (string-append "^" (or prefix ".*") "$"))) + (define tag-rx + (make-regexp + (string-append "([[:digit:]][^" delim-rx "[:punct:]]*" + "(" delim-rx "[^" delim-rx "]+)+" + ")" suffix-rx "$"))) + + (define (get-version tag) + (let ((tag-match (regexp-exec tag-rx tag))) + (and tag-match + (regexp-exec prefix-rx (match:prefix tag-match)) + (regexp-substitute/global + #f delim-rx (match:substring tag-match 1) + 'pre "." 'post)))) + + (define (entry) + '("alpha" "beta" "rc" "dev" "test"))) + + (let* ((tags (map (cut string-drop <> (string-length "refs/tags/")) + (ls-remote-refs url #:tags? #t))) + (versions->tags + (get-version-mapping (filter (negate pre-release?) tags) + #:prefix prefix + #:suffix suffix + #:delim delim))) + (cond + ((null? tags) + (git-no-tags-error)) + ((null? versions->tags) + (git-no-valid-tags-error)) + (else + (match (last versions->tags) + ((version . tag) + (values version tag))))))) + +(define (latest-git-tag-version package tag-prefix tag-suffix + tag-version-delimiter) + "Given a PACKAGE, the TAG-PREFIX, TAG-SUFFIX, and TAG-VERSION-DELIMITER +properties of PACKAGE, returns the latest version of PACKAGE." + (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c)) + (warning (or (package-field-location package 'source) + (package-location package)) + (G_ "~a for ~a~%") + (condition-message c) + (package-name package)) + #f) + ((eq? (exception-kind c) 'git-error) + (warning (or (package-field-location package 'source) + (package-location package)) + (G_ "failed to fetch Git repository for ~a~%") + (package-name package)) + #f)) + (let* ((source (package-source package)) + (url (git-reference-url (origin-uri source)))) + ;;(format #t "~a~%" (package-name package)) + (get-latest-tag url #:prefix tag-prefix #:suffix tag-suffix + #:delim tag-version-delimiter)))) + +(define (git-package? package) + "Whether the origin of PACKAGE is a Git repostiory." + (match (package-source package) + ((? origin? origin) + (and (eq? (origin-method origin) git-fetch) + (git-reference? (origin-uri origin)))) + (_ #f))) + +(define (latest-git-release package) + "Return the latest release of PACKAGE." + (let* ((name (package-name package)) + (properties (package-properties package)) + (tag-prefix (assq-ref properties 'tag-prefix)) + (tag-suffix (assq-ref properties 'tag-suffix)) + (tag-version-delimiter (assq-ref properties 'tag-version-delimite= r)) + (old-version (package-version package)) + (url (git-reference-url (origin-uri (package-source package)))) + (new-version (latest-git-tag-version package + tag-prefix + tag-suffix + tag-version-delimiter))) + + (if new-version + (upstream-source + (package name) + (version new-version) + (urls (list url))) + ;; No new release or no tags available. + #f))) + +(define %generic-git-updater + (upstream-updater + (name 'generic-git) + (description "Updater for packages hosted on Git repositories") + (pred git-package?) + (latest latest-git-release))) base-commit: 522a3bf99cbc21a9093f63280b9508cd69b94ff0 --=20 2.32.0 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Tue Sep 07 15:13:38 2021 Received: (at 50359) by debbugs.gnu.org; 7 Sep 2021 19:13:38 +0000 Received: from localhost ([127.0.0.1]:57771 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNgXJ-0005LB-6p for submit@debbugs.gnu.org; Tue, 07 Sep 2021 15:13:38 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:51562 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mNgXG-0005Kt-4E for 50359@debbugs.gnu.org; Tue, 07 Sep 2021 15:13:35 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631042005; bh=DLb7xkBOXXKVJSPIRcO86Zhb4/KB0Iu8cDZSok1Pkhk=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=i9smTY8/QkOYWKejtO5vC8Fy24djHjLWQYhEKwznR3WHQZ3AwM/8KLYNMJErr2fr+ Ky+4vIx/iRGF9oInQj9zfPgyH8LPq/DqUsIgPqYvEAuLgyiv+Lo+zRVmvSGy+bEVYB 1RzSz3b7JCBn5Mf5yVAZtmUXXEXm9m+8k7qovVW8= To: Sarah Morgensen Subject: Re: [bug#50359] [PATCH] import: Add 'generic-git' updater. In-Reply-To: <86pmtli4hn.fsf@mgsn.dev> References: <86k0jvkh5v.fsf@mgsn.dev> <87h7ez48d3.fsf@yoctocell.xyz> <86y28ai7ns.fsf@mgsn.dev> <87y28928vh.fsf@yoctocell.xyz> <86pmtli4hn.fsf@mgsn.dev> Date: Tue, 07 Sep 2021 21:13:22 +0200 Message-ID: <87wnnsyzal.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Mon, Sep 06 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >> Any trick you used to find all of there weird version numbers? :-) > > This monstrosity: > > rg -U -B4 --pcre2 '(?!\(let.*(\n.*){0,1})\(versio [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Mon, Sep 06 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >> Any trick you used to find all of there weird version numbers? :-) > > This monstrosity: > > rg -U -B4 --pcre2 '(?!\(let.*(\n.*){0,1})\(versio [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Mon, Sep 06 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >> Any trick you used to find all of there weird version numbers? :-) > > This monstrosity: > > rg -U -B4 --pcre2 '(?!\(let.*(\n.*){0,1})\(version "([^\n"]*[^0-9\.][^"= \n]*)".*(\n.*){0,10}commit.*version' gnu/packages > > and to show just the versions: > > rg -Uor '$2' --pcre2 --no-filename --no-line-number=20 Wow! I will try that and see for myself! :-) >>> IMO, just get rid of the delimiter. If we wanted to be *that* flexible, >>> we could make it so they provide a tag->version proc instead of (prefix, >>> suffix, delimiter). >> >> a =E2=80=98tag->version=E2=80=99 procedure would probably make things a = bit too >> complicated for the people writing package definitions. For example, >> having a delimiter would make it easy to match a tag like >> =E2=80=9C2021-01-01-release=E2=80=9D >> >> Delimiter is =E2=80=9C.=E2=80=9D (sorry if this hurts your eyes ;-)) >> >> scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^\\.[:pun= ct:]]+(\\.[^\\.[:punct:]]+)*).*$" "2021-01-01-release") 1) >> $28 =3D "2021" >> >> Delimiter is =E2=80=9C-=E2=80=9D >> >> scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^-[:punct= :]]+(-[^-[:punct:]]+)*).*$" "2021-01-01-release") 1) >> $29 =3D "2021-01-01-release" >> >> And then, setting the suffix to =E2=80=9C-release=E2=80=9D would match j= ust the version >> part. > > Right. I missed that. > > In that vein, should we keep the dashes in "2021-01-01" or convert them > to periods? Having periods would be more consistent, then could have a =E2=80=98date->version=E2=80=99 procedure that replaces the hyphens with do= ts and have (git-reference (url "https://git.example.org") (commit (date->version version))) > What about when a tag has underscores? Hmm, not sure about that, below is a list of packages I could find which had underscores as delimiters gnu/packages/graphics.scm 239: (commit "DIRECTFB_1_7_7"))) gnu/packages/gstreamer.scm 326: (commit "ESOUND_0_2_41"))) gnu/packages/java.scm 13925: (commit "jboss-transaction-api_1.2_spec-1.1.1.F= inal"))) They all seem to use periods in the =E2=80=98version=E2=80=99 field, though= , so I would say that the underscroes, should also be converted to periods. > What if a repo has tags in both formats? Then "3.0.1" would be > considered older than "2011-01-01". That=E2=80=99s tricky, there isn=E2=80=99t really a way to know how old =E2= =80=9C3.0.1=E2=80=9D is, without looking at the metadata of the tag. Maybe this is one of those corner cases which can=E2=80=99t really automatically determine the latest release. Should we have a =E2=80=98no-refresh?=E2=80=99 property to tell t= he refresh to not try to update the package? > Maybe we should just add an extra bit to detect a date format and only > consider it when there's no "proper versions"? That could be a good idea! > Aaaand I fell down a rabbit hole after that :) I've attached a patch > with what I've done. It still has lots of issues--it requires the tag > to contain at least one version delimiter, it requires the first > character of the version to be a number... it might not even be better > than before I touched it, and even so the added complexity might not be > worth it. But if you'd like to take it for a spin, I've attached it (it > applies straight on master). Great! I will try it out and see how it compares to my current WIP version. Not having characters in the first version number probably isn=E2=80=99t su= ch a big deal, most version that contain characters end with a character. E.g., =E2=80=9C1.2.3a=E2=80=9D is not to uncommon, but =E2=80=9Ca1.2.3=E2= =80=9D is rarely seen. > -- > Sarah > > From 08bd59a7fa1aa9735a1794672ce8d1f683d3d6db Mon Sep 17 00:00:00 2001 > Message-Id: <08bd59a7fa1aa9735a1794672ce8d1f683d3d6db.1630975873.git.iska= rian@mgsn.dev> > From: Xinglu Chen > Date: Fri, 3 Sep 2021 17:50:56 +0200 > Subject: [PATCH] import: Add 'generic-git' updater. > > * guix/import/git.scm: New file. > * doc/guix.texi (Invoking guix refresh): Document it. > * Makefile.am (MODULES): Register it. > --- > Makefile.am | 1 + > doc/guix.texi | 27 +++++++ > guix/git.scm | 33 ++++++++ > guix/import/git.scm | 191 ++++++++++++++++++++++++++++++++++++++++++++ > 4 files changed, 252 insertions(+) > create mode 100644 guix/import/git.scm > > diff --git a/Makefile.am b/Makefile.am > index 3c79760734..c4d3a456b1 100644 > --- a/Makefile.am > +++ b/Makefile.am > @@ -254,6 +254,7 @@ MODULES =3D \ > guix/import/egg.scm \ > guix/import/elpa.scm \ > guix/import/gem.scm \ > + guix/import/git.scm \ > guix/import/github.scm \ > guix/import/gnome.scm \ > guix/import/gnu.scm \ > diff --git a/doc/guix.texi b/doc/guix.texi > index 36a0c7f5ec..26afb1607a 100644 > --- a/doc/guix.texi > +++ b/doc/guix.texi > @@ -11920,6 +11920,33 @@ the updater for @uref{https://launchpad.net, Lau= nchpad} packages. > @item generic-html > a generic updater that crawls the HTML page where the source tarball of > the package is hosted, when applicable. > +@item generic-git > +a generic updater for packages hosted on Git repositories. It tries to > +be smart about parsing Git tag names, but if it is not able to parse the > +tag name and compare tags correctly, users can define the following > +properties for a package. > + > +@itemize > +@item @code{tag-prefix}: a regular expression for matching a prefix of > +the tag name. > + > +@item @code{tag-suffix}: a regular expression for matching a suffix of > +the tag name. > + > +@item @code{tag-version-delimiter}: a string used as the delimiter in > +the tag name for separating the numbers of the version. > +@end itemize > + > +@lisp > +(package > + (name "foo") > + ;; ... > + (properties > + '((tag-prefix . "^release0-") > + (tag-suffix . "[a-z]?$") > + (tag-version-delimiter . ":")))) > +@end lisp=20=20=20=20=20=20 > + > @end table >=20=20 > For instance, the following command only checks for updates of Emacs > diff --git a/guix/git.scm b/guix/git.scm > index 9c6f326c36..c5d0d2da8e 100644 > --- a/guix/git.scm > +++ b/guix/git.scm > @@ -56,6 +56,8 @@ > commit-difference > commit-relation >=20=20 > + ls-remote-refs > + > git-checkout > git-checkout? > git-checkout-url > @@ -556,6 +558,37 @@ objects: 'ancestor (meaning that OLD is an ancestor = of NEW), 'descendant, or > (if (set-contains? oldest new) > 'descendant > 'unrelated)))))) > + > +;; > +;;; Remote operations. > +;;; > + > +(define* (ls-remote-refs url #:key tags?) > + "Return the list of references advertised at Git repository URL. If T= AGS? > +is true, limit to only refs/tags." > + (define (ref? ref) > + ;; Like `git ls-remote --refs', only show actual references. > + (and (string-prefix? "refs/" ref) > + (not (string-suffix? "^{}" ref)))) > + > + (define (tag? ref) > + (string-prefix? "refs/tags/" ref)) > + > + (define (include? ref) > + (and (ref? ref) > + (or (not tags?) (tag? ref)))) > + > + (with-libgit2 > + (call-with-temporary-directory > + (lambda (cache-directory) > + (let* ((repository (repository-init cache-directory)) > + ;; Create an in-memory remote so we don't touch disk. > + (remote (remote-create-anonymous repository url))) > + (remote-connect remote) > + (remote-disconnect remote) > + (repository-close! repository) > + > + (filter include? (map remote-head-name (remote-ls remote)))))))) >=20=20 > > ;;; > diff --git a/guix/import/git.scm b/guix/import/git.scm > new file mode 100644 > index 0000000000..8568981af2 > --- /dev/null > +++ b/guix/import/git.scm > @@ -0,0 +1,191 @@ > +;;; GNU Guix --- Functional package management for GNU > +;;; Copyright =C2=A9 2021 Xinglu Chen > +;;; Copyright =C2=A9 2021 Sarah Morgensen > +;;; > +;;; This file is part of GNU Guix. > +;;; > +;;; GNU Guix is free software; you can redistribute it and/or modify it > +;;; under the terms of the GNU General Public License as published by > +;;; the Free Software Foundation; either version 3 of the License, or (at > +;;; your option) any later version. > +;;; > +;;; GNU Guix is distributed in the hope that it will be useful, but > +;;; WITHOUT ANY WARRANTY; without even the implied warranty of > +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +;;; GNU General Public License for more details. > +;;; > +;;; You should have received a copy of the GNU General Public License > +;;; along with GNU Guix. If not, see . > + > +(define-module (guix import git) > + #:use-module (guix build utils) > + #:use-module (guix diagnostics) > + #:use-module (guix git) > + #:use-module (guix git-download) > + #:use-module (guix i18n) > + #:use-module (guix packages) > + #:use-module (guix upstream) > + #:use-module (guix utils) > + #:use-module (ice-9 format) > + #:use-module (ice-9 match) > + #:use-module (ice-9 rdelim) > + #:use-module (ice-9 regex) > + #:use-module (srfi srfi-1) > + #:use-module (srfi srfi-26) > + #:use-module (srfi srfi-34) > + #:use-module (srfi srfi-35) > + #:use-module (srfi srfi-71) > + #:export (%generic-git-updater)) > + > +;;; Commentary: > +;;; > +;;; This module provides a generic package updater for packages hosted o= n Git > +;;; repositories. > +;;; > +;;; It tries to be smart about tag names, but if it is not automatically= able > +;;; to parse the tag names correctly, users can set the `tag-prefix', > +;;; `tag-suffix' and `tag-version-delimiter' properties of the package t= o make > +;;; the updater parse the Git tag name correctly. > +;;; > +;;; Code: > + > +;;; Errors & warnings > + > +(define-condition-type &git-no-valid-tags-error &error > + git-no-valid-tags-error?) > + > +(define (git-no-valid-tags-error) > + (raise (condition (&message (message "no valid tags found")) > + (&git-no-valid-tags-error)))) > + > +(define-condition-type &git-no-tags-error &error > + git-no-tags-error?) > + > +(define (git-no-tags-error) > + (raise (condition (&message (message "no tags were found")) > + (&git-no-tags-error)))) > + > + > +;;; Updater > + > +(define* (get-version-mapping tags #:key prefix suffix delim) > + (define (guess-delim) > + (let ((total (length tags)) > + (dots (reduce + 0 (map (cut string-count <> #\.) tags))) > + (dashes (reduce + 0 (map (cut string-count <> #\-) tags))) > + (underscores (reduce + 0 (map (cut string-count <> #\_) tags))= )) > + ;; (format #t "total: ~d, dots: ~d, dashes ~d, underscores ~d~%" > + ;; total dots dashes underscores) > + (cond > + ((> dots (* total 0.35)) ".") > + ((> dashes (* total 0.8)) "-") > + ((> underscores (* total 0.8)) "_") > + (else ".")))) These numbers seem rather arbitrary, how did you arrive at them? Also, AFAICS, versions without delimiters won=E2=80=99t be matched. > + (define delim-rx (regexp-quote (or delim (guess-delim)))) > + (define suffix-rx (or suffix "")) > + > + (define prefix-rx > + (make-regexp (string-append "^" (or prefix ".*") "$"))) Why is there a $ here? > + (define tag-rx > + (make-regexp > + (string-append "([[:digit:]][^" delim-rx "[:punct:]]*" > + "(" delim-rx "[^" delim-rx "]+)+" > + ")" suffix-rx "$"))) > + > + (define (get-version tag) > + (let ((tag-match (regexp-exec tag-rx tag))) > + (and tag-match > + (regexp-exec prefix-rx (match:prefix tag-match)) > + (regexp-substitute/global > + #f delim-rx (match:substring tag-match 1) > + 'pre "." 'post)))) > + > + (define (entry + (eq? (version-compare (car a) (car b)) '<)) > + > + (let ((mapping (fold alist-cons '() (map get-version tags) tags))) > + (stable-sort! (filter car mapping) entry + > +(define* (get-latest-tag url #:key prefix suffix delim) > + "Return the latest tag available from the Git repository at URL." > + (define (pre-release? tag) > + (any (cut string-contains tag <>) > + '("alpha" "beta" "rc" "dev" "test"))) This would only match lower-case characters, but sometimes upper-case characters are used. (define (pre-release? tag) (any (lambda (rx) (regexp-exec (make-regexp rx regexp/icase) tag)) '("alpha" "beta" "rc" "dev" "test"))) =20=20=20=20=20=20=20=20=20=20=20=20=20=20 > + (let* ((tags (map (cut string-drop <> (string-length "refs/tags/")) > + (ls-remote-refs url #:tags? #t))) > + (versions->tags > + (get-version-mapping (filter (negate pre-release?) tags) > + #:prefix prefix > + #:suffix suffix > + #:delim delim))) > + (cond > + ((null? tags) > + (git-no-tags-error)) > + ((null? versions->tags) > + (git-no-valid-tags-error)) > + (else > + (match (last versions->tags) > + ((version . tag) > + (values version tag))))))) > + > +(define (latest-git-tag-version package tag-prefix tag-suffix > + tag-version-delimiter) > + "Given a PACKAGE, the TAG-PREFIX, TAG-SUFFIX, and TAG-VERSION-DELIMITER > +properties of PACKAGE, returns the latest version of PACKAGE." > + (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c)) > + (warning (or (package-field-location package 'source) > + (package-location package)) > + (G_ "~a for ~a~%") > + (condition-message c) > + (package-name package)) > + #f) > + ((eq? (exception-kind c) 'git-error) > + (warning (or (package-field-location package 'source) > + (package-location package)) > + (G_ "failed to fetch Git repository for ~a~%") > + (package-name package)) > + #f)) > + (let* ((source (package-source package)) > + (url (git-reference-url (origin-uri source)))) > + ;;(format #t "~a~%" (package-name package)) > + (get-latest-tag url #:prefix tag-prefix #:suffix tag-suffix > + #:delim tag-version-delimiter)))) > + > +(define (git-package? package) > + "Whether the origin of PACKAGE is a Git repostiory." > + (match (package-source package) > + ((? origin? origin) > + (and (eq? (origin-method origin) git-fetch) > + (git-reference? (origin-uri origin)))) > + (_ #f))) > + > +(define (latest-git-release package) > + "Return the latest release of PACKAGE." > + (let* ((name (package-name package)) > + (properties (package-properties package)) > + (tag-prefix (assq-ref properties 'tag-prefix)) > + (tag-suffix (assq-ref properties 'tag-suffix)) > + (tag-version-delimiter (assq-ref properties 'tag-version-delimi= ter)) > + (old-version (package-version package)) > + (url (git-reference-url (origin-uri (package-source package)))) > + (new-version (latest-git-tag-version package > + tag-prefix > + tag-suffix > + tag-version-delimiter))) > + > + (if new-version > + (upstream-source > + (package name) > + (version new-version) > + (urls (list url))) > + ;; No new release or no tags available. > + #f))) > + > +(define %generic-git-updater > + (upstream-updater > + (name 'generic-git) > + (description "Updater for packages hosted on Git repositories") > + (pred git-package?) > + (latest latest-git-release))) > > base-commit: 522a3bf99cbc21a9093f63280b9508cd69b94ff0 > --=20 > 2.32.0 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmE3udIVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x59rgP/i9zonrR51YoZ4Dw68xfhW66m2op K7Xf9B45E+pAYKxo2R31HjO2B9CNYUXb5LBLAMvkqaH0KmP7Yv3LpBUpOm6EOmMh gtgTJTBlLetvjxrQmcOHBd6n/vIFp74wlukV17vPrxtlrQVCM50IBYvPcVuq8GtU eXQZhL3HYNc8K5ddKSRGwHNogZg3casFP3O2/x9J3wImFhIqA57IHan4n+mNAOUk ddUMhiypYNTM74gYQZ4tqspJNyWD7l7YNF2CA+mpsn+Vadz+tCALffKZTO6f1mQT ibuOV75ZbtuAcEkR+43o3dtWFamaQOuUZWLMBXFYU8iy91Suyqh3I1QazoKE0gXU 3hJO2x0+TdQV/bI8U/Hoj7nimXDbicJovFH9KCzOp2fPhcFxLCLwIMjc++BlMLx8 0TfkIZfHti962MlnYFgo0I0N/F0SiI6fFQBJ7L7i0nm3/el1wB23IIbjrNCQ9841 fLNw/Vz8QsO1xBNEFhrLhoFerzHLqmy+TggKVCzdlnlwPlrK5Y9cZZ1FIJq5LKli sirHcyfi22YL7B19sTRhqZYBY1Gttucykn1ttuRZg56JhJvPoDunoluZyTtYXUQt QkSpbjD/9VkyODCKgAjvm+3+e68uNlVePPCN5W/AroVPxw8oMjCeLakUTrKavM6w vl+l3fGzkXs14GR0 =d2MH -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed Sep 08 14:29:06 2021 Received: (at 50359) by debbugs.gnu.org; 8 Sep 2021 18:29:06 +0000 Received: from localhost ([127.0.0.1]:33546 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mO2Jl-0005oW-RI for submit@debbugs.gnu.org; Wed, 08 Sep 2021 14:29:06 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:55900 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mO2Ji-0005ny-Su for 50359@debbugs.gnu.org; Wed, 08 Sep 2021 14:29:04 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631125733; bh=2HyG349cNWVnzUPqUcYxl3f8KOxWIJNaD9SmLCyD2nY=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=jIpMh18dY+n2zCAWKPgSQqXNpv8mVairlZv2VcMCXstWNpQ6LxAU5GIHsQI7zXH1v y4btlj+tlIa4DEW4xwVl+0bCpXVwuAUSGzfcp/g25KCzaaremz4tSWwLIrbARhDjul iKdQ5COm1M6x2xHdiiDF7rTSoQ5DPVUF5HDehpL0= To: Sarah Morgensen Subject: Re: [bug#50359] [PATCH] import: Add 'generic-git' updater. In-Reply-To: <87wnnsyzal.fsf@yoctocell.xyz> References: <86k0jvkh5v.fsf@mgsn.dev> <87h7ez48d3.fsf@yoctocell.xyz> <86y28ai7ns.fsf@mgsn.dev> <87y28928vh.fsf@yoctocell.xyz> <86pmtli4hn.fsf@mgsn.dev> <87wnnsyzal.fsf@yoctocell.xyz> Date: Wed, 08 Sep 2021 20:28:38 +0200 Message-ID: <87mtomzzu1.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="==-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Tue, Sep 07 2021, Xinglu Chen wrote: > On Mon, Sep 06 2021, Sarah Morgensen wrote: > >> Hi, >> >> Xinglu Chen writes: >> >>> Any trick you used to find all of there weird version numbers? :-) >> >> This monstrosity [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Tue, Sep 07 2021, Xinglu Chen wrote: > On Mon, Sep 06 2021, Sarah Morgensen wrote: > >> Hi, >> >> Xinglu Chen writes: >> >>> Any trick you used to find all of there weird version numbers? :-) >> >> This monstrosity [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --==-=-= Content-Type: multipart/mixed; boundary="=-=-=" --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Tue, Sep 07 2021, Xinglu Chen wrote: > On Mon, Sep 06 2021, Sarah Morgensen wrote: > >> Hi, >> >> Xinglu Chen writes: >> >>> Any trick you used to find all of there weird version numbers? :-) >> >> This monstrosity: >> >> rg -U -B4 --pcre2 '(?!\(let.*(\n.*){0,1})\(version "([^\n"]*[^0-9\.][^= "\n]*)".*(\n.*){0,10}commit.*version' gnu/packages >> >> and to show just the versions: >> >> rg -Uor '$2' --pcre2 --no-filename --no-line-number=20 > > Wow! I will try that and see for myself! :-) > >>>> IMO, just get rid of the delimiter. If we wanted to be *that* flexibl= e, >>>> we could make it so they provide a tag->version proc instead of (prefi= x, >>>> suffix, delimiter). >>> >>> a =E2=80=98tag->version=E2=80=99 procedure would probably make things a= bit too >>> complicated for the people writing package definitions. For example, >>> having a delimiter would make it easy to match a tag like >>> =E2=80=9C2021-01-01-release=E2=80=9D >>> >>> Delimiter is =E2=80=9C.=E2=80=9D (sorry if this hurts your eyes ;-)) >>> >>> scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^\\.[:pu= nct:]]+(\\.[^\\.[:punct:]]+)*).*$" "2021-01-01-release") 1) >>> $28 =3D "2021" >>> >>> Delimiter is =E2=80=9C-=E2=80=9D >>> >>> scheme@(guile-user)> (match:substring (string-match "^[^0-9]*([^-[:punc= t:]]+(-[^-[:punct:]]+)*).*$" "2021-01-01-release") 1) >>> $29 =3D "2021-01-01-release" >>> >>> And then, setting the suffix to =E2=80=9C-release=E2=80=9D would match = just the version >>> part. >> >> Right. I missed that. >> >> In that vein, should we keep the dashes in "2021-01-01" or convert them >> to periods? > > Having periods would be more consistent, then could have a > =E2=80=98date->version=E2=80=99 procedure that replaces the hyphens with = dots and have > > (git-reference > (url "https://git.example.org") > (commit (date->version version))) > >> What about when a tag has underscores? > > Hmm, not sure about that, below is a list of packages I could find which = had > underscores as delimiters > > gnu/packages/graphics.scm > 239: (commit "DIRECTFB_1_7_7"))) > gnu/packages/gstreamer.scm > 326: (commit "ESOUND_0_2_41"))) > gnu/packages/java.scm > 13925: (commit "jboss-transaction-api_1.2_spec-1.1.1= .Final"))) > > They all seem to use periods in the =E2=80=98version=E2=80=99 field, thou= gh, so I would > say that the underscroes, should also be converted to periods. > >> What if a repo has tags in both formats? Then "3.0.1" would be >> considered older than "2011-01-01". > > That=E2=80=99s tricky, there isn=E2=80=99t really a way to know how old = =E2=80=9C3.0.1=E2=80=9D is, > without looking at the metadata of the tag. Maybe this is one of those > corner cases which can=E2=80=99t really automatically determine the latest > release. Should we have a =E2=80=98no-refresh?=E2=80=99 property to tell= the refresh to > not try to update the package? > >> Maybe we should just add an extra bit to detect a date format and only >> consider it when there's no "proper versions"? > > That could be a good idea! > >> Aaaand I fell down a rabbit hole after that :) I've attached a patch >> with what I've done. It still has lots of issues--it requires the tag >> to contain at least one version delimiter, it requires the first >> character of the version to be a number... it might not even be better >> than before I touched it, and even so the added complexity might not be >> worth it. But if you'd like to take it for a spin, I've attached it (it >> applies straight on master). > > Great! I will try it out and see how it compares to my current WIP > version. > > Not having characters in the first version number probably isn=E2=80=99t = such a > big deal, most version that contain characters end with a character. > E.g., =E2=80=9C1.2.3a=E2=80=9D is not to uncommon, but =E2=80=9Ca1.2.3=E2= =80=9D is rarely seen. I made some changes, to the patch and ran =E2=80=98guix refresh -t generic-= git | grep "^gnu/packages.*no valid=E2=80=99; I also modified the =E2=80=98github= =E2=80=99 updater so that it wouldn=E2=80=99t be used. =20=20 (define %github-updater (upstream-updater (name 'github) (description "Updater for GitHub packages") (pred (const #f)) ;this right here (latest latest-release))) Attached is a file with all the packages that didn=E2=80=99t have any valid tags, and with a short description of perharps why no valid tags were found. Something I noticed was the a lot of Julia package use a version scheme like this: (version "1.2.3+0") The "+0" is included in the version field and acts like a =E2=80=9Crevision= =E2=80=9D; I am not familiar with the Julia ecosystem, and I am not sure how we should handle this situation. The updated patch is also attached. --=-=-= Content-Type: application/octet-stream Content-Disposition: attachment; filename=no-valid-tags Content-Transfer-Encoding: base64 Ozs7IEN1c3RvbSBwcmVmaXggb3Igc3VmZml4IChzaG91bGQgYmUgc29sdmFibGUgYnkgdXMpCjs7 IGttc2Nvbi0gcHJlZml4CmdudS9wYWNrYWdlcy90ZXJtaW5hbHMuc2NtOjI4MToxNDogd2Fybmlu Zzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3Iga21zY29uCgo7OyB2MS4yLTxudW0+CmdudS9wYWNr YWdlcy9zdGF0aXN0aWNzLnNjbTo1NzAxOjU6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQg Zm9yIHItY29sb3JvdXQKCjs7IDxDSEFSPjxudW0+CmdudS9wYWNrYWdlcy9zZGwuc2NtOjY2Mjox NDogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3Igc2RsMi1jcwoKOzsgdGVzdC0gcHJl Zml4CmdudS9wYWNrYWdlcy9xdC5zY206MTE4Ojc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91 bmQgZm9yIHFpdGUKCjs7ICJ2IiBwcmVmaXgsICItc3RhYmxlIiBzdWZmaXgKZ251L3BhY2thZ2Vz L29wZW5jb2cuc2NtOjc2OjE0OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBhdG9t c3BhY2UKZ251L3BhY2thZ2VzL29wZW5jb2cuc2NtOjQ0OjE0OiB3YXJuaW5nOiBubyB2YWxpZCB0 YWdzIGZvdW5kIGZvciBjb2d1dGlsCgo7OyArPG51bT4gc3VmZml4CmdudS9wYWNrYWdlcy9vY2Ft bC5zY206NjAzOjEyOiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBvY2FtbC1tY2Nz Cgo7OyAidmVyc2lvbi0iIHByZWZpeApnbnUvcGFja2FnZXMvbWluZXRlc3Quc2NtOjYwMDo1OiB3 YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBtaW5ldGVzdC11bmlmaWVkLWludmVudG9y eQoKOzsgIm1hdmVuLXJlc291cmNlcy1wbHVnaW4tIiBwcmVmaXgKZ251L3BhY2thZ2VzL21hdmVu LnNjbToyOTgwOjEyOiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBtYXZlbi1yZXNv dXJjZXMtcGx1Z2luCgo7OyAidHhyLSIgcHJlZml4CmdudS9wYWNrYWdlcy9saXNwLnNjbTo5MTI6 NTogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgdHhyCgo7OyByZWxlYXNlLSBwcmVm aXgKZ251L3BhY2thZ2VzL2FsZ2VicmEuc2NtOjE0MjA6MTI6IHdhcm5pbmc6IG5vIHZhbGlkIHRh Z3MgZm91bmQgZm9yIG00cmkKZ251L3BhY2thZ2VzL2FsZ2VicmEuc2NtOjE1MTA6MTI6IHdhcm5p bmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIG00cmllCgo7OyBscmNhbGMtIHByZWZpeApnbnUv cGFja2FnZXMvYWxnZWJyYS5zY206MTU2NjoxMjogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3Vu ZCBmb3IgbHJjYWxjCgo7OyAzLjMuMDYtMSwgLTxudW0+IHN1ZmZpeApnbnUvcGFja2FnZXMvYWRt aW4uc2NtOjQwNDY6Mjogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgaW54aQpnbnUv cGFja2FnZXMvYWRtaW4uc2NtOjM5Nzc6Nzogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBm b3IgaW54aS1taW5pbWFsCgo7OyAtcmVsZWFzZS1EQVRFK1NPTUUtUkFORE9NLU5VTUJFUlMsIHJl ZnMvdGFncy8yLjAuMTgwLXJlbGVhc2UtMjAyMTAzMTUyMzE3MDcKZ251L3BhY2thZ2VzL2Jpb2lu Zm9ybWF0aWNzLnNjbToxMTE4OToxMjogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3Ig YmlvYmFtYmFtMgpnbnUvcGFja2FnZXMvYmlvaW5mb3JtYXRpY3Muc2NtOjExMTQ5OjEyOiB3YXJu aW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBsaWJtYXVzMgoKOzsgIitnaXQ8ZGF0ZT4iIHN1 ZmZpeApnbnUvcGFja2FnZXMvZ25vbWUuc2NtOjExODc4OjEyOiB3YXJuaW5nOiBubyB2YWxpZCB0 YWdzIGZvdW5kIGZvciBmZWVkYmFja2QKCjs7ICJmZXN0LXRlc3QtIiBwcmVmaXgKZ251L3BhY2th Z2VzL2phdmEuc2NtOjEwOTAwOjEyOiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBq YXZhLWZlc3QtdGVzdAoKOzsgIi1zdGFibGUiIHN1ZmZpeApnbnUvcGFja2FnZXMvZ2FtZS1kZXZl bG9wbWVudC5zY206MTcyMjoxMjogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgZ29k b3QKCjs7ICJ2IiBwcmVmaXgsICItZGViaWFuIiBzdWZmaXgKZ251L3BhY2thZ2VzL2dzdHJlYW1l ci5zY206OTU6NTogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3Igb3Blbm5pMgoKCjs7 ICs8bnVtPiBzdWZmaXgsIGJ1dCB0aGlzIGlzIHBhcnQgb2YgdGhlIGB2ZXJzaW9uJyBmaWVsZCBh cyB3ZWxsLCBub3QKOzsgc3VyZSB3aGF0IHRvIGRvIGFib3V0IHRoaXMKOzsKOzsgKHZlcnNpb24g IjEuNS4wKzAiKQpnbnUvcGFja2FnZXMvanVsaWEtamxsLnNjbTozNjY6Njogd2FybmluZzogbm8g dmFsaWQgdGFncyBmb3VuZCBmb3IganVsaWEtZnJpYmlkaS1qbGwKZ251L3BhY2thZ2VzL2p1bGlh LWpsbC5zY206MjUyMDo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxpYS14 b3JnLXh0cmFucy1qbGwKZ251L3BhY2thZ2VzL2p1bGlhLWpsbC5zY206MTgxMjo2OiB3YXJuaW5n OiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxpYS14b3JnLWxpYngxMS1qbGwKZ251L3BhY2th Z2VzL2p1bGlhLWpsbC5zY206MTE0OTo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZv ciBqdWxpYS1saWJ2b3JiaXMtamxsCmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjE0OTE6Njog d2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IganVsaWEtcXQ1YmFzZS1qbGwKZ251L3Bh Y2thZ2VzL2p1bGlhLWpsbC5zY206MTQxNTo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5k IGZvciBqdWxpYS1wY3JlLWpsbApnbnUvcGFja2FnZXMvanVsaWEtamxsLnNjbToxOTY0OjY6IHdh cm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGp1bGlhLXhvcmctbGlieGRtY3AtamxsCmdu dS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjE1ODE6Njogd2FybmluZzogbm8gdmFsaWQgdGFncyBm b3VuZCBmb3IganVsaWEtd2F5bGFuZC1wcm90b2NvbHMtamxsCmdudS9wYWNrYWdlcy9qdWxpYS1q bGwuc2NtOjY5NDo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxpYS1sYW1l LWpsbApnbnUvcGFja2FnZXMvanVsaWEtamxsLnNjbToyMjYxOjY6IHdhcm5pbmc6IG5vIHZhbGlk IHRhZ3MgZm91bmQgZm9yIGp1bGlhLXhvcmcteGNiLXV0aWwtamxsCmdudS9wYWNrYWdlcy9qdWxp YS1qbGwuc2NtOjE2MTg6Njogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IganVsaWEt eDI2NC1qbGwKZ251L3BhY2thZ2VzL2p1bGlhLWpsbC5zY206MTkyNjo2OiB3YXJuaW5nOiBubyB2 YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxpYS14b3JnLWxpYnhjdXJzb3ItamxsCmdudS9wYWNrYWdl cy9qdWxpYS1qbGwuc2NtOjkyMDo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBq dWxpYS1saWJncGctZXJyb3ItamxsCmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjg4Mjo2OiB3 YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxpYS1saWJnbHZuZC1qbGwKZ251L3Bh Y2thZ2VzL2p1bGlhLWpsbC5zY206NDg2OjY6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQg Zm9yIGp1bGlhLWdsaWItamxsCmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjE4NTA6Njogd2Fy bmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IganVsaWEteG9yZy1saWJ4YXUtamxsCmdudS9w YWNrYWdlcy9qdWxpYS1qbGwuc2NtOjEwNjo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5k IGZvciBqdWxpYS1jYWlyby1qbGwKZ251L3BhY2thZ2VzL2p1bGlhLWpsbC5zY206MjU1Njo2OiB3 YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxpYS14c2x0LWpsbApnbnUvcGFja2Fn ZXMvanVsaWEtamxsLnNjbToyMjk4OjY6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9y IGp1bGlhLXhvcmcteGNiLXV0aWwtaW1hZ2UtamxsCmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2Nt OjI0MDk6Njogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IganVsaWEteG9yZy14Y2It dXRpbC13bS1qbGwKZ251L3BhY2thZ2VzL2p1bGlhLWpsbC5zY206NjU3OjY6IHdhcm5pbmc6IG5v IHZhbGlkIHRhZ3MgZm91bmQgZm9yIGp1bGlhLWpwZWd0dXJiby1qbGwKZ251L3BhY2thZ2VzL2p1 bGlhLWpsbC5zY206MjA3NDo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxp YS14b3JnLWxpYnhpLWpsbApnbnUvcGFja2FnZXMvanVsaWEtamxsLnNjbToxMTg3OjY6IHdhcm5p bmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGp1bGlhLWx6by1qbGwKZ251L3BhY2thZ2VzL2p1 bGlhLWpsbC5zY206MjAwMDo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxp YS14b3JnLWxpYnhleHQtamxsCmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjg0Mzo2OiB3YXJu aW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxpYS1saWJnY3J5cHQtamxsCmdudS9wYWNr YWdlcy9qdWxpYS1qbGwuc2NtOjczMDo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZv ciBqdWxpYS1saWJhc3MtamxsCmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjE1NDI6Njogd2Fy bmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IganVsaWEtd2F5bGFuZC1qbGwKZ251L3BhY2th Z2VzL2p1bGlhLWpsbC5zY206MTExMTo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZv ciBqdWxpYS1saWJ1dWlkLWpsbApnbnUvcGFja2FnZXMvanVsaWEtamxsLnNjbToxMDcyOjY6IHdh cm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGp1bGlhLWxpYnRpZmYtamxsCmdudS9wYWNr YWdlcy9qdWxpYS1qbGwuc2NtOjIyMjQ6Njogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBm b3IganVsaWEteG9yZy1saWJ4cmVuZGVyLWpsbApnbnUvcGFja2FnZXMvanVsaWEtamxsLnNjbTo5 NTg6Njogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IganVsaWEtbGliaWNvbnYtamxs CmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjk5Njo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdz IGZvdW5kIGZvciBqdWxpYS1saWJtb3VudC1qbGwKZ251L3BhY2thZ2VzL2p1bGlhLWpsbC5zY206 MTAzNTo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxpYS1saWJwbmctamxs CmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjc3MDo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdz IGZvdW5kIGZvciBqdWxpYS1saWJmZGstYWFjLWpsbApnbnUvcGFja2FnZXMvanVsaWEtamxsLnNj bToxNzM1OjY6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGp1bGlhLXhtbDItamxs CmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjgwNjo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdz IGZvdW5kIGZvciBqdWxpYS1saWJmZmktamxsCmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjE4 ODY6Njogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IganVsaWEteG9yZy1saWJ4Y2It amxsCmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjI0NDY6Njogd2FybmluZzogbm8gdmFsaWQg dGFncyBmb3VuZCBmb3IganVsaWEteG9yZy14a2Jjb21wLWpsbApnbnUvcGFja2FnZXMvanVsaWEt amxsLnNjbToxNjU5OjY6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGp1bGlhLXgy NjUtamxsCmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjI0ODM6Njogd2FybmluZzogbm8gdmFs aWQgdGFncyBmb3VuZCBmb3IganVsaWEteG9yZy14a2V5Ym9hcmQtY29uZmlnLWpsbApnbnUvcGFj a2FnZXMvanVsaWEtamxsLnNjbTo0NDM6Njogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBm b3IganVsaWEtZ2xmdy1qbGwKZ251L3BhY2thZ2VzL2p1bGlhLWpsbC5zY206MjExMjo2OiB3YXJu aW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxpYS14b3JnLWxpYnhpbmVyYW1hLWpsbApn bnUvcGFja2FnZXMvanVsaWEtamxsLnNjbTo2Njo1OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZv dW5kIGZvciBqdWxpYS1iemlwMi1qbGwKZ251L3BhY2thZ2VzL2p1bGlhLWpsbC5zY206NTY5OjY6 IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGp1bGlhLWltYWdlbWFnaWNrLWpsbApn bnUvcGFja2FnZXMvanVsaWEtamxsLnNjbToxMjYzOjY6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3Mg Zm91bmQgZm9yIGp1bGlhLW9nZy1qbGwKZ251L3BhY2thZ2VzL2p1bGlhLWpsbC5zY206MjMyOjY6 IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGp1bGlhLWZmbXBlZy1qbGwKZ251L3Bh Y2thZ2VzL2p1bGlhLWpsbC5zY206MTY5NTo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5k IGZvciBqdWxpYS14a2Jjb21tb24tamxsCmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjI1OTk6 Njogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IganVsaWEtemxpYi1qbGwKZ251L3Bh Y2thZ2VzL2p1bGlhLWpsbC5zY206MjMzNTo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5k IGZvciBqdWxpYS14b3JnLXhjYi11dGlsLWtleXN5bXMtamxsCmdudS9wYWNrYWdlcy9qdWxpYS1q bGwuc2NtOjUzMDo1OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxpYS1ndW1i by1qbGwKZ251L3BhY2thZ2VzL2p1bGlhLWpsbC5zY206MjE4Njo2OiB3YXJuaW5nOiBubyB2YWxp ZCB0YWdzIGZvdW5kIGZvciBqdWxpYS14b3JnLWxpYnhyYW5kci1qbGwKZ251L3BhY2thZ2VzL2p1 bGlhLWpsbC5zY206MTk0OjY6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGp1bGlh LWV4cGF0LWpsbApnbnUvcGFja2FnZXMvanVsaWEtamxsLnNjbToxMzAwOjc6IHdhcm5pbmc6IG5v IHZhbGlkIHRhZ3MgZm91bmQgZm9yIGp1bGlhLW9wZW5zcGVjZnVuLWpsbApnbnUvcGFja2FnZXMv anVsaWEtamxsLnNjbTozMjQ6NTogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IganVs aWEtZnJlZXR5cGUyLWpsbApnbnUvcGFja2FnZXMvanVsaWEtamxsLnNjbToxMzQzOjY6IHdhcm5p bmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGp1bGlhLW9wZW5zc2wtamxsCmdudS9wYWNrYWdl cy9qdWxpYS1qbGwuc2NtOjEyMjU6NTogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3Ig anVsaWEtbWJlZHRscy1qbGwKZ251L3BhY2thZ2VzL2p1bGlhLWpsbC5zY206MTUyOjU6IHdhcm5p bmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGp1bGlhLWNvbXBpbGVyc3VwcG9ydGxpYnJhcmll cy1qbGwKZ251L3BhY2thZ2VzL2p1bGlhLWpsbC5zY206MjAzNzo2OiB3YXJuaW5nOiBubyB2YWxp ZCB0YWdzIGZvdW5kIGZvciBqdWxpYS14b3JnLWxpYnhmaXhlcy1qbGwKZ251L3BhY2thZ2VzL2p1 bGlhLWpsbC5zY206MjgxOjY6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGp1bGlh LWZvbnRjb25maWctamxsCmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjE3NzU6Njogd2Fybmlu Zzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IganVsaWEteG9yZy1saWJwdGhyZWFkLXN0dWJzLWps bApnbnUvcGFja2FnZXMvanVsaWEtamxsLnNjbToxMzc5OjY6IHdhcm5pbmc6IG5vIHZhbGlkIHRh Z3MgZm91bmQgZm9yIGp1bGlhLW9wdXMtamxsCmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjQw Mjo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxpYS1nZXR0ZXh0LWpsbApn bnUvcGFja2FnZXMvanVsaWEtamxsLnNjbToxNDUzOjY6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3Mg Zm91bmQgZm9yIGp1bGlhLXBpeG1hbi1qbGwKZ251L3BhY2thZ2VzL2p1bGlhLWpsbC5zY206MjM3 Mjo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxpYS14b3JnLXhjYi11dGls LXJlbmRlcnV0aWwtamxsCmdudS9wYWNrYWdlcy9qdWxpYS1qbGwuc2NtOjI2Mzc6Njogd2Fybmlu Zzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IganVsaWEtenN0ZC1qbGwKZ251L3BhY2thZ2VzL2p1 bGlhLWpsbC5zY206MjE0OTo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBqdWxp YS14b3JnLWxpYnhrYmZpbGUtamxsCgoMCjs7OyBvbmx5IHByZS1yZWxlYXNlcwpnbnUvcGFja2Fn ZXMvbGludXguc2NtOjEzMDY6Nzogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgcnRs ODgxMmF1LWFpcmNyYWNrLW5nLWxpbnV4LW1vZHVsZQpnbnUvcGFja2FnZXMvdmltLnNjbTo1OTY6 Nzogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgdmltLXNvbGFyaXplZApnbnUvcGFj a2FnZXMvbXVzaWMuc2NtOjIxMzM6MTI6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9y IHBvd2VydGFiZWRpdG9yCmdudS9wYWNrYWdlcy9tdXNpYy5zY206NDQ2MzoxNDogd2FybmluZzog bm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgcmtybHYyCmdudS9wYWNrYWdlcy9sdWEuc2NtOjI0ODox Mjogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgbHVhNS4xLXNvY2tldApnbnUvcGFj a2FnZXMvbHVhLnNjbToyNDg6MTI6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGx1 YTUuMi1zb2NrZXQKZ251L3BhY2thZ2VzL2d1aWxlLXh5ei5zY206Mzg2MzoxMjogd2FybmluZzog bm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgZ3VpbGUtYWMtZC1idXMKZ251L3BhY2thZ2VzL2dvbGFu Zy5zY206NDEzNjo3OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBnby1naXRodWIt Y29tLWJ0Y3N1aXRlLWJ0Y2QtYnRjZWMKZ251L3BhY2thZ2VzL2dhbWVzLnNjbTo3Mjc0OjU6IHdh cm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGNvbG9ib3QKCjs7IF9iZXRhPG51bT4gc3Vm Zml4LCByZWZzL3RhZ3MvdjAuMV9iZXRhMgpnbnUvcGFja2FnZXMvY29tcHRvbi5zY206NTA6Nzog d2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgY29tcHRvbgoKOzsgYWxtb3N0IGFsbCB0 YWdzIGFyZSBwcmUtcmVsZWFzZXMKOzsgcmVmcy90YWdzLzAuMC40LTE1Mi1hbHBoYQo7OyByZWZz L3RhZ3MvMC4wLjQtMTUyLWFscGhhXnt9Cjs7IHJlZnMvdGFncy8wLjAuNC0xNzItYWxwaGEKOzsg cmVmcy90YWdzLzAuMC40LTE3Mi1hbHBoYV57fQo7OyByZWZzL3RhZ3MvMC4xLjAtYWxwaGEKOzsg cmVmcy90YWdzLzAuMS4wLWFscGhhXnt9Cjs7IHJlZnMvdGFncy8wLjEuMS1hbHBoYQo7OyByZWZz L3RhZ3MvMC4xLjEtYWxwaGFee30KOzsgcmVmcy90YWdzLzAuMS4yLWFscGhhCjs7IHJlZnMvdGFn cy8wLjEuMi1hbHBoYV57fQo7OyByZWZzL3RhZ3MvMC4yLjAtYWxwaGEKOzsgcmVmcy90YWdzLzAu Mi4wLWFscGhhXnt9Cjs7IHJlZnMvdGFncy9wcmUtZXh0ZW5zaW9uCjs7IHJlZnMvdGFncy92MC4w LjItYWxwaGEKOzsgcmVmcy90YWdzL3YwLjAuMy1hbHBoYQo7OyByZWZzL3RhZ3MvdjAuMC40LWFs cGhhCjs7IHJlZnMvdGFncy92MC4wLjQtYWxwaGFee30KOzsgcmVmcy90YWdzL3YwLjEuMC1hbHBo YQo7OyByZWZzL3RhZ3MvdjAuMS4wLWFscGhhXnt9Cjs7IHJlZnMvdGFncy92MC4xLjItY2FuZGlk YXRlCjs7IHJlZnMvdGFncy92MC4xLjItY2FuZGlkYXRlXnt9CmdudS9wYWNrYWdlcy9ndWlsZS14 eXouc2NtOjM0ODg6MTI6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIG5vbWFkCgoM Cjs7OyBubyBkZWxpbWl0ZXIgb3IganVzdCBvbmUgdmVyc2lvbiBudW1iZXIKZ251L3BhY2thZ2Vz L21haWwuc2NtOjU4Nzo1OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBuZW9tdXR0 CmdudS9wYWNrYWdlcy9zeW5jLnNjbTo1MTg6MTI6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91 bmQgZm9yIGNhc3luYwpnbnUvcGFja2FnZXMvcHVsc2VhdWRpby5zY206MzE1OjEyOiB3YXJuaW5n OiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBwb255bWl4CmdudS9wYWNrYWdlcy9uZXR3b3JraW5n LnNjbToxMjY2OjEyOiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBpcHV0aWxzCmdu dS9wYWNrYWdlcy9tdXNpYy5zY206Njg4NDoxMjogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3Vu ZCBmb3IgYTJqbWlkaWQKZ251L3BhY2thZ2VzL2VtdWxhdG9ycy5zY206NTY3OjU6IHdhcm5pbmc6 IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGhpZ2FuCmdudS9wYWNrYWdlcy9lbWFjcy14eXouc2Nt OjExNDg5OjU6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGVtYWNzLWx1YS1tb2Rl CmdudS9wYWNrYWdlcy9kaWZmb3Njb3BlLnNjbTo3NjoxMjogd2FybmluZzogbm8gdmFsaWQgdGFn cyBmb3VuZCBmb3IgZGlmZm9zY29wZQpnbnUvcGFja2FnZXMvY3Vwcy5zY206Njc6MTQ6IHdhcm5p bmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGJybGFzZXIKZ251L3BhY2thZ2VzL2NvZGUuc2Nt OjU3NToxMjogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3Iga2NvdgpnbnUvcGFja2Fn ZXMvZ25vbWUteHl6LnNjbToxODU6NTogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3Ig cGFwaXJ1cy1pY29uLXRoZW1lCmdudS9wYWNrYWdlcy9nbm9tZS14eXouc2NtOjM2NzoxMjogd2Fy bmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgZ25vbWUtc2hlbGwtZXh0ZW5zaW9uLWdzY29u bmVjdApnbnUvcGFja2FnZXMvZ25vbWUteHl6LnNjbToyODk6NTogd2FybmluZzogbm8gdmFsaWQg dGFncyBmb3VuZCBmb3IgZ25vbWUtc2hlbGwtZXh0ZW5zaW9uLXRvcGljb25zLXJlZHV4CmdudS9w YWNrYWdlcy9nb2xhbmcuc2NtOjExMTY6NTogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBm b3IgZ28tZ2l0aHViLWNvbS1vcC1nby1sb2dnaW5nCmdudS9wYWNrYWdlcy9qYXZhLnNjbToxMDU3 OToxMjogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgamF2YS1qYXZheC1pbmplY3QK Z251L3BhY2thZ2VzL21haWwuc2NtOjE0MTU6MTI6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91 bmQgZm9yIG5vdG11Y2gtYWRkcmxvb2t1cC1jCgoKCgoKCgwKOzs7IChNb3N0IGxpa2VseSkgbm90 IHNvbHZhYmxlIGJ5IHVzCjs7IG9ubHkgb25lIHRhZyAiaW5pdGlhbC1jb21taXQiCmdudS9wYWNr YWdlcy9idWlsZC10b29scy5zY206MjA4OjE0OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5k IGZvciBnbgoKOzsgPz8/IHRoZSB3ZWlyZGVzdCB2ZXJzaW9uIHNjaGVtZT8/Pwo7OyByZWZzL3Rh Z3MvYnpyLXJldm5vLTc4OTQKOzsgcmVmcy90YWdzL21vc2VzLTMuMi4xMy0xNS1Ob3YtMjAxMgo7 OyByZWZzL3RhZ3MvbW92ZWQtdG8tdXJlLWJhc2VkLXIybAo7OyByZWZzL3RhZ3Mvb2Jzb2xldGUt QysrLVBMTgo7OyByZWZzL3RhZ3Mvb2Jzb2xldGUtUHl0aG9uLVBMTgo7OyByZWZzL3RhZ3Mvb2xk LWVtYm9kaW1lbnRfOC1ub3YtMjAxNQo7OyByZWZzL3RhZ3Mvb2xkLW9wZW5wc2ktSmluaHVhLUNo dWEtMjAxMQo7OyByZWZzL3RhZ3Mvb2xkLXBhdHRlcm4tbWluZXItMTEtSnVuLTIwMTkKOzsgcmVm cy90YWdzL3YwLjEuNC1zdGFibGUKZ251L3BhY2thZ2VzL29wZW5jb2cuc2NtOjIwOToxNDogd2Fy bmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3Igb3BlbmNvZwoKOzsgInF1aWNrbGlzcC0iIHBy ZWZpeCwgZGF0ZSBpbnN0ZWFkIG9mIHZlcnNpb24gbnVtYmVyCmdudS9wYWNrYWdlcy9saXNwLXh5 ei5zY206MzcwMzo3OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBzYmNsLXBvcnRh YmxlLXRocmVhZHMKCjs7IG5vIHZlcnNpb24gbnVtYmVyIGluIHRhZwpnbnUvcGFja2FnZXMvZ3Vp bGUteHl6LnNjbTozMjgzOjc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGd1aWxl LXNyZmktMTgwCmdudS9wYWNrYWdlcy9nb2xhbmcuc2NtOjYxOTM6MTQ6IHdhcm5pbmc6IG5vIHZh bGlkIHRhZ3MgZm91bmQgZm9yIGdvLWdpdGh1Yi1jb20tZ29sYW5nLWZyZWV0eXBlCmdudS9wYWNr YWdlcy9tZXNzYWdpbmcuc2NtOjE0OTU6MTQ6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQg Zm9yIGxpYnRveGNvcmUKZ251L3BhY2thZ2VzL25ldHdvcmtpbmcuc2NtOjM4MTQ6NTogd2Fybmlu Zzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgdmRlMgpnbnUvcGFja2FnZXMvbGlzcC14eXouc2Nt OjE1ODkyOjY6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGNsLWJvZGdlLWJsb2Jz LXN1cHBvcnQKZ251L3BhY2thZ2VzL2xpc3AteHl6LnNjbTozMTIwOjc6IHdhcm5pbmc6IG5vIHZh bGlkIHRhZ3MgZm91bmQgZm9yIGNsLWNmZmktYy1yZWYKZ251L3BhY2thZ2VzL2xpc3AteHl6LnNj bTo1OTc5Ojc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIHNiY2wtaWVlZS1mbG9h dHMKZ251L3BhY2thZ2VzL2VuZ2luZWVyaW5nLnNjbTo4MTQ6MTQ6IHdhcm5pbmc6IG5vIHZhbGlk IHRhZ3MgZm91bmQgZm9yIGxpYmZpdmUKZ251L3BhY2thZ2VzL2NwcC5zY206MjIxOjE0OiB3YXJu aW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciByY3QKCjs7ICJnby53ZWVrbHkuIiBwcmVmaXgs IGRhdGUgaW5zdGVhZCBvZiB2ZXJzaW9uCmdudS9wYWNrYWdlcy9nb2xhbmcuc2NtOjM5MDU6Nzog d2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgZ28tZ2l0aHViLWNvbS1taWNoaXdlbmQt Z29sYW5nLXByZXR0eQoKOzsgd2VpcmQgdmVyc2lvbiBzY2hlbWUKOzsgcmVmcy90YWdzL1YzCjs7 IHJlZnMvdGFncy92ZXJzaW9uXzEKOzsgcmVmcy90YWdzL3d5aGFzaF9hbHBoYQo7OyByZWZzL3Rh Z3Mvd3loYXNoX2ZpbmFsCjs7IHJlZnMvdGFncy93eWhhc2hfdjQKOzsgcmVmcy90YWdzL3d5aGFz aF92NQpnbnUvcGFja2FnZXMvZGlnZXN0LnNjbTozNjoxMjogd2FybmluZzogbm8gdmFsaWQgdGFn cyBmb3VuZCBmb3Igd3loYXNoCgoKCgo7OzsgY291bGRuJ3QgYmUgYm90aGVyZWQgdG8gY2hlY2su Li4KZ251L3BhY2thZ2VzL2xpc3AteHl6LnNjbTo2NTA4OjI6IHdhcm5pbmc6IG5vIHZhbGlkIHRh Z3MgZm91bmQgZm9yIGNsLXRyaXZpYQpnbnUvcGFja2FnZXMvbGlzcC14eXouc2NtOjEyMjE0Ojc6 IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGNsLW1jY2xpbQpnbnUvcGFja2FnZXMv bGlzcC14eXouc2NtOjEzNjMzOjc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGNs LWNsYXctdXRpbHMKZ251L3BhY2thZ2VzL2xpc3AteHl6LnNjbTozMTk1Ojc6IHdhcm5pbmc6IG5v IHZhbGlkIHRhZ3MgZm91bmQgZm9yIHNiY2wtcGFyZW5zY3JpcHQKZ251L3BhY2thZ2VzL2xpc3At eHl6LnNjbTo2NDY2Ojc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGNsLXRyaXZp YS50cml2aWFsCmdudS9wYWNrYWdlcy9saXNwLXh5ei5zY206NjUwODoyOiB3YXJuaW5nOiBubyB2 YWxpZCB0YWdzIGZvdW5kIGZvciBlY2wtdHJpdmlhCmdudS9wYWNrYWdlcy9saXNwLXh5ei5zY206 MTM1NjY6NTogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgY2wtY2xhdy1zdXBwb3J0 CmdudS9wYWNrYWdlcy9saXNwLXh5ei5zY206OTMyNjo3OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdz IGZvdW5kIGZvciBjbC1qcGVnCmdudS9wYWNrYWdlcy9saXNwLXh5ei5zY206MTM1NjY6NTogd2Fy bmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3Igc2JjbC1jbGF3LXN1cHBvcnQKZ251L3BhY2th Z2VzL2xpc3AteHl6LnNjbToxODM5NDo3OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZv ciBjbC10YWlscmVjCmdudS9wYWNrYWdlcy9saXNwLXh5ei5zY206MTM5OjY6IHdhcm5pbmc6IG5v IHZhbGlkIHRhZ3MgZm91bmQgZm9yIHNiY2wtYm9kZ2UtdXRpbGl0aWVzCmdudS9wYWNrYWdlcy9s aXNwLXh5ei5zY206MTU4NjA6Njogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgZWNs LWJvZGdlLW1hdGgKZ251L3BhY2thZ2VzL2xpc3AteHl6LnNjbToxMzU2Njo1OiB3YXJuaW5nOiBu byB2YWxpZCB0YWdzIGZvdW5kIGZvciBlY2wtY2xhdy1zdXBwb3J0CmdudS9wYWNrYWdlcy9saXNw LXh5ei5zY206MTc4Ojc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGNsLWJvZGdl LXF1ZXVlCmdudS9wYWNrYWdlcy9saXNwLXh5ei5zY206NjUwODoyOiB3YXJuaW5nOiBubyB2YWxp ZCB0YWdzIGZvdW5kIGZvciBzYmNsLXRyaXZpYQpnbnUvcGFja2FnZXMvbGlzcC14eXouc2NtOjY4 MDc6Nzogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3Igc2JjbC1zLXN5c2RlcHMKZ251 L3BhY2thZ2VzL2xpc3AteHl6LnNjbTo2NDY2Ojc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91 bmQgZm9yIGVjbC10cml2aWEudHJpdmlhbApnbnUvcGFja2FnZXMvbGlzcC14eXouc2NtOjUxOTo3 OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBjbC1maWFzY28KZ251L3BhY2thZ2Vz L2xpc3AteHl6LnNjbTo2ODA3Ojc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGVj bC1zLXN5c2RlcHMKZ251L3BhY2thZ2VzL2xpc3AteHl6LnNjbTo4OTI5Ojc6IHdhcm5pbmc6IG5v IHZhbGlkIHRhZ3MgZm91bmQgZm9yIGVjbC1jbC1hc3luYwpnbnUvcGFja2FnZXMvbGlzcC14eXou c2NtOjMxOTU6Nzogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3IgY2wtcGFyZW5zY3Jp cHQKZ251L3BhY2thZ2VzL2xpc3AteHl6LnNjbTo5MzI2Ojc6IHdhcm5pbmc6IG5vIHZhbGlkIHRh Z3MgZm91bmQgZm9yIGVjbC1jbC1qcGVnCmdudS9wYWNrYWdlcy9saXNwLXh5ei5zY206NTE5Ojc6 IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGVjbC1maWFzY28KZ251L3BhY2thZ2Vz L2xpc3AteHl6LnNjbTozNzAzOjc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGVj bC1wb3J0YWJsZS10aHJlYWRzCmdudS9wYWNrYWdlcy9saXNwLXh5ei5zY206MTIyMTQ6Nzogd2Fy bmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3Igc2JjbC1tY2NsaW0KZ251L3BhY2thZ2VzL2xp c3AteHl6LnNjbTo4NjI4Ojc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGNsLW1p c2MtZXh0ZW5zaW9ucwpnbnUvcGFja2FnZXMvbGlzcC14eXouc2NtOjEzNjMzOjc6IHdhcm5pbmc6 IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGVjbC1jbGF3LXV0aWxzCmdudS9wYWNrYWdlcy9saXNw LXh5ei5zY206NTk3OTo3OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBjbC1pZWVl LWZsb2F0cwpnbnUvcGFja2FnZXMvbGlzcC14eXouc2NtOjE1ODkyOjY6IHdhcm5pbmc6IG5vIHZh bGlkIHRhZ3MgZm91bmQgZm9yIGVjbC1ib2RnZS1ibG9icy1zdXBwb3J0CmdudS9wYWNrYWdlcy9s aXNwLXh5ei5zY206MzEyMDo3OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBlY2wt Y2ZmaS1jLXJlZgpnbnUvcGFja2FnZXMvbGlzcC14eXouc2NtOjU5Nzk6Nzogd2FybmluZzogbm8g dmFsaWQgdGFncyBmb3VuZCBmb3IgZWNsLWllZWUtZmxvYXRzCmdudS9wYWNrYWdlcy9saXNwLXh5 ei5zY206MTM5OjY6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGNsLWJvZGdlLXV0 aWxpdGllcwpnbnUvcGFja2FnZXMvbGlzcC14eXouc2NtOjE3ODo3OiB3YXJuaW5nOiBubyB2YWxp ZCB0YWdzIGZvdW5kIGZvciBlY2wtYm9kZ2UtcXVldWUKZ251L3BhY2thZ2VzL2xpc3AteHl6LnNj bTo5MzI2Ojc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIHNiY2wtY2wtanBlZwpn bnUvcGFja2FnZXMvbGlzcC14eXouc2NtOjUxOTo3OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZv dW5kIGZvciBzYmNsLWZpYXNjbwpnbnUvcGFja2FnZXMvbGlzcC14eXouc2NtOjE1ODkyOjY6IHdh cm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIHNiY2wtYm9kZ2UtYmxvYnMtc3VwcG9ydApn bnUvcGFja2FnZXMvbGlzcC14eXouc2NtOjEzOTo2OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZv dW5kIGZvciBlY2wtYm9kZ2UtdXRpbGl0aWVzCmdudS9wYWNrYWdlcy9saXNwLXh5ei5zY206MzE5 NTo3OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBlY2wtcGFyZW5zY3JpcHQKZ251 L3BhY2thZ2VzL2xpc3AteHl6LnNjbTozMTIwOjc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91 bmQgZm9yIHNiY2wtY2ZmaS1jLXJlZgpnbnUvcGFja2FnZXMvbGlzcC14eXouc2NtOjE3ODo3OiB3 YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBzYmNsLWJvZGdlLXF1ZXVlCmdudS9wYWNr YWdlcy9saXNwLXh5ei5zY206ODYyODo3OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZv ciBlY2wtbWlzYy1leHRlbnNpb25zCmdudS9wYWNrYWdlcy9saXNwLXh5ei5zY206MTU4NjA6Njog d2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBmb3Igc2JjbC1ib2RnZS1tYXRoCmdudS9wYWNr YWdlcy9saXNwLXh5ei5zY206ODYyODo3OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZv ciBzYmNsLW1pc2MtZXh0ZW5zaW9ucwpnbnUvcGFja2FnZXMvbGlzcC14eXouc2NtOjE1ODYwOjY6 IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGNsLWJvZGdlLW1hdGgKZ251L3BhY2th Z2VzL2xpc3AteHl6LnNjbToxMzYzMzo3OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZv ciBzYmNsLWNsYXctdXRpbHMKZ251L3BhY2thZ2VzL2xpc3AteHl6LnNjbTo2NDY2Ojc6IHdhcm5p bmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIHNiY2wtdHJpdmlhLnRyaXZpYWwKZ251L3BhY2th Z2VzL2xpc3AteHl6LnNjbToxODM5NDo3OiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZvdW5kIGZv ciBzYmNsLXRhaWxyZWMKZ251L3BhY2thZ2VzL2xpc3AteHl6LnNjbToxODM5NDo3OiB3YXJuaW5n OiBubyB2YWxpZCB0YWdzIGZvdW5kIGZvciBlY2wtdGFpbHJlYwpnbnUvcGFja2FnZXMvbGlzcC14 eXouc2NtOjEyMjE0Ojc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGVjbC1tY2Ns aW0KZ251L3BhY2thZ2VzL2xpc3AteHl6LnNjbTo4OTI5Ojc6IHdhcm5pbmc6IG5vIHZhbGlkIHRh Z3MgZm91bmQgZm9yIHNiY2wtY2wtYXN5bmMKZ251L3BhY2thZ2VzL2xpc3AteHl6LnNjbTo2ODA3 Ojc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGNsLXMtc3lzZGVwcwpnbnUvcGFj a2FnZXMvbGlzcC14eXouc2NtOjg5Mjk6Nzogd2FybmluZzogbm8gdmFsaWQgdGFncyBmb3VuZCBm b3IgY2wtYXN5bmMKZ251L3BhY2thZ2VzL2xpc3AteHl6LnNjbTozNzAzOjc6IHdhcm5pbmc6IG5v IHZhbGlkIHRhZ3MgZm91bmQgZm9yIGNsLXBvcnRhYmxlLXRocmVhZHMKZ251L3BhY2thZ2VzL2xp bnV4LnNjbToxMzA2Ojc6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIHJ0bDg4MTJh dS1haXJjcmFjay1uZy1saW51eC1tb2R1bGUKZ251L3BhY2thZ2VzL2xpbnV4LnNjbTo2Mjg0OjU6 IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIG1jZWxvZwpnbnUvcGFja2FnZXMva29k aS5zY206NDcwOjI6IHdhcm5pbmc6IG5vIHZhbGlkIHRhZ3MgZm91bmQgZm9yIGtvZGktd2F5bGFu ZApnbnUvcGFja2FnZXMva29kaS5zY206Mjc1OjEyOiB3YXJuaW5nOiBubyB2YWxpZCB0YWdzIGZv dW5kIGZvciBrb2RpCg== --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: attachment; filename=0001-import-Add-generic-git-updater.patch Content-Transfer-Encoding: quoted-printable From=20f924dbb835425f6b9a5796918125592870391405 Mon Sep 17 00:00:00 2001 Message-Id: From: Xinglu Chen Date: Fri, 3 Sep 2021 17:50:56 +0200 Subject: [PATCH] import: Add 'generic-git' updater. * guix/import/git.scm: New file. * doc/guix.texi (Invoking guix refresh): Document it. * Makefile.am (MODULES): Register it. * guix/git.scm (ls-remote-refs): New procedure. Co-authored-by: Sarah Morgensen =2D-- Makefile.am | 1 + doc/guix.texi | 27 ++++++ guix/git.scm | 33 +++++++ guix/import/git.scm | 217 ++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 278 insertions(+) create mode 100644 guix/import/git.scm diff --git a/Makefile.am b/Makefile.am index 3c79760734..c4d3a456b1 100644 =2D-- a/Makefile.am +++ b/Makefile.am @@ -254,6 +254,7 @@ MODULES =3D \ guix/import/egg.scm \ guix/import/elpa.scm \ guix/import/gem.scm \ + guix/import/git.scm \ guix/import/github.scm \ guix/import/gnome.scm \ guix/import/gnu.scm \ diff --git a/doc/guix.texi b/doc/guix.texi index 36a0c7f5ec..26afb1607a 100644 =2D-- a/doc/guix.texi +++ b/doc/guix.texi @@ -11920,6 +11920,33 @@ the updater for @uref{https://launchpad.net, Launc= hpad} packages. @item generic-html a generic updater that crawls the HTML page where the source tarball of the package is hosted, when applicable. +@item generic-git +a generic updater for packages hosted on Git repositories. It tries to +be smart about parsing Git tag names, but if it is not able to parse the +tag name and compare tags correctly, users can define the following +properties for a package. + +@itemize +@item @code{tag-prefix}: a regular expression for matching a prefix of +the tag name. + +@item @code{tag-suffix}: a regular expression for matching a suffix of +the tag name. + +@item @code{tag-version-delimiter}: a string used as the delimiter in +the tag name for separating the numbers of the version. +@end itemize + +@lisp +(package + (name "foo") + ;; ... + (properties + '((tag-prefix . "^release0-") + (tag-suffix . "[a-z]?$") + (tag-version-delimiter . ":")))) +@end lisp=20=20=20=20=20=20 + @end table =20 For instance, the following command only checks for updates of Emacs diff --git a/guix/git.scm b/guix/git.scm index 9c6f326c36..c5d0d2da8e 100644 =2D-- a/guix/git.scm +++ b/guix/git.scm @@ -56,6 +56,8 @@ commit-difference commit-relation =20 + ls-remote-refs + git-checkout git-checkout? git-checkout-url @@ -556,6 +558,37 @@ objects: 'ancestor (meaning that OLD is an ancestor of= NEW), 'descendant, or (if (set-contains? oldest new) 'descendant 'unrelated)))))) + +;; +;;; Remote operations. +;;; + +(define* (ls-remote-refs url #:key tags?) + "Return the list of references advertised at Git repository URL. If TAG= S? +is true, limit to only refs/tags." + (define (ref? ref) + ;; Like `git ls-remote --refs', only show actual references. + (and (string-prefix? "refs/" ref) + (not (string-suffix? "^{}" ref)))) + + (define (tag? ref) + (string-prefix? "refs/tags/" ref)) + + (define (include? ref) + (and (ref? ref) + (or (not tags?) (tag? ref)))) + + (with-libgit2 + (call-with-temporary-directory + (lambda (cache-directory) + (let* ((repository (repository-init cache-directory)) + ;; Create an in-memory remote so we don't touch disk. + (remote (remote-create-anonymous repository url))) + (remote-connect remote) + (remote-disconnect remote) + (repository-close! repository) + + (filter include? (map remote-head-name (remote-ls remote)))))))) =20 ;;; diff --git a/guix/import/git.scm b/guix/import/git.scm new file mode 100644 index 0000000000..52c98de197 =2D-- /dev/null +++ b/guix/import/git.scm @@ -0,0 +1,217 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix import git) + #:use-module (guix build utils) + #:use-module (guix diagnostics) + #:use-module (guix git) + #:use-module (guix git-download) + #:use-module (guix i18n) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module (ice-9 format) + #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 regex) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) + #:use-module (srfi srfi-34) + #:use-module (srfi srfi-35) + #:export (%generic-git-updater)) + +;;; Commentary: +;;; +;;; This module provides a generic package updater for packages hosted on = Git +;;; repositories. +;;; +;;; It tries to be smart about tag names, but if it is not automatically a= ble +;;; to parse the tag names correctly, users can set the `tag-prefix', +;;; `tag-suffix' and `tag-version-delimiter' properties of the package to = make +;;; the updater parse the Git tag name correctly. +;;; +;;; Code: + +;;; Errors & warnings + +(define-condition-type &git-no-valid-tags-error &error + git-no-valid-tags-error?) + +(define (git-no-valid-tags-error) + (raise (condition (&message (message "no valid tags found")) + (&git-no-valid-tags-error)))) + +(define-condition-type &git-no-tags-error &error + git-no-tags-error?) + +(define (git-no-tags-error) + (raise (condition (&message (message "no tags were found")) + (&git-no-tags-error)))) + + +;;; Updater + +(define %pre-release-words + '("alpha" "beta" "rc" "dev" "test")) + +(define* (get-version-mapping tags #:key prefix suffix delim pre-releases?) + (define (guess-delim) + (let ((total (length tags)) + (dots (reduce + 0 (map (cut string-count <> #\.) tags))) + (dashes (reduce + 0 (map (cut string-count <> #\-) tags))) + (underscores (reduce + 0 (map (cut string-count <> #\_) tags)))) + (display (format #t "total: ~d, dots: ~d, dashes ~d, underscores ~d~= %" + total dots dashes underscores)) + (cond + ((>=3D dots (* total 0.35)) ".") + ((>=3D dashes (* total 0.8)) "-") + ((>=3D underscores (* total 0.8)) "_") + (else "")))) + + (define delim-rx (regexp-quote (or delim (guess-delim)))) + (define suffix-rx + (string-append + (or suffix + (if pre-releases? + (string-append ".*(" (string-join %pre-release-words "|") ").= *") + "")) + "$")) +=20=20 + (define prefix-rx (string-append "^" (or prefix "[^[:digit:]]*"))) + (pk delim-rx) + + (define tag-rx + (string-append "([[:digit:]][^" delim-rx "[:punct:]]*" + "(" delim-rx "[^[:punct:]" delim-rx "]+)" + ;; If there is are no delimiters, it could mean that t= he + ;; version just contains one number (e.g., "2"), thus,= use + ;; "*" instead of "+" to match zero or more numbers. + (if (string=3D? delim-rx "") "*" "+") + ")" suffix-rx)) + + (define (get-version tag) + (let ((tag-match (regexp-exec (make-regexp tag-rx) tag))) + (pk tag) + (pk tag-match) + (pk tag-rx) + (and tag-match + (regexp-exec (make-regexp prefix-rx) (match:prefix tag-match)) + (regexp-substitute/global + #f delim-rx (match:substring tag-match) + ;; Don't insert "." if there aren't any delimiters in the first + ;; place. + 'pre (if (string=3D? delim-rx "") "" ".") 'post)))) + + (define (entry (string-length "refs/tags/")) + (ls-remote-refs url #:tags? #t))) + (versions->tags + (get-version-mapping (if pre-releases? + tags + (filter (negate pre-release?) tags)) + #:prefix prefix + #:suffix suffix + #:delim delim + #:pre-releases? pre-releases?))) + (display versions->tags) + (newline) + (cond + ((null? tags) + (git-no-tags-error)) + ((null? versions->tags) + (git-no-valid-tags-error)) + (else + (match (last versions->tags) + ((version . tag) + (values version tag))))))) + +(define (latest-git-tag-version package tag-prefix tag-suffix + tag-version-delimiter refresh-pre-releases= ?) + "Given a PACKAGE, the TAG-PREFIX, TAG-SUFFIX, TAG-VERSION-DELIMITER, and +REFRESH-PRE-RELEASES? properties of PACKAGE, returns the latest version of +PACKAGE." + (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c)) + (warning (or (package-field-location package 'source) + (package-location package)) + (G_ "~a for ~a~%") + (condition-message c) + (package-name package)) + #f) + ((eq? (exception-kind c) 'git-error) + (warning (or (package-field-location package 'source) + (package-location package)) + (G_ "failed to fetch Git repository for ~a~%") + (package-name package)) + #f)) + (let* ((source (package-source package)) + (url (git-reference-url (origin-uri source)))) + ;;(format #t "~a~%" (package-name package)) + (get-latest-tag url #:prefix tag-prefix #:suffix tag-suffix + #:delim tag-version-delimiter + #:pre-releases? refresh-pre-releases?)))) + +(define (git-package? package) + "Whether the origin of PACKAGE is a Git repostiory." + (match (package-source package) + ((? origin? origin) + (and (eq? (origin-method origin) git-fetch) + (git-reference? (origin-uri origin)))) + (_ #f))) + +(define (latest-git-release package) + "Return the latest release of PACKAGE." + (let* ((name (package-name package)) + (properties (package-properties package)) + (tag-prefix (assq-ref properties 'tag-prefix)) + (tag-suffix (assq-ref properties 'tag-suffix)) + (tag-version-delimiter (assq-ref properties 'tag-version-delimite= r)) + (refresh-pre-releases? (assq-ref properties 'refresh-pre-releases= ?)) + (old-version (package-version package)) + (url (git-reference-url (origin-uri (package-source package)))) + (new-version (latest-git-tag-version package + tag-prefix + tag-suffix + tag-version-delimiter + refresh-pre-releases?))) + + (if new-version + (upstream-source + (package name) + (version new-version) + (urls (list url))) + ;; No new release or no tags available. + #f))) + +(define %generic-git-updater + (upstream-updater + (name 'generic-git) + (description "Updater for packages hosted on Git repositories") + (pred git-package?) + (latest latest-git-release))) =2D-=20 2.33.0 --=-=-=-- --==-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmE5ANcVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5QjoP/A7+M+zpmxPhDcR69CHICY0j850j eljZ0arJ38IiGAonrsQDd7uGsSoFqFfwN5sks/J1HlpAUiYpkYIJ+TX9f5NzIEoR FKdKO+IDScFsoesBnp1JYg4YvUYq0JgktZaBEAL1a6LsLlRqJ7lVgE0oWXbpcuKh uKW3c50/QFglL91avSfLU/BPc1fSGrLF12f8+pE/yD5WivpUH5+/xv+apk2xiLzD +/c6KVGsVvA5CvMl2IEQ9riui0SKbnhw48HGRrMArvxbh/zAlN7gaYTOuRmwlOoR pM00dlNTZn3yJFu9TzaY05yhD2FFsNmZVH9FBfmy6h1AqJhjL92JuqiQnUfaPGVM oek6NE/JT7TRAVWEhbh0ks5m4pBppYyxDKseK/A3EF/ZqeYe+QQSQajS3W0zhDOc dtbeqrHuxgawrvTyT3PdzA7T3BCJRgtseISwie4a5sziZb8fxTORE3qHx3CTN+0f 50UiiLmC+FBdak4LpZnCnzwQsct2bFrFH1kvDLuze1OJ9OpK5xd9UWfXXHLRbSv0 bAwxO350RKPIg6+e2Bfw3NP+zmUUNyMVp/KkqVRxZ78lp4r6F/xkW4n64u4zZezR Oe9M4OWiVg+byX7DPybbJybeafNXIq/PWW46hZ4vrD5dgub2Iow7Ctzt9t16aiAr ywZ5GSLN7s5M/Hgc =xwBb -----END PGP SIGNATURE----- --==-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 10 04:36:29 2021 Received: (at 50359) by debbugs.gnu.org; 10 Sep 2021 08:36:29 +0000 Received: from localhost ([127.0.0.1]:36865 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOc1M-0004by-Mm for submit@debbugs.gnu.org; Fri, 10 Sep 2021 04:36:29 -0400 Received: from eggs.gnu.org ([209.51.188.92]:56760) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOc1K-0004bk-Rd for 50359@debbugs.gnu.org; Fri, 10 Sep 2021 04:36:27 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:37190) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mOc1A-0006OB-P7; Fri, 10 Sep 2021 04:36:17 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=47218 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mOc1A-0004Ru-1e; Fri, 10 Sep 2021 04:36:16 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Xinglu Chen Subject: Re: bug#50359: [PATCH] import: Add 'generic-git' updater. References: <86k0jvkh5v.fsf@mgsn.dev> <87h7ez48d3.fsf@yoctocell.xyz> <86y28ai7ns.fsf@mgsn.dev> <87y28928vh.fsf@yoctocell.xyz> <86pmtli4hn.fsf@mgsn.dev> <87wnnsyzal.fsf@yoctocell.xyz> <87mtomzzu1.fsf@yoctocell.xyz> Date: Fri, 10 Sep 2021 10:36:12 +0200 In-Reply-To: <87mtomzzu1.fsf@yoctocell.xyz> (Xinglu Chen's message of "Wed, 08 Sep 2021 20:28:38 +0200") Message-ID: <87o890ddz7.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.3 (/) X-Debbugs-Envelope-To: 50359 Cc: Sarah Morgensen , 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.3 (-) Hello, This looks very cool! Xinglu Chen skribis: > From f924dbb835425f6b9a5796918125592870391405 Mon Sep 17 00:00:00 2001 > Message-Id: > From: Xinglu Chen > Date: Fri, 3 Sep 2021 17:50:56 +0200 > Subject: [PATCH] import: Add 'generic-git' updater. > > * guix/import/git.scm: New file. > * doc/guix.texi (Invoking guix refresh): Document it. > * Makefile.am (MODULES): Register it. > * guix/git.scm (ls-remote-refs): New procedure. > > Co-authored-by: Sarah Morgensen Overall LGTM; comments below. > diff --git a/doc/guix.texi b/doc/guix.texi > index 36a0c7f5ec..26afb1607a 100644 > --- a/doc/guix.texi > +++ b/doc/guix.texi > @@ -11920,6 +11920,33 @@ the updater for @uref{https://launchpad.net, Lau= nchpad} packages. > @item generic-html > a generic updater that crawls the HTML page where the source tarball of > the package is hosted, when applicable. > +@item generic-git Please add a newline above. > +@lisp > +(package > + (name "foo") > + ;; ... > + (properties > + '((tag-prefix . "^release0-") > + (tag-suffix . "[a-z]?$") > + (tag-version-delimiter . ":")))) > +@end lisp=20=20=20=20=20=20 Very nice. Perhaps s/tag-/release-tag-/ for clarity? > +(define* (ls-remote-refs url #:key tags?) > + "Return the list of references advertised at Git repository URL. If T= AGS? > +is true, limit to only refs/tags." To remain consistent with existing naming conventions, I=E2=80=99d call it =E2=80=98remote-refs=E2=80=99. > + (with-libgit2 > + (call-with-temporary-directory > + (lambda (cache-directory) > + (let* ((repository (repository-init cache-directory)) > + ;; Create an in-memory remote so we don't touch disk. > + (remote (remote-create-anonymous repository url))) Too bad we need to create an empty repo; hopefully it costs next to nothing though. > + (remote-connect remote) > + (remote-disconnect remote) > + (repository-close! repository) > + > + (filter include? (map remote-head-name (remote-ls remote)))))))) Use =E2=80=98filter-map=E2=80=99. > +(define* (get-version-mapping tags #:key prefix suffix delim pre-release= s?) Please add a docstring and remove =E2=80=98get-=E2=80=99 from the name. :-) > + (define (guess-delim) > + (let ((total (length tags)) > + (dots (reduce + 0 (map (cut string-count <> #\.) tags))) > + (dashes (reduce + 0 (map (cut string-count <> #\-) tags))) > + (underscores (reduce + 0 (map (cut string-count <> #\_) tags))= )) > + (display (format #t "total: ~d, dots: ~d, dashes ~d, underscores ~= d~%" > + total dots dashes underscores)) Leftover? (Also display + format.) Please spell out =E2=80=98delimiter=E2=80=99 (info "(guix) Formatting Code"= ). > + (cond > + ((>=3D dots (* total 0.35)) ".") > + ((>=3D dashes (* total 0.8)) "-") > + ((>=3D underscores (* total 0.8)) "_") > + (else "")))) That=E2=80=99s a fancy heuristic. :-) > + (let ((mapping (fold alist-cons '() (map get-version tags) tags))) > + (stable-sort! (filter car mapping) entry +(define* (get-latest-tag url #:key prefix suffix delim pre-releases?) > + "Return the latest tag available from the Git repository at URL." Maybe =E2=80=9Cthe tag corresponding to the latest version=E2=80=9D. s/get-latest-tag/latest-tag/ > + (define (pre-release? tag) > + (any (lambda (rx) (regexp-exec (make-regexp rx regexp/icase) tag)) > + %pre-release-words)) Better call =E2=80=98make-regexp=E2=80=99 only once; so you could change =E2=80=98%pre-release-words=E2=80=99 to be a list of regexp objects instead= of a list of strings. > +(define (latest-git-tag-version package tag-prefix tag-suffix > + tag-version-delimiter refresh-pre-releas= es?) > + "Given a PACKAGE, the TAG-PREFIX, TAG-SUFFIX, TAG-VERSION-DELIMITER, a= nd > +REFRESH-PRE-RELEASES? properties of PACKAGE, returns the latest version= of > +PACKAGE." Maybe s/refresh-pre-releases?/accept-pre-preleases?/ Since this procedure takes a package, it probably doesn=E2=80=99t need the = other arguments: it can extract them from the package properties, rather than doing it at the call site. > +(define (latest-git-release package) > + "Return the latest release of PACKAGE." > + (let* ((name (package-name package)) > + (properties (package-properties package)) > + (tag-prefix (assq-ref properties 'tag-prefix)) > + (tag-suffix (assq-ref properties 'tag-suffix)) > + (tag-version-delimiter (assq-ref properties 'tag-version-delimi= ter)) > + (refresh-pre-releases? (assq-ref properties 'refresh-pre-releas= es?)) > + (old-version (package-version package)) > + (url (git-reference-url (origin-uri (package-source package)))) > + (new-version (latest-git-tag-version package > + tag-prefix > + tag-suffix > + tag-version-delimiter > + refresh-pre-releases?))) > + > + (if new-version > + (upstream-source > + (package name) > + (version new-version) > + (urls (list url))) > + ;; No new release or no tags available. > + #f))) Simply: (and new-version (upstream-source =E2=80=A6)). It would have been nice to have tests. I think testing =E2=80=98latest-git-release=E2=80=99 should be feasible without too much ha= ssle using the (guix tests git) infrastructure, as is done in tests/git.scm, with a package referring to a locally-created repo using a git-reference with a file:// URL. Could you send an updated patch? Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 10 09:24:16 2021 Received: (at 50359) by debbugs.gnu.org; 10 Sep 2021 13:24:16 +0000 Received: from localhost ([127.0.0.1]:37260 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOgVs-0006oh-CH for submit@debbugs.gnu.org; Fri, 10 Sep 2021 09:24:16 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:32840 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOgVl-0006oN-Dg for 50359@debbugs.gnu.org; Fri, 10 Sep 2021 09:24:14 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631280241; bh=dA/Av/IPp+kgHuUFg4yuSP5fWmSIuSvL7H81Ahln9XU=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=SqoVnPZ8gwW5ffYbExWKbL2MUVKaazlkzjU5Oqcir88ToTs3VH3ROEeZYwcDFM4tQ n9stD1FGmfyHwPqviIMKHcY/Y6WJUBwYNrub5jkxXUqNyTe9l/UZbiI+ecL8uTFHDj EbH4cWejKq+d3yVbfVcGLvXFDmKvqb+1HFcDLFDY= To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: [bug#50359] [PATCH] import: Add 'generic-git' updater. In-Reply-To: <87o890ddz7.fsf_-_@gnu.org> References: <86k0jvkh5v.fsf@mgsn.dev> <87h7ez48d3.fsf@yoctocell.xyz> <86y28ai7ns.fsf@mgsn.dev> <87y28928vh.fsf@yoctocell.xyz> <86pmtli4hn.fsf@mgsn.dev> <87wnnsyzal.fsf@yoctocell.xyz> <87mtomzzu1.fsf@yoctocell.xyz> <87o890ddz7.fsf_-_@gnu.org> Date: Fri, 10 Sep 2021 15:23:46 +0200 Message-ID: <87bl50zhr1.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Fri, Sep 10 2021, Ludovic Courtès wrote: > Hello, > > This looks very cool! Thanks for taking a look! Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: Sarah Morgensen , 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Fri, Sep 10 2021, Ludovic Courtès wrote: > Hello, > > This looks very cool! Thanks for taking a look! Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Fri, Sep 10 2021, Ludovic Court=C3=A8s wrote: > Hello, > > This looks very cool! Thanks for taking a look! It=E2=80=99s still a WIP, but I think it=E2=80=99s getting there. :-) > Xinglu Chen skribis: > >> From f924dbb835425f6b9a5796918125592870391405 Mon Sep 17 00:00:00 2001 >> Message-Id: >> From: Xinglu Chen >> Date: Fri, 3 Sep 2021 17:50:56 +0200 >> Subject: [PATCH] import: Add 'generic-git' updater. >> >> * guix/import/git.scm: New file. >> * doc/guix.texi (Invoking guix refresh): Document it. >> * Makefile.am (MODULES): Register it. >> * guix/git.scm (ls-remote-refs): New procedure. >> >> Co-authored-by: Sarah Morgensen > > Overall LGTM; comments below. > >> diff --git a/doc/guix.texi b/doc/guix.texi >> index 36a0c7f5ec..26afb1607a 100644 >> --- a/doc/guix.texi >> +++ b/doc/guix.texi >> @@ -11920,6 +11920,33 @@ the updater for @uref{https://launchpad.net, La= unchpad} packages. >> @item generic-html >> a generic updater that crawls the HTML page where the source tarball of >> the package is hosted, when applicable. >> +@item generic-git > > Please add a newline above. Noted. >> +@lisp >> +(package >> + (name "foo") >> + ;; ... >> + (properties >> + '((tag-prefix . "^release0-") >> + (tag-suffix . "[a-z]?$") >> + (tag-version-delimiter . ":")))) >> +@end lisp=20=20=20=20=20=20 > > Very nice. Perhaps s/tag-/release-tag-/ for clarity? Good idea. >> +(define* (ls-remote-refs url #:key tags?) >> + "Return the list of references advertised at Git repository URL. If = TAGS? >> +is true, limit to only refs/tags." > > To remain consistent with existing naming conventions, I=E2=80=99d call it > =E2=80=98remote-refs=E2=80=99. > >> + (with-libgit2 >> + (call-with-temporary-directory >> + (lambda (cache-directory) >> + (let* ((repository (repository-init cache-directory)) >> + ;; Create an in-memory remote so we don't touch disk. >> + (remote (remote-create-anonymous repository url))) > > Too bad we need to create an empty repo; hopefully it costs next to > nothing though. > >> + (remote-connect remote) >> + (remote-disconnect remote) >> + (repository-close! repository) >> + >> + (filter include? (map remote-head-name (remote-ls remote)))))))) > > Use =E2=80=98filter-map=E2=80=99. > >> +(define* (get-version-mapping tags #:key prefix suffix delim pre-releas= es?) > > Please add a docstring and remove =E2=80=98get-=E2=80=99 from the name. = :-) > >> + (define (guess-delim) >> + (let ((total (length tags)) >> + (dots (reduce + 0 (map (cut string-count <> #\.) tags))) >> + (dashes (reduce + 0 (map (cut string-count <> #\-) tags))) >> + (underscores (reduce + 0 (map (cut string-count <> #\_) tags)= ))) >> + (display (format #t "total: ~d, dots: ~d, dashes ~d, underscores = ~d~%" >> + total dots dashes underscores)) > > Leftover? (Also display + format.) Yep. :-) > Please spell out =E2=80=98delimiter=E2=80=99 (info "(guix) Formatting Cod= e"). > >> + (cond >> + ((>=3D dots (* total 0.35)) ".") >> + ((>=3D dashes (* total 0.8)) "-") >> + ((>=3D underscores (* total 0.8)) "_") >> + (else "")))) > > That=E2=80=99s a fancy heuristic. :-) Yeah, it was suggested by Sarah, and in my testing it seems to work pretty well. :-) >> + (let ((mapping (fold alist-cons '() (map get-version tags) tags))) >> + (stable-sort! (filter car mapping) entry > It=E2=80=99s perhaps clearer written like this: > > (stable-sort (filter-map (lambda (tag) > (let ((version (get-version tag))) > (and version (cons version tag)))) > tags) > entry> +(define* (get-latest-tag url #:key prefix suffix delim pre-releases?) >> + "Return the latest tag available from the Git repository at URL." > > Maybe =E2=80=9Cthe tag corresponding to the latest version=E2=80=9D. Yeah, as the latest tag might not correspond to a release... > s/get-latest-tag/latest-tag/ > >> + (define (pre-release? tag) >> + (any (lambda (rx) (regexp-exec (make-regexp rx regexp/icase) tag)) >> + %pre-release-words)) > > Better call =E2=80=98make-regexp=E2=80=99 only once; so you could change > =E2=80=98%pre-release-words=E2=80=99 to be a list of regexp objects inste= ad of a list of > strings. Noted. >> +(define (latest-git-tag-version package tag-prefix tag-suffix >> + tag-version-delimiter refresh-pre-relea= ses?) >> + "Given a PACKAGE, the TAG-PREFIX, TAG-SUFFIX, TAG-VERSION-DELIMITER, = and >> +REFRESH-PRE-RELEASES? properties of PACKAGE, returns the latest versio= n of >> +PACKAGE." > > Maybe s/refresh-pre-releases?/accept-pre-preleases?/ =E2=80=98accept-pre-releases?=E2=80=99 ;-) > Since this procedure takes a package, it probably doesn=E2=80=99t need th= e other > arguments: it can extract them from the package properties, rather than > doing it at the call site. Good point. >> +(define (latest-git-release package) >> + "Return the latest release of PACKAGE." >> + (let* ((name (package-name package)) >> + (properties (package-properties package)) >> + (tag-prefix (assq-ref properties 'tag-prefix)) >> + (tag-suffix (assq-ref properties 'tag-suffix)) >> + (tag-version-delimiter (assq-ref properties 'tag-version-delim= iter)) >> + (refresh-pre-releases? (assq-ref properties 'refresh-pre-relea= ses?)) >> + (old-version (package-version package)) >> + (url (git-reference-url (origin-uri (package-source package)))) >> + (new-version (latest-git-tag-version package >> + tag-prefix >> + tag-suffix >> + tag-version-delimiter >> + refresh-pre-releases?))) >> + >> + (if new-version >> + (upstream-source >> + (package name) >> + (version new-version) >> + (urls (list url))) >> + ;; No new release or no tags available. >> + #f))) > > Simply: (and new-version (upstream-source =E2=80=A6)). > > It would have been nice to have tests. I think testing > =E2=80=98latest-git-release=E2=80=99 should be feasible without too much = hassle using > the (guix tests git) infrastructure, as is done in tests/git.scm, with a > package referring to a locally-created repo using a git-reference with a > file:// URL. Thanks for the pointers! I will look into it. > Could you send an updated patch? Sure! Thanks for the review! :-) --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmE7XGMVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5WHkQALL76RuYAn3al9OovUTR6OM0PY80 icGunDKkIgsAPba0SWL+3SmohhwfKwtaKlOn7+LSCdTNmV9D9USS/58oGCwQ8z+o HX3SCHLGhAP66pHPbT/LKv5AZL2yvt/bPP6N8VKGuwoihWF3zVMmH2yjDl8ux8se hN4Orie8H1b8u2g+edApe7CpZWIX8K2caxy7yDKuyh7YeMOMtPgroy/03o2ucA0m odIRZXuTr4m/YD6O6gqKn7tRaIfzFPIMfF2FiZqHAB+g+pdRMdLFxLFA5QaW+3+W J3fRe4YQA08lA/YOobb1sOVT357pT4jVpAK9N3E6yulNil+D/jQfCkAhDulCDRLP aEO6SWntd8lBBDOvWnbUSuvZvQOKNED9qIOj/OXYmcfNBRze2YUpmchLiGv12Mai xrQfp6xOplPtDTci5ZYYSp/Y8ygMn+a6ClzA6e6rdru2BMYVjpvGe1fjvzz7laeP dFBQOpiblgL7K/xVF/xia0LvHmjFKDpm6zwccGZ9p+F/cNUeU4Mh+so3Y4V/Zjvk PWjktr9Yx+yK5Ue9bbLkEyXPHUAzDWRzz8p85FcKcIOkWUTKFHRWdqZK3wlB8hJ/ 9MastbBgN3dIIC9fBTWA0MSPOQN0VtUt8pkFxSxuZiYzVsADTblUbASYB/zWKeAY 92cDAVngc5i36LDp =RW8F -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 10 12:20:37 2021 Received: (at 50359) by debbugs.gnu.org; 10 Sep 2021 16:20:37 +0000 Received: from localhost ([127.0.0.1]:39056 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOjGW-0005YI-Tu for submit@debbugs.gnu.org; Fri, 10 Sep 2021 12:20:37 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:37004 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOjGU-0005Y3-Q4 for 50359@debbugs.gnu.org; Fri, 10 Sep 2021 12:20:35 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631290825; bh=uT6ySe8o8ok3i8wheFT5owmUIcivPHnPmsMQpTOrNPU=; h=From:To:Subject:In-Reply-To:References:Date; b=cUWTgaHAeCQhBsIa+3dcc0F7MaGTJoTFuY8ldZgE8sQ9wusEe6E+oqWx+5AieRfYS /kTE19At5wznCfdBRbfAEa/skDb9+IGBrn/r3Tyv+2XZqFHVC9/14R7/ehS5/fWc8R gETl5RNU9d72UJzllf/kEIkaiTGefKAU+GdHFK/M= To: 50359@debbugs.gnu.org Subject: [PATCH 0/3] Add 'generic-git' updater. In-Reply-To: References: Message-Id: Date: Fri, 10 Sep 2021 18:20:23 +0200 MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Changes since v1: * Add ‘remote-refs’ procedure to (guix git) (written by Sarah Morgensen). Add tests for it too. * Make the updater try to guess the delimiter if none was provided (also written by Sarah). Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.9 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Changes since v1: * Add ‘remote-refs’ procedure to (guix git) (written by Sarah Morgensen). Add tests for it too. * Make the updater try to guess the delimiter if none was provided (also written by Sarah). Content analysis details: (1.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Changes since v1: * Add =E2=80=98remote-refs=E2=80=99 procedure to (guix git) (written by Sar= ah Morgensen). Add tests for it too. * Make the updater try to guess the delimiter if none was provided (also written by Sarah). * Honor the =E2=80=98accept-pre-releases?=E2=80=99 property to include pre-= releases when looking for tags. * Various regexp improvements. * Add tests for the updater. * Some fixes to (guix tests git). Xinglu Chen (3): tests: git: Don't read from the users global Git config file. tests: git: Make 'tag' directive non-interactive. import: Add 'generic-git' updater. Makefile.am | 2 + doc/guix.texi | 32 +++++++ guix/git.scm | 37 ++++++++ guix/import/git.scm | 218 +++++++++++++++++++++++++++++++++++++++++++ guix/tests/git.scm | 6 +- tests/git.scm | 26 ++++++ tests/import-git.scm | 204 ++++++++++++++++++++++++++++++++++++++++ 7 files changed, 523 insertions(+), 2 deletions(-) create mode 100644 guix/import/git.scm create mode 100644 tests/import-git.scm base-commit: 9875f9bca3976bf3576eab9be42164fde454597e =2D-=20 2.33.0 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmE7hccVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5fKwP/ilcewnTalQmrsgdluSaqBHUIReu PparJfqAWqkKbrwgQOxxo7Y9Qm+qqc6OPpp97IGJ2lciWtnnF5Dz9UOi37hfuNdi xJDWedHesXVOPx4UlHN71c0679MxW/kbmpnRBrmS64/dvzgBk5LYj3W9BxhJSSKa lm90431vkZrdn9yf7CCaF2yuYCJQ37Z1OD4EXbvJlwOj+urdJJPL2+bXMxOQK37b SOO/lRH0+d1/M2TOcKEnYTvak/3VkhNvLI3y4NYJj0OkEJnkYBMXiKQ8ualH/f17 UX7BAcZTg1Dm9G7/i0vafa9AjepHrzYTCJnjrXiU53ERi+8kwhO08ocFB47N8W+q 04DiWGtxCwIwEWvHHsdi4bn2+K3cVc3wFUiVcXaXpLOYeG31fQ2826Sj0Gl3BpCd vlG9uqQvYks2/REFJtL2D7ue7G0S16NdpjIayP6kL6r8INvMdwYNIoeIePpg/OsB ou/bMHhXGShnRon10Gx5O+1N4k+5o6OSZe/3/Bkx3Bt5YeazHaZSyTJHaCSYRPUn gBI6KAvUG62lfCATPgGLhUfvYiMgu079K9xQIvjYDhHiLCsGVudJVcywY8RGBeRe PK+dgpxreG7TEHPUH1g122UpK58r4LNx8o7DYp122uEqjzDYt5fOWh+ewMxKNW7y CEZsJ+Xc9rtMxTN/ =KcN9 -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 10 12:21:29 2021 Received: (at 50359) by debbugs.gnu.org; 10 Sep 2021 16:21:29 +0000 Received: from localhost ([127.0.0.1]:39064 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOjHN-0005an-64 for submit@debbugs.gnu.org; Fri, 10 Sep 2021 12:21:29 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:37028 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOjHK-0005aU-VP for 50359@debbugs.gnu.org; Fri, 10 Sep 2021 12:21:27 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631290879; bh=D1fehBRccifXBpnZDbd2xCB6H0KHAxZb9ggk4wYluzo=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=VzHe2oGMqjxV6gRWm3m2hRccE4+9farsMJ69LkE8NdVQ/jCig24BtW+Jc45ca8Bne l9lAfXeTPlbjaYyOoCFXoI+nRRfRyoU/Z8pOn2Gm1lGF/Ru3u1dDtaM0NPBbjU67gq rziRMTdkN9oeUilib174bvDBOf/Qd49Ko8cv7Ijg= To: 50359@debbugs.gnu.org Subject: [PATCH 1/3] tests: git: Don't read from the users global Git config file. In-Reply-To: References: Message-Id: <04db258c6eb1b90dca9f2b32a7bc40ed8e39c471.1631290349.git.public@yoctocell.xyz> Date: Fri, 10 Sep 2021 18:21:17 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: * guix/tests/git (populate-git-repository): Set the GIT_CONFIG_GLOBAL environment variable to the temporary Git config file. --- Without this, Git would try to sign the commits when running tests/git. [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , Sarah Morgensen X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.9 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: * guix/tests/git (populate-git-repository): Set the GIT_CONFIG_GLOBAL environment variable to the temporary Git config file. --- Without this, Git would try to sign the commits when running tests/git. [...] Content analysis details: (1.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps * guix/tests/git (populate-git-repository): Set the GIT_CONFIG_GLOBAL environment variable to the temporary Git config file. --- Without this, Git would try to sign the commits when running tests/git.scm, this was beacuse I had set =E2=80=98gpgSign =3D true=E2=80= =99 in ~/.config/git/config. Setting the environment variable would remove this impurity. guix/tests/git.scm | 2 ++ 1 file changed, 2 insertions(+) diff --git a/guix/tests/git.scm b/guix/tests/git.scm index b8e5f7e643..e11541e83b 100644 --- a/guix/tests/git.scm +++ b/guix/tests/git.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2019, 2020 Ludovic Court=C3=A8s +;;; Copyright =C2=A9 2021 Xinglu Chen ;;; ;;; This file is part of GNU Guix. ;;; @@ -53,6 +54,7 @@ Return DIRECTORY on success." (with-environment-variables `(("GIT_CONFIG_NOSYSTEM" "1") ("GIT_ATTR_NOSYSTEM" "1") + ("GIT_CONFIG_GLOBAL" ,(string-append home "/.gitconfig")) ("HOME" ,home)) (apply invoke (git-command) "-C" directory command args))))) --=20 2.33.0 From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 10 12:21:34 2021 Received: (at 50359) by debbugs.gnu.org; 10 Sep 2021 16:21:34 +0000 Received: from localhost ([127.0.0.1]:39067 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOjHS-0005b8-D9 for submit@debbugs.gnu.org; Fri, 10 Sep 2021 12:21:34 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:37040 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOjHQ-0005al-VA for 50359@debbugs.gnu.org; Fri, 10 Sep 2021 12:21:33 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631290885; bh=dcGafiE09F0lvz11QdAsd9lHlaLZq97KCXKZkOyaaJ4=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=a/iuZOJiR5lcoTA4R1FQ3KzfcJtqOguytFqboetM8r2KE+Wb71lGRIBQJqOA3IkyZ B48zgld6EkKMoBoY5Vfoqa/6uMeA1jnqbgou21If2wpUryGbotyeajk9O9QGn3016/ Ydmy08qRAsTKF1GvYv/f+I/ELc3q3twcGgOwvuRs= To: 50359@debbugs.gnu.org Subject: [PATCH 2/3] tests: git: Make 'tag' directive non-interactive. In-Reply-To: References: Message-Id: <29b4b579f8fe8fc3406a48224c5af28d9e70b865.1631290349.git.public@yoctocell.xyz> Date: Fri, 10 Sep 2021 18:21:25 +0200 MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: When running 'git tag TAGNAME', Git will open up the user's default text editor to make them write a message. This is not desirable when running tests. * guix/tests/git.scm (populate-git-repository): Make the 'tag' directive take an additional argument, and pass it to the '-m' flag. --- guix/tests/git.scm | 4 ++-- 1 file changed, 2 insertions(+), 2 d [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , Sarah Morgensen X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.9 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: When running 'git tag TAGNAME', Git will open up the user's default text editor to make them write a message. This is not desirable when running tests. * guix/tests/git.scm (populate-git-repository): Make the 'tag' directive take an additional argument, and pass it to the '-m' flag. --- guix/tests/git.scm | 4 ++-- 1 file changed, 2 insertions(+), 2 d [...] Content analysis details: (1.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps When running 'git tag TAGNAME', Git will open up the user's default text editor to make them write a message. This is not desirable when running tests. * guix/tests/git.scm (populate-git-repository): Make the 'tag' directive take an additional argument, and pass it to the '-m' flag. --- guix/tests/git.scm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/guix/tests/git.scm b/guix/tests/git.scm index e11541e83b..e8d4946e87 100644 --- a/guix/tests/git.scm +++ b/guix/tests/git.scm @@ -87,8 +87,8 @@ Return DIRECTORY on success." ((('commit text ('signer fingerprint)) rest ...) (git "commit" "-m" text (string-append "--gpg-sign=" fingerprint)) (loop rest)) - ((('tag name) rest ...) - (git "tag" name) + ((('tag name text) rest ...) + (git "tag" "-m" text name) (loop rest)) ((('branch name) rest ...) (git "branch" name) -- 2.33.0 From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 10 12:21:45 2021 Received: (at 50359) by debbugs.gnu.org; 10 Sep 2021 16:21:45 +0000 Received: from localhost ([127.0.0.1]:39070 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOjHc-0005bV-Ji for submit@debbugs.gnu.org; Fri, 10 Sep 2021 12:21:45 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:37048 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mOjHW-0005az-6X for 50359@debbugs.gnu.org; Fri, 10 Sep 2021 12:21:43 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631290890; bh=oLzOkXkhGcUbIClE8yUyURsLSov7O8LsDzC18JfE0lI=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=frdrsRGRkIXVUBikuAd5Xo1n4l49fHjJ5/tzM+W0Jhmg1y9beC9KJQvOFLQRQ5WiD Si/I4zdrAJ0yWIdmAPViXmcr7mbIGUfdD2hRCQcoe4KA32jsUvUcSWkwM8+PDv8eE1 ADVs1bM5GD/qR911pWLNR9pvX1sdXpmZA6ZNjapA= To: 50359@debbugs.gnu.org Subject: [PATCH 3/3] import: Add 'generic-git' updater. In-Reply-To: References: Message-Id: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> Date: Fri, 10 Sep 2021 18:21:30 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: * guix/git.scm (ls-remote-refs): New procedure. * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. * guix/import/git.scm: New file. * doc/guix.texi (Invoking guix refresh): Document [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 T_SPF_HELO_TEMPERROR SPF: test of HELO record failed (temperror) 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , Sarah Morgensen X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.9 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: * guix/git.scm (ls-remote-refs): New procedure. * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. * guix/import/git.scm: New file. * doc/guix.texi (Invoking guix refresh): Document [...] Content analysis details: (1.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps * guix/git.scm (ls-remote-refs): New procedure. * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. * guix/import/git.scm: New file. * doc/guix.texi (Invoking guix refresh): Document it. * tests/import-git.scm: New test file. * Makefile.am (MODULES, SCM_TESTS): Register the new files. Co-authored-by: Sarah Morgensen --- Makefile.am | 2 + doc/guix.texi | 32 +++++++ guix/git.scm | 37 ++++++++ guix/import/git.scm | 218 +++++++++++++++++++++++++++++++++++++++++++ tests/git.scm | 26 ++++++ tests/import-git.scm | 204 ++++++++++++++++++++++++++++++++++++++++ 6 files changed, 519 insertions(+) create mode 100644 guix/import/git.scm create mode 100644 tests/import-git.scm diff --git a/Makefile.am b/Makefile.am index dd40a5ad9c..c71d9a29e2 100644 --- a/Makefile.am +++ b/Makefile.am @@ -254,6 +254,7 @@ MODULES =3D \ guix/import/egg.scm \ guix/import/elpa.scm \ guix/import/gem.scm \ + guix/import/git.scm \ guix/import/github.scm \ guix/import/gnome.scm \ guix/import/gnu.scm \ @@ -473,6 +474,7 @@ SCM_TESTS =3D \ tests/graph.scm \ tests/gremlin.scm \ tests/hackage.scm \ + tests/import-git.scm \ tests/import-utils.scm \ tests/inferior.scm \ tests/lint.scm \ diff --git a/doc/guix.texi b/doc/guix.texi index 220499503d..dbaa000006 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -11921,6 +11921,38 @@ the updater for @uref{https://launchpad.net, Launc= hpad} packages. @item generic-html a generic updater that crawls the HTML page where the source tarball of the package is hosted, when applicable. + +@item generic-git +a generic updater for packages hosted on Git repositories. It tries to +be smart about parsing Git tag names, but if it is not able to parse the +tag name and compare tags correctly, users can define the following +properties for a package. + +@itemize +@item @code{release-tag-prefix}: a regular expression for matching a prefi= x of +the tag name. + +@item @code{release-tag-suffix}: a regular expression for matching a suffi= x of +the tag name. + +@item @code{release-tag-version-delimiter}: a string used as the delimiter= in +the tag name for separating the numbers of the version. +@end itemize + +@lisp +(package + (name "foo") + ;; ... + (properties + '((release-tag-prefix . "^release0-") + (release-tag-suffix . "[a-z]?$") + (release-tag-version-delimiter . ":")))) +@end lisp + +By default, the updater will ignore pre-releases; to make it also look +for pre-releases, set the @code{accept-pre-releases?} property to +@code{#t}. + @end table =20 For instance, the following command only checks for updates of Emacs diff --git a/guix/git.scm b/guix/git.scm index acc48fd12f..dc3d3afd02 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -57,6 +57,8 @@ commit-difference commit-relation =20 + remote-refs + git-checkout git-checkout? git-checkout-url @@ -571,6 +573,41 @@ objects: 'ancestor (meaning that OLD is an ancestor of= NEW), 'descendant, or (if (set-contains? oldest new) 'descendant 'unrelated)))))) + +;; +;;; Remote operations. +;;; + +(define* (remote-refs url #:key tags?) + "Return the list of references advertised at Git repository URL. If TAG= S? +is true, limit to only refs/tags." + (define (ref? ref) + ;; Like `git ls-remote --refs', only show actual references. + (and (string-prefix? "refs/" ref) + (not (string-suffix? "^{}" ref)))) + + (define (tag? ref) + (string-prefix? "refs/tags/" ref)) + + (define (include? ref) + (and (ref? ref) + (or (not tags?) (tag? ref)))) + + (with-libgit2 + (call-with-temporary-directory + (lambda (cache-directory) + (let* ((repository (repository-init cache-directory)) + ;; Create an in-memory remote so we don't touch disk. + (remote (remote-create-anonymous repository url))) + (remote-connect remote) + (remote-disconnect remote) + (repository-close! repository) + + (filter-map (lambda (remote) + (let ((name (remote-head-name remote))) + (and (include? name) + name))) + (remote-ls remote))))))) =20 ;;; diff --git a/guix/import/git.scm b/guix/import/git.scm new file mode 100644 index 0000000000..b69f9d70f2 --- /dev/null +++ b/guix/import/git.scm @@ -0,0 +1,218 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix import git) + #:use-module (guix build utils) + #:use-module (guix diagnostics) + #:use-module (guix git) + #:use-module (guix git-download) + #:use-module (guix i18n) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module (ice-9 format) + #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 regex) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) + #:use-module (srfi srfi-34) + #:use-module (srfi srfi-35) + #:export (%generic-git-updater + + ;; For tests. + latest-git-tag-version)) + +;;; Commentary: +;;; +;;; This module provides a generic package updater for packages hosted on = Git +;;; repositories. +;;; +;;; It tries to be smart about tag names, but if it is not automatically a= ble +;;; to parse the tag names correctly, users can set the `release-tag-prefi= x', +;;; `release-tag-suffix' and `release-tag-version-delimiter' properties of= the +;;; package to make the updater parse the Git tag name correctly. +;;; +;;; Code: + +;;; Errors & warnings + +(define-condition-type &git-no-valid-tags-error &error + git-no-valid-tags-error?) + +(define (git-no-valid-tags-error) + (raise (condition (&message (message "no valid tags found")) + (&git-no-valid-tags-error)))) + +(define-condition-type &git-no-tags-error &error + git-no-tags-error?) + +(define (git-no-tags-error) + (raise (condition (&message (message "no tags were found")) + (&git-no-tags-error)))) + + +;;; Updater + +(define %pre-release-words + '("alpha" "beta" "rc" "dev" "test")) + +(define %pre-release-rx + (map (cut make-regexp <> regexp/icase) %pre-release-words)) + +(define* (version-mapping tags #:key prefix suffix delim pre-releases?) + "Given a list of Git TAGS, return a association list where the car is the +version corresponding to the tag, and the cdr is the name of the tag." + (define (guess-delimiter) + (let ((total (length tags)) + (dots (reduce + 0 (map (cut string-count <> #\.) tags))) + (dashes (reduce + 0 (map (cut string-count <> #\-) tags))) + (underscores (reduce + 0 (map (cut string-count <> #\_) tags)))) + (cond + ((>=3D dots (* total 0.35)) ".") + ((>=3D dashes (* total 0.8)) "-") + ((>=3D underscores (* total 0.8)) "_") + (else "")))) + + (define delim-rx (regexp-quote (or delim (guess-delimiter)))) + (define suffix-rx (string-append (or suffix "") "$")) + +=20=20 + (define prefix-rx (string-append "^" (or prefix "[^[:digit:]]*"))) + (define pre-release-rx + (if pre-releases? + (string-append ".*(" (string-join %pre-release-words "|") ").*") + "")) + + (define tag-rx + (string-append prefix-rx "([[:digit:]][^" delim-rx "[:punct:]]*" + "(" delim-rx "[^[:punct:]" delim-rx "]+)" + ;; If there is are no delimiters, it could mean that the + ;; version just contains one number (e.g., "2"), thus, = use + ;; "*" instead of "+" to match zero or more numbers. + (if (string=3D? delim-rx "") "*" "+") + pre-release-rx ")" suffix-rx)) + + (define (get-version tag) + (let ((tag-match (regexp-exec (make-regexp tag-rx) tag))) + (and tag-match + (regexp-substitute/global + #f delim-rx (match:substring tag-match 1) + ;; Don't insert "." if there aren't any delimiters in the first + ;; place. + 'pre (if (string=3D? delim-rx "") "" ".") 'post)))) + + (define (entry tag) + %pre-release-rx)) + + (let* ((tags (map (cut string-drop <> (string-length "refs/tags/")) + (remote-refs url #:tags? #t))) + (versions->tags + (version-mapping (if pre-releases? + tags + (filter (negate pre-release?) tags)) + #:prefix prefix + #:suffix suffix + #:delim delim + #:pre-releases? pre-releases?))) + (cond + ((null? tags) + (git-no-tags-error)) + ((null? versions->tags) + (git-no-valid-tags-error)) + (else + (match (last versions->tags) + ((version . tag) + (values version tag))))))) + +(define (latest-git-tag-version package) + "Given a PACKAGE, return the latest version of it, or #f if the latest v= ersion +could not be determined." + (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c)) + (warning (or (package-field-location package 'source) + (package-location package)) + (G_ "~a for ~a~%") + (condition-message c) + (package-name package)) + #f) + ((eq? (exception-kind c) 'git-error) + (warning (or (package-field-location package 'source) + (package-location package)) + (G_ "failed to fetch Git repository for ~a~%") + (package-name package)) + #f)) + (let* ((source (package-source package)) + (url (git-reference-url (origin-uri source))) + (properties (package-properties package)) + (tag-prefix (assq-ref properties 'release-tag-prefix)) + (tag-suffix (assq-ref properties 'release-tag-suffix)) + (tag-version-delimiter (assq-ref properties 'release-tag-versio= n-delimiter)) + (refresh-pre-releases? (assq-ref properties 'accept-pre-release= s?))) + (latest-tag url + #:prefix tag-prefix + #:suffix tag-suffix + #:delim tag-version-delimiter + #:pre-releases? refresh-pre-releases?)))) + +(define (git-package? package) + "Whether the origin of PACKAGE is a Git repostiory." + (match (package-source package) + ((? origin? origin) + (and (eq? (origin-method origin) git-fetch) + (git-reference? (origin-uri origin)))) + (_ #f))) + +(define (latest-git-release package) + "Return the latest release of PACKAGE." + (let* ((name (package-name package)) + (old-version (package-version package)) + (url (git-reference-url (origin-uri (package-source package)))) + (new-version (latest-git-tag-version package))) + + (and new-version + (upstream-source + (package name) + (version new-version) + (urls (list url)))))) + +(define %generic-git-updater + (upstream-updater + (name 'generic-git) + (description "Updater for packages hosted on Git repositories") + (pred git-package?) + (latest latest-git-release))) diff --git a/tests/git.scm b/tests/git.scm index aa4f03ca62..1f4fbb9adb 100644 --- a/tests/git.scm +++ b/tests/git.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2019, 2020 Ludovic Court=C3=A8s +;;; Copyright =C2=A9 2021 Xinglu Chen . + +(define-module (test-import-git) + #:use-module (git) + #:use-module (guix git) + #:use-module (guix tests) + #:use-module (guix packages) + #:use-module (guix import git) + #:use-module (guix git-download) + #:use-module (guix tests git) + #:use-module (guix build utils) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-64)) + +;; Test the (guix import git) tools. + +(test-begin "git") + +(define* (make-package directory version #:optional (properties '())) + (dummy-package "test-package" + (version version) + (properties properties) + (source + (origin + (method git-fetch) + (uri (git-reference + (url (string-append "file://" directory)) + (commit version))) + (sha256 + (base32 + "0000000000000000000000000000000000000000000000000000")))))) + +(test-equal "latest-git-tag-version: no custom prefix, suffix, and delimit= er" + "1.0.1" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "1.0.1" "Release 1.0.1")) + (let ((package (make-package directory "1.0.0"))) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: custom prefix, no suffix and delimite= r" + "1.0.1" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "prefix-1.0.1" "Release 1.0.1")) + (let ((package (make-package directory "1.0.0" + '((release-tag-prefix . "prefix-"))))) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: custom suffix, no prefix and delimite= r" + "1.0.1" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "1.0.1-suffix-123" "Release 1.0.1")) + (let ((package (make-package directory "1.0.0" + '((release-tag-suffix . "-suffix-[0-9]*")= )))) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: custom delimiter, no prefix and suffi= x" + "2021.09.07" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "2021-09-07" "Release 2021-09-07")) + (let ((package (make-package directory "2021-09-06" + '((release-tag-version-delimiter . "-")))= )) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: empty delimiter, no prefix and suffix" + "20210907" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "20210907" "Release 20210907")) + (let ((package (make-package directory "20210906" + '((release-tag-version-delimiter . ""))))) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: custom prefix and suffix, no delimite= r" + "2.0.0" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "Release-2.0.0suffix-1" "Release 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((release-tag-prefix . "Release-") + (release-tag-suffix . "suffix-[0-9]")))= )) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: custom prefix, suffix, and delimiter" + "2.0.0" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "Release-2_0_0suffix-1" "Release 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((release-tag-prefix . "Release-") + (release-tag-suffix . "suffix-[0-9]") + (release-tag-version-delimiter . "_")))= )) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: only pre-releases available" + #f + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "2.0.0-rc1" "Release candidate for 2.0.0")) + (let ((package (make-package directory "1.0.0"))) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: accept pre-releases" + "2.0.0-rc1" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "2.0.0-rc1" "Release candidate for 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((accept-pre-releases? . #t))))) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: accept pre-releases, and custom prefi= x" + "2.0.0-rc1" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "version-2.0.0-rc1" "Release candidate for 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((accept-pre-releases? . #t) + (release-tag-prefix . "version-"))))) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: accept pre-releases, and custom suffi= x" + "2.0.0-rc1" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "2.0.0-rc1-suffix" "Release candidate for 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((accept-pre-releases? . #t) + (release-tag-suffix . "-suffix"))))) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: accept pre-releases, and custom suffi= x and prefix" + "2.0.0-alpha" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "prefix123-2.0.0-alpha-suffix" "Alpha release for 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((accept-pre-releases? . #t) + (release-tag-prefix . "prefix[0-9]{3}-") + (release-tag-suffix . "-suffix"))))) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: accept pre-releases, and custom suffi= x, prefix, and delimiter" + "2.0.0-alpha" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "prefix123-2_0_0-alpha-suffix" "Alpha release for 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((accept-pre-releases? . #t) + (release-tag-prefix . "prefix[0-9]{3}-") + (release-tag-suffix . "-suffix") + (release-tag-version-delimiter . "_")))= )) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: no tags found" + #f + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit")) + (let ((package (make-package directory "1.0.0"))) + (latest-git-tag-version package)))) + +(test-equal "latest-git-tag-version: no valid tags found" + #f + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "Test" "Test tag")) + (let ((package (make-package directory "1.0.0"))) + (latest-git-tag-version package)))) + +(test-end "git") --=20 2.33.0 From debbugs-submit-bounces@debbugs.gnu.org Mon Sep 13 04:03:31 2021 Received: (at 50359) by debbugs.gnu.org; 13 Sep 2021 08:03:32 +0000 Received: from localhost ([127.0.0.1]:44163 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mPgw7-00036e-MO for submit@debbugs.gnu.org; Mon, 13 Sep 2021 04:03:31 -0400 Received: from eggs.gnu.org ([209.51.188.92]:60514) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mPgw5-00036L-NA for 50359@debbugs.gnu.org; Mon, 13 Sep 2021 04:03:30 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:57106) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mPgvy-0008Dp-SV; Mon, 13 Sep 2021 04:03:22 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=47238 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mPgvy-00017K-Gc; Mon, 13 Sep 2021 04:03:22 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Xinglu Chen Subject: Re: [PATCH 2/3] tests: git: Make 'tag' directive non-interactive. References: <29b4b579f8fe8fc3406a48224c5af28d9e70b865.1631290349.git.public@yoctocell.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 27 Fructidor an 229 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 13 Sep 2021 10:03:20 +0200 In-Reply-To: <29b4b579f8fe8fc3406a48224c5af28d9e70b865.1631290349.git.public@yoctocell.xyz> (Xinglu Chen's message of "Fri, 10 Sep 2021 18:21:25 +0200") Message-ID: <87bl4wc37b.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.3 (/) X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org, Sarah Morgensen X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.3 (-) Hello, Xinglu Chen skribis: > When running 'git tag TAGNAME', Git will open up the user's default text > editor to make them write a message. This is not desirable when running > tests. > > * guix/tests/git.scm (populate-git-repository): Make the 'tag' directive = take > an additional argument, and pass it to the '-m' flag. [...] > - ((('tag name) rest ...) > - (git "tag" name) > + ((('tag name text) rest ...) > + (git "tag" "-m" text name) LGTM, but you need to update at least tests/channels.scm accordingly. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Mon Sep 13 04:07:13 2021 Received: (at 50359) by debbugs.gnu.org; 13 Sep 2021 08:07:13 +0000 Received: from localhost ([127.0.0.1]:44175 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mPgzg-0003Cw-MV for submit@debbugs.gnu.org; Mon, 13 Sep 2021 04:07:12 -0400 Received: from eggs.gnu.org ([209.51.188.92]:33508) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mPgze-0003Ci-Du for 50359@debbugs.gnu.org; Mon, 13 Sep 2021 04:07:11 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:57238) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mPgzY-0003AR-Py; Mon, 13 Sep 2021 04:07:04 -0400 Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=47240 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mPgzY-00029Y-Fx; Mon, 13 Sep 2021 04:07:04 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Xinglu Chen Subject: Re: [PATCH 3/3] import: Add 'generic-git' updater. References: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 27 Fructidor an 229 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Mon, 13 Sep 2021 10:07:03 +0200 In-Reply-To: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> (Xinglu Chen's message of "Fri, 10 Sep 2021 18:21:30 +0200") Message-ID: <875yv4c314.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.3 (/) X-Debbugs-Envelope-To: 50359 Cc: 50359@debbugs.gnu.org, Sarah Morgensen X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.3 (-) Xinglu Chen skribis: > * guix/git.scm (ls-remote-refs): New procedure. > * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. > * guix/import/git.scm: New file. > * doc/guix.texi (Invoking guix refresh): Document it. > * tests/import-git.scm: New test file. > * Makefile.am (MODULES, SCM_TESTS): Register the new files. > > Co-authored-by: Sarah Morgensen Nice, thanks for writing the tests! > +++ b/tests/git.scm > @@ -1,5 +1,6 @@ > ;;; GNU Guix --- Functional package management for GNU > ;;; Copyright =C2=A9 2019, 2020 Ludovic Court=C3=A8s > +;;; Copyright =C2=A9 2021 Xinglu Chen ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -161,4 +162,29 @@ > (commit-relation master1 merge) > (commit-relation merge master1)))))) >=20=20 > +(test-equal "remote-refs" > + '("refs/heads/develop" "refs/heads/master" > + "refs/tags/v1.0" "refs/tags/v1.1") > + (with-temporary-git-repository directory > + '((add "a.txt" "A") > + (commit "First commit") > + (tag "v1.0" "release-1.0") > + (branch "develop") > + (checkout "develop") > + (add "b.txt" "B") > + (commit "Second commit") > + (tag "v1.1" "release-1.1")) > + (remote-refs directory))) [...] > +(test-equal "latest-git-tag-version: no custom prefix, suffix, and delim= iter" > + "1.0.1" > + (with-temporary-git-repository directory > + '((add "a.txt" "A") > + (commit "First commit") > + (tag "1.0.1" "Release 1.0.1")) > + (let ((package (make-package directory "1.0.0"))) > + (latest-git-tag-version package)))) I think that for each of these tests that uses the =E2=80=98git=E2=80=99 co= mmand under the hood, you=E2=80=99ll need something like what =E2=80=98tests/git.scm=E2= =80=99 does: (unless (which (git-command)) (test-skip 1)) (test-equal =E2=80=A6) It=E2=80=99d admittedly annoying to have this boilerplate, but I can=E2=80= =99t think of a better solution. Could you send an updated version? Then we=E2=80=99ll be all set! Thank you, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Wed Sep 15 04:44:41 2021 Received: (at 50359) by debbugs.gnu.org; 15 Sep 2021 08:44:41 +0000 Received: from localhost ([127.0.0.1]:50831 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mQQX3-0000r5-4S for submit@debbugs.gnu.org; Wed, 15 Sep 2021 04:44:41 -0400 Received: from out2.migadu.com ([188.165.223.204]:54869) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mQQX0-0000qv-SA for 50359@debbugs.gnu.org; Wed, 15 Sep 2021 04:44:40 -0400 MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1631695477; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/560hck3gl4aDjQNlJGLO5RXT9aQriQgkBb0VWZc53k=; b=jfkmYfJ1AQBid/roEbS9JshRdMUEFvHjvLCgRARFtMi/kw+BRKy5cn3firqZgVzdUwpimg 5XbGgm/JDzgdRwSLpcHra562cqhHA0npRoJyFNSh6IDWSwmHHhoS6QGpLU/3QM/bs0NUvp QLEk/kuwseIn+BzK6ni78bmNSW6u3P4= Date: Wed, 15 Sep 2021 08:44:35 +0000 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: iskarian@mgsn.dev Message-ID: <44519ae5dfdf3702dc73903b80314a6a@mgsn.dev> Subject: Re: [PATCH 3/3] import: Add 'generic-git' updater. To: "Xinglu Chen" , 50359@debbugs.gnu.org In-Reply-To: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> References: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: iskarian@mgsn.dev X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi, September 10, 2021 9:21 AM, "Xinglu Chen" wrote: > * guix/git.scm (ls-remote-refs): New procedure. > * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. > * guix/import/git.scm: New file. > * doc/guix.texi (Invoking guix refresh): D [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [188.165.223.204 listed in wl.mailspike.net] -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-Debbugs-Envelope-To: 50359 Cc: =?utf-8?B?THVkb3ZpYyBDb3VydMOocw==?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) Hi,=0A=0ASeptember 10, 2021 9:21 AM, "Xinglu Chen" = wrote:=0A=0A> * guix/git.scm (ls-remote-refs): New procedure.=0A> * test= s/git.scm ("remote-refs" "remote-refs: only tags"): New tests.=0A> * guix= /import/git.scm: New file.=0A> * doc/guix.texi (Invoking guix refresh): D= ocument it.=0A> * tests/import-git.scm: New test file.=0A> * Makefile.am = (MODULES, SCM_TESTS): Register the new files.=0A> =0A> Co-authored-by: Sa= rah Morgensen =0A=0AOverall this is looking good. Tha= nk you for adding tests (for remote-refs as well!), much appreciated. It= looks like you've done some good polishing. I see a few nits, which I'l= l point out in a separate email when I'm not on mobile. I'll also give i= t a good test.=0A=0ABut... I've been thinking about the overall approach = for a couple days, because I'm not very happy with the complexity of my h= euristic.=0A=0AThere can be a lot of weird tags in a repository--look at = the one for xf86-video-intel for example. My heuristic attempts to captu= re the assumption that repostories tend to move from using "_" or "-" to = "." but it does fail to account for moving to or from dates (because date= s don't compare with normal versions).=0A=0AI also realized that we are n= ot using a very useful piece of information--the previous version/tag com= bo. I expect that in the vast majority of cases, the version delimiter f= or the newest version will be the same as the version delimiter for the l= ast known version. (Perhaps the prefix as well?) Can we use this inform= ation to make our guesses better? What do you think?=0A=0ADespite saying= all that, it's probably better to not try to get it perfect on the first= go--we can always adjust the internals later. We just want to avoid sho= wing bogus updates.=0A=0A(Later, I think I'll put together a dataset of t= ags and current versions to see if I can test how well a particular algor= ithm works.)=0A=0A--=0ASarah (mobile) From debbugs-submit-bounces@debbugs.gnu.org Wed Sep 15 07:59:35 2021 Received: (at 50359) by debbugs.gnu.org; 15 Sep 2021 11:59:36 +0000 Received: from localhost ([127.0.0.1]:51174 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mQTZf-0008FS-Ko for submit@debbugs.gnu.org; Wed, 15 Sep 2021 07:59:35 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:55490 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mQTZe-0008FG-Gz for 50359@debbugs.gnu.org; Wed, 15 Sep 2021 07:59:35 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631707166; bh=ZX+NNxys7tt8H0ouS4ghq4wpDvK3m7uJ8o1mi9v3u2U=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=lu6s1iYmRv9l0rnC5t6Kp55+Wt7/bx63x9/mxrGxAgWr78LZi2/zMXEKkbRjbs3oj SzaxF6CyB9VH5IaWFFK/Yb4xsatvi7EVYNxFcrHKbv/cOjP4cE3TR5n3RKOyBgWq4U oJ9FQ2FDqkxiKScu9T8mhYdTBqLrRH+O8BXmgePM= To: iskarian@mgsn.dev, 50359@debbugs.gnu.org Subject: Re: [bug#50359] [PATCH 3/3] import: Add 'generic-git' updater. In-Reply-To: <44519ae5dfdf3702dc73903b80314a6a@mgsn.dev> References: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> <44519ae5dfdf3702dc73903b80314a6a@mgsn.dev> Date: Wed, 15 Sep 2021 13:59:22 +0200 Message-ID: <87sfy6rqw5.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Wed, Sep 15 2021, iskarian@mgsn.dev wrote: > Hi, > > September 10, 2021 9:21 AM, "Xinglu Chen" wrote: > >> * guix/git.scm (ls-remote-refs): New procedure. >> * tests/git.scm ("remote-refs" "remote-refs: only tags"): New [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Wed, Sep 15 2021, iskarian@mgsn.dev wrote: > Hi, > > September 10, 2021 9:21 AM, "Xinglu Chen" wrote: > >> * guix/git.scm (ls-remote-refs): New procedure. >> * tests/git.scm ("remote-refs" "remote-refs: only tags"): New [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Wed, Sep 15 2021, iskarian@mgsn.dev wrote: > Hi, > > September 10, 2021 9:21 AM, "Xinglu Chen" wrote: > >> * guix/git.scm (ls-remote-refs): New procedure. >> * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. >> * guix/import/git.scm: New file. >> * doc/guix.texi (Invoking guix refresh): Document it. >> * tests/import-git.scm: New test file. >> * Makefile.am (MODULES, SCM_TESTS): Register the new files. >>=20 >> Co-authored-by: Sarah Morgensen > > Overall this is looking good. Thank you for adding tests (for > remote-refs as well!), much appreciated. It looks like you've done > some good polishing. I see a few nits, which I'll point out in a > separate email when I'm not on mobile. I'll also give it a good test. > > But... I've been thinking about the overall approach for a couple > days, because I'm not very happy with the complexity of my heuristic. > > There can be a lot of weird tags in a repository--look at the one for > xf86-video-intel for example. My heuristic attempts to capture the > assumption that repostories tend to move from using "_" or "-" to "." > but it does fail to account for moving to or from dates (because dates > don't compare with normal versions). But if a repo moved from using versions to tags, or vice-versa, we still wouldn=E2=80=99t know if say =E2=80=9C3.0.1=E2=80=9D is newer than =E2=80= =9C2021.03.02=E2=80=9D. We would have to know when the =E2=80=9C3.0.1=E2=80=9D tag was created. Maybe we could have a =E2=80=98release-tag-date-scheme?=E2=80=99 property, = that way we could just try to match dates? > I also realized that we are not using a very useful piece of > information--the previous version/tag combo. I expect that in the > vast majority of cases, the version delimiter for the newest version > will be the same as the version delimiter for the last known version. > (Perhaps the prefix as well?) Can we use this information to make our > guesses better? What do you think? That sounds like a good idea. What should happen if the delimiter from the previous version/tag combo is different from the one that the =E2=80=98guess-delimiter=E2=80=99 procedure returns? Should the one from t= he previous version/tag combo take precedence. > Despite saying all that, it's probably better to not try to get it > perfect on the first go--we can always adjust the internals later. We > just want to avoid showing bogus updates. Yeah, we can always improve things later. =20=20 > (Later, I think I'll put together a dataset of tags and current > versions to see if I can test how well a particular algorithm works.) Cool, looking forward to that. :-) --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmFB4BoVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5F/sQAKkyjf7qyW/HkRcAfoZG3oGyTfLi 5DNEy2XtqrxV07gZNy9pJOhdz+x+utSchutn33xRjJLo0N5eg9uj9FNTyjVsTpMV 6P31JCLU3fB6fQZiStONuHTksTLP3YrKoNMR31BYRCQ7EQ6EysxAbCLApcz/E+vn jHQvPcLxIeZj4NfTw9qH3jCpk3JKEn+fyEu50H/XXxE/loXVXCSGxP1aWKSCo0Ii +TpFS6CSYLCHjxW5M0j0JOVrFliFhqUV3L+EKr6ooTDt0FmZyWf1/iD5DBH9D+Vm 0iD8wiZ0sjRRqL8955en44NnEU+g63iTBdR3UvX8pOLQZn/SbBHLzQdi3Gpvx2HF BTvEm3wmsRQlt18e5yyn43wmxofJfksQI6/sYEF8ULDWzqC3rKiZL1r7Vt2+I9pn 1vW7YC8Zy9mYWYx9vDdWai8i9ltIrxXkSSZKmiLXDlNB0cq1LSDoQv4OQXmuUiMA rcilETL3RNT1RdeFGGzPhmTHEHlMnpZG1ZCtgUiG8tjJkRRSz2fvL25M09i89SLK 9pLGUYwXoYJeykQ7rhcUnLPekHzPveC+AnY7JbpFwpXqlVI7wTNceVG4NXXxjEY0 vRxNiMGrSwPg1K7uRTDangD5rLFndt7nq5ANhUj1B4p30jiemoWwQM5xJsAO3ER0 b1afQNrw2MAsEgk3 =5CXA -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 16 05:09:49 2021 Received: (at 50359) by debbugs.gnu.org; 16 Sep 2021 09:09:49 +0000 Received: from localhost ([127.0.0.1]:53807 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mQnOu-0007OJ-Nf for submit@debbugs.gnu.org; Thu, 16 Sep 2021 05:09:49 -0400 Received: from out0.migadu.com ([94.23.1.103]:33137) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mQnOn-0007O5-Qm for 50359@debbugs.gnu.org; Thu, 16 Sep 2021 05:09:47 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1631783379; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=+4Td9ov4xszFVhGp2zWH36ghExMeFtjA/PLoyjKvFGo=; b=V1lRUYIkBVYPIrNh38PMkwtxAIO5d5rpIfzBP5kiJSTcrI5yvZNPLwlxByXbq2G9TMM0Oh Jj4a1F1YrMIfNojwBp02Dmee8k1Wg1nVIo5liypWQKYGqJMY2MVhptVi1wXe9mbp5WAbuq m947wjhYsSNEOLyZq5z/tYdvmJVSZ0Q= From: Sarah Morgensen To: Xinglu Chen Subject: Re: [bug#50359] [PATCH 3/3] import: Add 'generic-git' updater. References: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> Date: Thu, 16 Sep 2021 02:09:32 -0700 In-Reply-To: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> (Xinglu Chen's message of "Fri, 10 Sep 2021 18:21:30 +0200 (5 days, 15 hours, 34 minutes ago)") Message-ID: <86czp8j38z.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: text/plain X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: iskarian@mgsn.dev X-Spam-Score: 1.3 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hello, Here are my promised nits. Xinglu Chen writes: Content analysis details: (1.3 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [94.23.1.103 listed in wl.mailspike.net] -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/, low trust [94.23.1.103 listed in list.dnswl.org] 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.0 T_SPF_TEMPERROR SPF: test of record failed (temperror) X-Debbugs-Envelope-To: 50359 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.3 (/) Hello, Here are my promised nits. Xinglu Chen writes: > * guix/git.scm (ls-remote-refs): New procedure. > * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. > * guix/import/git.scm: New file. > * doc/guix.texi (Invoking guix refresh): Document it. > * tests/import-git.scm: New test file. > * Makefile.am (MODULES, SCM_TESTS): Register the new files. > > Co-authored-by: Sarah Morgensen Again, much thanks for writing tests. > +@item generic-git > +a generic updater for packages hosted on Git repositories. It tries to > +be smart about parsing Git tag names, but if it is not able to parse the > +tag name and compare tags correctly, users can define the following > +properties for a package. > + > +@itemize > +@item @code{release-tag-prefix}: a regular expression for matching a prefix of > +the tag name. > + > +@item @code{release-tag-suffix}: a regular expression for matching a suffix of > +the tag name. > + > +@item @code{release-tag-version-delimiter}: a string used as the delimiter in > +the tag name for separating the numbers of the version. > +@end itemize > + > +@lisp > +(package > + (name "foo") > + ;; ... > + (properties > + '((release-tag-prefix . "^release0-") > + (release-tag-suffix . "[a-z]?$") > + (release-tag-version-delimiter . ":")))) > +@end lisp > + > +By default, the updater will ignore pre-releases; to make it also look > +for pre-releases, set the @code{accept-pre-releases?} property to > +@code{#t}. Should this be itemized above? > + > +;; > +;;; Remote operations. > +;;; > + > +(define* (remote-refs url #:key tags?) > + "Return the list of references advertised at Git repository URL. If TAGS? > +is true, limit to only refs/tags." > + (define (ref? ref) > + ;; Like `git ls-remote --refs', only show actual references. > + (and (string-prefix? "refs/" ref) > + (not (string-suffix? "^{}" ref)))) > + > + (define (tag? ref) > + (string-prefix? "refs/tags/" ref)) > + > + (define (include? ref) > + (and (ref? ref) > + (or (not tags?) (tag? ref)))) > + > + (with-libgit2 > + (call-with-temporary-directory > + (lambda (cache-directory) > + (let* ((repository (repository-init cache-directory)) > + ;; Create an in-memory remote so we don't touch disk. > + (remote (remote-create-anonymous repository url))) > + (remote-connect remote) > + (remote-disconnect remote) > + (repository-close! repository) > + > + (filter-map (lambda (remote) > + (let ((name (remote-head-name remote))) > + (and (include? name) > + name))) > + (remote-ls remote))))))) I discovered that this can segfault unless 'remote-disconnect' and possibly 'repository-close!' are called *after* copying the data out. I've attached a diff for this. > + > +;;; Updater > + > +(define %pre-release-words > + '("alpha" "beta" "rc" "dev" "test")) I found a few packages that use "pre" as well. > + > +(define %pre-release-rx > + (map (cut make-regexp <> regexp/icase) %pre-release-words)) > + > +(define* (version-mapping tags #:key prefix suffix delim pre-releases?) > + "Given a list of Git TAGS, return a association list where the car is the ^ an > +version corresponding to the tag, and the cdr is the name of the tag." > + (define (guess-delimiter) > + (let ((total (length tags)) > + (dots (reduce + 0 (map (cut string-count <> #\.) tags))) > + (dashes (reduce + 0 (map (cut string-count <> #\-) tags))) > + (underscores (reduce + 0 (map (cut string-count <> #\_) tags)))) > + (cond > + ((>= dots (* total 0.35)) ".") > + ((>= dashes (* total 0.8)) "-") > + ((>= underscores (* total 0.8)) "_") > + (else "")))) > + > + (define delim-rx (regexp-quote (or delim (guess-delimiter)))) > + (define suffix-rx (string-append (or suffix "") "$")) > + > + > + (define prefix-rx (string-append "^" (or prefix "[^[:digit:]]*"))) > + (define pre-release-rx > + (if pre-releases? > + (string-append ".*(" (string-join %pre-release-words "|") ").*") > + "")) > + > + (define tag-rx > + (string-append prefix-rx "([[:digit:]][^" delim-rx "[:punct:]]*" > + "(" delim-rx "[^[:punct:]" delim-rx "]+)" > + ;; If there is are no delimiters, it could mean that the ^ no "is" > + ;; version just contains one number (e.g., "2"), thus, use > + ;; "*" instead of "+" to match zero or more numbers. > + (if (string=? delim-rx "") "*" "+") Good catch. > + pre-release-rx ")" suffix-rx)) > + > + (define (get-version tag) > + (let ((tag-match (regexp-exec (make-regexp tag-rx) tag))) > + (and tag-match > + (regexp-substitute/global > + #f delim-rx (match:substring tag-match 1) > + ;; Don't insert "." if there aren't any delimiters in the first Nit: "if there were no delimiters", to be consistent with above comment. > + ;; place. > + 'pre (if (string=? delim-rx "") "" ".") 'post)))) One issue with returning a different delimiter than the package currently uses is that the automatic updater won't really work as-is. Hmmm. When things are modified so the updater gets both the version and the git-reference, it should be able to reverse-engineer things well enough there. I imagine this is really only going to be an issue with dates currently written as "2017-01-01", anyway. I'll put my comments on that in reply to the other email. > + > + (define (entry + (eq? (version-compare (car a) (car b)) '<)) > + > + (stable-sort (filter-map (lambda (tag) > + (let ((version (get-version tag))) > + (and version (cons version tag)))) > + tags) > + entry + > +(define* (latest-tag url #:key prefix suffix delim pre-releases?) > + "Return the latest tag available from the Git repository at URL." This returns two values (in preparation for the above-mentioned switch), so maybe something like "Return the latest version and corresponding tag available from..." > + (define (pre-release? tag) > + (any (cut regexp-exec <> tag) > + %pre-release-rx)) > + > + (let* ((tags (map (cut string-drop <> (string-length "refs/tags/")) Should be "cute" so string-length is only evaluated once -- though it's probably optimized like that anyway. > + (remote-refs url #:tags? #t))) > + (versions->tags > + (version-mapping (if pre-releases? > + tags > + (filter (negate pre-release?) tags)) > + #:prefix prefix > + #:suffix suffix > + #:delim delim > + #:pre-releases? pre-releases?))) > + (cond > + ((null? tags) > + (git-no-tags-error)) > + ((null? versions->tags) > + (git-no-valid-tags-error)) > + (else > + (match (last versions->tags) > + ((version . tag) > + (values version tag))))))) > + > +(define (latest-git-tag-version package) > + "Given a PACKAGE, return the latest version of it, or #f if the latest version > +could not be determined." > + (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c)) > + (warning (or (package-field-location package 'source) > + (package-location package)) > + (G_ "~a for ~a~%") > + (condition-message c) > + (package-name package)) > + #f) > + ((eq? (exception-kind c) 'git-error) > + (warning (or (package-field-location package 'source) > + (package-location package)) > + (G_ "failed to fetch Git repository for ~a~%") > + (package-name package)) > + #f)) > + (let* ((source (package-source package)) > + (url (git-reference-url (origin-uri source))) > + (properties (package-properties package)) > + (tag-prefix (assq-ref properties 'release-tag-prefix)) > + (tag-suffix (assq-ref properties 'release-tag-suffix)) > + (tag-version-delimiter (assq-ref properties 'release-tag-version-delimiter)) > + (refresh-pre-releases? (assq-ref properties 'accept-pre-releases?))) > + (latest-tag url > + #:prefix tag-prefix > + #:suffix tag-suffix > + #:delim tag-version-delimiter > + #:pre-releases? refresh-pre-releases?)))) This is entirely a style preference, so only take this suggestion if you like it :) (let* ((source (package-source package)) (url (git-reference-url (origin-uri source))) (property (cute assq-ref (package-properties package) <>))) (latest-tag url #:prefix (property 'release-tag-prefix) #:suffix (property 'release-tag-suffix) #:delim (property 'release-tag-version-delimiter) #:pre-releases? (property 'accept-pre-releases?))))) > + > +(define (git-package? package) > + "Whether the origin of PACKAGE is a Git repostiory." "Return true if PACKAGE is..." > + (match (package-source package) > + ((? origin? origin) > + (and (eq? (origin-method origin) git-fetch) > + (git-reference? (origin-uri origin)))) > + (_ #f))) > + > +(define (latest-git-release package) > + "Return the latest release of PACKAGE." "Return an for the latest...", to match the other updaters. > + (let* ((name (package-name package)) > + (old-version (package-version package)) > + (url (git-reference-url (origin-uri (package-source package)))) > + (new-version (latest-git-tag-version package))) > + > + (and new-version > + (upstream-source > + (package name) > + (version new-version) > + (urls (list url)))))) > + > +(define %generic-git-updater > + (upstream-updater > + (name 'generic-git) > + (description "Updater for packages hosted on Git repositories") > + (pred git-package?) > + (latest latest-git-release))) I tested this updater on all packages in .scm files starting with f through z, and I found the following packages with possibly bogus updates: --8<---------------cut here---------------start------------->8--- javaxom luakit ocproxy pitivi eid-mw libhomfly gnuradio welle-io racket-minimal milkytracker cl-portal kodi-cli openjdk java-bouncycastle hurd opencsg povray gpsbabel go stepmania ocaml-mcl many minetest packages (minetest will have its own updater, though) ocaml4.07-core-kernel, ocamlbuild and many other ocaml packages (they seem to be covered by the github updater) --8<---------------cut here---------------end--------------->8--- The following packages suggest a version -> date update, which may or may not be bogus: --8<---------------cut here---------------start------------->8--- cataclysm-dda autotrace lbalgtk nheko libqalculate cl-antik cl-antik-base cl-hu.dwim.stefil cl-stefil cl-gsll sbcl-cl-gserver --8<---------------cut here---------------end--------------->8--- -- Sarah From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 16 05:46:18 2021 Received: (at 50359) by debbugs.gnu.org; 16 Sep 2021 09:46:18 +0000 Received: from localhost ([127.0.0.1]:53862 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mQnyD-0000vY-SJ for submit@debbugs.gnu.org; Thu, 16 Sep 2021 05:46:18 -0400 Received: from out1.migadu.com ([91.121.223.63]:37057) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mQnyA-0000s3-J4 for 50359@debbugs.gnu.org; Thu, 16 Sep 2021 05:46:16 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1631785572; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=l1BG7SqJ9jopQRULDRoJuwtszJ5w4cSIwejutiGBDDg=; b=VDR6iEJjNqcrioAgV9Q9htR0Oj1S5vcvfB6O2XF3PJq46jlNN0dBNFD4sI3aOsfW0052Pb i+TykrhZRQuDUuJaIxzBZQCgS/qprl8dFtG23Q5NQ8ghDtf31sPWG49xg2hJvwpU5KzBBB aHt+e2D88dGOvMGn+r9poqXBVtoe3bo= From: Sarah Morgensen To: Xinglu Chen Subject: Re: [bug#50359] [PATCH 3/3] import: Add 'generic-git' updater. References: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> <44519ae5dfdf3702dc73903b80314a6a@mgsn.dev> <87sfy6rqw5.fsf@yoctocell.xyz> Date: Thu, 16 Sep 2021 02:46:10 -0700 In-Reply-To: <87sfy6rqw5.fsf@yoctocell.xyz> (Xinglu Chen's message of "Wed, 15 Sep 2021 13:59:22 +0200 (20 hours, 25 minutes, 52 seconds ago)") Message-ID: <86a6kcj1jx.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: iskarian@mgsn.dev X-Spam-Score: 1.3 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi, Xinglu Chen writes: > On Wed, Sep 15 2021, iskarian@mgsn.dev wrote: > >> Hi, >> >> September 10, 2021 9:21 AM, "Xinglu Chen" wrote: >> >>> * guix/git.scm (ls-remote-refs): New procedure. >>> * test [...] Content analysis details: (1.3 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/, low trust [91.121.223.63 listed in list.dnswl.org] -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [91.121.223.63 listed in wl.mailspike.net] -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-Debbugs-Envelope-To: 50359 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.3 (/) Hi, Xinglu Chen writes: > On Wed, Sep 15 2021, iskarian@mgsn.dev wrote: > >> Hi, >> >> September 10, 2021 9:21 AM, "Xinglu Chen" wrote: >> >>> * guix/git.scm (ls-remote-refs): New procedure. >>> * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. >>> * guix/import/git.scm: New file. >>> * doc/guix.texi (Invoking guix refresh): Document it. >>> * tests/import-git.scm: New test file. >>> * Makefile.am (MODULES, SCM_TESTS): Register the new files. >>>=20 >>> Co-authored-by: Sarah Morgensen >> >> Overall this is looking good. Thank you for adding tests (for >> remote-refs as well!), much appreciated. It looks like you've done >> some good polishing. I see a few nits, which I'll point out in a >> separate email when I'm not on mobile. I'll also give it a good test. >> >> But... I've been thinking about the overall approach for a couple >> days, because I'm not very happy with the complexity of my heuristic. >> >> There can be a lot of weird tags in a repository--look at the one for >> xf86-video-intel for example. My heuristic attempts to capture the >> assumption that repostories tend to move from using "_" or "-" to "." >> but it does fail to account for moving to or from dates (because dates >> don't compare with normal versions). > > But if a repo moved from using versions to tags, or vice-versa, we still > wouldn=E2=80=99t know if say =E2=80=9C3.0.1=E2=80=9D is newer than =E2=80= =9C2021.03.02=E2=80=9D. We would have > to know when the =E2=80=9C3.0.1=E2=80=9D tag was created. You're right; I thought of that afterwards. > Maybe we could have a =E2=80=98release-tag-date-scheme?=E2=80=99 property= , that way we > could just try to match dates? That seems like it might be the only way to handle it in some cases (if they have both versions and dates with a "." delimiter). (Though, we are actually interested in the *lack* of a date scheme. If they use a date scheme now, other versions will be disregarded, so we're fine; but if they use versions now and used a date scheme before, the versions will be discarded.) >> I also realized that we are not using a very useful piece of >> information--the previous version/tag combo. I expect that in the >> vast majority of cases, the version delimiter for the newest version >> will be the same as the version delimiter for the last known version. >> (Perhaps the prefix as well?) Can we use this information to make our >> guesses better? What do you think? > > That sounds like a good idea. What should happen if the delimiter from > the previous version/tag combo is different from the one that the > =E2=80=98guess-delimiter=E2=80=99 procedure returns? Should the one from= the previous > version/tag combo take precedence. Consider: prefix :=3D 'tag-prefix or guess-prefix-from-current-version+tag or defau= lt delim :=3D 'tag-delim or guess-delim-from-current-version+tag or guess-de= limiter suffix :=3D 'tag-suffix or default This should cover: 1. Format stayed the same (including date formats) 2. Format changed from (git-version ...) to proper version This does not otherwise cover a complete change in format, such as "_" -> ".", date(-) -> version, or version -> date(.), for which I could argue requiring a manual update is reasonable. It also does not cover when the tags have both versions and dates with the same delimiter. Though it would be nice to see when such updates are available, is it worth some bogus results? Are false positives better or false negatives better? Unless you/we want to pursue one or both of the above changes now, the latest patch LGTM (modulo my nits). Thanks for your work, -- Sarah From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 16 08:48:41 2021 Received: (at 50359) by debbugs.gnu.org; 16 Sep 2021 12:48:41 +0000 Received: from localhost ([127.0.0.1]:54090 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mQqoi-0007Dt-Dp for submit@debbugs.gnu.org; Thu, 16 Sep 2021 08:48:41 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:34060 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mQqoc-0007Db-JV for 50359@debbugs.gnu.org; Thu, 16 Sep 2021 08:48:38 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631796506; bh=sqik6FOfVFPmKqsgSwYkJN3vEqN6ixjpqVUDzhm8ND0=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=tQv9FBimJ1wLBh9WLfDFmP/YmwXiM+Mvd8+7wC8vL1VM7N3AmAGC0eV0jktt/0t69 3WlotXBBhG7SVjw1VtHSZpBtFBTTwWucq2mXwa47bqMJIcj7Gs9tsuUCuDxs7pZTYo pIZyG9E5iTLEpBA1l5BWyl+Qj8G/0QtQLyYEGKXQ= To: Sarah Morgensen Subject: Re: [bug#50359] [PATCH 3/3] import: Add 'generic-git' updater. In-Reply-To: <86czp8j38z.fsf@mgsn.dev> References: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> <86czp8j38z.fsf@mgsn.dev> Date: Thu, 16 Sep 2021 14:48:23 +0200 Message-ID: <87h7ekr8iw.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Thu, Sep 16 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >> On Wed, Sep 15 2021, iskarian@mgsn.dev wrote: >> >>> Hi, >>> >>> September 10, 2021 9:21 AM, "Xinglu Chen" wrote: >>> > [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Thu, Sep 16 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >> On Wed, Sep 15 2021, iskarian@mgsn.dev wrote: >> >>> Hi, >>> >>> September 10, 2021 9:21 AM, "Xinglu Chen" wrote: >>> > [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Thu, Sep 16 2021, Sarah Morgensen wrote: > Hi, > > Xinglu Chen writes: > >> On Wed, Sep 15 2021, iskarian@mgsn.dev wrote: >> >>> Hi, >>> >>> September 10, 2021 9:21 AM, "Xinglu Chen" wrote: >>> >>>> * guix/git.scm (ls-remote-refs): New procedure. >>>> * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. >>>> * guix/import/git.scm: New file. >>>> * doc/guix.texi (Invoking guix refresh): Document it. >>>> * tests/import-git.scm: New test file. >>>> * Makefile.am (MODULES, SCM_TESTS): Register the new files. >>>>=20 >>>> Co-authored-by: Sarah Morgensen >>> >>> Overall this is looking good. Thank you for adding tests (for >>> remote-refs as well!), much appreciated. It looks like you've done >>> some good polishing. I see a few nits, which I'll point out in a >>> separate email when I'm not on mobile. I'll also give it a good test. >>> >>> But... I've been thinking about the overall approach for a couple >>> days, because I'm not very happy with the complexity of my heuristic. >>> >>> There can be a lot of weird tags in a repository--look at the one for >>> xf86-video-intel for example. My heuristic attempts to capture the >>> assumption that repostories tend to move from using "_" or "-" to "." >>> but it does fail to account for moving to or from dates (because dates >>> don't compare with normal versions). >> >> But if a repo moved from using versions to tags, or vice-versa, we still >> wouldn=E2=80=99t know if say =E2=80=9C3.0.1=E2=80=9D is newer than =E2= =80=9C2021.03.02=E2=80=9D. We would have >> to know when the =E2=80=9C3.0.1=E2=80=9D tag was created. > > You're right; I thought of that afterwards. > >> Maybe we could have a =E2=80=98release-tag-date-scheme?=E2=80=99 propert= y, that way we >> could just try to match dates? > > That seems like it might be the only way to handle it in some cases (if > they have both versions and dates with a "." delimiter). It doesn=E2=80=99t have to be =E2=80=9C.=E2=80=9D delimiter though; if they= both have the same delimiter it would be difficult to distinguish a version from a date, e.g., =E2=80=9C1-2-3=E2=80=9D vs =E2=80=9C2021-03-23=E2=80=9D. > (Though, we are actually interested in the *lack* of a date scheme. > If they use a date scheme now, other versions will be disregarded, so > we're fine; but if they use versions now and used a date scheme > before, the versions will be discarded.) I am not sure what you are trying to say, could you elaborate? >>> I also realized that we are not using a very useful piece of >>> information--the previous version/tag combo. I expect that in the >>> vast majority of cases, the version delimiter for the newest version >>> will be the same as the version delimiter for the last known version. >>> (Perhaps the prefix as well?) Can we use this information to make our >>> guesses better? What do you think? >> >> That sounds like a good idea. What should happen if the delimiter from >> the previous version/tag combo is different from the one that the >> =E2=80=98guess-delimiter=E2=80=99 procedure returns? Should the one fro= m the previous >> version/tag combo take precedence. > > Consider: > > prefix :=3D 'tag-prefix or guess-prefix-from-current-version+tag or def= ault > delim :=3D 'tag-delim or guess-delim-from-current-version+tag or guess-= delimiter > suffix :=3D 'tag-suffix or default > > This should cover: > > 1. Format stayed the same (including date formats) > 2. Format changed from (git-version ...) to proper version > > This does not otherwise cover a complete change in format, such as "_" > -> ".", date(-) -> version, or version -> date(.), for which I could > argue requiring a manual update is reasonable. Yeah, it=E2=80=99s not really possible to automatically detect those kind of changes. > It also does not cover when the tags have both versions and dates with > the same delimiter. > > Though it would be nice to see when such updates are available, is it > worth some bogus results? Are false positives better or false negatives > better? Hmm, good question! If in the future we have some kind of bot that automatically runs =E2=80=98guix refresh -u=E2=80=99, builds the updated pa= ckage, and send a patch to the mailing list, not having false positives might be more important. We could also have a =E2=80=98disable-tag-updater?=E2=80= =99 property to disable the updater for packages which gives false positive, or maybe that will result in to many properties. > Unless you/we want to pursue one or both of the above changes now, the > latest patch LGTM (modulo my nits). I would prefer to wait a bit with the improvements mentioned above. The current patch has been in the works a week or two already, so it=E2=80=99s probably a good idea to get it merged, and try to solve the less important issues later. :-) > Thanks for your work, You are welcome! And thanks for taking the time to test and review the work. On Thu, Sep 16 2021, Sarah Morgensen wrote (again): > Hello, > > Here are my promised nits. > > Xinglu Chen writes: > >> * guix/git.scm (ls-remote-refs): New procedure. >> * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. >> * guix/import/git.scm: New file. >> * doc/guix.texi (Invoking guix refresh): Document it. >> * tests/import-git.scm: New test file. >> * Makefile.am (MODULES, SCM_TESTS): Register the new files. >> >> Co-authored-by: Sarah Morgensen > > Again, much thanks for writing tests. > >> +@item generic-git >> +a generic updater for packages hosted on Git repositories. It tries to >> +be smart about parsing Git tag names, but if it is not able to parse the >> +tag name and compare tags correctly, users can define the following >> +properties for a package. >> + >> +@itemize >> +@item @code{release-tag-prefix}: a regular expression for matching a pr= efix of >> +the tag name. >> + >> +@item @code{release-tag-suffix}: a regular expression for matching a su= ffix of >> +the tag name. >> + >> +@item @code{release-tag-version-delimiter}: a string used as the delimi= ter in >> +the tag name for separating the numbers of the version. >> +@end itemize >> + >> +@lisp >> +(package >> + (name "foo") >> + ;; ... >> + (properties >> + '((release-tag-prefix . "^release0-") >> + (release-tag-suffix . "[a-z]?$") >> + (release-tag-version-delimiter . ":")))) >> +@end lisp >> + >> +By default, the updater will ignore pre-releases; to make it also look >> +for pre-releases, set the @code{accept-pre-releases?} property to >> +@code{#t}. > > Should this be itemized above? That=E2=80=99s probably a good idea, since it is related to how the tag wil= l be parsed. >> + >> +;; >> +;;; Remote operations. >> +;;; >> + >> +(define* (remote-refs url #:key tags?) >> + "Return the list of references advertised at Git repository URL. If = TAGS? >> +is true, limit to only refs/tags." >> + (define (ref? ref) >> + ;; Like `git ls-remote --refs', only show actual references. >> + (and (string-prefix? "refs/" ref) >> + (not (string-suffix? "^{}" ref)))) >> + >> + (define (tag? ref) >> + (string-prefix? "refs/tags/" ref)) >> + >> + (define (include? ref) >> + (and (ref? ref) >> + (or (not tags?) (tag? ref)))) >> + >> + (with-libgit2 >> + (call-with-temporary-directory >> + (lambda (cache-directory) >> + (let* ((repository (repository-init cache-directory)) >> + ;; Create an in-memory remote so we don't touch disk. >> + (remote (remote-create-anonymous repository url))) >> + (remote-connect remote) >> + (remote-disconnect remote) >> + (repository-close! repository) >> + >> + (filter-map (lambda (remote) >> + (let ((name (remote-head-name remote))) >> + (and (include? name) >> + name))) >> + (remote-ls remote))))))) > > I discovered that this can segfault unless 'remote-disconnect' and > possibly 'repository-close!' are called *after* copying the data out. > I've attached a diff for this. I don=E2=80=99t see a diff attached; maybe you forgot? :-) >> + >> +;;; Updater >> + >> +(define %pre-release-words >> + '("alpha" "beta" "rc" "dev" "test")) > > I found a few packages that use "pre" as well. Good catch, I noticed that as well when doing some more testing. >> + >> +(define %pre-release-rx >> + (map (cut make-regexp <> regexp/icase) %pre-release-words)) >> + >> +(define* (version-mapping tags #:key prefix suffix delim pre-releases?) >> + "Given a list of Git TAGS, return a association list where the car is= the > ^ an > >> +version corresponding to the tag, and the cdr is the name of the tag." >> + (define (guess-delimiter) >> + (let ((total (length tags)) >> + (dots (reduce + 0 (map (cut string-count <> #\.) tags))) >> + (dashes (reduce + 0 (map (cut string-count <> #\-) tags))) >> + (underscores (reduce + 0 (map (cut string-count <> #\_) tags)= ))) >> + (cond >> + ((>=3D dots (* total 0.35)) ".") >> + ((>=3D dashes (* total 0.8)) "-") >> + ((>=3D underscores (* total 0.8)) "_") >> + (else "")))) >> + >> + (define delim-rx (regexp-quote (or delim (guess-delimiter)))) >> + (define suffix-rx (string-append (or suffix "") "$")) >> + >> +=20=20 >> + (define prefix-rx (string-append "^" (or prefix "[^[:digit:]]*"))) >> + (define pre-release-rx >> + (if pre-releases? >> + (string-append ".*(" (string-join %pre-release-words "|") ").*") >> + "")) >> + >> + (define tag-rx >> + (string-append prefix-rx "([[:digit:]][^" delim-rx "[:punct:]]*" >> + "(" delim-rx "[^[:punct:]" delim-rx "]+)" >> + ;; If there is are no delimiters, it could mean that= the > ^ no "is" > >> + ;; version just contains one number (e.g., "2"), thu= s, use >> + ;; "*" instead of "+" to match zero or more numbers. >> + (if (string=3D? delim-rx "") "*" "+") > > Good catch. > >> + pre-release-rx ")" suffix-rx)) >> + >> + (define (get-version tag) >> + (let ((tag-match (regexp-exec (make-regexp tag-rx) tag))) >> + (and tag-match >> + (regexp-substitute/global >> + #f delim-rx (match:substring tag-match 1) >> + ;; Don't insert "." if there aren't any delimiters in the f= irst > > Nit: "if there were no delimiters", to be consistent with above comment. Noted. >> + ;; place. >> + 'pre (if (string=3D? delim-rx "") "" ".") 'post)))) > > One issue with returning a different delimiter than the package > currently uses is that the automatic updater won't really work as-is. Good point;, the tag name would be incorrect in those cases. > Hmmm. When things are modified so the updater gets both the version and > the git-reference, it should be able to reverse-engineer things well > enough there. Ah, looks like (guix upstream) needs some work. :-) > I imagine this is really only going to be an issue with dates currently > written as "2017-01-01", anyway. I'll put my comments on that in reply > to the other email. > >> + >> + (define (entry> + (eq? (version-compare (car a) (car b)) '<)) >> + >> + (stable-sort (filter-map (lambda (tag) >> + (let ((version (get-version tag))) >> + (and version (cons version tag)))) >> + tags) >> + entry> + >> +(define* (latest-tag url #:key prefix suffix delim pre-releases?) >> + "Return the latest tag available from the Git repository at URL." > > This returns two values (in preparation for the above-mentioned switch), > so maybe something like "Return the latest version and corresponding tag > available from..." Good catch. >> + (define (pre-release? tag) >> + (any (cut regexp-exec <> tag) >> + %pre-release-rx)) >> + >> + (let* ((tags (map (cut string-drop <> (string-length "refs/tags/")) > > Should be "cute" so string-length is only evaluated once -- though it's > probably optimized like that anyway. Good catch, I keep forgetting that =E2=80=98cute=E2=80=99 exists. :-) >> + (remote-refs url #:tags? #t))) >> + (versions->tags >> + (version-mapping (if pre-releases? >> + tags >> + (filter (negate pre-release?) tags)) >> + #:prefix prefix >> + #:suffix suffix >> + #:delim delim >> + #:pre-releases? pre-releases?))) >> + (cond >> + ((null? tags) >> + (git-no-tags-error)) >> + ((null? versions->tags) >> + (git-no-valid-tags-error)) >> + (else >> + (match (last versions->tags) >> + ((version . tag) >> + (values version tag))))))) >> + >> +(define (latest-git-tag-version package) >> + "Given a PACKAGE, return the latest version of it, or #f if the lates= t version >> +could not be determined." >> + (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c)) >> + (warning (or (package-field-location package 'source) >> + (package-location package)) >> + (G_ "~a for ~a~%") >> + (condition-message c) >> + (package-name package)) >> + #f) >> + ((eq? (exception-kind c) 'git-error) >> + (warning (or (package-field-location package 'source) >> + (package-location package)) >> + (G_ "failed to fetch Git repository for ~a~%") >> + (package-name package)) >> + #f)) >> + (let* ((source (package-source package)) >> + (url (git-reference-url (origin-uri source))) >> + (properties (package-properties package)) >> + (tag-prefix (assq-ref properties 'release-tag-prefix)) >> + (tag-suffix (assq-ref properties 'release-tag-suffix)) >> + (tag-version-delimiter (assq-ref properties 'release-tag-ver= sion-delimiter)) >> + (refresh-pre-releases? (assq-ref properties 'accept-pre-rele= ases?))) >> + (latest-tag url >> + #:prefix tag-prefix >> + #:suffix tag-suffix >> + #:delim tag-version-delimiter >> + #:pre-releases? refresh-pre-releases?)))) > > This is entirely a style preference, so only take this suggestion if you > like it :) > > (let* ((source (package-source package)) > (url (git-reference-url (origin-uri source))) > (property (cute assq-ref (package-properties package) <>))) > (latest-tag url > #:prefix (property 'release-tag-prefix) > #:suffix (property 'release-tag-suffix) > #:delim (property 'release-tag-version-delimiter) > #:pre-releases? (property 'accept-pre-releases?))))) That does look cleaner, thanks for the suggestion! >> + >> +(define (git-package? package) >> + "Whether the origin of PACKAGE is a Git repostiory." > > "Return true if PACKAGE is..." =E2=80=9CPACKAGE is a Git repository.=E2=80=9D doesn=E2=80=99t really sound= right, maybe =E2=80=9Cif PACKAGE is hosted on a Git repository=E2=80=9D? >> + (match (package-source package) >> + ((? origin? origin) >> + (and (eq? (origin-method origin) git-fetch) >> + (git-reference? (origin-uri origin)))) >> + (_ #f))) >> + >> +(define (latest-git-release package) >> + "Return the latest release of PACKAGE." > > "Return an for the latest...", to match the other > updaters. > >> + (let* ((name (package-name package)) >> + (old-version (package-version package)) >> + (url (git-reference-url (origin-uri (package-source package)))) >> + (new-version (latest-git-tag-version package))) >> + >> + (and new-version >> + (upstream-source >> + (package name) >> + (version new-version) >> + (urls (list url)))))) >> + >> +(define %generic-git-updater >> + (upstream-updater >> + (name 'generic-git) >> + (description "Updater for packages hosted on Git repositories") >> + (pred git-package?) >> + (latest latest-git-release))) > > I tested this updater on all packages in .scm files starting with f > through z, and I found the following packages with possibly bogus > updates: > > --8<---------------cut here---------------start------------->8--- > javaxom I assume you meant =E2=80=98java-xom=E2=80=99 :-) That=E2=80=99s a weird scheme; setting the delimiter to =E2=80=9C.=E2=80=9D= doesn=E2=80=99t help since it thinks that =E2=80=9C127=E2=80=9D is greater than =E2=80=9C1.3.7=E2=80= =9D. > luakit > ocproxy > pitivi =E2=80=98pitivi=E2=80=99 has a pretty weird version string to begin with; i= t may be better to change it to the date: =E2=80=9C0.999.0-2021-05.0=E2=80=9D -> =E2= =80=9C2021-05.0=E2=80=9D. > eid-mw > libhomfly > gnuradio > welle-io Setting the delimiter to "." fixes the issue. > racket-minimal Setting the prefix to "v" fixes this. > milkytracker > cl-portal > kodi-cli > openjdk > java-bouncycastle > hurd > opencsg Setting the suffix to "-release" fixes this. > povray > gpsbabel Setting the prefix to "gpsbabel_" fixes this. > go > stepmania > ocaml-mcl > > many minetest packages (minetest will have its own updater, though) > > ocaml4.07-core-kernel, ocamlbuild and many other ocaml packages > (they seem to be covered by the github updater) > --8<---------------cut here---------------end--------------->8--- Hmm, =E2=80=98guix refresh=E2=80=99 says that =E2=80=98ocamlbuild=E2=80=99 = is already the latest version. But you are right, many of the packages are already taken care of by the =E2=80=98github=E2=80=99 updater. > The following packages suggest a version -> date update, which may or > may not be bogus: > > --8<---------------cut here---------------start------------->8--- > cataclysm-dda > autotrace > lbalgtk > nheko > libqalculate > cl-antik > cl-antik-base > cl-hu.dwim.stefil > cl-stefil > cl-gsll > sbcl-cl-gserver > --8<---------------cut here---------------end--------------->8--- Thanks for taking the time to find these false positive! --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmFDPRcVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5+L0P/jci8tK0hLt7DucS+KeZSQHncSzx fumt9Ln41wxv2WmnnLMb0id8slP+7dSsolk4TXzx+Udly500HLVTNuioLJWV1S86 nvl4bO01DtDz5u48oht6YpRTVjshepAS7mpY39UubGP0aF137ljICsL03uQltuoU 7S5pZggZSlwAfFZtIwpmkrzi+xLMcIMLto2Q+fefe7f2+oKz3XkvZqHhs9hNuCLt TAglODmOgs2fo2A6QBp4K/3CfcMiZphGvPpWUOsTKQ+bFMFnhRD44qBl6q9cyY1J JauSz9nC24nWgOqDhkx3SFxcsDKpc5Oep27aOGSnNS+qrDI1qDsXtBwsfDPbeM28 jH44J9qnzxu69r0trAx5xpLp5sF6YV5GQk49MXandRtsMUChcy4P+w+kg83oR+y/ dYPgFjvG2pDh46EorgO+Yu7T7ND1XcMlrQ4HZfWBwB+nzkhhocMxsPGnOuJHVxhO vmZ42EtrC1XyE2zgtnk7nCZlcrgVyPqpnBpJzeyPd1iLSYVX+fYDP2W1X0YkoZLg YoAiy9Vf3XoU69iiEZmP26nVJUFoTCiK5vUmla6u8mQuGZgZf4ozMF5dRR5xErCl 2WMqgguZGq5DtOIvhH8hvwvvB5LfMNyL8Bu2LyP7mBBR3wTf08iQAT+inBuQSg/b kBypjToJW+j6qFU/ =dF62 -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Sep 16 19:42:47 2021 Received: (at 50359) by debbugs.gnu.org; 16 Sep 2021 23:42:48 +0000 Received: from localhost ([127.0.0.1]:57873 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR11j-0004gZ-GR for submit@debbugs.gnu.org; Thu, 16 Sep 2021 19:42:47 -0400 Received: from out2.migadu.com ([188.165.223.204]:34083) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR11f-0004gP-SU for 50359@debbugs.gnu.org; Thu, 16 Sep 2021 19:42:45 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1631835762; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=BS09ZLZPxojuNp4q9Or7B6VHTO2+qDBBXy5mFsL59N0=; b=kni2L2m0ReE2FRvBp97aIE9Wbos+i7+/cRHFHm9TXc3vmkTYYs0IwoUUA2hetSSX3aj8t2 3vfpVygdHsMYFmJpSPMw+bglPhy9OIdxVpGP9Vo6TioT6+go5wPLv+nQfm0la+Jk7e1vQq 18NE+A1cxvgzzgHjX3V/ara84EAmEac= From: Sarah Morgensen To: Xinglu Chen Subject: Re: [bug#50359] [PATCH 3/3] import: Add 'generic-git' updater. References: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> <86czp8j38z.fsf@mgsn.dev> <87h7ekr8iw.fsf@yoctocell.xyz> Date: Thu, 16 Sep 2021 16:42:38 -0700 In-Reply-To: <87h7ekr8iw.fsf@yoctocell.xyz> (Xinglu Chen's message of "Thu, 16 Sep 2021 14:48:23 +0200 (10 hours, 1 minute, 2 seconds ago)") Message-ID: <867dfghytt.fsf@mgsn.dev> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Migadu-Flow: FLOW_OUT X-Migadu-Auth-User: iskarian@mgsn.dev X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Xinglu Chen writes: >>> Maybe we could have a ‘release-tag-date-scheme?’ property, that way we >>> could just try to match dates? >> >> That seems like it might be the only way to handle it in some cases (if >> they [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [188.165.223.204 listed in wl.mailspike.net] X-Debbugs-Envelope-To: 50359 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Xinglu Chen writes: >>> Maybe we could have a =E2=80=98release-tag-date-scheme?=E2=80=99 proper= ty, that way we >>> could just try to match dates? >> >> That seems like it might be the only way to handle it in some cases (if >> they have both versions and dates with a "." delimiter). > > It doesn=E2=80=99t have to be =E2=80=9C.=E2=80=9D delimiter though; if th= ey both have the same > delimiter it would be difficult to distinguish a version from a date, > e.g., =E2=80=9C1-2-3=E2=80=9D vs =E2=80=9C2021-03-23=E2=80=9D. Sure, but I haven't seen the former :) >> (Though, we are actually interested in the *lack* of a date scheme. >> If they use a date scheme now, other versions will be disregarded, so >> we're fine; but if they use versions now and used a date scheme >> before, the versions will be discarded.) > > I am not sure what you are trying to say, could you elaborate? Just that the important case is disallowing dates when 'release-tag-date-scheme? is #f. If the tags of a repo are: 12.1 12.2 13.0 13.4 2018.01.01 2018.05.05 and we do nothing, the 2018.05.05 tag will be selected. This is correct if we do want dates, but incorrect if we don't (in which case we would set 'tag-version-date-scheme? to #f to get the correct result). >> Though it would be nice to see when such updates are available, is it >> worth some bogus results? Are false positives better or false negatives >> better? > > Hmm, good question! If in the future we have some kind of bot that > automatically runs =E2=80=98guix refresh -u=E2=80=99, builds the updated = package, and > send a patch to the mailing list, not having false positives might be > more important. We could also have a =E2=80=98disable-tag-updater?=E2=80= =99 property to > disable the updater for packages which gives false positive, or maybe > that will result in to many properties. For these packages, it would probably easier to just use the existing tag- properties. In fact, instead of this or the date-scheme above, a 'tag-version-regex' would cover both cases. In fact, we could replace 'tag-version-delimiter' with 'tag-version-regex' and instead provide convencience functions such as (untested): (define (version-regex delim) (let ((delim-rx (regexp-quote delim))) (string-append "([[:digit:]][^" delim-rx "[:punct:]]*" "(" delim-rx "[^[:punct:]" delim-rx "]+)" (if (string=3D? delim-rx "") "*" "+")))) (define* (version-date-regex (delim ".")) (let ((delim-rx (regexp-quote delim))) (string-append "([0-9]{4}" delim-rx "(0[1-9]|11|12)" delim-rx "(0[1-9]|[1-2][0-9])"))) WDYT? >> Unless you/we want to pursue one or both of the above changes now, the >> latest patch LGTM (modulo my nits). > > I would prefer to wait a bit with the improvements mentioned above. The > current patch has been in the works a week or two already, so it=E2=80=99s > probably a good idea to get it merged, and try to solve the less > important issues later. :-) Sounds good to me, then! >> I discovered that this can segfault unless 'remote-disconnect' and >> possibly 'repository-close!' are called *after* copying the data out. >> I've attached a diff for this. > > I don=E2=80=99t see a diff attached; maybe you forgot? :-) > I've actually attached it this time :) >>> + >>> +(define (git-package? package) >>> + "Whether the origin of PACKAGE is a Git repostiory." >> >> "Return true if PACKAGE is..." > > =E2=80=9CPACKAGE is a Git repository.=E2=80=9D doesn=E2=80=99t really sou= nd right, maybe =E2=80=9Cif > PACKAGE is hosted on a Git repository=E2=80=9D?' Sorry, yes, that's what I meant, or "Return true if the origin..."; I was just suggesting making it a full sentence. >> I tested this updater on all packages in .scm files starting with f >> through z, and I found the following packages with possibly bogus >> updates: >> >> --8<---------------cut here---------------start------------->8--- >> javaxom > > I assume you meant =E2=80=98java-xom=E2=80=99 :-) > > That=E2=80=99s a weird scheme; setting the delimiter to =E2=80=9C.=E2=80= =9D doesn=E2=80=99t help since > it thinks that =E2=80=9C127=E2=80=9D is greater than =E2=80=9C1.3.7=E2=80= =9D. 'tag-version-regex would allow fixing this ;) > >> luakit >> ocproxy >> pitivi > > =E2=80=98pitivi=E2=80=99 has a pretty weird version string to begin with;= it may be > better to change it to the date: =E2=80=9C0.999.0-2021-05.0=E2=80=9D -> = =E2=80=9C2021-05.0=E2=80=9D. > >> eid-mw >> libhomfly >> gnuradio >> welle-io > > Setting the delimiter to "." fixes the issue. > >> racket-minimal > > Setting the prefix to "v" fixes this. > >> milkytracker >> cl-portal >> kodi-cli >> openjdk >> java-bouncycastle >> hurd >> opencsg > > Setting the suffix to "-release" fixes this. > >> povray >> gpsbabel > > Setting the prefix to "gpsbabel_" fixes this. > >> go >> stepmania >> ocaml-mcl >> >> many minetest packages (minetest will have its own updater, though) >> >> ocaml4.07-core-kernel, ocamlbuild and many other ocaml packages >> (they seem to be covered by the github updater) >> --8<---------------cut here---------------end--------------->8--- I'm glad to see that these are easily fixed with the properties, though! That's some good validation. Now I just have to give the (guix upstream) some attention... -- Sarah --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=generic-git-fix-segfault.patch Content-Description: Fix undeterministic segfaults in remote-refs. diff --git a/guix/git.scm b/guix/git.scm index dc3d3afd02..bbff4fc890 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -593,6 +593,11 @@ is true, limit to only refs/tags." (and (ref? ref) (or (not tags?) (tag? ref)))) + (define (remote-head->ref remote) + (let ((name (remote-head-name remote))) + (and (include? name) + name))) + (with-libgit2 (call-with-temporary-directory (lambda (cache-directory) @@ -600,14 +605,13 @@ is true, limit to only refs/tags." ;; Create an in-memory remote so we don't touch disk. (remote (remote-create-anonymous repository url))) (remote-connect remote) - (remote-disconnect remote) - (repository-close! repository) - - (filter-map (lambda (remote) - (let ((name (remote-head-name remote))) - (and (include? name) - name))) - (remote-ls remote))))))) + + (let* ((remote-heads (remote-ls remote)) + (refs (filter-map remote-head->ref remote-heads))) + ;; Wait until we're finished with the repository before closing it. + (remote-disconnect remote) + (repository-close! repository) + refs)))))) ;;; --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 17 03:49:12 2021 Received: (at 50359) by debbugs.gnu.org; 17 Sep 2021 07:49:12 +0000 Received: from localhost ([127.0.0.1]:58374 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR8cR-0000wY-Hw for submit@debbugs.gnu.org; Fri, 17 Sep 2021 03:49:11 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:60596 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR8cP-0000wJ-Rw for 50359@debbugs.gnu.org; Fri, 17 Sep 2021 03:49:10 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631864941; bh=RkqjigjBO+ib8clJXIDbwCx8ki4ITq+5anaKC2HP7ik=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=KyzwyKO2DRMDnq2BnTQ4TORkXyDwyirGLlrZ+oz5lWjgfeByYINVQepIicAbdHA6a ONEQeD6bISVVqMEExfrUvw/PJL6sK6ICZVIjpXXssQFCMsnY/L/JqO2DQPjkP04n4j puJmtWJlOXSqcKtJP+rrA9Q2+r6GZC4iwpoSZ2Kg= To: Sarah Morgensen Subject: Re: [bug#50359] [PATCH 3/3] import: Add 'generic-git' updater. In-Reply-To: <867dfghytt.fsf@mgsn.dev> References: <5d10dd1e65b0a65ada4a8102310c10de42f53e8d.1631290349.git.public@yoctocell.xyz> <86czp8j38z.fsf@mgsn.dev> <87h7ekr8iw.fsf@yoctocell.xyz> <867dfghytt.fsf@mgsn.dev> Date: Fri, 17 Sep 2021 09:48:49 +0200 Message-ID: <8735q3r6am.fsf@yoctocell.xyz> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Thu, Sep 16 2021, Sarah Morgensen wrote: > Xinglu Chen writes: > >>>> Maybe we could have a ‘release-tag-date-scheme?’ property, that way we >>>> could just try to match dates? >>> >>> That seems like it might be t [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , 50359@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: On Thu, Sep 16 2021, Sarah Morgensen wrote: > Xinglu Chen writes: > >>>> Maybe we could have a ‘release-tag-date-scheme?’ property, that way we >>>> could just try to match dates? >>> >>> That seems like it might be t [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 1.0 BULK_RE_SUSP_NTLD Precedence bulk and RE: from a suspicious TLD -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On Thu, Sep 16 2021, Sarah Morgensen wrote: > Xinglu Chen writes: > >>>> Maybe we could have a =E2=80=98release-tag-date-scheme?=E2=80=99 prope= rty, that way we >>>> could just try to match dates? >>> >>> That seems like it might be the only way to handle it in some cases (if >>> they have both versions and dates with a "." delimiter). >> >> It doesn=E2=80=99t have to be =E2=80=9C.=E2=80=9D delimiter though; if t= hey both have the same >> delimiter it would be difficult to distinguish a version from a date, >> e.g., =E2=80=9C1-2-3=E2=80=9D vs =E2=80=9C2021-03-23=E2=80=9D. > > Sure, but I haven't seen the former :) > >>> (Though, we are actually interested in the *lack* of a date scheme. >>> If they use a date scheme now, other versions will be disregarded, so >>> we're fine; but if they use versions now and used a date scheme >>> before, the versions will be discarded.) >> >> I am not sure what you are trying to say, could you elaborate? > > Just that the important case is disallowing dates when > 'release-tag-date-scheme? is #f. > > If the tags of a repo are: > > 12.1 > 12.2 > 13.0 > 13.4 > 2018.01.01 > 2018.05.05 > > and we do nothing, the 2018.05.05 tag will be selected. This is correct > if we do want dates, but incorrect if we don't (in which case we would > set 'tag-version-date-scheme? to #f to get the correct result). Ah, that makes sense. :-) >>> Though it would be nice to see when such updates are available, is it >>> worth some bogus results? Are false positives better or false negatives >>> better? >> >> Hmm, good question! If in the future we have some kind of bot that >> automatically runs =E2=80=98guix refresh -u=E2=80=99, builds the updated= package, and >> send a patch to the mailing list, not having false positives might be >> more important. We could also have a =E2=80=98disable-tag-updater?=E2= =80=99 property to >> disable the updater for packages which gives false positive, or maybe >> that will result in to many properties. > > For these packages, it would probably easier to just use the existing > tag- properties. In fact, instead of this or the date-scheme above, > a 'tag-version-regex' would cover both cases. > > In fact, we could replace 'tag-version-delimiter' with > 'tag-version-regex' and instead provide convencience functions such as > (untested): > > (define (version-regex delim) > (let ((delim-rx (regexp-quote delim))) > (string-append "([[:digit:]][^" delim-rx "[:punct:]]*" > "(" delim-rx "[^[:punct:]" delim-rx "]+)" > (if (string=3D? delim-rx "") "*" "+")))) > > (define* (version-date-regex (delim ".")) > (let ((delim-rx (regexp-quote delim))) > (string-append "([0-9]{4}" delim-rx "(0[1-9]|11|12)" > delim-rx "(0[1-9]|[1-2][0-9])"))) > > WDYT? That sounds like a good idea! Where would we put these procedures, (guix packages)? >>> Unless you/we want to pursue one or both of the above changes now, the >>> latest patch LGTM (modulo my nits). >> >> I would prefer to wait a bit with the improvements mentioned above. The >> current patch has been in the works a week or two already, so it=E2=80= =99s >> probably a good idea to get it merged, and try to solve the less >> important issues later. :-) > > Sounds good to me, then! > >>> I discovered that this can segfault unless 'remote-disconnect' and >>> possibly 'repository-close!' are called *after* copying the data out. >>> I've attached a diff for this. >> >> I don=E2=80=99t see a diff attached; maybe you forgot? :-) >> > > I've actually attached it this time :) > >>>> + >>>> +(define (git-package? package) >>>> + "Whether the origin of PACKAGE is a Git repostiory." >>> >>> "Return true if PACKAGE is..." >> >> =E2=80=9CPACKAGE is a Git repository.=E2=80=9D doesn=E2=80=99t really so= und right, maybe =E2=80=9Cif >> PACKAGE is hosted on a Git repository=E2=80=9D?' > > Sorry, yes, that's what I meant, or "Return true if the origin..."; I > was just suggesting making it a full sentence. > >>> I tested this updater on all packages in .scm files starting with f >>> through z, and I found the following packages with possibly bogus >>> updates: >>> >>> --8<---------------cut here---------------start------------->8--- >>> javaxom >> >> I assume you meant =E2=80=98java-xom=E2=80=99 :-) >> >> That=E2=80=99s a weird scheme; setting the delimiter to =E2=80=9C.=E2=80= =9D doesn=E2=80=99t help since >> it thinks that =E2=80=9C127=E2=80=9D is greater than =E2=80=9C1.3.7=E2= =80=9D. > > 'tag-version-regex would allow fixing this ;) > >> >>> luakit >>> ocproxy >>> pitivi >> >> =E2=80=98pitivi=E2=80=99 has a pretty weird version string to begin with= ; it may be >> better to change it to the date: =E2=80=9C0.999.0-2021-05.0=E2=80=9D -> = =E2=80=9C2021-05.0=E2=80=9D. >> >>> eid-mw >>> libhomfly >>> gnuradio >>> welle-io >> >> Setting the delimiter to "." fixes the issue. >> >>> racket-minimal >> >> Setting the prefix to "v" fixes this. >> >>> milkytracker >>> cl-portal >>> kodi-cli >>> openjdk >>> java-bouncycastle >>> hurd >>> opencsg >> >> Setting the suffix to "-release" fixes this. >> >>> povray >>> gpsbabel >> >> Setting the prefix to "gpsbabel_" fixes this. >> >>> go >>> stepmania >>> ocaml-mcl >>> >>> many minetest packages (minetest will have its own updater, though) >>> >>> ocaml4.07-core-kernel, ocamlbuild and many other ocaml packages >>> (they seem to be covered by the github updater) >>> --8<---------------cut here---------------end--------------->8--- > > I'm glad to see that these are easily fixed with the properties, though! > That's some good validation. Yeah, it=E2=80=99s looking pretty good. :-) > Now I just have to give the (guix upstream) some attention... > > -- > Sarah > > diff --git a/guix/git.scm b/guix/git.scm > index dc3d3afd02..bbff4fc890 100644 > --- a/guix/git.scm > +++ b/guix/git.scm > @@ -593,6 +593,11 @@ is true, limit to only refs/tags." > (and (ref? ref) > (or (not tags?) (tag? ref)))) >=20=20 > + (define (remote-head->ref remote) > + (let ((name (remote-head-name remote))) > + (and (include? name) > + name))) > + > (with-libgit2 > (call-with-temporary-directory > (lambda (cache-directory) > @@ -600,14 +605,13 @@ is true, limit to only refs/tags." > ;; Create an in-memory remote so we don't touch disk. > (remote (remote-create-anonymous repository url))) > (remote-connect remote) > - (remote-disconnect remote) > - (repository-close! repository) > - > - (filter-map (lambda (remote) > - (let ((name (remote-head-name remote))) > - (and (include? name) > - name))) > - (remote-ls remote))))))) > + > + (let* ((remote-heads (remote-ls remote)) > + (refs (filter-map remote-head->ref remote-heads))) > + ;; Wait until we're finished with the repository before closin= g it. > + (remote-disconnect remote) > + (repository-close! repository) > + refs)))))) >=20=20 > > ;;; --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmFESGEVHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5PYkP/RWAH42qmxWT38TjfWw7DG3v7WXY gqpFQEvdnW7AFtjB6xBviU11NFF/20zuGX8z6SSibydOoj3lcuxW2WunOcoMOQuw n1yDrpQ/8EkSOkqIMqt+5zBGEocPyjwu+ggdqFuttPfSmXyRQp831woY0lGYalQc MtP6jqFdYlliShONpLaTrMOT8l8md1FpJXX8PVZUukFHgwHFhrLZLlEVCggEsZAk yxkjzIX4Hs/TMT/MY0b46/R2p9ocWHO8jcOlHhxerehRHt3IWB1SHn48ziVHXDou 166/uwJRGWPsAYn0en0UdEv9bPgOCABE+bGDBd4G41UFEYB9Ags3MW2/OoEAGmzC cWW5EhVXQFBx+EKOrYiHUuK1khh4ZMrM7Igb0tvUi7oce38exk9LM3GSWldEUKmG 0Bkhfmc6Glb6ytKiBY2f1RkhSjQ1qZt+BUjT/bCf/9JprXexPwgdEaZJ72qgQnSC QrJd90D76X3/P+JbXB4wgtvnJKhYQI4Jcf4rDIS5b4sxNrjhVd7H8P30rB0g8IYB c29b5DwlmC/GnZi/gGWpgMatFFA443MW6JF4huZOzUaTopaqF2keXXj9YU5LmPI0 CeNMFM7fKC/Asvp2QDNTUFJJM81YPRL2tG6JuAi0rTooXNo5dQxT22kf/mQPdA3D MsuVbuwHe8ZJ5wvL =5SH8 -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 17 04:04:27 2021 Received: (at 50359) by debbugs.gnu.org; 17 Sep 2021 08:04:27 +0000 Received: from localhost ([127.0.0.1]:58393 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR8rD-0001K0-91 for submit@debbugs.gnu.org; Fri, 17 Sep 2021 04:04:27 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:60966 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR8rA-0001Jn-Jc for 50359@debbugs.gnu.org; Fri, 17 Sep 2021 04:04:25 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631865856; bh=3/1QAwDfGvCkAb5h7ewSepFyjRMpev4Km5eLg5NeaYs=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=JH9FnL4Q1rH5G6rMjS34DsjfKzAnlv/1dzDHjkn1xZFrhl9KDKGbLj/LBtJibXm0n ivWQnOCYBP86GFbs+HJ63ShVQEOuSrqmsv9cI3UsaFdkYb39XgSliU/itLTGb5/6yX YoHoTlGMH4Cy08TCZPdS2oEjQsT1eRjUuQ3HOELs= To: 50359@debbugs.gnu.org Subject: [PATCH v3 0/3] Add 'generic-git' updater In-Reply-To: References: Message-Id: Date: Fri, 17 Sep 2021 10:04:15 +0200 MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Changes since v2: * Address the feedback by Ludovic and Sarah. One problem I noticed was that when ‘accept-pre-releases?’ is #t, a tag like “1-2-3-alpha” would turn into “1.2.3.alpha”, but I think the correct version string would be “1.2.3-alpha”. Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , Sarah Morgensen X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.9 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Changes since v2: * Address the feedback by Ludovic and Sarah. One problem I noticed was that when ‘accept-pre-releases?’ is #t, a tag like “1-2-3-alpha” would turn into “1.2.3.alpha”, but I think the correct version string would be “1.2.3-alpha”. Content analysis details: (1.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Changes since v2: * Address the feedback by Ludovic and Sarah. One problem I noticed was that when =E2=80=98accept-pre-releases?=E2=80=99 = is #t, a tag like =E2=80=9C1-2-3-alpha=E2=80=9D would turn into =E2=80=9C1.2.3.alpha=E2= =80=9D, but I think the correct version string would be =E2=80=9C1.2.3-alpha=E2=80=9D. I solved the problem by making the pre-release part a separate regexp group, and then appending the pre-release part after extracting the version from the tag. That way, the =E2=80=9C-=E2=80=9D in =E2=80=9C-alpha= =E2=80=9D would not be interpreted as a version delimiter. I also added a new test for testing this. One of the tests in tests/channels.scm is failing; I am not sure why. Before the first and second patches were applied, 6 of them were failing for me, so I guess it=E2=80=99s an improvement. However, on IRC, Ludovic s= aid that all of them were passing (prior to apply my patches). It would be great if people could run the tests before and after applying patches, and see if they pass. Xinglu Chen (3): tests: git: Don't read from the users global Git config file. tests: git: Make 'tag' directive non-interactive. import: Add 'generic-git' updater. Makefile.am | 2 + doc/guix.texi | 34 ++++++ guix/git.scm | 41 ++++++++ guix/import/git.scm | 225 +++++++++++++++++++++++++++++++++++++++ guix/tests/git.scm | 6 +- tests/channels.scm | 2 +- tests/git.scm | 28 +++++ tests/import-git.scm | 245 +++++++++++++++++++++++++++++++++++++++++++ 8 files changed, 580 insertions(+), 3 deletions(-) create mode 100644 guix/import/git.scm create mode 100644 tests/import-git.scm base-commit: 33bc3fb2a5f30a6e21f1b8d6d43867d921bd951c =2D-=20 2.33.0 --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJJBAEBCAAzFiEEAVhh4yyK5+SEykIzrPUJmaL7XHkFAmFES/8VHHB1YmxpY0B5 b2N0b2NlbGwueHl6AAoJEKz1CZmi+1x5McIP/At0XwVS5YLZMbunOsHxMC1/bEcM iD8qc4gatT7bnM3Aa4GF9lKRWcqlTZWWLeIiOX1Omg+GR0ZTxngaRFMGbhkptMG6 iDc4wzRee+sR2rD37fUeL8Q7F3n58+OGUK8BHWTy93Sp4XrSaP7RoXxoelxVejO4 HwsEIAo6VaIHaV4SHj4G0rvELlglYtl+87rC5DPFECRRfs37r4IjF289xXwMzH87 ghqOk2Kuu3mTvMVn1S270gpiN4NWrQYNn/l8lVHkzY9aMUCczmd534FN6HabRUfH XiB3nWXoXYb6ncOGZa7CYNdQrBvMvZrlBGW76MrcFYaXuS088KDjQHIhb8qvYvA1 JA/aejKHzZixJd/eTR0LioHmImVpors6NPb1IBv6BUywf1nXlGnQa/22JijKCKdb A/EqgZ/JN7pSjD+CtqE28E9BWMenwr8Ouz77RENJdzA/k1mqwuuMuGVKNM9vk/5P qpmpo93jeU6JXXAba4GVy9/np+IkDls12j1JEa7cTHhVOpBqqsnnmdCcByRcv8VK wyvyoliikk2MJrOucI5IJnz1E83QRZP+4qgGtsor3cu59bwdYkmn238lu8KlD8CP 22RnIeZ4Y9V9fZC5bq5K+Jf5pAfCoIf2T426YFYGv6r7Zu9U0y0QVoIHZA+e5/Uz OvoEsf4xHHsE41Mn =lomJ -----END PGP SIGNATURE----- --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 17 04:04:45 2021 Received: (at 50359) by debbugs.gnu.org; 17 Sep 2021 08:04:45 +0000 Received: from localhost ([127.0.0.1]:58396 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR8rV-0001KY-Hp for submit@debbugs.gnu.org; Fri, 17 Sep 2021 04:04:45 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:60980 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR8rT-0001KJ-Tz for 50359@debbugs.gnu.org; Fri, 17 Sep 2021 04:04:44 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631865876; bh=zYxeRFVEIpLu7CKw5H+EQM5bjfyRX3CQxzCiN72vIBk=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=CIQAHIXJ4Lmcz9YUe7Eze+j3jth1E3c6bZxTyzBFkyZhe+jCoVyKd5b7Dz4jQfx26 JdbJ90nIswK8ECXowry+DsB49i8S7/jyfoM/IWu30q1xUc04OrJdqQYlxdgOAHQCcK tIH74I+3tAEGbFTF4PURb61JAyODWJSk1YQwR3P8= To: 50359@debbugs.gnu.org Subject: [PATCH v3 1/3] tests: git: Don't read from the users global Git config file. In-Reply-To: References: Message-Id: <981f87c7596e6eb62b49536224d76904c70ba022.1631865317.git.public@yoctocell.xyz> Date: Fri, 17 Sep 2021 10:04:36 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: * guix/tests/git (populate-git-repository): Set the GIT_CONFIG_GLOBAL environment variable to the temporary Git config file. --- guix/tests/git.scm | 2 ++ 1 file changed, 2 insertions(+) diff --git a/guix/tests/git.scm b/guix/tests/git.scm index b8e5f7e643..e11541e83b 100644 --- a/guix/tests/git.scm +++ b/guix/tests/git.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package managemen [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , Sarah Morgensen X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.9 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: * guix/tests/git (populate-git-repository): Set the GIT_CONFIG_GLOBAL environment variable to the temporary Git config file. --- guix/tests/git.scm | 2 ++ 1 file changed, 2 insertions(+) diff --git a/guix/tests/git.scm b/guix/tests/git.scm index b8e5f7e643..e11541e83b 100644 --- a/guix/tests/git.scm +++ b/guix/tests/git.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package managemen [...] Content analysis details: (1.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps * guix/tests/git (populate-git-repository): Set the GIT_CONFIG_GLOBAL environment variable to the temporary Git config file. --- guix/tests/git.scm | 2 ++ 1 file changed, 2 insertions(+) diff --git a/guix/tests/git.scm b/guix/tests/git.scm index b8e5f7e643..e11541e83b 100644 --- a/guix/tests/git.scm +++ b/guix/tests/git.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2019, 2020 Ludovic Court=C3=A8s +;;; Copyright =C2=A9 2021 Xinglu Chen ;;; ;;; This file is part of GNU Guix. ;;; @@ -53,6 +54,7 @@ Return DIRECTORY on success." (with-environment-variables `(("GIT_CONFIG_NOSYSTEM" "1") ("GIT_ATTR_NOSYSTEM" "1") + ("GIT_CONFIG_GLOBAL" ,(string-append home "/.gitconfig")) ("HOME" ,home)) (apply invoke (git-command) "-C" directory command args))))) --=20 2.33.0 From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 17 04:04:54 2021 Received: (at 50359) by debbugs.gnu.org; 17 Sep 2021 08:04:54 +0000 Received: from localhost ([127.0.0.1]:58400 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR8rd-0001Ky-PB for submit@debbugs.gnu.org; Fri, 17 Sep 2021 04:04:53 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:60992 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR8rb-0001Kg-TB for 50359@debbugs.gnu.org; Fri, 17 Sep 2021 04:04:52 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631865884; bh=vyyOjLmJdXrHvS8lKxM+aYMGF8KOAjIWUtn+DFvdEf4=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=gGBDxFDrNV/CIMLHSKvPoP2sk2y0NT8XkTWvlXD6JspoKffVe0K8ryclt1qsoc+B+ VHT51w56RmOf8y8vK4578zUfjTXscwNicUjB6JTcPP6zW6OOZ0STX8hRjVf0v+kwfj 8lIK8PZx8xlQWGGJ8wvSYJBzURDdc4OoKmj782GI= To: 50359@debbugs.gnu.org Subject: [PATCH v3 2/3] tests: git: Make 'tag' directive non-interactive. In-Reply-To: References: Message-Id: Date: Fri, 17 Sep 2021 10:04:44 +0200 MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: When running 'git tag TAGNAME', Git will open up the user's default text editor to make them write a message. This is not desirable when running tests. * guix/tests/git.scm (populate-git-repository): Make the 'tag' directive take an additional argument, and pass it to the '-m' flag. * tests/channels.scm ("channel-news, one entry"): Adjust accordingly [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , Sarah Morgensen X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.9 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: When running 'git tag TAGNAME', Git will open up the user's default text editor to make them write a message. This is not desirable when running tests. * guix/tests/git.scm (populate-git-repository): Make the 'tag' directive take an additional argument, and pass it to the '-m' flag. * tests/channels.scm ("channel-news, one entry"): Adjust accordingly [...] Content analysis details: (1.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps When running 'git tag TAGNAME', Git will open up the user's default text editor to make them write a message. This is not desirable when running tests. * guix/tests/git.scm (populate-git-repository): Make the 'tag' directive take an additional argument, and pass it to the '-m' flag. * tests/channels.scm ("channel-news, one entry"): Adjust accordingly. --- guix/tests/git.scm | 4 ++-- tests/channels.scm | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/guix/tests/git.scm b/guix/tests/git.scm index e11541e83b..e8d4946e87 100644 --- a/guix/tests/git.scm +++ b/guix/tests/git.scm @@ -87,8 +87,8 @@ Return DIRECTORY on success." ((('commit text ('signer fingerprint)) rest ...) (git "commit" "-m" text (string-append "--gpg-sign=" fingerprint)) (loop rest)) - ((('tag name) rest ...) - (git "tag" name) + ((('tag name text) rest ...) + (git "tag" "-m" text name) (loop rest)) ((('branch name) rest ...) (git "branch" name) diff --git a/tests/channels.scm b/tests/channels.scm index 0264369d9e..8f7ff1e7a8 100644 --- a/tests/channels.scm +++ b/tests/channels.scm @@ -329,7 +329,7 @@ (commit "first commit") (add "src/a.txt" "A") (commit "second commit") - (tag "tag-for-first-news-entry") + (tag "tag-for-first-news-entry" "First news entry!") (add "news.scm" ,(lambda (repository) (let ((previous -- 2.33.0 From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 17 04:05:01 2021 Received: (at 50359) by debbugs.gnu.org; 17 Sep 2021 08:05:01 +0000 Received: from localhost ([127.0.0.1]:58403 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR8rk-0001LI-1I for submit@debbugs.gnu.org; Fri, 17 Sep 2021 04:05:01 -0400 Received: from h87-96-130-155.cust.a3fiber.se ([87.96.130.155]:32770 helo=mail.yoctocell.xyz) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mR8rh-0001Kr-Iu for 50359@debbugs.gnu.org; Fri, 17 Sep 2021 04:04:58 -0400 From: Xinglu Chen DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=yoctocell.xyz; s=mail; t=1631865890; bh=D+CxAc00K/mVZ8z7HTG3jJ/4j5dpwTNXJ80qjLI5fiQ=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=lm099p7P3vnGrQL8yp0BDioakvkQRa0hNlPfz+z7mlWgIurd3t7ALGevxKavLjCHs eXTZ7oYJXK69A7eEYEF7ELt72xfvNitDnbV0uNXmFmkMDsGsvTwWkAxBfrFuZL3Ed4 nlWyUy1PdegTW9VYKEFV5nEBtJbubB0BAO+bheyM= To: 50359@debbugs.gnu.org Subject: [PATCH v3 3/3] import: Add 'generic-git' updater. In-Reply-To: References: Message-Id: <78cfc99b42371c9c21189f805030f11ca7e78861.1631865317.git.public@yoctocell.xyz> Date: Fri, 17 Sep 2021 10:04:49 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 2.9 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: * guix/git.scm (ls-remote-refs): New procedure. * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. * guix/import/git.scm: New file. * doc/guix.texi (Invoking guix refresh): Document [...] Content analysis details: (2.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps X-Debbugs-Envelope-To: 50359 Cc: Ludovic =?utf-8?Q?Court=C3=A8s?= , Sarah Morgensen X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.9 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: * guix/git.scm (ls-remote-refs): New procedure. * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. * guix/import/git.scm: New file. * doc/guix.texi (Invoking guix refresh): Document [...] Content analysis details: (1.9 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_PASS SPF: sender matches SPF record 2.0 PDS_OTHER_BAD_TLD Untrustworthy TLDs [URI: yoctocell.xyz (xyz)] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.5 FROM_SUSPICIOUS_NTLD From abused NTLD 0.4 RDNS_DYNAMIC Delivered to internal network by host with dynamic-looking rDNS -1.0 MAILING_LIST_MULTI Multiple indicators imply a widely-seen list manager 0.0 PDS_RDNS_DYNAMIC_FP RDNS_DYNAMIC with FP steps * guix/git.scm (ls-remote-refs): New procedure. * tests/git.scm ("remote-refs" "remote-refs: only tags"): New tests. * guix/import/git.scm: New file. * doc/guix.texi (Invoking guix refresh): Document it. * tests/import-git.scm: New test file. * Makefile.am (MODULES, SCM_TESTS): Register the new files. Co-authored-by: Sarah Morgensen --- Makefile.am | 2 + doc/guix.texi | 34 ++++++ guix/git.scm | 41 ++++++++ guix/import/git.scm | 225 +++++++++++++++++++++++++++++++++++++++ tests/git.scm | 28 +++++ tests/import-git.scm | 245 +++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 575 insertions(+) create mode 100644 guix/import/git.scm create mode 100644 tests/import-git.scm diff --git a/Makefile.am b/Makefile.am index 299bc0f7fb..f3bdc7448e 100644 --- a/Makefile.am +++ b/Makefile.am @@ -254,6 +254,7 @@ MODULES =3D \ guix/import/egg.scm \ guix/import/elpa.scm \ guix/import/gem.scm \ + guix/import/git.scm \ guix/import/github.scm \ guix/import/gnome.scm \ guix/import/gnu.scm \ @@ -473,6 +474,7 @@ SCM_TESTS =3D \ tests/graph.scm \ tests/gremlin.scm \ tests/hackage.scm \ + tests/import-git.scm \ tests/import-utils.scm \ tests/inferior.scm \ tests/lint.scm \ diff --git a/doc/guix.texi b/doc/guix.texi index 2fc9687910..6436e83a7c 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -11928,6 +11928,40 @@ the updater for @uref{https://launchpad.net, Launc= hpad} packages. @item generic-html a generic updater that crawls the HTML page where the source tarball of the package is hosted, when applicable. + +@item generic-git +a generic updater for packages hosted on Git repositories. It tries to +be smart about parsing Git tag names, but if it is not able to parse the +tag name and compare tags correctly, users can define the following +properties for a package. + +@itemize +@item @code{release-tag-prefix}: a regular expression for matching a prefi= x of +the tag name. + +@item @code{release-tag-suffix}: a regular expression for matching a suffi= x of +the tag name. + +@item @code{release-tag-version-delimiter}: a string used as the delimiter= in +the tag name for separating the numbers of the version. + +@item @code{accept-pre-releases}: by default, the updater will ignore +pre-releases; to make it also look for pre-releases, set the this +property to @code{#t}. + +@end itemize + +@lisp +(package + (name "foo") + ;; ... + (properties + '((release-tag-prefix . "^release0-") + (release-tag-suffix . "[a-z]?$") + (release-tag-version-delimiter . ":")))) +@end lisp + + @end table =20 For instance, the following command only checks for updates of Emacs diff --git a/guix/git.scm b/guix/git.scm index acc48fd12f..bbff4fc890 100644 --- a/guix/git.scm +++ b/guix/git.scm @@ -57,6 +57,8 @@ commit-difference commit-relation =20 + remote-refs + git-checkout git-checkout? git-checkout-url @@ -571,6 +573,45 @@ objects: 'ancestor (meaning that OLD is an ancestor of= NEW), 'descendant, or (if (set-contains? oldest new) 'descendant 'unrelated)))))) + +;; +;;; Remote operations. +;;; + +(define* (remote-refs url #:key tags?) + "Return the list of references advertised at Git repository URL. If TAG= S? +is true, limit to only refs/tags." + (define (ref? ref) + ;; Like `git ls-remote --refs', only show actual references. + (and (string-prefix? "refs/" ref) + (not (string-suffix? "^{}" ref)))) + + (define (tag? ref) + (string-prefix? "refs/tags/" ref)) + + (define (include? ref) + (and (ref? ref) + (or (not tags?) (tag? ref)))) + + (define (remote-head->ref remote) + (let ((name (remote-head-name remote))) + (and (include? name) + name))) + + (with-libgit2 + (call-with-temporary-directory + (lambda (cache-directory) + (let* ((repository (repository-init cache-directory)) + ;; Create an in-memory remote so we don't touch disk. + (remote (remote-create-anonymous repository url))) + (remote-connect remote) + + (let* ((remote-heads (remote-ls remote)) + (refs (filter-map remote-head->ref remote-heads))) + ;; Wait until we're finished with the repository before closing = it. + (remote-disconnect remote) + (repository-close! repository) + refs)))))) =20 ;;; diff --git a/guix/import/git.scm b/guix/import/git.scm new file mode 100644 index 0000000000..1eb219f3fe --- /dev/null +++ b/guix/import/git.scm @@ -0,0 +1,225 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright =C2=A9 2021 Xinglu Chen +;;; Copyright =C2=A9 2021 Sarah Morgensen +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix import git) + #:use-module (guix build utils) + #:use-module (guix diagnostics) + #:use-module (guix git) + #:use-module (guix git-download) + #:use-module (guix i18n) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module (ice-9 format) + #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 regex) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) + #:use-module (srfi srfi-34) + #:use-module (srfi srfi-35) + #:export (%generic-git-updater + + ;; For tests. + latest-git-tag-version)) + +;;; Commentary: +;;; +;;; This module provides a generic package updater for packages hosted on = Git +;;; repositories. +;;; +;;; It tries to be smart about tag names, but if it is not automatically a= ble +;;; to parse the tag names correctly, users can set the `release-tag-prefi= x', +;;; `release-tag-suffix' and `release-tag-version-delimiter' properties of= the +;;; package to make the updater parse the Git tag name correctly. +;;; +;;; Possible improvements: +;;; +;;; * More robust method for trying to guess the delimiter. Maybe look at= the +;;; previous version/tag combo to determine the delimiter. +;;; +;;; * Differentiate between "normal" versions, e.g., 1.2.3, and dates, e.g= ., +;;; 2021.12.31. Honor a `release-tag-date-scheme?' property? +;;; +;;; Code: + +;;; Errors & warnings + +(define-condition-type &git-no-valid-tags-error &error + git-no-valid-tags-error?) + +(define (git-no-valid-tags-error) + (raise (condition (&message (message "no valid tags found")) + (&git-no-valid-tags-error)))) + +(define-condition-type &git-no-tags-error &error + git-no-tags-error?) + +(define (git-no-tags-error) + (raise (condition (&message (message "no tags were found")) + (&git-no-tags-error)))) + + +;;; Updater + +(define %pre-release-words + '("alpha" "beta" "rc" "dev" "test" "pre")) + +(define %pre-release-rx + (map (lambda (word) + (make-regexp (string-append ".+" word) regexp/icase)) + %pre-release-words)) + +(define* (version-mapping tags #:key prefix suffix delim pre-releases?) + "Given a list of Git TAGS, return an association list where the car is t= he +version corresponding to the tag, and the cdr is the name of the tag." + (define (guess-delimiter) + (let ((total (length tags)) + (dots (reduce + 0 (map (cut string-count <> #\.) tags))) + (dashes (reduce + 0 (map (cut string-count <> #\-) tags))) + (underscores (reduce + 0 (map (cut string-count <> #\_) tags)))) + (cond + ((>=3D dots (* total 0.35)) ".") + ((>=3D dashes (* total 0.8)) "-") + ((>=3D underscores (* total 0.8)) "_") + (else "")))) + + (define delim-rx (regexp-quote (or delim (guess-delimiter)))) + (define suffix-rx (string-append (or suffix "") "$")) + (define prefix-rx (string-append "^" (or prefix "[^[:digit:]]*"))) + (define pre-release-rx + (if pre-releases? + (string-append "(.*(" (string-join %pre-release-words "|") ").*)") + "")) + + (define tag-rx + (string-append prefix-rx "([[:digit:]][^" delim-rx "[:punct:]]*" + "(" delim-rx "[^[:punct:]" delim-rx "]+)" + ;; If there are no delimiters, it could mean that the + ;; version just contains one number (e.g., "2"), thus, = use + ;; "*" instead of "+" to match zero or more numbers. + (if (string=3D? delim-rx "") "*" "+") ")" + ;; We don't want the pre-release stuff (e.g., "-alpha")= be + ;; part of the first group; otherwise, the "-" in "-alp= ha" + ;; might be interpreted as a delimiter, and thus replac= ed + ;; with "." + pre-release-rx suffix-rx)) + + + + (define (get-version tag) + (let ((tag-match (regexp-exec (make-regexp tag-rx) tag))) + (and=3D> (and tag-match + (regexp-substitute/global + #f delim-rx (match:substring tag-match 1) + ;; If there were no delimiters, don't insert ".". + 'pre (if (string=3D? delim-rx "") "" ".") 'post)) + (lambda (version) + (if pre-releases? + (string-append version (match:substring tag-match 3)) + version))))) + + (define (entry tag) + %pre-release-rx)) + + (let* ((tags (map (cut string-drop <> (string-length "refs/tags/")) + (remote-refs url #:tags? #t))) + (versions->tags + (version-mapping (if pre-releases? + tags + (filter (negate pre-release?) tags)) + #:prefix prefix + #:suffix suffix + #:delim delim + #:pre-releases? pre-releases?))) + (cond + ((null? tags) + (git-no-tags-error)) + ((null? versions->tags) + (git-no-valid-tags-error)) + (else + (match (last versions->tags) + ((version . tag) + (values version tag))))))) + +(define (latest-git-tag-version package) + "Given a PACKAGE, return the latest version of it, or #f if the latest v= ersion +could not be determined." + (guard (c ((or (git-no-tags-error? c) (git-no-valid-tags-error? c)) + (warning (or (package-field-location package 'source) + (package-location package)) + (G_ "~a for ~a~%") + (condition-message c) + (package-name package)) + #f) + ((eq? (exception-kind c) 'git-error) + (warning (or (package-field-location package 'source) + (package-location package)) + (G_ "failed to fetch Git repository for ~a~%") + (package-name package)) + #f)) + (let* ((source (package-source package)) + (url (git-reference-url (origin-uri source))) + (property (cute assq-ref (package-properties package) <>))) + (latest-tag url + #:prefix (property 'release-tag-prefix) + #:suffix (property 'release-tag-suffix) + #:delim (property 'release-tag-version-delimiter) + #:pre-releases? (property 'accept-pre-releases?))))) + +(define (git-package? package) + "Return true if PACKAGE is hosted on a Git repository." + (match (package-source package) + ((? origin? origin) + (and (eq? (origin-method origin) git-fetch) + (git-reference? (origin-uri origin)))) + (_ #f))) + +(define (latest-git-release package) + "Return an for the latest release of PACKAGE." + (let* ((name (package-name package)) + (old-version (package-version package)) + (url (git-reference-url (origin-uri (package-source package)))) + (new-version (latest-git-tag-version package))) + + (and new-version + (upstream-source + (package name) + (version new-version) + (urls (list url)))))) + +(define %generic-git-updater + (upstream-updater + (name 'generic-git) + (description "Updater for packages hosted on Git repositories") + (pred git-package?) + (latest latest-git-release))) diff --git a/tests/git.scm b/tests/git.scm index aa4f03ca62..d0646bbc85 100644 --- a/tests/git.scm +++ b/tests/git.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2019, 2020 Ludovic Court=C3=A8s +;;; Copyright =C2=A9 2021 Xinglu Chen . + +(define-module (test-import-git) + #:use-module (git) + #:use-module (guix git) + #:use-module (guix tests) + #:use-module (guix packages) + #:use-module (guix import git) + #:use-module (guix git-download) + #:use-module (guix tests git) + #:use-module (guix build utils) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-64)) + +;; Test the (guix import git) tools. + +(test-begin "git") + +(define* (make-package directory version #:optional (properties '())) + (dummy-package "test-package" + (version version) + (properties properties) + (source + (origin + (method git-fetch) + (uri (git-reference + (url (string-append "file://" directory)) + (commit version))) + (sha256 + (base32 + "0000000000000000000000000000000000000000000000000000")))))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: no custom prefix, suffix, and delimit= er" + "1.0.1" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "1.0.1" "Release 1.0.1")) + (let ((package (make-package directory "1.0.0"))) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: custom prefix, no suffix and delimite= r" + "1.0.1" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "prefix-1.0.1" "Release 1.0.1")) + (let ((package (make-package directory "1.0.0" + '((release-tag-prefix . "prefix-"))))) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: custom suffix, no prefix and delimite= r" + "1.0.1" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "1.0.1-suffix-123" "Release 1.0.1")) + (let ((package (make-package directory "1.0.0" + '((release-tag-suffix . "-suffix-[0-9]*")= )))) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: custom delimiter, no prefix and suffi= x" + "2021.09.07" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "2021-09-07" "Release 2021-09-07")) + (let ((package (make-package directory "2021-09-06" + '((release-tag-version-delimiter . "-")))= )) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: empty delimiter, no prefix and suffix" + "20210907" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "20210907" "Release 20210907")) + (let ((package (make-package directory "20210906" + '((release-tag-version-delimiter . ""))))) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: custom prefix and suffix, no delimite= r" + "2.0.0" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "Release-2.0.0suffix-1" "Release 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((release-tag-prefix . "Release-") + (release-tag-suffix . "suffix-[0-9]")))= )) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: custom prefix, suffix, and delimiter" + "2.0.0" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "Release-2_0_0suffix-1" "Release 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((release-tag-prefix . "Release-") + (release-tag-suffix . "suffix-[0-9]") + (release-tag-version-delimiter . "_")))= )) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: only pre-releases available" + #f + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "2.0.0-rc1" "Release candidate for 2.0.0")) + (let ((package (make-package directory "1.0.0"))) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: accept pre-releases" + "2.0.0-rc1" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "2.0.0-rc1" "Release candidate for 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((accept-pre-releases? . #t))))) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: accept pre-releases, and custom prefi= x" + "2.0.0-rc1" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "version-2.0.0-rc1" "Release candidate for 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((accept-pre-releases? . #t) + (release-tag-prefix . "version-"))))) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: accept pre-releases, and custom suffi= x" + "2.0.0-rc1" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "2.0.0-rc1-suffix" "Release candidate for 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((accept-pre-releases? . #t) + (release-tag-suffix . "-suffix"))))) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: accept pre-releases, delimiter confli= cts with pre-release part" + "2.0.0_alpha" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "2_0_0_alpha" "Alpha release for 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((accept-pre-releases? . #t) + (release-tag-version-delimiter . "_")))= )) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: accept pre-releases, and custom suffi= x and prefix" + "2.0.0-alpha" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "prefix123-2.0.0-alpha-suffix" "Alpha release for 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((accept-pre-releases? . #t) + (release-tag-prefix . "prefix[0-9]{3}-") + (release-tag-suffix . "-suffix"))))) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: accept pre-releases, and custom suffi= x, prefix, and delimiter" + "2.0.0-alpha" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "prefix123-2-0-0-alpha-suffix" "Alpha release for 2.0.0")) + (let ((package (make-package directory "1.0.0" + '((accept-pre-releases? . #t) + (release-tag-prefix . "prefix[0-9]{3}-") + (release-tag-suffix . "-suffix") + (release-tag-version-delimiter . "-")))= )) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: accept pre-releases, no delimiter, an= d custom suffix, prefix" + "2alpha" + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "prefix123-2alpha-suffix" "Alpha release for version 2")) + (let ((package (make-package directory "1.0.0" + '((accept-pre-releases? . #t) + (release-tag-prefix . "prefix[0-9]{3}-") + (release-tag-suffix . "-suffix") + (release-tag-version-delimiter . ""))))) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: no tags found" + #f + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit")) + (let ((package (make-package directory "1.0.0"))) + (latest-git-tag-version package)))) + +(unless (which (git-command)) (test-skip 1)) +(test-equal "latest-git-tag-version: no valid tags found" + #f + (with-temporary-git-repository directory + '((add "a.txt" "A") + (commit "First commit") + (tag "Test" "Test tag")) + (let ((package (make-package directory "1.0.0"))) + (latest-git-tag-version package)))) + +(test-end "git") --=20 2.33.0 From debbugs-submit-bounces@debbugs.gnu.org Sat Sep 18 13:47:44 2021 Received: (at 50359-done) by debbugs.gnu.org; 18 Sep 2021 17:47:44 +0000 Received: from localhost ([127.0.0.1]:36806 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mReRE-0000ej-0b for submit@debbugs.gnu.org; Sat, 18 Sep 2021 13:47:44 -0400 Received: from eggs.gnu.org ([209.51.188.92]:59206) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mReRC-0000e9-7J for 50359-done@debbugs.gnu.org; Sat, 18 Sep 2021 13:47:42 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:60564) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mReR6-0000Pt-6w; Sat, 18 Sep 2021 13:47:36 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=35824 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mReR5-0003Mr-V4; Sat, 18 Sep 2021 13:47:36 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Xinglu Chen Subject: Re: [PATCH v3 0/3] Add 'generic-git' updater References: X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: Jour du =?utf-8?Q?G=C3=A9nie?= de =?utf-8?Q?l'Ann?= =?utf-8?Q?=C3=A9e?= 229 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sat, 18 Sep 2021 19:47:34 +0200 In-Reply-To: (Xinglu Chen's message of "Fri, 17 Sep 2021 10:04:15 +0200") Message-ID: <871r5lu66h.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.3 (/) X-Debbugs-Envelope-To: 50359-done Cc: Sarah Morgensen , 50359-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.3 (-) Hello! Xinglu Chen skribis: > Changes since v2: > > * Address the feedback by Ludovic and Sarah. > > One problem I noticed was that when =E2=80=98accept-pre-releases?=E2=80= =99 is #t, a tag > like =E2=80=9C1-2-3-alpha=E2=80=9D would turn into =E2=80=9C1.2.3.alpha= =E2=80=9D, but I think the > correct version string would be =E2=80=9C1.2.3-alpha=E2=80=9D. > > I solved the problem by making the pre-release part a separate regexp > group, and then appending the pre-release part after extracting the > version from the tag. That way, the =E2=80=9C-=E2=80=9D in =E2=80=9C-alp= ha=E2=80=9D would not be > interpreted as a version delimiter. I also added a new test for testing > this. I think that=E2=80=99s fine; this is all guesswork anyway, and there are al= ways cases where we=E2=80=99ll get it wrong. What=E2=80=99s useful though is te= sts to guard against regressions in the heuristics. > One of the tests in tests/channels.scm is failing; I am not sure why. > Before the first and second patches were applied, 6 of them were failing > for me, so I guess it=E2=80=99s an improvement. However, on IRC, Ludovic= said > that all of them were passing (prior to apply my patches). It would > be great if people could run the tests before and after applying > patches, and see if they pass. Turns out that the failure in tests/channels.scm was a real bug: =E2=80=98channel-news-entry-commit=E2=80=99 was not resolving annotated tag= s correctly (it would return the ID of the tag instead of the ID of the commit pointed to by the tag). Fixed in 778c1fb4eabbb48c05f6c7555c89466d5249ebce. > tests: git: Don't read from the users global Git config file. > tests: git: Make 'tag' directive non-interactive. > import: Add 'generic-git' updater. Applied! I changed the second patch to preserve support for non-annotated tags and to leave tests/channels.scm unchanged. BTW, =E2=80=9Cgit tag xyz=E2=80=9D is not interactive AFAICS. Thanks Sarah & Xinglu for this work! Let=E2=80=99s update our packages! := -) Ludo=E2=80=99. From unknown Sun Jun 15 08:40:16 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Sun, 17 Oct 2021 11:24:06 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator