From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 13 16:43:43 2021 Received: (at submit) by debbugs.gnu.org; 13 Mar 2021 21:43:43 +0000 Received: from localhost ([127.0.0.1]:60823 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC2w-0000eW-RC for submit@debbugs.gnu.org; Sat, 13 Mar 2021 16:43:43 -0500 Received: from lists.gnu.org ([209.51.188.17]:47488) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC2t-0000eN-L5 for submit@debbugs.gnu.org; Sat, 13 Mar 2021 16:43:41 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:43738) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lLC2t-0004aK-Db for guix-patches@gnu.org; Sat, 13 Mar 2021 16:43:39 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:42755) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLC2t-0001Eb-69; Sat, 13 Mar 2021 16:43:39 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=39270 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lLC2s-0004ju-Og; Sat, 13 Mar 2021 16:43:38 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: guix-patches@gnu.org Subject: [PATCH 0/7] Add 'generic-html' updater Date: Sat, 13 Mar 2021 22:43:26 +0100 Message-Id: <20210313214326.28052-1-ludo@gnu.org> X-Mailer: git-send-email 2.30.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! These patches allow ‘guix refresh’ coverage to go from 78% to 88% as reported by ‘guix refresh --list-updaters’ (both are probably slightly overestimated) by adding a new ‘generic-html’ updater. The updater crawls the web page where the package’s source tarball is stored, using Guile-Lib’s (htmlprag), which we depend on since commit 02e2e093e858e8a0ca7bd66c1f1f6fd0a1705edb. Among other things, it handles freedesktop.org packages. Feedback welcome! Thanks, Ludo’. Ludovic Courtès (7): gnu-maintenance: Use (htmlprag) for 'latest-html-release'. gnu-maintenance: 'latest-html-release' considers non-relative URLs. gnu-maintenance: 'release-file?' rejects checksum files. gnu-maintenance: 'latest-html-release' can determine signature file name. gnu-maintenance: 'latest-html-release' better computes version number. gnu-maintenance: Add 'generic-html' updater. gnu: hwloc: Add 'release-monitoring-url' property. doc/guix.texi | 6 +- gnu/packages/mpi.scm | 6 ++ guix/gnu-maintenance.scm | 136 ++++++++++++++++++++++++++++----------- 3 files changed, 108 insertions(+), 40 deletions(-) -- 2.30.1 From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 13 16:46:34 2021 Received: (at 47126) by debbugs.gnu.org; 13 Mar 2021 21:46:34 +0000 Received: from localhost ([127.0.0.1]:60829 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5h-0000k0-9U for submit@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:33 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54478) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5g-0000jh-6l for 47126@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:32 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:42777) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLC5a-0002qP-GG; Sat, 13 Mar 2021 16:46:26 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=39284 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lLC5a-0004wi-2U; Sat, 13 Mar 2021 16:46:26 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 47126@debbugs.gnu.org Subject: [PATCH 1/7] gnu-maintenance: Use (htmlprag) for 'latest-html-release'. Date: Sat, 13 Mar 2021 22:46:14 +0100 Message-Id: <20210313214620.28186-1-ludo@gnu.org> X-Mailer: git-send-email 2.30.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 47126 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) * guix/gnu-maintenance.scm (html->sxml): Remove. Autoload (htmlprag) instead. * doc/guix.texi (Requirements): Mention 'guix refresh' for the Guile-Lib dependency. --- doc/guix.texi | 3 ++- guix/gnu-maintenance.scm | 13 +------------ 2 files changed, 3 insertions(+), 13 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 4cf241c56a..97094a7d0a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -865,7 +865,8 @@ the @code{crate} importer (@pxref{Invoking guix import}). @item @uref{https://www.nongnu.org/guile-lib/doc/ref/htmlprag/, Guile-Lib} for -the @code{go} importer (@pxref{Invoking guix import}). +the @code{go} importer (@pxref{Invoking guix import}) and for some of +the ``updaters'' (@pxref{Invoking guix refresh}). @item When @url{http://www.bzip.org, libbz2} is available, diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm index 9e393d18cd..febed57c3a 100644 --- a/guix/gnu-maintenance.scm +++ b/guix/gnu-maintenance.scm @@ -38,6 +38,7 @@ #:use-module (guix upstream) #:use-module (guix packages) #:autoload (zlib) (call-with-gzip-input-port) + #:autoload (htmlprag) (html->sxml) ;from Guile-Lib #:export (gnu-package-name gnu-package-mundane-name gnu-package-copyright-holder @@ -447,18 +448,6 @@ hosted on ftp.gnu.org, or not under that name (this is the case for ;;; Latest HTTP release. ;;; -(define (html->sxml port) - "Read HTML from PORT and return the corresponding SXML tree." - (let ((str (get-string-all port))) - (catch #t - (lambda () - ;; XXX: This is the poor developer's HTML-to-XML converter. It's good - ;; enough for directory listings at but if - ;; needed we could resort to (htmlprag) from Guile-Lib. - (call-with-input-string (string-replace-substring str "
" "
") - xml->sxml)) - (const '(html))))) ;parse error - (define (html-links sxml) "Return the list of links found in SXML, the SXML tree of an HTML page." (let loop ((sxml sxml) -- 2.30.1 From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 13 16:46:37 2021 Received: (at 47126) by debbugs.gnu.org; 13 Mar 2021 21:46:37 +0000 Received: from localhost ([127.0.0.1]:60834 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5l-0000kT-12 for submit@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:37 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54482) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5h-0000jj-0G for 47126@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:33 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:42779) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLC5b-0002qt-7v; Sat, 13 Mar 2021 16:46:27 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=39284 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lLC5a-0004wi-PG; Sat, 13 Mar 2021 16:46:27 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 47126@debbugs.gnu.org Subject: [PATCH 2/7] gnu-maintenance: 'latest-html-release' considers non-relative URLs. Date: Sat, 13 Mar 2021 22:46:15 +0100 Message-Id: <20210313214620.28186-2-ludo@gnu.org> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210313214620.28186-1-ludo@gnu.org> References: <20210313214620.28186-1-ludo@gnu.org> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 47126 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) * guix/gnu-maintenance.scm (latest-html-release): Allow for URL to be an arbitrary URL rather than a relative URL reference. --- guix/gnu-maintenance.scm | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm index febed57c3a..98d326e500 100644 --- a/guix/gnu-maintenance.scm +++ b/guix/gnu-maintenance.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès +;;; Copyright © 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ludovic Courtès ;;; Copyright © 2012, 2013 Nikita Karetnikov ;;; Copyright © 2021 Simon Tournier ;;; @@ -479,19 +479,21 @@ return the corresponding signature URL, or #f it signatures are unavailable." (port (http-fetch/cached uri #:ttl 3600)) (sxml (html->sxml port))) (define (url->release url) - (and (string=? url (basename url)) ;relative reference? - (release-file? package url) - (let-values (((name version) - (package-name->name+version - (tarball-sans-extension url) - #\-))) - (upstream-source - (package name) - (version version) - (urls (list (string-append base-url directory "/" url))) - (signature-urls - (list (file->signature - (string-append base-url directory "/" url)))))))) + (let* ((base (basename url)) + (url (if (string=? base url) + (string-append base-url directory "/" url) + url))) + (and (release-file? package base) + (let-values (((name version) + (package-name->name+version + (tarball-sans-extension base) + #\-))) + (upstream-source + (package name) + (version version) + (urls (list url)) + (signature-urls + (list (file->signature url)))))))) (define candidates (filter-map url->release (html-links sxml))) -- 2.30.1 From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 13 16:46:41 2021 Received: (at 47126) by debbugs.gnu.org; 13 Mar 2021 21:46:41 +0000 Received: from localhost ([127.0.0.1]:60837 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5l-0000kW-Dm for submit@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:41 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54492) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5j-0000jp-1g for 47126@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:35 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:42780) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLC5c-0002rF-40; Sat, 13 Mar 2021 16:46:29 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=39284 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lLC5b-0004wi-HP; Sat, 13 Mar 2021 16:46:27 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 47126@debbugs.gnu.org Subject: [PATCH 3/7] gnu-maintenance: 'release-file?' rejects checksum files. Date: Sat, 13 Mar 2021 22:46:16 +0100 Message-Id: <20210313214620.28186-3-ludo@gnu.org> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210313214620.28186-1-ludo@gnu.org> References: <20210313214620.28186-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 47126 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) * guix/gnu-maintenance.scm (release-file?): Reject ".md5sum", ".sha1sum", and ".sha256sum". --- guix/gnu-maintenance.scm | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm index 98d326e500..a8b24fa336 100644 --- a/guix/gnu-maintenance.scm +++ b/guix/gnu-maintenance.scm @@ -247,7 +247,9 @@ network to check in GNU's database." (define (release-file? project file) "Return #f if FILE is not a release tarball of PROJECT, otherwise return true." - (and (not (member (file-extension file) '("sig" "sign" "asc"))) + (and (not (member (file-extension file) + '("sig" "sign" "asc" + "md5sum" "sha1sum" "sha256sum"))) (and=> (regexp-exec %tarball-rx file) (lambda (match) ;; Filter out unrelated files, like `guile-www-1.1.1'. -- 2.30.1 From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 13 16:46:41 2021 Received: (at 47126) by debbugs.gnu.org; 13 Mar 2021 21:46:41 +0000 Received: from localhost ([127.0.0.1]:60840 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5p-0000ks-8t for submit@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:41 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54488) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5j-0000jo-1f for 47126@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:35 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:42782) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLC5d-0002ra-Ck; Sat, 13 Mar 2021 16:46:29 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=39284 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lLC5c-0004wi-DF; Sat, 13 Mar 2021 16:46:29 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 47126@debbugs.gnu.org Subject: [PATCH 4/7] gnu-maintenance: 'latest-html-release' can determine signature file name. Date: Sat, 13 Mar 2021 22:46:17 +0100 Message-Id: <20210313214620.28186-4-ludo@gnu.org> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210313214620.28186-1-ludo@gnu.org> References: <20210313214620.28186-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 47126 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) * guix/gnu-maintenance.scm (latest-html-release): #:file->signature defaults to #f. [file->signature/guess]: New procedure. [url->release]: Use it when FILE->SIGNATURE is #f. Introduce 'links' variable. (url-prefix-rewrite): Check whether URL is true before calling 'string-prefix?'. (latest-savannah-release): Adjust comment about detached signatures. --- guix/gnu-maintenance.scm | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm index a8b24fa336..3bffa4d11e 100644 --- a/guix/gnu-maintenance.scm +++ b/guix/gnu-maintenance.scm @@ -470,16 +470,29 @@ hosted on ftp.gnu.org, or not under that name (this is the case for #:key (base-url "https://kernel.org/pub") (directory (string-append "/" package)) - (file->signature (cut string-append <> ".sig"))) + file->signature) "Return an for the latest release of PACKAGE (a string) on SERVER under DIRECTORY, or #f. BASE-URL should be the URL of an HTML page, typically a directory listing as found on 'https://kernel.org/pub'. -FILE->SIGNATURE must be a procedure; it is passed a source file URL and must -return the corresponding signature URL, or #f it signatures are unavailable." - (let* ((uri (string->uri (string-append base-url directory "/"))) - (port (http-fetch/cached uri #:ttl 3600)) - (sxml (html->sxml port))) +When FILE->SIGNATURE is omitted or #f, guess the detached signature file name, +if any. Otherwise, FILE->SIGNATURE must be a procedure; it is passed a source +file URL and must return the corresponding signature URL, or #f it signatures +are unavailable." + (let* ((uri (string->uri (string-append base-url directory "/"))) + (port (http-fetch/cached uri #:ttl 3600)) + (sxml (html->sxml port)) + (links (delete-duplicates (html-links sxml)))) + (define (file->signature/guess url) + (let ((base (basename url))) + (any (lambda (link) + (any (lambda (extension) + (and (string=? (string-append base extension) + (basename link)) + (string-append url extension))) + '(".asc" ".sig" ".sign"))) + links))) + (define (url->release url) (let* ((base (basename url)) (url (if (string=? base url) @@ -495,10 +508,10 @@ return the corresponding signature URL, or #f it signatures are unavailable." (version version) (urls (list url)) (signature-urls - (list (file->signature url)))))))) + (list ((or file->signature file->signature/guess) url)))))))) (define candidates - (filter-map url->release (html-links sxml))) + (filter-map url->release links)) (close-port port) (match candidates @@ -614,7 +627,7 @@ releases are on gnu.org." (define (url-prefix-rewrite old new) "Return a one-argument procedure that rewrites URL prefix OLD to NEW." (lambda (url) - (if (string-prefix? old url) + (if (and url (string-prefix? old url)) (string-append new (string-drop url (string-length old))) url))) @@ -646,9 +659,8 @@ releases are on gnu.org." (directory (dirname (uri-path uri))) (rewrite (url-prefix-rewrite %savannah-base "mirror://savannah"))) - ;; Note: We use the default 'file->signature', which adds ".sig", but not - ;; all projects on Savannah follow that convention: some use ".asc" and - ;; perhaps some lack signatures altogether. + ;; Note: We use the default 'file->signature', which adds ".sig", ".asc", + ;; or whichever detached signature naming scheme PACKAGE uses. (and=> (latest-html-release package #:base-url %savannah-base #:directory directory) -- 2.30.1 From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 13 16:46:42 2021 Received: (at 47126) by debbugs.gnu.org; 13 Mar 2021 21:46:42 +0000 Received: from localhost ([127.0.0.1]:60842 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5p-0000l2-Md for submit@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:41 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54508) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5k-0000js-VH for 47126@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:38 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:42783) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLC5e-0002rl-61; Sat, 13 Mar 2021 16:46:31 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=39284 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lLC5d-0004wi-MB; Sat, 13 Mar 2021 16:46:29 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 47126@debbugs.gnu.org Subject: [PATCH 5/7] gnu-maintenance: 'latest-html-release' better computes version number. Date: Sat, 13 Mar 2021 22:46:18 +0100 Message-Id: <20210313214620.28186-5-ludo@gnu.org> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210313214620.28186-1-ludo@gnu.org> References: <20210313214620.28186-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 47126 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) * guix/gnu-maintenance.scm (latest-html-release): Use 'tarball->version' rather than 'package-name->name+version' to extract the version number. This fixes problems with packages like 'netsurf' and 'libdom' that have "-src" in their tarball name, where "src" would be taken as the new version number. --- guix/gnu-maintenance.scm | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm index 3bffa4d11e..5aa16acfde 100644 --- a/guix/gnu-maintenance.scm +++ b/guix/gnu-maintenance.scm @@ -499,12 +499,9 @@ are unavailable." (string-append base-url directory "/" url) url))) (and (release-file? package base) - (let-values (((name version) - (package-name->name+version - (tarball-sans-extension base) - #\-))) + (let ((version (tarball->version base))) (upstream-source - (package name) + (package package) (version version) (urls (list url)) (signature-urls -- 2.30.1 From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 13 16:46:42 2021 Received: (at 47126) by debbugs.gnu.org; 13 Mar 2021 21:46:42 +0000 Received: from localhost ([127.0.0.1]:60844 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5p-0000l9-WB for submit@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:42 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54516) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5n-0000kD-KD for 47126@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:39 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:42786) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLC5i-0002t6-DF; Sat, 13 Mar 2021 16:46:34 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=39284 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lLC5g-0004wi-1u; Sat, 13 Mar 2021 16:46:33 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 47126@debbugs.gnu.org Subject: [PATCH 7/7] gnu: hwloc: Add 'release-monitoring-url' property. Date: Sat, 13 Mar 2021 22:46:20 +0100 Message-Id: <20210313214620.28186-7-ludo@gnu.org> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210313214620.28186-1-ludo@gnu.org> References: <20210313214620.28186-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 47126 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) * gnu/packages/mpi.scm (hwloc-1)[properties]: New field. --- gnu/packages/mpi.scm | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/gnu/packages/mpi.scm b/gnu/packages/mpi.scm index 53ee6ef1cd..a8ebd8aeb8 100644 --- a/gnu/packages/mpi.scm +++ b/gnu/packages/mpi.scm @@ -66,6 +66,12 @@ (sha256 (base32 "0za1b9lvrm3rhn0lrxja5f64r0aq1qs4m0pxn1ji2mbi8ndppyyx")))) + + (properties + ;; Tell the 'generic-html' updater to monitor this URL for updates. + `((release-monitoring-url + . "https://www-lb.open-mpi.org/software/hwloc/current"))) + (build-system gnu-build-system) (outputs '("out" ;'lstopo' & co., depends on Cairo, libx11, etc. "lib" ;small closure -- 2.30.1 From debbugs-submit-bounces@debbugs.gnu.org Sat Mar 13 16:46:57 2021 Received: (at 47126) by debbugs.gnu.org; 13 Mar 2021 21:46:58 +0000 Received: from localhost ([127.0.0.1]:60846 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5y-0000la-B4 for submit@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:57 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54512) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lLC5l-0000jt-3D for 47126@debbugs.gnu.org; Sat, 13 Mar 2021 16:46:48 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:42784) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lLC5f-0002sL-Ov; Sat, 13 Mar 2021 16:46:31 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=39284 helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lLC5e-0004wi-FV; Sat, 13 Mar 2021 16:46:30 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: 47126@debbugs.gnu.org Subject: [PATCH 6/7] gnu-maintenance: Add 'generic-html' updater. Date: Sat, 13 Mar 2021 22:46:19 +0100 Message-Id: <20210313214620.28186-6-ludo@gnu.org> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210313214620.28186-1-ludo@gnu.org> References: <20210313214620.28186-1-ludo@gnu.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 47126 Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.3 (/) This brings total updater coverage, as reported by 'guix refresh --list-updaters', from 78% to 88.3%. Among many other things, it covers freedesktop.org packages. * guix/gnu-maintenance.scm (html-updatable-package?) (latest-html-updatable-release): New procedures. (%generic-html-updater): New variable. * doc/guix.texi (Invoking guix refresh): Document it. --- doc/guix.texi | 3 +++ guix/gnu-maintenance.scm | 58 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 60 insertions(+), 1 deletion(-) diff --git a/doc/guix.texi b/doc/guix.texi index 97094a7d0a..89c8c58295 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -11693,6 +11693,9 @@ the updater for @uref{https://www.stackage.org, Stackage} packages. the updater for @uref{https://crates.io, Crates} packages. @item launchpad the updater for @uref{https://launchpad.net, Launchpad} packages. +@item generic-html +a generic updater that crawls the HTML page where the source tarball of +the package is hosted, when applicable. @end table For instance, the following command only checks for updates of Emacs diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm index 5aa16acfde..ced5497b37 100644 --- a/guix/gnu-maintenance.scm +++ b/guix/gnu-maintenance.scm @@ -28,6 +28,7 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) + #:use-module (srfi srfi-34) #:use-module (rnrs io ports) #:use-module (system foreign) #:use-module (guix http-client) @@ -66,7 +67,8 @@ %gnu-ftp-updater %savannah-updater %xorg-updater - %kernel.org-updater)) + %kernel.org-updater + %generic-html-updater)) ;;; Commentary: ;;; @@ -697,6 +699,53 @@ releases are on gnu.org." #:file->signature file->signature) (cut adjusted-upstream-source <> rewrite)))) +(define html-updatable-package? + ;; Return true if the given package may be handled by the generic HTML + ;; updater. + (let ((hosting-sites '("github.com" "github.io" "gitlab.com" + "notabug.org" "sr.ht" + "gforge.inria.fr" "gitlab.inria.fr" + "ftp.gnu.org" "download.savannah.gnu.org" + "pypi.org" "crates.io" "rubygems.org" + "bioconductor.org"))) + (url-predicate (lambda (url) + (match (string->uri url) + (#f #f) + (uri + (let ((scheme (uri-scheme uri)) + (host (uri-host uri))) + (and (memq scheme '(http https)) + (not (member host hosting-sites)))))))))) + +(define (latest-html-updatable-release package) + "Return the latest release of PACKAGE. Do that by crawling the HTML page of +the directory containing its source tarball." + (let* ((uri (string->uri + (match (origin-uri (package-source package)) + ((? string? url) url) + ((url _ ...) url)))) + (custom (assoc-ref (package-properties package) + 'release-monitoring-url)) + (base (or custom + (string-append (symbol->string (uri-scheme uri)) + "://" (uri-host uri)))) + (directory (if custom + "" + (dirname (uri-path uri)))) + (package (package-upstream-name package))) + (catch #t + (lambda () + (guard (c ((http-get-error? c) #f)) + (latest-html-release package + #:base-url base + #:directory directory))) + (lambda (key . args) + ;; Return false and move on upon connection failures. + (unless (memq key '(gnutls-error tls-certificate-error + system-error)) + (apply throw key args)) + #f)))) + (define %gnu-updater ;; This is for everything at ftp.gnu.org. (upstream-updater @@ -737,4 +786,11 @@ releases are on gnu.org." (pred (url-prefix-predicate "mirror://kernel.org/")) (latest latest-kernel.org-release))) +(define %generic-html-updater + (upstream-updater + (name 'generic-html) + (description "Updater that crawls HTML pages.") + (pred html-updatable-package?) + (latest latest-html-updatable-release))) + ;;; gnu-maintenance.scm ends here -- 2.30.1 From debbugs-submit-bounces@debbugs.gnu.org Wed Mar 17 06:19:12 2021 Received: (at 47126) by debbugs.gnu.org; 17 Mar 2021 10:19:12 +0000 Received: from localhost ([127.0.0.1]:42117 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lMTGi-0006A6-At for submit@debbugs.gnu.org; Wed, 17 Mar 2021 06:19:12 -0400 Received: from mail.zaclys.net ([178.33.93.72]:57773) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lMTGe-00069j-5I for 47126@debbugs.gnu.org; Wed, 17 Mar 2021 06:19:11 -0400 Received: from guix-xps.local (82-64-145-38.subs.proxad.net [82.64.145.38]) (authenticated bits=0) by mail.zaclys.net (8.14.7/8.14.7) with ESMTP id 12HAJ1XX019373 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for <47126@debbugs.gnu.org>; Wed, 17 Mar 2021 11:19:02 +0100 DMARC-Filter: OpenDMARC Filter v1.3.2 mail.zaclys.net 12HAJ1XX019373 Authentication-Results: mail.zaclys.net; dmarc=fail (p=reject dis=none) header.from=zaclys.net Authentication-Results: mail.zaclys.net; spf=fail smtp.mailfrom=lle-bout@zaclys.net DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zaclys.net; s=default; t=1615976342; bh=kL/xAFIvbDLaGi1+VDF0PNBziBwwNwdO5L9hdFNEbSU=; h=Subject:From:To:Date:From; b=BEl9vAREmeboykV21yNzHZRGztSJ/chR3cLQi15ChnjvoAxcM2fhpkW+3yzWymTEj A+mnj8Y7kdjYod3Yl3RLv4EvMlDNSI+U2HMkuUvWZmijdWw3aJguXqvhM03mv/nsS8 4gJTfpLQ+seaL2GqtnsTY6lA25lR03U/8PRgYx/E= Message-ID: <5a2391930ed890f8cf61da88ccda6df9bb874630.camel@zaclys.net> Subject: [PATCH 0/7] Add 'generic-html' updater From: =?ISO-8859-1?Q?L=E9o?= Le Bouter To: 47126@debbugs.gnu.org Date: Wed, 17 Mar 2021 11:18:57 +0100 Content-Type: multipart/signed; micalg="pgp-sha512"; protocol="application/pgp-signature"; boundary="=-it3/iALI7cLpury7ul9G" User-Agent: Evolution 3.34.2 MIME-Version: 1.0 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 47126 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) --=-it3/iALI7cLpury7ul9G Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello! That's awesome thanks a lot Ludo!! I am wondering, does this handle cases where there's a subfolder with version and then another tarball with version as well? Like GNOME for example:=20 https://download.gnome.org/sources/NetworkManager/1.31/NetworkManager-1.31.= 1.tar.xz I see this is a generic solution, I see you made available some options to customize per-package as needed but can we get as precise/reliable as Debian's watch/uscan with that? Or if I understand correctly, we should always point it to a page where the link for the latest release is always published? That last thing really sounds nice! L=C3=A9o --=-it3/iALI7cLpury7ul9G Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEFIvLi9gL+xax3g6RRaix6GvNEKYFAmBR15EACgkQRaix6GvN EKYmFxAAsgGHKw+x7IWA4NJMYWvLo0WhY1Ss3wt0CbnOeNDW3aaDV1S39WCs5zPO 5GpYp1Y/jo1PGDgilJL5V/Aof8McoRQi1PttVTmfUDAg2n3q4Uv9eKJlKfEFPopZ JgxYE/1Y9b+6qjNagBT921AgNpuL0JVdsftdhcOEmmr2ROLTSMLBvHiKc9uCXPDv e/hqIVYfXtf6p+EJIwAbEruqhPuLFuPcmOKoEm8hd6T/0cBYoRt36g+uQ8Dkyil5 YEPR61XbGo+A6n88uoTmUrCmGvuiQ+uhvI513PlvTrOPtPYvRQf23r5vWii8Dazj az1BVBH3EnKkZRXhPjE9IG+mwxPq/MWzaZsuyKP5qBwDGZmSrzpzarbaYso4y4La NdbpCmUMdLYVvK9F/o2UB6P9wyMXH6HCtvZUg86WiylhGnceMj65bZdrnmqWlQAo f9O6WTwdXU6vD4RgjBpINqsPH2iqGEFkKDP68ZznNTqjAlEbC5vOoWVVIkO32VmI ZtEf2sQggUFDO9ez+78VgY7he4YXeqfzenDwfyFhSRIe+LKGy0oNbrYuizSkbviy vOy1YblmEQnE+vG1d7Uwy7CWANBFPSEUJIHZdKrutV2NPH6CexgpktEjFcoeSTu2 PfR/BfIEoIbZMN/4GVwhwcakc08njQV0c7L4JTqQcm2swr/GdAU= =faOe -----END PGP SIGNATURE----- --=-it3/iALI7cLpury7ul9G-- From debbugs-submit-bounces@debbugs.gnu.org Wed Mar 17 09:52:57 2021 Received: (at 47126) by debbugs.gnu.org; 17 Mar 2021 13:52:57 +0000 Received: from localhost ([127.0.0.1]:42382 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lMWbZ-0005Tq-8x for submit@debbugs.gnu.org; Wed, 17 Mar 2021 09:52:57 -0400 Received: from eggs.gnu.org ([209.51.188.92]:56726) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lMWbV-0005Tb-Ud for 47126@debbugs.gnu.org; Wed, 17 Mar 2021 09:52:55 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:36340) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lMWbQ-0005kp-8q; Wed, 17 Mar 2021 09:52:48 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=47058 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lMWbO-0006Wo-67; Wed, 17 Mar 2021 09:52:47 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: =?utf-8?Q?L=C3=A9o?= Le Bouter Subject: Re: bug#47126: [PATCH 0/7] Add 'generic-html' updater References: <20210313214326.28052-1-ludo@gnu.org> <5a2391930ed890f8cf61da88ccda6df9bb874630.camel@zaclys.net> Date: Wed, 17 Mar 2021 14:52:44 +0100 In-Reply-To: <5a2391930ed890f8cf61da88ccda6df9bb874630.camel@zaclys.net> (=?utf-8?Q?=22L=C3=A9o?= Le Bouter"'s message of "Wed, 17 Mar 2021 11:18:57 +0100") Message-ID: <87lfal51lf.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 47126 Cc: 47126@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Hi L=C3=A9o, L=C3=A9o Le Bouter skribis: > That's awesome thanks a lot Ludo!! Just pushed this series as fe96f64110676f28b948f0d31a1726501abdae0e. Unleash your update powers, comrades! :-) > I am wondering, does this handle cases where there's a subfolder with > version and then another tarball with version as well? > > Like GNOME for example:=20 > https://download.gnome.org/sources/NetworkManager/1.31/NetworkManager-1.3= 1.1.tar.xz > > I see this is a generic solution, I see you made available some options > to customize per-package as needed but can we get as precise/reliable > as Debian's watch/uscan with that? There=E2=80=99s a =E2=80=98gnome=E2=80=99 updater for GNOME: https://guix.gnu.org/manual/en/html_node/Invoking-guix-refresh.html And yes, it actually works. :-) In the case of NetworkManager, there=E2=80=99s a bug right now: --8<---------------cut here---------------start------------->8--- $ guix refresh network-manager ni sekvas la redirektigon al 'https://download.gnome.org/sources/NetworkMan= ager/cache.json'... ni sekvas la redirektigon al 'https://fr2.rpmfind.net/linux/gnome.org/sourc= es/NetworkManager/cache.json'... gnu/packages/gnome.scm:7648:13: network-manager would be upgraded from 1.24= .0 to rc2 --8<---------------cut here---------------end--------------->8--- I=E2=80=99ll see what=E2=80=99s up. But otherwise =E2=80=98guix refresh -t= gnome=E2=80=99 produces sensible results. At any rate, updaters sometimes bitrot, produce buggy results as in the example above. Please do use =E2=80=98guix refresh=E2=80=99 and report any= issues! Also, there are still ~12% of packages for which none of the updaters apply. We should investigate and see how we can bring that down to zero. Thanks for your feedback! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Wed Mar 17 09:53:10 2021 Received: (at control) by debbugs.gnu.org; 17 Mar 2021 13:53:10 +0000 Received: from localhost ([127.0.0.1]:42386 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lMWbl-0005Uf-It for submit@debbugs.gnu.org; Wed, 17 Mar 2021 09:53:10 -0400 Received: from eggs.gnu.org ([209.51.188.92]:56760) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lMWbk-0005UU-CM for control@debbugs.gnu.org; Wed, 17 Mar 2021 09:53:08 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:36344) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lMWbf-0005uP-4s for control@debbugs.gnu.org; Wed, 17 Mar 2021 09:53:03 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=47144 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1lMWbe-0006Z5-MK for control@debbugs.gnu.org; Wed, 17 Mar 2021 09:53:02 -0400 Date: Wed, 17 Mar 2021 14:53:01 +0100 Message-Id: <87k0q551ky.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #47126 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) tags 47126 fixed close 47126 quit From unknown Wed Aug 20 06:39:28 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 15 Apr 2021 11:24:07 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator