GNU bug report logs - #60520
[PATCH 0/4] Tests for 'guix refresh'

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 60520 in the body.
You can then email your comments to 60520 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: guix-patches <at> gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 0/4] Tests for 'guix refresh'
Date: Tue,  3 Jan 2023 17:44:39 +0100

Hi!

Following the discussion at
<https://lists.gnu.org/archive/html/guix-devel/2022-12/msg00311.html>,
I thought it’s about time to have tests for the ‘guix refresh’ CLI
(so far we had tests for importers, for (guix gnu-maintenance), and
for (guix upstream), but that did not cover the logic in ‘guix refresh’
itself as we’ve seen).

The strategy is to define a “pseudo updater”: the test defines an
environment variable that specifies which package updates the updater
should simulate, and then we go on using ‘guix refresh’.

Thoughts?

Ludo’.

Ludovic Courtès (4):
  gnu-maintenance: Factorize 'false-if-networking-failure'.
  gnu-maintenance: 'gnu' and 'gnu-ftp' predicates catch networking
    errors.
  import: stackage: Catch networking errors in predicate.
  refresh: Add CLI tests.

 Makefile.am              |   4 +-
 guix/gnu-maintenance.scm |  31 ++++-------
 guix/import/stackage.scm |  15 ++---
 guix/import/test.scm     |  88 +++++++++++++++++++++++++++++
 guix/import/utils.scm    |  24 +++++++-
 tests/guix-refresh.sh    | 117 +++++++++++++++++++++++++++++++++++++++
 6 files changed, 251 insertions(+), 28 deletions(-)
 create mode 100644 guix/import/test.scm
 create mode 100644 tests/guix-refresh.sh


base-commit: 473692b812b4ab4267d9bddad0fb27787d2112ff
-- 
2.38.1

Message #8 received at 60520 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 60520 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 2/4] gnu-maintenance: 'gnu' and 'gnu-ftp' predicates catch
 networking errors.
Date: Tue,  3 Jan 2023 18:03:17 +0100

Previously, in a networking-less environment such as 'guix shell -C -D
guix', 'guix refresh --list-updaters' would crash due to a
'gettaddrinfo-error' exception in these predicates.

* guix/gnu-maintenance.scm (%gnu-updater)[pred]: Wrap in
'false-if-networking-error'.
(%gnu-ftp-updater)[pred]: Likewise.
---
 guix/gnu-maintenance.scm | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm
index 0418c297f2..0aa70243b5 100644
--- a/guix/gnu-maintenance.scm
+++ b/guix/gnu-maintenance.scm
@@ -883,7 +883,8 @@ (define %gnu-updater
   (upstream-updater
    (name 'gnu)
    (description "Updater for GNU packages")
-   (pred gnu-hosted?)
+   (pred (lambda (package)
+           (false-if-networking-error (gnu-hosted? package))))
    (import import-gnu-release)))
 
 (define %gnu-ftp-updater
@@ -893,8 +894,9 @@ (define %gnu-ftp-updater
    (name 'gnu-ftp)
    (description "Updater for GNU packages only available via FTP")
    (pred (lambda (package)
-           (and (not (gnu-hosted? package))
-                (pure-gnu-package? package))))
+           (false-if-networking-error
+            (and (not (gnu-hosted? package))
+                 (pure-gnu-package? package)))))
    (import import-release*)))
 
 (define %savannah-updater
-- 
2.38.1

Message #11 received at 60520 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 60520 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 3/4] import: stackage: Catch networking errors in predicate.
Date: Tue,  3 Jan 2023 18:03:18 +0100

* guix/import/stackage.scm (stackage-lts-package?): Wrap body in
'false-if-networking-error'.
---
 guix/import/stackage.scm | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/guix/import/stackage.scm b/guix/import/stackage.scm
index 70d3e271f4..c0284e48a4 100644
--- a/guix/import/stackage.scm
+++ b/guix/import/stackage.scm
@@ -3,7 +3,7 @@
 ;;; Copyright © 2018 Ricardo Wurmus <rekado <at> elephly.net>
 ;;; Copyright © 2020 Martin Becze <mjbecze <at> riseup.net>
 ;;; Copyright © 2021 Xinglu Chem <public <at> yoctocell.xyz>
-;;; Copyright © 2021 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2021, 2023, 2023 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2022 Hartmut Goebel <h.goebel <at> crazy-compilers.com>
 ;;;
 ;;; This file is part of GNU Guix.
@@ -170,12 +170,13 @@ (define latest-lts-release
 (define (stackage-lts-package? package)
   "Return whether PACKAGE is available on the default Stackage LTS release."
   (and (hackage-package? package)
-       (let ((packages (stackage-lts-packages
-                        (stackage-lts-info-fetch %default-lts-version)))
-             (hackage-name (guix-package->hackage-name package)))
-         (find (lambda (package)
-                 (string=? (stackage-package-name package) hackage-name))
-               packages))))
+       (false-if-networking-error
+        (let ((packages (stackage-lts-packages
+                         (stackage-lts-info-fetch %default-lts-version)))
+              (hackage-name (guix-package->hackage-name package)))
+          (find (lambda (package)
+                  (string=? (stackage-package-name package) hackage-name))
+                packages)))))
 
 (define %stackage-updater
   (upstream-updater
-- 
2.38.1

Message #14 received at 60520 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 60520 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 1/4] gnu-maintenance: Factorize 'false-if-networking-failure'.
Date: Tue,  3 Jan 2023 18:03:16 +0100

* guix/import/utils.scm (call-with-networking-exception-handler): New
procedure.
(false-if-networking-error): New macro.
* guix/gnu-maintenance.scm (import-html-updatable-release): Use it
instead of inline code.
---
 guix/gnu-maintenance.scm | 23 +++++++----------------
 guix/import/utils.scm    | 24 +++++++++++++++++++++++-
 2 files changed, 30 insertions(+), 17 deletions(-)

diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm
index 8e60e52ea0..0418c297f2 100644
--- a/guix/gnu-maintenance.scm
+++ b/guix/gnu-maintenance.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2010-2022 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2010-2023 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2012, 2013 Nikita Karetnikov <nikita <at> karetnikov.org>
 ;;; Copyright © 2021 Simon Tournier <zimon.toutoune <at> gmail.com>
 ;;; Copyright © 2022 Maxime Devos <maximedevos <at> telenet.be>
@@ -43,6 +43,7 @@ (define-module (guix gnu-maintenance)
   #:use-module (guix records)
   #:use-module (guix upstream)
   #:use-module (guix packages)
+  #:autoload   (guix import utils) (false-if-networking-error)
   #:autoload   (zlib) (call-with-gzip-input-port)
   #:autoload   (htmlprag) (html->sxml)            ;from Guile-Lib
   #:export (gnu-package-name
@@ -871,21 +872,11 @@ (define* (import-html-updatable-release package #:key (version #f))
                         ""
                         (dirname (uri-path uri))))
          (package   (package-upstream-name package)))
-    (catch #t
-      (lambda ()
-        (guard (c ((http-get-error? c) #f))
-          (import-html-release package
-                               #:version version
-                               #:base-url base
-                               #:directory directory)))
-      (lambda (key . args)
-        ;; Return false and move on upon connection failures and bogus HTTP
-        ;; servers.
-        (unless (memq key '(gnutls-error tls-certificate-error
-                                         system-error getaddrinfo-error
-                                         bad-header bad-header-component))
-          (apply throw key args))
-        #f))))
+    (false-if-networking-error
+     (import-html-release package
+                          #:version version
+                          #:base-url base
+                          #:directory directory))))
 
 (define %gnu-updater
   ;; This is for everything at ftp.gnu.org.
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 41311cb86e..72795d2c61 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2012, 2013, 2018, 2019, 2020 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2012, 2013, 2018, 2019, 2020, 2023 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2016 Jelle Licht <jlicht <at> fsfe.org>
 ;;; Copyright © 2016 David Craven <david <at> craven.ch>
 ;;; Copyright © 2017, 2019, 2020, 2022 Ricardo Wurmus <rekado <at> elephly.net>
@@ -54,10 +54,12 @@ (define-module (guix import utils)
   #:use-module (srfi srfi-9)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-26)
+  #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-71)
   #:export (factorize-uri
 
             flatten
+            false-if-networking-error
 
             url-fetch
             guix-hash-url
@@ -122,6 +124,26 @@ (define (flatten lst)
      (cons elem memo)))
    '() lst))
 
+(define (call-with-networking-exception-handler thunk)
+  "Invoke THUNK, returning #f if one of the usual networking exception is
+thrown."
+  (catch #t
+    (lambda ()
+      (guard (c ((http-get-error? c) #f))
+        (thunk)))
+    (lambda (key . args)
+      ;; Return false and move on upon connection failures and bogus HTTP
+      ;; servers.
+      (unless (memq key '(gnutls-error tls-certificate-error
+                                       system-error getaddrinfo-error
+                                       bad-header bad-header-component))
+        (apply throw key args))
+      #f)))
+
+(define-syntax-rule (false-if-networking-error exp)
+  "Evaluate EXP, returning #f if a networking-related exception is thrown."
+  (call-with-networking-exception-handler (lambda () exp)))
+
 (define (url-fetch url file-name)
   "Save the contents of URL to FILE-NAME.  Return #f on failure."
   (parameterize ((current-output-port (current-error-port)))
-- 
2.38.1

Message #17 received at 60520 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: 60520 <at> debbugs.gnu.org
Cc: Ludovic Courtès <ludo <at> gnu.org>
Subject: [PATCH 4/4] refresh: Add CLI tests.
Date: Tue,  3 Jan 2023 18:03:19 +0100

* guix/import/test.scm, tests/guix-refresh.sh: New files.
* Makefile.am (MODULES, SH_TESTS): Add them.
---
 Makefile.am           |   4 +-
 guix/import/test.scm  |  88 +++++++++++++++++++++++++++++++
 tests/guix-refresh.sh | 117 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 208 insertions(+), 1 deletion(-)
 create mode 100644 guix/import/test.scm
 create mode 100644 tests/guix-refresh.sh

diff --git a/Makefile.am b/Makefile.am
index 8b026b6da6..d9d23eec88 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -1,5 +1,5 @@
 # GNU Guix --- Functional package management for GNU
-# Copyright © 2012-2022 Ludovic Courtès <ludo <at> gnu.org>
+# Copyright © 2012-2023 Ludovic Courtès <ludo <at> gnu.org>
 # Copyright © 2013 Andreas Enge <andreas <at> enge.fr>
 # Copyright © 2015, 2017 Alex Kost <alezost <at> gmail.com>
 # Copyright © 2016, 2018 Mathieu Lirzin <mthl <at> gnu.org>
@@ -289,6 +289,7 @@ MODULES =					\
   guix/import/print.scm				\
   guix/import/pypi.scm				\
   guix/import/stackage.scm			\
+  guix/import/test.scm   			\
   guix/import/texlive.scm   			\
   guix/import/utils.scm				\
   guix/scripts.scm				\
@@ -595,6 +596,7 @@ SH_TESTS =					\
   tests/guix-authenticate.sh			\
   tests/guix-environment.sh			\
   tests/guix-environment-container.sh		\
+  tests/guix-refresh.sh				\
   tests/guix-shell.sh				\
   tests/guix-shell-export-manifest.sh		\
   tests/guix-graph.sh				\
diff --git a/guix/import/test.scm b/guix/import/test.scm
new file mode 100644
index 0000000000..767dcd5b61
--- /dev/null
+++ b/guix/import/test.scm
@@ -0,0 +1,88 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Ludovic Courtès <ludo <at> gnu.org>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import test)
+  #:use-module (srfi srfi-1)
+  #:use-module (srfi srfi-26)
+  #:use-module (web uri)
+  #:use-module (guix packages)
+  #:use-module (guix upstream)
+  #:use-module ((guix utils) #:select (version-prefix?))
+  #:use-module (ice-9 vlist)
+  #:use-module (ice-9 match)
+  #:export (%test-updater))
+
+;;; Commentary:
+;;;
+;;; This module defines a pseudo updater whose sole purpose is to allow
+;;; testing of the whole 'guix refresh' command.
+;;;
+;;; Code:
+
+(define test-target-version
+  ;; VHash that maps package names to version/URL tuples.
+  (make-parameter
+   (or (and=> (getenv "GUIX_TEST_UPDATER_TARGETS")
+              (lambda (str)
+                (alist->vhash (call-with-input-string str read))))
+       vlist-null)))
+
+(define (available-updates package)
+  "Return the list of available <upstream-source> records for PACKAGE."
+  (vhash-fold* (lambda (version+updates result)
+                 (match version+updates
+                   ((version (updates ...))
+                    (if (version-prefix? version
+                                         (package-version package))
+                        (append (map (match-lambda
+                                       ((version url)
+                                        (upstream-source
+                                         (package (package-name package))
+                                         (version version)
+                                         (urls (list url)))))
+                                     updates)
+                                result)
+                        result))))
+               '()
+               (package-name package)
+               (test-target-version)))
+
+(define (test-package? package)
+  "Return true if PACKAGE has pseudo updates available."
+  (and (not (vlist-null? (test-target-version)))  ;cheap test
+       (pair? (available-updates package))))
+
+(define* (import-release package #:key (version #f))
+  "Return the <upstream-source> record denoting either the latest version of
+PACKAGE or VERSION."
+  (match (available-updates package)
+    (() #f)
+    ((sources ...)
+     (if version
+         (find (lambda (source)
+                 (string=? (upstream-source-version source)
+                           version))
+               sources)
+         (first sources)))))
+
+(define %test-updater
+  (upstream-updater
+   (name 'test)
+   (description "Pseudo updater for testing purposes.")
+   (pred test-package?)
+   (import import-release)))
diff --git a/tests/guix-refresh.sh b/tests/guix-refresh.sh
new file mode 100644
index 0000000000..de094a6c1d
--- /dev/null
+++ b/tests/guix-refresh.sh
@@ -0,0 +1,117 @@
+# GNU Guix --- Functional package management for GNU
+# Copyright © 2023 Ludovic Courtès <ludo <at> gnu.org>
+#
+# This file is part of GNU Guix.
+#
+# GNU Guix is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or (at
+# your option) any later version.
+#
+# GNU Guix is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GNU Guix.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# Test the 'guix refresh' command-line utility.
+#
+
+guix refresh --version
+
+manifest="t-guix-refresh-manifest-$$.scm"
+module_dir="t-guix-refresh-modules-$$"
+trap 'rm -f "$manifest"; rm -rf "$module_dir"' EXIT
+
+# Tell the 'test' updater what to simulate.
+export GUIX_TEST_UPDATER_TARGETS
+idutils_version="$(guix package -A ^idutils$ | cut -f2)"
+GUIX_TEST_UPDATER_TARGETS='
+  (("guile" "3" (("12.5" "file:///dev/null")
+                 ("1.6.4" "file:///dev/null")))
+   ("libreoffice" "" (("1.0" "file:///dev/null")))
+   ("idutils" "" (("'$idutils_version'" "file:///dev/null")))
+   ("the-test-package" "" (("5.5" "file://'$PWD/$module_dir'/source"))))'
+
+# No newer version available.
+! guix refresh -t test idutils
+case "$(guix refresh -t test idutils 2>&1)" in
+    *"$idutils_version"*"already the latest version"*) true;;
+    *) false;;
+esac
+! guix refresh -t test libreoffice
+case "$(guix refresh -t test libreoffice 2>&1)" in
+    *"greater than the latest known version"*"1.0"*) true;;
+    *) false;;
+esac
+
+# Various ways to specify packages.
+cat > "$manifest" <<EOF
+(specifications->manifest (list "guile <at> 3.0"))
+EOF
+default_IFS="$IFS"
+IFS=X
+for spec in "guile"					\
+	    "guile <at> 3.0"					\
+	    "-eX(@ (gnu packages guile) guile-3.0)"	\
+	    "-mX$manifest"				\
+	    "guileX-r"					\
+	    "-sXcore"
+do
+    guix refresh -t test $spec
+    case "$(guix refresh -t test $spec 2>&1)" in
+	*"would be upgraded"*"12.5"*)
+	    true;;
+	*)
+	    false;;
+    esac
+done
+IFS="$default_IFS"
+
+# Actually updating.
+mkdir "$module_dir"
+echo hello > "$module_dir/source"
+cat > "$module_dir/sample.scm"<<EOF
+(define-module (sample)
+  #:use-module (guix packages)
+  #:use-module (guix download)
+  #:use-module (gnu packages base))
+
+(define-public my-thing
+  (package
+    (inherit hello)
+    (name "the-test-package")
+    (version "4.3")
+    (source (origin
+              (method url-fetch)
+              (uri (string-append "mirror://gnu/hello/hello-" version
+                                  ".tar.gz"))
+              (sha256
+               (base32
+                "086vqwk2wl8zfs47sq2xpjc9k066ilmb8z6dn0q6ymwjzlm196cd"))))))
+EOF
+guix refresh -t test -L "$module_dir" the-test-package
+guix refresh -t test -L "$module_dir" the-test-package -u
+grep 'version "5.5"' "$module_dir/sample.scm"
+grep "$(guix hash -H sha256 -f nix-base32 "$module_dir/source")" "$module_dir/sample.scm"
+
+# Specifying a target version.
+! guix refresh -t test guile=2.0.0
+case "$(guix refresh -t test guile=2.0.0 2>&1)" in
+    *"failed to find"*"2.0.0"*) true;;
+    *) false;;
+esac
+for spec in "guile=1.6.4" "guile <at> 3=1.6.4"
+do
+    guix refresh -t test "$spec"
+    case "$(guix refresh -t test "$spec" 2>&1)" in
+	*"would be downgraded"*"1.6.4"*) true;;
+	*) false;;
+    esac
+done
+
+# Listing updaters.  This should work whether or not networking is available.
+guix refresh --list-updaters
-- 
2.38.1

Message #20 received at 60520 <at> debbugs.gnu.org (full text, mbox):

From: Hartmut Goebel <h.goebel <at> crazy-compilers.com>
To: Ludovic Courtès <ludo <at> gnu.org>, 60520 <at> debbugs.gnu.org
Cc: Ricardo Wurmus <rekado <at> elephly.net>
Subject: Re: [bug#60520] [PATCH 0/4] Tests for 'guix refresh'
Date: Tue, 3 Jan 2023 19:51:07 +0100

[Message part 1 (text/plain, inline)]

Am 03.01.23 um 17:44 schrieb Ludovic Courtès:
> Thoughts?

Thanks for implementing this,  It was terribly missing.

Two mini remarks:

+IFS=X

What about using a symbol like underscore?

+ "guileX-r" \

For symetry I'd write "-r guile".

-- 
Regards
Hartmut Goebel

| Hartmut Goebel          |h.goebel <at> crazy-compilers.com                |
|www.crazy-compilers.com  | compilers which you thought are impossible |

[Message part 2 (text/html, inline)]

Message #25 received at 60520-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Hartmut Goebel <h.goebel <at> crazy-compilers.com>
Cc: Ricardo Wurmus <rekado <at> elephly.net>, 60520-done <at> debbugs.gnu.org
Subject: Re: bug#60520: [PATCH 0/4] Tests for 'guix refresh'
Date: Sun, 08 Jan 2023 16:16:37 +0100

Hi,

Hartmut Goebel <h.goebel <at> crazy-compilers.com> skribis:

> Two mini remarks:
>
> +IFS=X
>
> What about using a symbol like underscore?
>
> + "guileX-r" \
>
> For symetry I'd write "-r guile".

Good ideas.  I made those changes and pushed:

  04f247be81 * refresh: Add CLI tests.
  2e9c0e1ff6 * import: stackage: Catch networking errors in predicate.
  472dd29f37 * gnu-maintenance: 'gnu' and 'gnu-ftp' predicates catch networking errors.
  f3edf29c67 * gnu-maintenance: Factorize 'false-if-networking-failure'.

Thanks,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.