GNU bug report logs - #74151
[PATCH 0/7] Add anonip system test.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 74151 in the body.
You can then email your comments to 74151 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: guix-patches <at> gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH 0/7] Add anonip system test.
Date: Fri,  1 Nov 2024 15:20:38 +0900

This was developed as part of investigating the source of bug #59181 ([berlin]
web services fail to start on reboot due to anonip).

There is no issue having anonip run and the system reboot (as should be
expected as /var/run is supposed to be cleared thus the named pipes should be
recreated every boot).  This points to a race with nginx that would "win"
creating its log files under /var/run/anonip/* before the anonip service does.

It'll need further investigating in that direction.

Maxim Cournoyer (7):
  doc: Use @table @code for anonip-configuration doc.
  services: anonip: Add 'debug?' configuration field.
  system/vm: Fix virtual-machine bug.
  tests: web: Have the retry-on-error throw on exhausted attempts.
  services: herd: Export 'eval-there' in API.
  build: marionette: Make it possible to reboot VM during tests.
  tests: Add anonip system test.

 doc/guix.texi            |  21 ++++---
 gnu/build/marionette.scm |   2 +-
 gnu/services/herd.scm    |   1 +
 gnu/services/web.scm     |   8 ++-
 gnu/system/vm.scm        |   9 ++-
 gnu/tests/web.scm        | 117 ++++++++++++++++++++++++++++++++++++++-
 6 files changed, 142 insertions(+), 16 deletions(-)


base-commit: 4009d1de954d694cb11af391d4113d29c5c1379d
-- 
2.46.0

Message #8 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH 1/7] doc: Use @table @code for anonip-configuration doc.
Date: Fri,  1 Nov 2024 16:11:37 +0900

* doc/guix.texi (Log Rotation): Use @table @code for anonip-configuration doc.

Change-Id: Ib6f8136715ecabc81d53dc13a14f4b111302c5c9
---
 doc/guix.texi | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 5768e195b0..b519b483fe 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20788,30 +20788,30 @@ Log Rotation
 
 The following optional settings may be provided:
 
-@table @asis
-@item @code{skip-private?}
+@table @code
+@item skip-private?
 When @code{#true} do not mask addresses in private ranges.
 
-@item @code{column}
+@item column
 A 1-based indexed column number.  Assume IP address is in the specified
 column (default is 1).
 
-@item @code{replacement}
+@item replacement
 Replacement string in case address parsing fails, e.g. @code{"0.0.0.0"}.
 
-@item @code{ipv4mask}
+@item ipv4mask
 Number of bits to mask in IPv4 addresses.
 
-@item @code{ipv6mask}
+@item ipv6mask
 Number of bits to mask in IPv6 addresses.
 
-@item @code{increment}
+@item increment
 Increment the IP address by the given number.  By default this is zero.
 
-@item @code{delimiter}
+@item delimiter
 Log delimiter string.
 
-@item @code{regex}
+@item regex
 Regular expression for detecting IP addresses.  Use this instead of @code{column}.
 @end table
 @end deftp
-- 
2.46.0

Message #11 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH 2/7] services: anonip: Add 'debug?' configuration field.
Date: Fri,  1 Nov 2024 16:11:38 +0900

* gnu/services/web.scm (<anonip-configuration>) [debug?]: New field.
(anonip-shepherd-service): Honor it.
* doc/guix.texi (Log Rotation) <anonip-configuration>: Document it.

Change-Id: Iaf57b5992808374b069a55c34a9adfdfe52b046c
---
 doc/guix.texi        | 3 +++
 gnu/services/web.scm | 8 ++++++--
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index b519b483fe..51a543d690 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20789,6 +20789,9 @@ Log Rotation
 The following optional settings may be provided:
 
 @table @code
+@item debug?
+Print debug messages when @code{#true}.
+
 @item skip-private?
 When @code{#true} do not mask addresses in private ranges.
 
diff --git a/gnu/services/web.scm b/gnu/services/web.scm
index cc6f4e6d9b..cf3515bf70 100644
--- a/gnu/services/web.scm
+++ b/gnu/services/web.scm
@@ -220,6 +220,7 @@ (define-module (gnu services web)
             anonip-configuration-anonip
             anonip-configuration-input
             anonip-configuration-output
+            anonip-configuration-debug?
             anonip-configuration-skip-private?
             anonip-configuration-column
             anonip-configuration-replacement
@@ -1448,6 +1449,8 @@ (define-record-type* <anonip-configuration>
                      (default anonip))
   (input             anonip-configuration-input)       ;string
   (output            anonip-configuration-output)      ;string
+  (debug?            anonip-configuration-debug?       ;boolean
+                     (default #f))
   (skip-private?     anonip-configuration-skip-private? ;boolean
                      (default #f))
   (column            anonip-configuration-column       ;number
@@ -1503,14 +1506,15 @@ (define (anonip-shepherd-service config)
                                     "/bin/anonip")
                      (string-append "--input=" #$input)
                      (string-append "--output=" #$output))
+               (if #$(anonip-configuration-debug? config)
+                   '("--debug") (list))
                (if #$(anonip-configuration-skip-private? config)
                    '("--skip-private") (list))
                '#$(optional anonip-configuration-column "--column")
                '#$(optional anonip-configuration-ipv4mask "--ipv4mask")
                '#$(optional anonip-configuration-ipv6mask "--ipv6mask")
                '#$(optional anonip-configuration-increment "--increment")
-               '#$(optional anonip-configuration-replacement
-                            "--replacement")
+               '#$(optional anonip-configuration-replacement "--replacement")
                '#$(optional anonip-configuration-delimiter "--delimiter")
                '#$(optional anonip-configuration-regex "--regex"))
               ;; Run in a UTF-8 locale
-- 
2.46.0

Message #14 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH 3/7] system/vm: Fix virtual-machine bug.
Date: Fri,  1 Nov 2024 16:11:39 +0900

The virtual-machine syntax would not accept a single operating-system field,
which was puzzling.

* gnu/system/vm.scm (virtual-machine): Add a pattern matching a single literal
'operating-system' field and value.

Change-Id: If207fd71df3a3f763b2e63229eafa82f63e80773
---
 gnu/system/vm.scm | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index a2743453e7..4a400056d7 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -422,10 +422,14 @@ (define-record-type* <virtual-machine> %virtual-machine
                     (default #f)))
 
 (define-syntax virtual-machine
-  (syntax-rules ()
+  (syntax-rules (operating-system)
     "Declare a virtual machine running the specified OS, with the given
 options."
-    ((_ os)                                       ;shortcut
+    ((_ (operating-system os))
+     ;; Also accept the long form (virtual-machine (operating-system os)), for
+     ;; correctness.
+     (%virtual-machine (operating-system os)))
+    ((_ os)                             ;shortcut
      (%virtual-machine (operating-system os)))
     ((_ fields ...)
      (%virtual-machine fields ...))))
-- 
2.46.0

Message #17 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH 4/7] tests: web: Have the retry-on-error throw on exhausted
 attempts.
Date: Fri,  1 Nov 2024 16:11:40 +0900

Previously, it'd simply return the error arguments, which would not always
fail a 'test-assert' test, for example.

* gnu/tests/web.scm (retry-on-error): Re-throw the error when the attempts
have been exhausted.

Change-Id: Ic1468d8ba23f0f78377e90d19bccb2878dc56f0e
---
 gnu/tests/web.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index a071e05e1d..5c50322cef 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -83,7 +83,7 @@ (define retry-on-error
            return-value)
           ((#f . error-args)
            (if (>= attempt times)
-               error-args
+               (apply throw error-args)
                (begin
                  (sleep delay)
                  (loop (+ 1 attempt)))))))))
-- 
2.46.0

Message #20 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH 5/7] services: herd: Export 'eval-there' in API.
Date: Fri,  1 Nov 2024 16:11:41 +0900

* gnu/services/herd.scm (gnu): Export 'eval-there', which is useful to
evaluate Scheme expressions.

Change-Id: Iff1db56e0847fc5886ac9fe594d677b363e7e6e7
---
 gnu/services/herd.scm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/gnu/services/herd.scm b/gnu/services/herd.scm
index 4b47acf72a..c185fc2cec 100644
--- a/gnu/services/herd.scm
+++ b/gnu/services/herd.scm
@@ -54,6 +54,7 @@ (define-module (gnu services herd)
             with-shepherd-action
             current-service
             current-services
+            eval-there
             unload-services
             unload-service
             load-services
-- 
2.46.0

Message #23 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH 6/7] build: marionette: Make it possible to reboot VM during
 tests.
Date: Fri,  1 Nov 2024 16:11:42 +0900

* gnu/build/marionette.scm (make-marionette): Add 'reconnect=1' socket
parameter.
* gnu/system/vm.scm (common-qemu-options): Remove '-no-reboot' option.

Change-Id: I5e100543ddddba0aea3ebe4e2f5cb8b0261c0d73
---
 gnu/build/marionette.scm | 2 +-
 gnu/system/vm.scm        | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/gnu/build/marionette.scm b/gnu/build/marionette.scm
index 0b0a8a70d8..af211eff43 100644
--- a/gnu/build/marionette.scm
+++ b/gnu/build/marionette.scm
@@ -108,7 +108,7 @@ (define* (make-marionette command
     (list "-nographic"
           "-monitor" (string-append "unix:" socket-directory "/monitor")
           "-chardev" (string-append "socket,id=repl,path=" socket-directory
-                                    "/repl")
+                                    "/repl,reconnect=1")
           "-chardev" (string-append "socket,id=qga,server=on,wait=off,path="
                                     socket-directory "/qemu-ga")
 
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4a400056d7..1e3f72c7b2 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -227,7 +227,6 @@ (define* (common-qemu-options image shared-fs
             '("-enable-kvm")
             '())
 
-     "-no-reboot"
      "-object" "rng-random,filename=/dev/urandom,id=guix-vm-rng"
      "-device" "virtio-rng-pci,rng=guix-vm-rng"
 
-- 
2.46.0

Message #26 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH 7/7] tests: Add anonip system test.
Date: Fri,  1 Nov 2024 16:11:43 +0900

* gnu/tests/web.scm (%test-anonip): New test.
(%anonip-os): New variables.
(run-anonip-test): New procedure.

Change-Id: Ieed210a784dbdeee8a498e74b6c0e31cb72cd9b8
---
 gnu/tests/web.scm | 115 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 115 insertions(+)

diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index 5c50322cef..f7a5659dc5 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -4,6 +4,7 @@
 ;;; Copyright © 2017, 2018 Clément Lassieur <clement <at> lassieur.org>
 ;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby <at> inria.fr>
 ;;; Copyright © 2018 Marius Bakke <mbakke <at> fastmail.com>
+;;; Copyright © 2024 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -33,6 +34,7 @@ (define-module (gnu tests web)
   #:use-module (gnu services networking)
   #:use-module (gnu services shepherd)
   #:use-module (gnu services mail)
+  #:use-module (gnu packages base)
   #:use-module (gnu packages databases)
   #:use-module (gnu packages guile-xyz)
   #:use-module (gnu packages patchutils)
@@ -52,6 +54,7 @@ (define-module (gnu tests web)
             %test-php-fpm
             %test-hpcguix-web
             %test-tailon
+            %test-anonip
             %test-patchwork
             %test-agate))
 
@@ -509,6 +512,118 @@ (define %test-tailon
    (description "Connect to a running Tailon server.")
    (value (run-tailon-test))))
 
+
+;;;
+;;; Anonip
+;;;
+(define %anonip-os
+  ;; Operating system under test.
+  (simple-operating-system
+   (service anonip-service-type
+            (anonip-configuration
+             (input "/var/run/anonip/http.access.log")
+             (output "/var/log/anonip/http.access.log")
+             (debug? #t)))))
+
+(define (run-anonip-test)
+  (define os
+    (marionette-operating-system
+     %anonip-os
+     #:imported-modules '((gnu services herd)
+                          (guix combinators))))
+
+  (define vm
+    (virtual-machine
+     (operating-system os)
+     ;; We are interested in verifying if anonip still launches following a
+     ;; reboot; thus make the base image writable.
+     (volatile? #f)))
+
+  (define test
+    (with-imported-modules '((gnu build marionette))
+      #~(begin
+          (use-modules (ice-9 match)
+                       (srfi srfi-64)
+                       (gnu build marionette))
+
+          (define marionette
+            (make-marionette (list #$vm)))
+
+          (test-runner-current (system-test-runner #$output))
+          (test-begin "anonip")
+
+          (test-assert "service is running"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (wait-for-service 'anonip-/var/log/anonip/http.access.log))
+             marionette))
+
+          (test-assert "service can be restarted"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (restart-service 'anonip-/var/log/anonip/http.access.log)
+                (wait-for-service 'anonip-/var/log/anonip/http.access.log))
+             marionette))
+
+          (test-assert "ip addresses are anonymized"
+            (marionette-eval
+             '(begin
+                (use-modules (ice-9 textual-ports))
+                (call-with-output-file "/var/run/anonip/http.access.log"
+                  (lambda (port)
+                    (display "192.168.100.200 - - \
+[30/Oct/2024:14:57:44 +0100] GET /xxx.narinfo HTTP/1.1\" 200 1065 \
+\"-\" \"GNU Guile\"\n" port)
+                    (display "2001:0db8:85a3:0000:0000:8a2e:0370:7334 - - \
+[30/Oct/2024:14:57:44 +0100] \"GET /xxx.narinfo HTTP/1.1\" 200 1065 \
+\"-\" \"GNU Guile\"\n" port)))
+                (#$retry-on-error
+                 (lambda ()
+                   (call-with-input-file "/var/log/anonip/http.access.log"
+                     (lambda (port)
+                       (let ((content (get-string-all port)))
+                         ;; The expected values are taken from anonip's test
+                         ;; suite (see its test_module.py file).
+                         (or (and (string-contains content "192.168.96.0")
+                                  (string-contains content "2001:db8:85a0::"))
+                             (error "could not find expected anonymized IPs"
+                                    content))))))
+                 #:times 20
+                 #:delay 1))
+             marionette))
+
+          (test-assert "service is running after reboot"
+            (begin
+              (marionette-eval
+               '(begin
+                  (use-modules (gnu services herd))
+                  (eval-there '(begin
+                                 (use-modules (shepherd system))
+                                 (sync) ;ensure the log is fully written
+                                 (reboot))))
+               marionette)
+              ;; Note: a distinct marionette-eval call is needed here; if
+              ;; included in the previous one issuing the reboot,
+              ;; 'wait-for-service' would apparently run before the system had
+              ;; rebooted (and succeed), which would defeat the test.
+              (marionette-eval
+               '(begin
+                  (use-modules (gnu services herd))
+                  (wait-for-service 'anonip-/var/log/anonip/http.access.log))
+               marionette)))
+
+          (test-end))))
+
+  (gexp->derivation "anonip-test" test))
+
+(define %test-anonip
+  (system-test
+   (name "anonip")
+   (description "Anonymize logs via Anonip")
+   (value (run-anonip-test))))
+
 
 ;;;
 ;;; Patchwork
-- 
2.46.0

Message #29 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH v2 1/9] doc: Use @table @code for anonip-configuration doc.
Date: Fri,  1 Nov 2024 21:39:21 +0900

* doc/guix.texi (Log Rotation): Use @table @code for anonip-configuration doc.

Change-Id: Ib6f8136715ecabc81d53dc13a14f4b111302c5c9
---
 doc/guix.texi | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 5768e195b0..b519b483fe 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20788,30 +20788,30 @@ Log Rotation
 
 The following optional settings may be provided:
 
-@table @asis
-@item @code{skip-private?}
+@table @code
+@item skip-private?
 When @code{#true} do not mask addresses in private ranges.
 
-@item @code{column}
+@item column
 A 1-based indexed column number.  Assume IP address is in the specified
 column (default is 1).
 
-@item @code{replacement}
+@item replacement
 Replacement string in case address parsing fails, e.g. @code{"0.0.0.0"}.
 
-@item @code{ipv4mask}
+@item ipv4mask
 Number of bits to mask in IPv4 addresses.
 
-@item @code{ipv6mask}
+@item ipv6mask
 Number of bits to mask in IPv6 addresses.
 
-@item @code{increment}
+@item increment
 Increment the IP address by the given number.  By default this is zero.
 
-@item @code{delimiter}
+@item delimiter
 Log delimiter string.
 
-@item @code{regex}
+@item regex
 Regular expression for detecting IP addresses.  Use this instead of @code{column}.
 @end table
 @end deftp

base-commit: 4009d1de954d694cb11af391d4113d29c5c1379d
-- 
2.46.0

Message #32 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH v2 2/9] services: anonip: Add 'debug?' configuration field.
Date: Fri,  1 Nov 2024 21:39:22 +0900

* gnu/services/web.scm (<anonip-configuration>) [debug?]: New field.
(anonip-shepherd-service): Honor it.
* doc/guix.texi (Log Rotation) <anonip-configuration>: Document it.

Change-Id: Iaf57b5992808374b069a55c34a9adfdfe52b046c
---
 doc/guix.texi        | 3 +++
 gnu/services/web.scm | 8 ++++++--
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index b519b483fe..51a543d690 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -20789,6 +20789,9 @@ Log Rotation
 The following optional settings may be provided:
 
 @table @code
+@item debug?
+Print debug messages when @code{#true}.
+
 @item skip-private?
 When @code{#true} do not mask addresses in private ranges.
 
diff --git a/gnu/services/web.scm b/gnu/services/web.scm
index cc6f4e6d9b..cf3515bf70 100644
--- a/gnu/services/web.scm
+++ b/gnu/services/web.scm
@@ -220,6 +220,7 @@ (define-module (gnu services web)
             anonip-configuration-anonip
             anonip-configuration-input
             anonip-configuration-output
+            anonip-configuration-debug?
             anonip-configuration-skip-private?
             anonip-configuration-column
             anonip-configuration-replacement
@@ -1448,6 +1449,8 @@ (define-record-type* <anonip-configuration>
                      (default anonip))
   (input             anonip-configuration-input)       ;string
   (output            anonip-configuration-output)      ;string
+  (debug?            anonip-configuration-debug?       ;boolean
+                     (default #f))
   (skip-private?     anonip-configuration-skip-private? ;boolean
                      (default #f))
   (column            anonip-configuration-column       ;number
@@ -1503,14 +1506,15 @@ (define (anonip-shepherd-service config)
                                     "/bin/anonip")
                      (string-append "--input=" #$input)
                      (string-append "--output=" #$output))
+               (if #$(anonip-configuration-debug? config)
+                   '("--debug") (list))
                (if #$(anonip-configuration-skip-private? config)
                    '("--skip-private") (list))
                '#$(optional anonip-configuration-column "--column")
                '#$(optional anonip-configuration-ipv4mask "--ipv4mask")
                '#$(optional anonip-configuration-ipv6mask "--ipv6mask")
                '#$(optional anonip-configuration-increment "--increment")
-               '#$(optional anonip-configuration-replacement
-                            "--replacement")
+               '#$(optional anonip-configuration-replacement "--replacement")
                '#$(optional anonip-configuration-delimiter "--delimiter")
                '#$(optional anonip-configuration-regex "--regex"))
               ;; Run in a UTF-8 locale
-- 
2.46.0

Message #35 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH v2 3/9] system/vm: Fix virtual-machine bug.
Date: Fri,  1 Nov 2024 21:39:23 +0900

The virtual-machine syntax would not accept a single operating-system field,
which was puzzling.

* gnu/system/vm.scm (virtual-machine): Add a pattern matching a single literal
'operating-system' field and value.

Change-Id: If207fd71df3a3f763b2e63229eafa82f63e80773
---
 gnu/system/vm.scm | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index a2743453e7..4a400056d7 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -422,10 +422,14 @@ (define-record-type* <virtual-machine> %virtual-machine
                     (default #f)))
 
 (define-syntax virtual-machine
-  (syntax-rules ()
+  (syntax-rules (operating-system)
     "Declare a virtual machine running the specified OS, with the given
 options."
-    ((_ os)                                       ;shortcut
+    ((_ (operating-system os))
+     ;; Also accept the long form (virtual-machine (operating-system os)), for
+     ;; correctness.
+     (%virtual-machine (operating-system os)))
+    ((_ os)                             ;shortcut
      (%virtual-machine (operating-system os)))
     ((_ fields ...)
      (%virtual-machine fields ...))))
-- 
2.46.0

Message #38 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH v2 4/9] tests: web: Have the retry-on-error throw on exhausted
 attempts.
Date: Fri,  1 Nov 2024 21:39:24 +0900

Previously, it'd simply return the error arguments, which would not always
fail a 'test-assert' test, for example.

* gnu/tests/web.scm (retry-on-error): Re-throw the error when the attempts
have been exhausted.

Change-Id: Ic1468d8ba23f0f78377e90d19bccb2878dc56f0e
---
 gnu/tests/web.scm | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index a071e05e1d..5c50322cef 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -83,7 +83,7 @@ (define retry-on-error
            return-value)
           ((#f . error-args)
            (if (>= attempt times)
-               error-args
+               (apply throw error-args)
                (begin
                  (sleep delay)
                  (loop (+ 1 attempt)))))))))
-- 
2.46.0

Message #41 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH v2 6/9] build: marionette: Make it possible to reboot VM
 during tests.
Date: Fri,  1 Nov 2024 21:39:26 +0900

* gnu/build/marionette.scm (make-marionette): Add 'reconnect=1' socket
parameter.
* gnu/system/vm.scm (common-qemu-options): Remove '-no-reboot' option.

Change-Id: I5e100543ddddba0aea3ebe4e2f5cb8b0261c0d73
---
 gnu/build/marionette.scm | 2 +-
 gnu/system/vm.scm        | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/gnu/build/marionette.scm b/gnu/build/marionette.scm
index 0b0a8a70d8..af211eff43 100644
--- a/gnu/build/marionette.scm
+++ b/gnu/build/marionette.scm
@@ -108,7 +108,7 @@ (define* (make-marionette command
     (list "-nographic"
           "-monitor" (string-append "unix:" socket-directory "/monitor")
           "-chardev" (string-append "socket,id=repl,path=" socket-directory
-                                    "/repl")
+                                    "/repl,reconnect=1")
           "-chardev" (string-append "socket,id=qga,server=on,wait=off,path="
                                     socket-directory "/qemu-ga")
 
diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 4a400056d7..1e3f72c7b2 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -227,7 +227,6 @@ (define* (common-qemu-options image shared-fs
             '("-enable-kvm")
             '())
 
-     "-no-reboot"
      "-object" "rng-random,filename=/dev/urandom,id=guix-vm-rng"
      "-device" "virtio-rng-pci,rng=guix-vm-rng"
 
-- 
2.46.0

Message #44 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH v2 5/9] services: herd: Export 'eval-there' in API.
Date: Fri,  1 Nov 2024 21:39:25 +0900

* gnu/services/herd.scm (gnu): Export 'eval-there', which is useful to
evaluate Scheme expressions.

Change-Id: Iff1db56e0847fc5886ac9fe594d677b363e7e6e7
---
 gnu/services/herd.scm | 1 +
 1 file changed, 1 insertion(+)

diff --git a/gnu/services/herd.scm b/gnu/services/herd.scm
index 4b47acf72a..c185fc2cec 100644
--- a/gnu/services/herd.scm
+++ b/gnu/services/herd.scm
@@ -54,6 +54,7 @@ (define-module (gnu services herd)
             with-shepherd-action
             current-service
             current-services
+            eval-there
             unload-services
             unload-service
             load-services
-- 
2.46.0

Message #47 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH v2 7/9] tests: Add anonip system test.
Date: Fri,  1 Nov 2024 21:39:27 +0900

* gnu/tests/web.scm (%test-anonip): New test.
(%anonip-os): New variables.
(run-anonip-test): New procedure.

Change-Id: Ieed210a784dbdeee8a498e74b6c0e31cb72cd9b8
---
 gnu/tests/web.scm | 122 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 122 insertions(+)

diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index 5c50322cef..6ef32138ed 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -4,6 +4,7 @@
 ;;; Copyright © 2017, 2018 Clément Lassieur <clement <at> lassieur.org>
 ;;; Copyright © 2018 Pierre-Antoine Rouby <pierre-antoine.rouby <at> inria.fr>
 ;;; Copyright © 2018 Marius Bakke <mbakke <at> fastmail.com>
+;;; Copyright © 2024 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -33,6 +34,7 @@ (define-module (gnu tests web)
   #:use-module (gnu services networking)
   #:use-module (gnu services shepherd)
   #:use-module (gnu services mail)
+  #:use-module (gnu packages base)
   #:use-module (gnu packages databases)
   #:use-module (gnu packages guile-xyz)
   #:use-module (gnu packages patchutils)
@@ -52,6 +54,7 @@ (define-module (gnu tests web)
             %test-php-fpm
             %test-hpcguix-web
             %test-tailon
+            %test-anonip
             %test-patchwork
             %test-agate))
 
@@ -509,6 +512,125 @@ (define %test-tailon
    (description "Connect to a running Tailon server.")
    (value (run-tailon-test))))
 
+
+;;;
+;;; Anonip
+;;;
+(define %anonip-os
+  ;; Operating system under test.
+  (simple-operating-system
+   (service anonip-service-type
+            (anonip-configuration
+             (input "/var/run/anonip/access.log")
+             (output "/var/log/anonip/access.log")
+             (debug? #t)))))
+
+(define (run-anonip-test)
+  (define os
+    (marionette-operating-system
+     %anonip-os
+     #:imported-modules '((gnu services herd)
+                          (guix combinators))))
+
+  (define vm
+    (virtual-machine
+     (operating-system os)
+     ;; We are interested in verifying if anonip still launches following a
+     ;; reboot; thus make the base image writable.
+     (volatile? #f)))
+
+  (define test
+    (with-imported-modules '((gnu build marionette))
+      #~(begin
+          (use-modules (ice-9 match)
+                       (srfi srfi-64)
+                       (gnu build marionette))
+
+          (define marionette
+            (make-marionette (list #$vm)))
+
+          (test-runner-current (system-test-runner #$output))
+          (test-begin "anonip")
+
+          (test-assert "service is running"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (wait-for-service 'anonip-/var/log/anonip/access.log))
+             marionette))
+
+          (test-assert "service can be restarted"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (restart-service 'anonip-/var/log/anonip/access.log)
+                (wait-for-service 'anonip-/var/log/anonip/access.log))
+             marionette))
+
+          (test-assert "ip addresses are anonymized"
+            (marionette-eval
+             '(begin
+                (use-modules (ice-9 textual-ports))
+                (call-with-output-file "/var/run/anonip/access.log"
+                  (lambda (port)
+                    (display "192.168.100.200 - - \
+[30/Oct/2024:14:57:44 +0100] GET /xxx.narinfo HTTP/1.1\" 200 1065 \
+\"-\" \"GNU Guile\"\n" port)
+                    (display "2001:0db8:85a3:0000:0000:8a2e:0370:7334 - - \
+[30/Oct/2024:14:57:44 +0100] \"GET /xxx.narinfo HTTP/1.1\" 200 1065 \
+\"-\" \"GNU Guile\"\n" port)))
+                (#$retry-on-error
+                 (lambda ()
+                   (call-with-input-file "/var/log/anonip/access.log"
+                     (lambda (port)
+                       (let ((content (get-string-all port)))
+                         ;; The expected values are taken from anonip's test
+                         ;; suite (see its test_module.py file).
+                         (or (and (string-contains content "192.168.96.0")
+                                  (string-contains content "2001:db8:85a0::"))
+                             (error "could not find expected anonymized IPs"
+                                    content))))))
+                 #:times 20
+                 #:delay 1))
+             marionette))
+
+          (test-assert "service is running after reboot"
+            (begin
+              (marionette-eval
+               '(begin
+                  (use-modules (gnu services herd))
+                  (eval-there '(begin
+                                 (use-modules (shepherd system))
+                                 (sync) ;ensure the log is fully written
+                                 (reboot))))
+               marionette)
+              ;; Note: a distinct marionette-eval call is needed here; if
+              ;; included in the previous one issuing the reboot,
+              ;; 'wait-for-service' would apparently run before the system had
+              ;; rebooted (and succeed), which would defeat the test.
+              (marionette-eval
+               '(begin
+                  (use-modules (gnu services herd))
+                  (wait-for-service 'anonip-/var/log/anonip/access.log))
+               marionette)))
+
+          (test-assert "service can be stopped"
+            (marionette-eval
+             '(begin
+                (use-modules (gnu services herd))
+                (stop-service 'anonip-/var/log/anonip/access.log))
+             marionette))
+
+          (test-end))))
+
+  (gexp->derivation "anonip-test" test))
+
+(define %test-anonip
+  (system-test
+   (name "anonip")
+   (description "Anonymize logs via Anonip")
+   (value (run-anonip-test))))
+
 
 ;;;
 ;;; Patchwork
-- 
2.46.0

Message #50 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH v2 8/9] tests: web: Add nginx+anonip test.
Date: Fri,  1 Nov 2024 21:39:28 +0900

This test reproduces the problem reported in bug #59181 ("[berlin] web
services fail to start on reboot due to anonip").

* gnu/tests/web.scm (run-webserver-test) <extra-args>: New keyword argument.
(%nginx+anonip-os, nginx-anonip-tests, %test-nginx+anonip): New variables.

Change-Id: I7bbc8fb9f2ab33ce89bf1c0945d7ddbddf26a7ea
---
New commit in v2

 gnu/tests/web.scm | 43 ++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 40 insertions(+), 3 deletions(-)

diff --git a/gnu/tests/web.scm b/gnu/tests/web.scm
index 6ef32138ed..33e0a697a8 100644
--- a/gnu/tests/web.scm
+++ b/gnu/tests/web.scm
@@ -50,6 +50,7 @@ (define-module (gnu tests web)
   #:use-module (ice-9 match)
   #:export (%test-httpd
             %test-nginx
+            %test-nginx+anonip
             %test-varnish
             %test-php-fpm
             %test-hpcguix-web
@@ -91,9 +92,11 @@ (define retry-on-error
                  (sleep delay)
                  (loop (+ 1 attempt)))))))))
 
-(define* (run-webserver-test name test-os #:key (log-file #f) (http-port 8080))
-  "Run tests in %NGINX-OS, which has nginx running and listening on
-HTTP-PORT."
+(define* (run-webserver-test name test-os #:key (log-file #f) (http-port 8080)
+                             extra-tests)
+  "Run tests in %NGINX-OS, which has nginx running and listening on HTTP-PORT.
+EXTRA-TESTS should be a sexp of gexp containing extra code to run as part of
+the tests."
   (define os
     (marionette-operating-system
      test-os
@@ -154,6 +157,8 @@ (define* (run-webserver-test name test-os #:key (log-file #f) (http-port 8080))
                       marionette)))
                  '())
 
+          #$extra-tests
+
           (test-end))))
 
   (gexp->derivation (string-append name "-test") test))
@@ -209,6 +214,38 @@ (define %test-nginx
    (value (run-webserver-test name %nginx-os
                               #:log-file "/var/log/nginx/access.log"))))
 
+(define %nginx+anonip-os
+  (simple-operating-system
+   (service dhcp-client-service-type)
+   (service anonip-service-type
+            (anonip-configuration
+             (input "/var/run/anonip/access.log")
+             (output "/var/log/anonip/access.log")
+             (debug? #t)))
+   (service nginx-service-type
+            (nginx-configuration
+             (log-directory "/var/run/anonip/")
+             (server-blocks %nginx-servers)
+             (shepherd-requirement '(anonip-/var/log/anonip/access.log))))
+   (simple-service 'make-http-root activation-service-type
+                   %make-http-root)))
+
+(define nginx-anonip-tests
+  #~(test-assert "anonip service is running"
+      (marionette-eval
+       '(begin
+          (use-modules (gnu services herd))
+          (wait-for-service 'anonip-/var/log/anonip/access.log))
+       marionette)))
+
+(define %test-nginx+anonip
+  (system-test
+   (name "nginx+anonip")
+   (description "Run a NGINX server with logs anonymized by Anonip")
+   (value (run-webserver-test "nginx" %nginx+anonip-os
+                              #:log-file "/var/log/anonip/access.log"
+                              #:extra-tests nginx-anonip-tests))))
+
 
 ;;;
 ;;; Varnish
-- 
2.46.0

Message #53 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: 74151 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Subject: [PATCH v2 9/9] services: web: Fix race between nginx activation and
 anonip.
Date: Fri,  1 Nov 2024 21:39:29 +0900

* gnu/services/web.scm (anonip-shepherd-service): Recreate the input file when
it's not a FIFO.

Fixes: <https://issues.guix.gnu.org/59181>
Change-Id: I8ba87f9fc48ecfd515e34bdee9e2949a2a559f9c
---
New commit in v2

 gnu/services/web.scm | 64 ++++++++++++++++++++------------------------
 1 file changed, 29 insertions(+), 35 deletions(-)

diff --git a/gnu/services/web.scm b/gnu/services/web.scm
index cf3515bf70..4cf7c68997 100644
--- a/gnu/services/web.scm
+++ b/gnu/services/web.scm
@@ -1499,41 +1499,35 @@ (define (anonip-shepherd-service config)
        "Anonimyze the given log file location with anonip.")
       (start
        #~(lambda ()
-           (define (spawn)
-             (fork+exec-command
-              (append
-               (list #$(file-append (anonip-configuration-anonip config)
-                                    "/bin/anonip")
-                     (string-append "--input=" #$input)
-                     (string-append "--output=" #$output))
-               (if #$(anonip-configuration-debug? config)
-                   '("--debug") (list))
-               (if #$(anonip-configuration-skip-private? config)
-                   '("--skip-private") (list))
-               '#$(optional anonip-configuration-column "--column")
-               '#$(optional anonip-configuration-ipv4mask "--ipv4mask")
-               '#$(optional anonip-configuration-ipv6mask "--ipv6mask")
-               '#$(optional anonip-configuration-increment "--increment")
-               '#$(optional anonip-configuration-replacement "--replacement")
-               '#$(optional anonip-configuration-delimiter "--delimiter")
-               '#$(optional anonip-configuration-regex "--regex"))
-              ;; Run in a UTF-8 locale
-              #:environment-variables
-              (list (string-append "GUIX_LOCPATH="
-                                   #$(libc-utf8-locales-for-target)
-                                   "/lib/locale")
-                    "LC_ALL=en_US.utf8")))
-
-           (let ((stat (stat #$input #f)))
-             (cond ((not stat)
-                    (mknod #$input 'fifo #o600 0)
-                    (spawn))
-                   ((eq? 'fifo (stat:type stat))
-                    (spawn))
-                   (else
-                    (format #t "'~a' is not a FIFO; bailing out~%"
-                            #$input)
-                    #f)))))
+           ;; Always attempt to recreate the named pipe, as activation scripts
+           ;; such as that of nginx may have created plain files in its place
+           ;; (see: https://issues.guix.gnu.org/59181).
+           (false-if-exception (delete-file #$input))
+           (mknod #$input 'fifo #o600 0)
+
+           (fork+exec-command
+            (append
+             (list #$(file-append (anonip-configuration-anonip config)
+                                  "/bin/anonip")
+                   (string-append "--input=" #$input)
+                   (string-append "--output=" #$output))
+             (if #$(anonip-configuration-debug? config)
+                 '("--debug") (list))
+             (if #$(anonip-configuration-skip-private? config)
+                 '("--skip-private") (list))
+             '#$(optional anonip-configuration-column "--column")
+             '#$(optional anonip-configuration-ipv4mask "--ipv4mask")
+             '#$(optional anonip-configuration-ipv6mask "--ipv6mask")
+             '#$(optional anonip-configuration-increment "--increment")
+             '#$(optional anonip-configuration-replacement "--replacement")
+             '#$(optional anonip-configuration-delimiter "--delimiter")
+             '#$(optional anonip-configuration-regex "--regex"))
+            ;; Run in a UTF-8 locale
+            #:environment-variables
+            (list (string-append "GUIX_LOCPATH="
+                                 #$(libc-utf8-locales-for-target)
+                                 "/lib/locale")
+                  "LC_ALL=en_US.utf8"))))
       (stop #~(make-kill-destructor))))))
 
 (define anonip-service-type
-- 
2.46.0

Message #56 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 74151 <at> debbugs.gnu.org
Subject: Re: [bug#74151] [PATCH v2 6/9] build: marionette: Make it possible
 to reboot VM during tests.
Date: Wed, 20 Nov 2024 10:29:20 +0100

Hi!

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> * gnu/build/marionette.scm (make-marionette): Add 'reconnect=1' socket
> parameter.
> * gnu/system/vm.scm (common-qemu-options): Remove '-no-reboot' option.
>
> Change-Id: I5e100543ddddba0aea3ebe4e2f5cb8b0261c0d73

[...]

> +++ b/gnu/system/vm.scm
> @@ -227,7 +227,6 @@ (define* (common-qemu-options image shared-fs
>              '("-enable-kvm")
>              '())
>  
> -     "-no-reboot"

I believe the reason we had ‘-no-reboot’ is because otherwise QEMU would
restart the VM when we halt it (?).  I’m not sure actually what the
exact scenario was.

Anyway, if you confirm that other system tests are unaffected by this
change, this is probably good.

Ludo’.

Message #59 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
Cc: 74151 <at> debbugs.gnu.org
Subject: Re: [bug#74151] [PATCH 0/7] Add anonip system test.
Date: Wed, 20 Nov 2024 10:31:15 +0100

Hi,

Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:

> This was developed as part of investigating the source of bug #59181 ([berlin]
> web services fail to start on reboot due to anonip).
>
> There is no issue having anonip run and the system reboot (as should be
> expected as /var/run is supposed to be cleared thus the named pipes should be
> recreated every boot).  This points to a race with nginx that would "win"
> creating its log files under /var/run/anonip/* before the anonip service does.
>
> It'll need further investigating in that direction.

This is nice work, LGTM!

This did not elucidate the cause of #59181, right?

Thanks,
Ludo’.

Message #62 received at 74151 <at> debbugs.gnu.org (full text, mbox):

From: Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 74151 <at> debbugs.gnu.org
Subject: Re: [bug#74151] [PATCH 0/7] Add anonip system test.
Date: Sun, 24 Nov 2024 21:07:10 +0900

Hi Ludovic,

Ludovic Courtès <ludo <at> gnu.org> writes:

> Hi,
>
> Maxim Cournoyer <maxim.cournoyer <at> gmail.com> skribis:
>
>> This was developed as part of investigating the source of bug #59181 ([berlin]
>> web services fail to start on reboot due to anonip).
>>
>> There is no issue having anonip run and the system reboot (as should be
>> expected as /var/run is supposed to be cleared thus the named pipes should be
>> recreated every boot).  This points to a race with nginx that would "win"
>> creating its log files under /var/run/anonip/* before the anonip service does.
>>
>> It'll need further investigating in that direction.
>
> This is nice work, LGTM!
>
> This did not elucidate the cause of #59181, right?

It did!  The cause was the activation snippet of nginx, which runs the
nginx check command, which creates every file involved to check if it
has sufficient permissions.

The files would then exist before anonip had a chance to create them.

The fix has already been merged, so a reboot following a recent
reconfigure on berlin should show all our services up and running.

I'll run the system tests again and push it this doesn't break any.

-- 
Thanks,
Maxim

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.