GNU bug report logs - #74389
[PATCH] services: add cloud-init service

Previous Next

To reply to this bug, email your comments to 74389 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Alexander Joss <alex <at> infiniteadaptability.org>
To: guix-patches <at> gnu.org
Cc: Alexander Joss <alex <at> infiniteadaptability.org>
Subject: [PATCH] services: add cloud-init service
Date: Sun, 17 Nov 2024 06:26:05 +0000

* gnu/services/cloud-init.scm: add cloud-init service
* gnu/system/examples: add cloud-init-image.tmpl

Change-Id: I28fe295c1dbbab7ea7df65f6b764c7d795e58d77
---
 gnu/services/cloud-init.scm               | 137 ++++++++++++++++++++++
 gnu/system/examples/cloud-init-image.tmpl |  63 ++++++++++
 2 files changed, 200 insertions(+)
 create mode 100644 gnu/services/cloud-init.scm
 create mode 100644 gnu/system/examples/cloud-init-image.tmpl

diff --git a/gnu/services/cloud-init.scm b/gnu/services/cloud-init.scm
new file mode 100644
index 0000000000..d6362f70a7
--- /dev/null
+++ b/gnu/services/cloud-init.scm
@@ -0,0 +1,137 @@
+(define-module (gnu services cloud-init)
+  #:use-module (gnu packages bash)
+  #:use-module (gnu packages python-web)
+  #:use-module (gnu services)
+  #:use-module (gnu services shepherd)
+  #:use-module (guix gexp)
+  #:use-module (guix records)
+  #:export (cloud-init-configuration cloud-init-service
+                                     cloud-init-service-type))
+
+(define-record-type* <cloud-init-configuration> cloud-init-configuration
+                     make-cloud-init-configuration
+  cloud-init-configuration?
+
+  (cloud-init cloud-init-configuration-cloud-init ;file-like
+              (default python-cloud-init))
+  (init-modules cloud-init-configuration-init-modules ;list of symbols
+                (default '(seed_random growpart
+                                       resizefs
+                                       disk_setup
+                                       mounts
+                                       set_hostname
+                                       update_hostname
+                                       users_groups
+                                       ssh
+                                       set_passwords)))
+  (config-modules cloud-init-configuration-config-modules ;list of symbols
+                  (default '()))
+  (final-modules cloud-init-configuration-final-modules ;list of symbols
+                 (default '(ssh_authkey_fingerprints)))
+  (extra-configuration-files
+   cloud-init-configuration-extra-configuration-files ;list of file-likes
+   (default '())))
+
+(define %cloud-dir
+  "/etc/cloud")
+
+(define %cloud-cfg
+  (string-append %cloud-dir "/cloud.cfg"))
+
+(define %cloud-run
+  (mixed-text-file "run.sh"
+                   "#!"
+                   (file-append bash "/bin/bash")
+                   "\n\nset -euo pipefail\n\n"
+                   (file-append python-cloud-init "/bin/cloud-init")
+                   " init --local\n"
+                   (file-append python-cloud-init "/bin/cloud-init")
+                   " init\n"
+                   (file-append python-cloud-init "/bin/cloud-init")
+                   " modules --mode config\n"
+                   (file-append python-cloud-init "/bin/cloud-init")
+                   " modules --mode final\n"))
+
+(define %cloud-cfg-d
+  (string-append %cloud-dir "/cloud.cfg.d"))
+
+(define (cloud-init-initialization init-modules config-modules final-modules
+                                   extra)
+  "Return the gexp to initialize the cloud-init configuration files"
+  #~(begin
+      (use-modules (srfi srfi-1)
+                   (srfi srfi-2)
+                   (guix build utils))
+
+      (define reduce-modules
+        (lambda (mods)
+          (string-join (map (lambda (mod)
+                              (string-append "\n  - "
+                                             (symbol->string mod))) mods))))
+
+      (mkdir-p #$%cloud-cfg-d)
+
+      (copy-file #$%cloud-run
+                 (string-append #$%cloud-dir "/run.sh"))
+      (chmod (string-append #$%cloud-dir "/run.sh") #o755)
+
+      (unless (null? '(#$@extra))
+        (for-each (lambda (file)
+                    (symlink (cadr file)
+                             (string-append #$%cloud-cfg-d "/"
+                                            (car file))))
+                  '(#$@extra)))
+
+      (call-with-output-file #$%cloud-cfg
+        (lambda (p)
+          (unless (null? '(#$@init-modules))
+            (display (string-append "cloud_init_modules:"
+                                    (reduce-modules '(#$@init-modules)) "\n\n")
+                     p))
+          (unless (null? '(#$@config-modules))
+            (display (string-append "cloud_config_modules:"
+                                    (reduce-modules '(#$@config-modules))
+                                    "\n\n") p))
+          (unless (null? '(#$@final-modules))
+            (display (string-append "cloud_final_modules:"
+                                    (reduce-modules '(#$@final-modules))
+                                    "\n\n") p))))))
+
+(define (cloud-init-activation config)
+  "Return the activation gexp for CONFIG."
+  #~(begin
+      (use-modules (guix build utils))
+      #$(cloud-init-initialization (cloud-init-configuration-init-modules
+                                    config)
+                                   (cloud-init-configuration-config-modules
+                                    config)
+                                   (cloud-init-configuration-final-modules
+                                    config)
+                                   (cloud-init-configuration-extra-configuration-files
+                                    config))))
+
+(define (cloud-init-service config)
+  "Return a <cloud-init-service> for cloud-init with CONFIG."
+  (define cloud-init
+    (cloud-init-configuration-cloud-init config))
+
+  (list (shepherd-service (documentation "cloud-init service")
+                          (provision '(cloud-init))
+                          (requirement '(networking))
+                          (one-shot? #t)
+                          (start #~(fork+exec-command (list (string-append #$%cloud-dir
+                                                             "/run.sh"))
+                                                      #:log-file (string-append
+                                                                  "/var/log/cloud-init.log")
+                                                      #:environment-variables '
+                                                      ("PATH=/run/current-system/profile/bin:/run/current-system/profile/sbin:"))))))
+
+(define cloud-init-service-type
+  (service-type (name 'cloud-init)
+                (default-value (cloud-init-configuration))
+                (description "cloud init")
+                (extensions (list (service-extension
+                                   shepherd-root-service-type
+                                   cloud-init-service)
+                                  (service-extension activation-service-type
+                                                     cloud-init-activation)))))
diff --git a/gnu/system/examples/cloud-init-image.tmpl b/gnu/system/examples/cloud-init-image.tmpl
new file mode 100644
index 0000000000..e2e69e8691
--- /dev/null
+++ b/gnu/system/examples/cloud-init-image.tmpl
@@ -0,0 +1,63 @@
+;; This vm image is meant to be used as an image template
+;; to be deployed on cloud providers that use cloud-init.
+
+(use-modules (gnu)
+             (guix)
+             (guix gexp)
+             (srfi srfi-1))
+(use-service-modules cloud-init base networking ssh)
+(use-package-modules admin bootloaders package-management python-web ssh)
+
+(operating-system
+  (host-name "gnu")
+  (timezone "Etc/UTC")
+  (locale "en_US.utf8")
+  (keyboard-layout (keyboard-layout "us"))
+
+  (firmware '())
+
+  ;; Below we assume /dev/vda is the VM's hard disk.
+  ;; Adjust as needed.
+  (bootloader (bootloader-configuration
+                (bootloader grub-bootloader)
+                (targets '("/dev/vda"))
+                (terminal-outputs '(console))))
+  (file-systems (cons (file-system
+                        (mount-point "/")
+                        (device "/dev/vda1")
+                        (type "ext4")) %base-file-systems))
+
+  ;; The cloud-utils packages provides some utilities to allow
+  ;; us to piggyback off ubuntu's cloud-init modules/integrations
+  ;; without having to write guix specific functionality.
+  ;;
+  ;; The python-cloud-init package is not strictly required to be
+  ;; in system-wide packages.
+  (packages (append (list cloud-utils python-cloud-init) %base-packages))
+
+  (services
+   (append (list (service cloud-init-service-type)
+                 ;; An example of extra configuration files. This specific
+                 ;; file is required for properly running cloud-init on DigitalOcean
+                 ;; (cloud-init-configuration (extra-configuration-files `
+                 ;; (("99-digitalocean.cfg" ,
+                 ;; (plain-file
+                 ;; "99-digitalocean.cfg"
+                 ;; "datasource_list: [ ConfigDrive, DigitalOcean, NoCloud, None ]"))))))
+                 
+                 (service network-manager-service-type)
+                 (service wpa-supplicant-service-type)
+                 (service openssh-service-type
+                          (openssh-configuration (openssh openssh-sans-x)
+                                                 (permit-root-login #t))))
+           %base-services
+           ;; Uncomment the following and replace the above to automatically add your guix
+           ;; signing key to the vm for easy reconfiguration.
+           ;; (modify-services %base-services
+           ;; (guix-service-type config =>
+           ;; (guix-configuration (inherit config)
+           ;; (authorized-keys (append
+           ;; (list (local-file
+           ;; "/etc/guix/signing-key.pub"))
+           ;; %default-authorized-guix-keys)))))))
+           )))

base-commit: 0e1ffbc7f5f060f89c890472377a6102f27f6e9b
-- 
2.46.0

Message #8 received at 74389 <at> debbugs.gnu.org (full text, mbox):

From: Ricardo Wurmus <rekado <at> elephly.net>
To: 74389 <at> debbugs.gnu.org
Subject: [PATCH] services: add cloud-init service
Date: Thu, 21 Nov 2024 19:25:47 +0100

Hi Alexander,

thank you for the patch!

I'm interested in a cloud-init service because I think we need a way to
have Guix System images be configured dynamically when using them in AWS
or in Openstack environments.  The dynamic configuration I'm most
interested in is user credentials and networking.

Last I looked into this, the Python library for cloud init had explicit
support for various distributions to modify system configuration files
and run tools (e.g. to configure and bring up networking, or to set the
hostname).  It did not generate a central configuration file that could
be processed.  It just triggered explicit per-distro actions.

We couldn't actually use any of that stuff for Guix System. What I
investigated a few months back was to see if there was a way to generate
some sort of configuration file that our services could then consult.

Sadly there was no such thing as a central configuration file containing
all necessary bits of information.  From what I can tell cloud-utils
does not have what it takes to configure networking or user accounts.
It's probably useful for other bits, though I haven't ever used those.

Could you please expand on what your service does---and what it cannot
or does not aim to do---and how it would be used?

Thanks!

-- 
Ricardo

Message #11 received at 74389 <at> debbugs.gnu.org (full text, mbox):

From: Alex <alex <at> infiniteadaptability.org>
To: "74389 <at> debbugs.gnu.org" <74389 <at> debbugs.gnu.org>
Subject: [PATCH] services: add cloud-init service
Date: Fri, 22 Nov 2024 19:20:13 +0000

Ricardo,

This patch is just a MVP for implementing a cloud-init service. The goal primarily was to get the networking, disk and filesystem mounts, and ssh authentication working properly.

From a high level this patch implements a cloud-init-service which uses cloud-init functionality to gather information from it's run-time environment and then run cloud-init modules in order to properly configure the vm. It provides a configuration interface for choosing which modules to run as well as a mechanism to add extra configuration files.

I took a few major shortcuts in order to get a working image.

The first is that instead of writing guix specific functionality for cloud-init modules, I decided to attempt to use the ubuntu (which I believe is the default) tools. This is why the cloud-utils package as well as the network-manager-service-type are included in the vm image template I created as part of this patch.

The second shortcut is that I didn't see a good way to guix-ify the cloud-init service definitions/runtime environment (i.e. I used the /etc/cloud directory to handle most of the configuration files). I'd prefer if configuration lives in the store, but the configuration files and the gathered data live in locations which I didn't see a way to overwrite easily. I'm sure this exists, but would require more digging into the cloud-init source code and/or patches that would need to be pushed upstream.

The final shortcut is the actual running of the cloud-init-service. I didn't see a good way to *wait* for a process to complete, i.e. to block other services until the current service completes. It looks to me that running a long-running one-shot service (each stage of cloud-init typically takes a few seconds to run) doesn't block any service which requires it's provisions. Maybe there's some shepherd nuances that I'm not aware of. I ended up creating a computed file which produces a shell script which runs each cloud-init stage in sequence as a workaround to a more robust solution.

I added some extra comments with instructions on how to add extra cloud-init configuration files. This is required for cloud-init to run properly on DigitalOcean (the order of the data sources needs to be changed from the default). It might be necessary for other p

I've tested this on digital ocean and gotten it work properly (and then used guix deploy to update the configuration of the image). 

NOTE: there are some comments in the template file that would need to be uncommented to build the exact image I used.

Using this patch it should be easy to create an image for testing using:

./pre-inst-env guix system image -t qcow2 gnu/system/examples/cloud-init-image.tmpl

I would appreciate further testers and any suggestions for improvement.

Thanks!

Alex

Message #14 received at 74389 <at> debbugs.gnu.org (full text, mbox):

From: Steve George <steve <at> futurile.net>
To: 74389 <at> debbugs.gnu.org
Cc: alex <at> infiniteadaptability.org, andrew <at> trop.in
Subject: RE: services: add cloud-init service (for guix)
Date: Tue, 8 Apr 2025 14:25:44 +0100

Hi Alex/Ricardo,

In 74389 you're discussing adding a cloud-init service. It might be worth noting that I think Andrew has something in RDE that might be reusable. Andrew, is it upstreamable?

Just a thought,

Futurile / Steve

Message #17 received at 74389 <at> debbugs.gnu.org (full text, mbox):

From: Andrew Tropin <andrew <at> trop.in>
To: Steve George <steve <at> futurile.net>, 74389 <at> debbugs.gnu.org
Cc: alex <at> infiniteadaptability.org
Subject: Re: [bug#74389] services: add cloud-init service (for guix)
Date: Wed, 09 Apr 2025 09:05:42 +0700

[Message part 1 (text/plain, inline)]

On 2025-04-08 14:25, Steve George wrote:

> Hi Alex/Ricardo,
>
> In 74389 you're discussing adding a cloud-init service. It might be
> worth noting that I think Andrew has something in RDE that might be
> reusable. Andrew, is it upstreamable?

We have a partial implementation of cloud init, that covers basic needs,
but I would say the implementation is clean and complete:

https://git.sr.ht/~abcdw/rde/tree/a4a8329c/src/rde/system/services/cloud-init.scm#L1

-- 
Best regards,
Andrew Tropin

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 74389 <at> debbugs.gnu.org (full text, mbox):

From: Alex <alex <at> infiniteadaptability.org>
To: "andrew <at> trop.in" <andrew <at> trop.in>
Cc: "74389 <at> debbugs.gnu.org" <74389 <at> debbugs.gnu.org>,
 "steve <at> futurile.net" <steve <at> futurile.net>
Subject: Re: [bug#74389] services: add cloud-init service (for guix)
Date: Thu, 10 Apr 2025 01:20:01 +0000

Hi all:

Mine is usable as well, I've got an image built from a checkout I've used to stand up a vm on digitalocean. I took a quick look at that patch series and it looks like much more work was done on the configuration side and also there was some better understanding of resize partition functionality. I basically piggybacked off ubuntu's tools. My implementation's networking only worked on digitalocean when network-manager-service-type and wpa-supplicant-service-type were provided for some reason that I haven't debugged yet.

My work on this has stalled (been busy with day job), but I'm still going to need it by early 2026 when my current infrastructure expires, so would love to help get something into guix if possible.

Alex

-------- Original Message --------
On 4/8/25 7:05 PM, Andrew Tropin <andrew <at> trop.in> wrote:

>  On 2025-04-08 14:25, Steve George wrote:
>  
>  > Hi Alex/Ricardo,
>  >
>  > In 74389 you're discussing adding a cloud-init service. It might be
>  > worth noting that I think Andrew has something in RDE that might be
>  > reusable. Andrew, is it upstreamable?
>  
>  We have a partial implementation of cloud init, that covers basic needs,
>  but I would say the implementation is clean and complete:
>  
>  https://git.sr.ht/~abcdw/rde/tree/a4a8329c/src/rde/system/services/cloud-init.scm#L1
>  
>  -- 
>  Best regards,
>  Andrew Tropin
>

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.