From unknown Sun Aug 17 01:49:36 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#60573] [PATCH] doc: cookbook: Add "Installing Guix on a Cluster" chapter. Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-Sender: "Debbugs-submit" Resent-CC: rekado@elephly.net, guix-patches@gnu.org Resent-Date: Thu, 05 Jan 2023 11:48:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 60573 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 60573@debbugs.gnu.org Cc: Ludovic =?UTF-8?Q?Court=C3=A8s?= , Ricardo Wurmus X-Debbugs-Original-To: guix-patches@gnu.org X-Debbugs-Original-Xcc: Ricardo Wurmus Received: via spool by submit@debbugs.gnu.org id=B.167291925731215 (code B ref -1); Thu, 05 Jan 2023 11:48:01 +0000 Received: (at submit) by debbugs.gnu.org; 5 Jan 2023 11:47:37 +0000 Received: from localhost ([127.0.0.1]:50422 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pDOid-00087M-L6 for submit@debbugs.gnu.org; Thu, 05 Jan 2023 06:47:36 -0500 Received: from lists.gnu.org ([209.51.188.17]:54462) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pDOib-000879-10 for submit@debbugs.gnu.org; Thu, 05 Jan 2023 06:47:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pDOia-0002cv-ON for guix-patches@gnu.org; Thu, 05 Jan 2023 06:47:32 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pDOiZ-0000jr-KN; Thu, 05 Jan 2023 06:47:31 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=xXq7B9e0NHmuJylimEr/jCrTbn8vOOP5ukm8oi664TM=; b=WpjFN3zNXGorqe PKD3cSbfxh6cuCur6pH4QeUacg4TxX9LIOCWJ82LM3Lrzk1EMfRh5Tk9Eh2eqhMywGPd+cIE4sWUc OSA2mHSauR8WvIMNv6wqFba/L44J8y2RfCS4eMzIKVHSGjp2AXgUprzLewMsKRXHmlMNNGpiF0i1h Dlj8XwzGu6+9uy1j8pIc+QJysfbBfp64CpaJT1POtmmmCd6Tl0+qmirBKKEbuEP1noA71WuREmE/d xbD1esbYAy2BHgW0Cgn8XCXpfwf+6sTfDg65a1XyRl7QILVD7oIkoYIgeB36r8s3bJtlf7qXqdAb1 CJS6k8iV4UqwyjTidD8A==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201] helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pDOiZ-0002zU-82; Thu, 05 Jan 2023 06:47:31 -0500 From: Ludovic =?UTF-8?Q?Court=C3=A8s?= Date: Thu, 5 Jan 2023 12:47:23 +0100 Message-Id: <20230105114723.14928-1-ludo@gnu.org> X-Mailer: git-send-email 2.38.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) From: Ludovic Courtès This is derived from the article at , with clarifications and updates. * doc/guix-cookbook.texi (Installing Guix on a Cluster): New chapter. --- doc/guix-cookbook.texi | 433 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 414 insertions(+), 19 deletions(-) Hello! This turns into a living document that we can keep up-to-date. Comments? Suggestions? Ludo'. diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi index 795e7d3b25..5e7209c9c5 100644 --- a/doc/guix-cookbook.texi +++ b/doc/guix-cookbook.texi @@ -21,7 +21,8 @@ Copyright @copyright{} 2020 Brice Waegeneire@* Copyright @copyright{} 2020 André Batista@* Copyright @copyright{} 2020 Christine Lemmer-Webber@* Copyright @copyright{} 2021 Joshua Branson@* -Copyright @copyright{} 2022 Maxim Cournoyer* +Copyright @copyright{} 2022 Maxim Cournoyer@* +Copyright @copyright{} 2023 Ludovic Courtès Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or @@ -73,8 +74,9 @@ Weblate} (@pxref{Translating Guix,,, guix, GNU Guix reference manual}). * Packaging:: Packaging tutorials * System Configuration:: Customizing the GNU System * Containers:: Isolated environments and nested systems -* Advanced package management:: Power to the users! +* Advanced package management:: Power to the users! * Environment management:: Control environment +* Installing Guix on a Cluster:: High-performance computing. * Acknowledgments:: Thanks! * GNU Free Documentation License:: The license of this document. @@ -83,28 +85,45 @@ Weblate} (@pxref{Translating Guix,,, guix, GNU Guix reference manual}). @detailmenu --- The Detailed Node Listing --- -Scheme tutorials - -* A Scheme Crash Course:: Learn the basics of Scheme - Packaging -* Packaging Tutorial:: Let's add a package to Guix! +* Packaging Tutorial:: A tutorial on how to add packages to Guix. System Configuration -* Auto-Login to a Specific TTY:: Automatically Login a User to a Specific TTY -* Customizing the Kernel:: Creating and using a custom Linux kernel on Guix System. -* Guix System Image API:: Customizing images to target specific platforms. -* Using security keys:: How to use security keys with Guix System. -* Connecting to Wireguard VPN:: Connecting to a Wireguard VPN. -* Customizing a Window Manager:: Handle customization of a Window manager on Guix System. -* Running Guix on a Linode Server:: Running Guix on a Linode Server. Running Guix on a Linode Server -* Setting up a bind mount:: Setting up a bind mount in the file-systems definition. -* Getting substitutes from Tor:: Configuring Guix daemon to get substitutes through Tor. -* Setting up NGINX with Lua:: Configuring NGINX web-server to load Lua modules. +* Auto-Login to a Specific TTY:: Automatically Login a User to a Specific TTY +* Customizing the Kernel:: Creating and using a custom Linux kernel on Guix System. +* Guix System Image API:: Customizing images to target specific platforms. +* Using security keys:: How to use security keys with Guix System. +* Connecting to Wireguard VPN:: Connecting to a Wireguard VPN. +* Customizing a Window Manager:: Handle customization of a Window manager on Guix System. +* Running Guix on a Linode Server:: Running Guix on a Linode Server +* Setting up a bind mount:: Setting up a bind mount in the file-systems definition. +* Getting substitutes from Tor:: Configuring Guix daemon to get substitutes through Tor. +* Setting up NGINX with Lua:: Configuring NGINX web-server to load Lua modules. * Music Server with Bluetooth Audio:: Headless music player with Bluetooth output. +Containers + +* Guix Containers:: Perfectly isolated environments +* Guix System Containers:: A system inside your system + +Advanced package management + +* Guix Profiles in Practice:: Strategies for multiple profiles and manifests. + +Environment management + +* Guix environment via direnv:: Setup Guix environment with direnv + +Installing Guix on a Cluster + +* Setting Up a Head Node:: The node that runs the daemon. +* Setting Up Compute Nodes:: Client nodes. +* Cluster Network Access:: Dealing with network access restrictions. +* Cluster Disk Usage:: Disk usage considerations. +* Cluster Security Considerations:: Keeping the cluster secure. + @end detailmenu @end menu @@ -3637,6 +3656,380 @@ will have predefined environment variables and procedures. Run @command{direnv allow} to setup the environment for the first time. + +@c ********************************************************************* +@node Installing Guix on a Cluster +@chapter Installing Guix on a Cluster + +@cindex cluster installation +@cindex high-performance computing, HPC +@cindex HPC, high-performance computing +Guix is appealing to scientists and @acronym{HPC, high-performance +computing} practitioners: it makes it easy to deploy potentially complex +software stacks, and it lets you do so in a reproducible fashion---you +can redeploy the exact same software on different machines and at +different points in time. + +In this chapter we look at how a cluster sysadmin can install Guix for +system-wide use, such that it can be used on all the cluster nodes, and +discuss the various tradeoffs@footnote{This chapter is adapted from a +@uref{https://hpc.guix.info/blog/2017/11/installing-guix-on-a-cluster/, +blog post published on the Guix-HPC web site in 2017}.}. + +@quotation Note +Here we assume that the cluster is running a GNU/Linux distro other than +Guix System and that we are going to install Guix on top of it. +@end quotation + +@menu +* Setting Up a Head Node:: The node that runs the daemon. +* Setting Up Compute Nodes:: Client nodes. +* Cluster Network Access:: Dealing with network access restrictions. +* Cluster Disk Usage:: Disk usage considerations. +* Cluster Security Considerations:: Keeping the cluster secure. +@end menu + +@node Setting Up a Head Node +@section Setting Up a Head Node + +The recommended approach is to set up one @emph{head node} running +@command{guix-daemon} and exporting @file{/gnu/store} over NFS to +compute nodes. + +Remember that @command{guix-daemon} is responsible for spawning build +processes and downloads on behalf of clients (@pxref{Invoking +guix-daemon,,, guix, GNU Guix Reference Manual}), and more generally +accessing @file{/gnu/store}, which contains all the package binaries +built by all the users (@pxref{The Store,,, guix, GNU Guix Reference +Manual}). ``Client'' here refers to all the Guix commands that users +see, such as @code{guix install}. On a cluster, these commands may be +running on the compute nodes and we'll want them to talk to the head +node's @code{guix-daemon} instance. + +To begin with, the head node can be installed following the usual binary +installation instructions (@pxref{Binary Installation,,, guix, GNU Guix +Reference Manual}). Thanks to the installation script, this should be +quick. Once installation is complete, we need to make some adjustments. + +Since we want @code{guix-daemon} to be reachable not just from the head +node but also from the compute nodes, we need to arrange so that it +listens for connections over TCP/IP. To do that, we'll edit the systemd +startup file for @command{guix-daemon}, +@file{/etc/systemd/system/guix-daemon.service}, and add a +@code{--listen} argument to the @code{ExecStart} line so that it looks +something like this: + +@example +ExecStart=/var/guix/profiles/per-user/root/current-guix/bin/guix-daemon --build-users-group=guixbuild --listen=/var/guix/daemon-socket/socket --listen=0.0.0.0 +@end example + +For these changes to take effect, the service needs to be restarted: + +@example +systemctl daemon-reload +systemctl restart guix-daemon +@end example + +@quotation Note +The @code{--listen=0.0.0.0} bit means that @code{guix-daemon} will +process @emph{all} incoming TCP connections on port 44146 +(@pxref{Invoking guix-daemon,,, guix, GNU Guix Reference Manual}). This +is usually fine in a cluster setup where the head node is reachable +exclusively from the cluster's local area network---you don't want that +to be exposed to the Internet! +@end quotation + +The next step is to define our NFS exports in +@uref{https://linux.die.net/man/5/exports,@file{/etc/exports}} by adding +something along these lines: + +@example +/gnu/store *(ro) +/var/guix *(rw, async) +/var/log/guix *(ro) +@end example + +The @file{/gnu/store} directory can be exported read-only since only +@command{guix-daemon} on the master node will ever modify it. +@file{/var/guix} contains @emph{user profiles} as managed by @code{guix +package}; thus, to allow users to install packages with @code{guix +package}, this must be read-write. + +Users can create as many profiles as they like in addition to the +default profile, @file{~/.guix-profile}. For instance, @code{guix +package -p ~/dev/python-dev -i python} installs Python in a profile +reachable from the @code{~/dev/python-dev} symlink. To make sure that +this profile is protected from garbage collection---i.e., that Python +will not be removed from @file{/gnu/store} while this profile exists---, +@emph{home directories should be mounted on the head node} as well so +that @code{guix-daemon} knows about these non-standard profiles and +avoids collecting software they refer to. + +It may be a good idea to periodically remove unused bits from +@file{/gnu/store} by running @command{guix gc} (@pxref{Invoking guix +gc,,, guix, GNU Guix Reference Manual}). This can be done by adding a +crontab entry on the head node: + +@example +root@@master# crontab -e +@end example + +@noindent +... with something like this: + +@example +# Every day at 5AM, run the garbage collector to make sure +# at least 10 GB are free on /gnu/store. +0 5 * * 1 /usr/local/bin/guix gc -F10G +@end example + +We're done with the head node! Let's look at compute nodes now. + +@node Setting Up Compute Nodes +@section Setting Up Compute Nodes + +First of all, we need compute nodes to mount those NFS directories that +the head node exports. This can be done by adding the following lines +to @uref{https://linux.die.net/man/5/fstab,@file{/etc/fstab}}: + +@example +@var{head-node}:/gnu/store /gnu/store nfs defaults,_netdev,vers=3 0 0 +@var{head-node}:/var/guix /var/guix nfs defaults,_netdev,vers=3 0 0 +@var{head-node}:/var/log/guix /var/log/guix nfs defaults,_netdev,vers=3 0 0 +@end example + +@noindent +... where @var{head-node} is the name or IP address of your head node. +From there on, assuming the mount points exist, you should be able to +mount each of these on the compute nodes. + +Next, we need to provide a default @command{guix} command that users can +run when they first connect to the cluster (eventually they will invoke +@command{guix pull}, which will provide them with their ``own'' +@command{guix} command). Similar to what the binary installation script +did on the head node, we'll store that in @file{/usr/local/bin}: + +@example +mkdir -p /usr/local/bin +ln -s /var/guix/profiles/per-user/root/current-guix/bin/guix \ + /usr/local/bin/guix +@end example + +We then need to tell @code{guix} to talk to the daemon running on our +master node, by adding these lines to @code{/etc/profile}: + +@example +GUIX_DAEMON_SOCKET="guix://@var{head-node}" +export GUIX_DAEMON_SOCKET +@end example + +To avoid warnings and make sure @code{guix} uses the right locale, we +need to tell it to use locale data provided by Guix (@pxref{Application +Setup,,, guix, GNU Guix Reference Manual}): + +@example +GUIX_LOCPATH=/var/guix/profiles/per-user/root/guix-profile/lib/locale +export GUIX_LOCPATH + +# Here we must use a valid locale name. Try "ls $GUIX_LOCPATH/*" +# to see what names can be used. +LC_ALL=fr_FR.utf8 +export LC_ALL +@end example + +For convenience, @code{guix package} automatically generates +@file{~/.guix-profile/etc/profile}, which defines all the environment +variables necessary to use the packages---@code{PATH}, +@code{C_INCLUDE_PATH}, @code{PYTHONPATH}, etc. Thus it's a good idea to +source it from @code{/etc/profile}: + +@example +GUIX_PROFILE="$HOME/.guix-profile" +if [ -f "$GUIX_PROFILE/etc/profile" ]; then + . "$GUIX_PROFILE/etc/profile" +fi +@end example + +Last but not least, Guix provides command-line completion notably for +Bash and zsh. In @code{/etc/bashrc}, consider adding this line: + +@verbatim +. /var/guix/profiles/per-user/root/current-guix/etc/bash_completion.d/guix +@end verbatim + +Voilà! + +You can check that everything's in place by logging in on a compute node +and running: + +@example +guix install hello +@end example + +The daemon on the head node should download pre-built binaries on your +behalf and unpack them in @file{/gnu/store}, and @command{guix install} +should create @file{~/.guix-profile} containing the +@file{~/.guix-profile/bin/hello} command. + +@node Cluster Network Access +@section Network Access + +Guix requires network access to download source code and pre-built +binaries. The good news is that only the head node needs that since +compute nodes simply delegate to it. + +It is customary for cluster nodes to have access at best to a +@emph{white list} of hosts. Our head node needs at least +@code{ci.guix.gnu.org} in this white list since this is where it gets +pre-built binaries from by default, for all the packages that are in +Guix proper. + +Incidentally, @code{ci.guix.gnu.org} also serves as a +@emph{content-addressed mirror} of the source code of those packages. +Consequently, it is sufficient to have @emph{only} +@code{ci.guix.gnu.org} in that white list. + +Software packages maintained in a separate repository such as one of the +various @uref{https://hpc.guix.info/channels, HPC channels} are of +course unavailable from @code{ci.guix.gnu.org}. For these packages, you +may want to extend the white list such that source and pre-built +binaries (assuming this-party servers provide binaries for these +packages) can be downloaded. As a last resort, users can always +download source on their workstation and add it to the cluster's +@file{/gnu/store}, like this: + +@verbatim +GUIX_DAEMON_SOCKET=ssh://compute-node.example.org \ + guix download http://starpu.gforge.inria.fr/files/starpu-1.2.3/starpu-1.2.3.tar.gz +@end verbatim + +The above command downloads @code{starpu-1.2.3.tar.gz} @emph{and} sends +it to the cluster's @code{guix-daemon} instance over SSH. + +Air-gapped clusters require more work. At the moment, our suggestion +would be to download all the necessary source code on a workstation +running Guix. For instance, using the @option{--sources} option of +@command{guix build} (@pxref{Invoking guix build,,, guix, GNU Guix +Reference Manual}), the example below downloads all the source code the +@code{openmpi} package depends on: + +@example +$ guix build --sources=transitive openmpi + +@dots{} + +/gnu/store/xc17sm60fb8nxadc4qy0c7rqph499z8s-openmpi-1.10.7.tar.bz2 +/gnu/store/s67jx92lpipy2nfj5cz818xv430n4b7w-gcc-5.4.0.tar.xz +/gnu/store/npw9qh8a46lrxiwh9xwk0wpi3jlzmjnh-gmp-6.0.0a.tar.xz +/gnu/store/hcz0f4wkdbsvsdky3c0vdvcawhdkyldb-mpfr-3.1.5.tar.xz +/gnu/store/y9akh452n3p4w2v631nj0injx7y0d68x-mpc-1.0.3.tar.gz +/gnu/store/6g5c35q8avfnzs3v14dzl54cmrvddjm2-glibc-2.25.tar.xz +/gnu/store/p9k48dk3dvvk7gads7fk30xc2pxsd66z-hwloc-1.11.8.tar.bz2 +/gnu/store/cry9lqidwfrfmgl0x389cs3syr15p13q-gcc-5.4.0.tar.xz +/gnu/store/7ak0v3rzpqm2c5q1mp3v7cj0rxz0qakf-libfabric-1.4.1.tar.bz2 +/gnu/store/vh8syjrsilnbfcf582qhmvpg1v3rampf-rdma-core-14.tar.gz +… +@end example + +(In case you're wondering, that's more than 320@ MiB of +@emph{compressed} source code.) + +We can then make a big archive containing all of this (@pxref{Invoking +guix archive,,, guix, GNU Guix Reference Manual}): + +@verbatim +$ guix archive --export \ + `guix build --sources=transitive openmpi` \ + > openmpi-source-code.nar +@end verbatim + +@dots{} and we can eventually transfer that archive to the cluster on +removable storage and unpack it there: + +@verbatim +$ guix archive --import < openmpi-source-code.nar +@end verbatim + +This process has to be repeated every time new source code needs to be +brought to the cluster. + +As we write this, the research institutes involved in Guix-HPC do not +have air-gapped clusters though. If you have experience with such +setups, we would like to hear feedback and suggestions. + +@node Cluster Disk Usage +@section Disk Usage + +@cindex disk usage, on a cluster +A common concern of sysadmins' is whether this is all going to eat a lot +of disk space. If anything, if something is going to exhaust disk +space, it's going to be scientific data sets rather than compiled +software---that's our experience with almost ten years of Guix usage on +HPC clusters. Nevertheless, it's worth taking a look at how Guix +contributes to disk usage. + +First, having several versions or variants of a given package in +@file{/gnu/store} does not necessarily cost much, because +@command{guix-daemon} implements deduplication of identical files, and +package variants are likely to have a number of common files. + +As mentioned above, we recommend having a cron job to run @code{guix gc} +periodically, which removes @emph{unused} software from +@file{/gnu/store}. However, there's always a possibility that users will +keep lots of software in their profiles, or lots of old generations of +their profiles, which is ``live'' and cannot be deleted from the +viewpoint of @command{guix gc}. + +The solution to this is for users to regularly remove old generations of +their profile. For instance, the following command removes generations +that are more than two-month old: + +@example +guix package --delete-generations=2m +@end example + +Likewise, it's a good idea to invite users to regularly upgrade their +profile, which can reduce the number of variants of a given piece of +software stored in @file{/gnu/store}: + +@example +guix pull +guix upgrade +@end example + +As a last resort, it is always possible for sysadmins to do some of this +on behalf of their users. Nevertheless, one of the strengths of Guix is +the freedom and control users get on their software environment, so we +strongly recommend leaving users in control. + +@node Cluster Security Considerations +@section Security Considerations + +@cindex security, on a cluster +On an HPC cluster, Guix is typically used to manage scientific software. +Security-critical software such as the operating system kernel and +system services such as @code{sshd} and the batch scheduler remain under +control of sysadmins. + +The Guix project has a good track record delivering security updates in +a timely fashion (@pxref{Security Updates,,, guix, GNU Guix Reference +Manual}). To get security updates, users have to run @code{guix pull && +guix upgrade}. + +Because Guix uniquely identifies software variants, it is easy to see if +a vulnerable piece of software is in use. For instance, to check whether +the glibc@ 2.25 variant without the mitigation patch against +``@uref{https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt,Stack +Clash}'', one can check whether user profiles refer to it at all: + +@example +guix gc --referrers /gnu/store/…-glibc-2.25 +@end example + +This will report whether profiles exist that refer to this specific +glibc variant. + + @c ********************************************************************* @node Acknowledgments @chapter Acknowledgments @@ -3658,8 +4051,10 @@ information on these fine people. The @file{THANKS} file lists people who have helped by reporting bugs, taking care of the infrastructure, providing artwork and themes, making suggestions, and more---thank you! -This document includes adapted sections from articles that have previously -been published on the Guix blog at @uref{https://guix.gnu.org/blog}. +This document includes adapted sections from articles that have +previously been published on the Guix blog at +@uref{https://guix.gnu.org/blog} and on the Guix-HPC blog at +@uref{https://hpc.guix.info/blog}. @c ********************************************************************* base-commit: 473692b812b4ab4267d9bddad0fb27787d2112ff -- 2.38.1 From unknown Sun Aug 17 01:49:36 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Subject: bug#60573: closed (Re: bug#60573: [PATCH] doc: cookbook: Add "Installing Guix on a Cluster" chapter.) Message-ID: References: <87mt6rr5wq.fsf@gnu.org> <20230105114723.14928-1-ludo@gnu.org> X-Gnu-PR-Message: they-closed 60573 X-Gnu-PR-Package: guix-patches X-Gnu-PR-Keywords: patch Reply-To: 60573@debbugs.gnu.org Date: Mon, 09 Jan 2023 17:10:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1673284202-13692-1" This is a multi-part message in MIME format... ------------=_1673284202-13692-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #60573: [PATCH] doc: cookbook: Add "Installing Guix on a Cluster" chapter. which was filed against the guix-patches package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 60573@debbugs.gnu.org. --=20 60573: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D60573 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1673284202-13692-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 60573-done) by debbugs.gnu.org; 9 Jan 2023 17:09:51 +0000 Received: from localhost ([127.0.0.1]:38061 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pEveh-0003YN-BW for submit@debbugs.gnu.org; Mon, 09 Jan 2023 12:09:51 -0500 Received: from eggs.gnu.org ([209.51.188.92]:54108) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pEveg-0003YA-CB for 60573-done@debbugs.gnu.org; Mon, 09 Jan 2023 12:09:50 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pEvea-0001Yx-Rb; Mon, 09 Jan 2023 12:09:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To: From; bh=vjEI26fFC/wSn2gl8Mu3t92hSpfIm2ivkVg50bpU+p0=; b=GC4sfOSTmUDgM5Ug731l B7LvkA5uJg+/VU77mNom+IsxMjuD57x0U5pMkAHdjNOvQcBUhgp2PEsqgRVbMmZrWH/6476B6+m/5 osOiD8/0F972ezPYaJ89RrjqeRtewbTwp+tw/14gDjx/zlizGRAFIunocdv1wlDJ/v7Z0XnH5Xfyr 8fo4cboyFsWiYqx7wEcYy0F8nPdscu5u+lDxMSlCBbFnIVA227EVNPm1jgVeozMGc2JpJO12dbOC2 LM8NfKPUJTPTgpXQtX9nqhF4PJ4WjN9/kuPIL0yWwVM0f4/7FDqZj8MOlRY6Q6GyHaMmpNDknVzIR JMVLHAfNZhAx2w==; Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pEveZ-0004Gw-Ul; Mon, 09 Jan 2023 12:09:44 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 60573-done@debbugs.gnu.org Subject: Re: bug#60573: [PATCH] doc: cookbook: Add "Installing Guix on a Cluster" chapter. References: <20230105114723.14928-1-ludo@gnu.org> Date: Mon, 09 Jan 2023 18:09:41 +0100 In-Reply-To: <20230105114723.14928-1-ludo@gnu.org> ("Ludovic =?utf-8?Q?Cou?= =?utf-8?Q?rt=C3=A8s=22's?= message of "Thu, 5 Jan 2023 12:47:23 +0100") Message-ID: <87mt6rr5wq.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 60573-done Cc: Ricardo Wurmus X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Ludovic Court=C3=A8s skribis: > From: Ludovic Court=C3=A8s > > This is derived from the article at > , with > clarifications and updates. > > * doc/guix-cookbook.texi (Installing Guix on a Cluster): New chapter. Pushed as 47c1de22df30aa6c4a2f0d8249be93c3c7bc0022! Ludo'. ------------=_1673284202-13692-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 5 Jan 2023 11:47:37 +0000 Received: from localhost ([127.0.0.1]:50422 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pDOid-00087M-L6 for submit@debbugs.gnu.org; Thu, 05 Jan 2023 06:47:36 -0500 Received: from lists.gnu.org ([209.51.188.17]:54462) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pDOib-000879-10 for submit@debbugs.gnu.org; Thu, 05 Jan 2023 06:47:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pDOia-0002cv-ON for guix-patches@gnu.org; Thu, 05 Jan 2023 06:47:32 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pDOiZ-0000jr-KN; Thu, 05 Jan 2023 06:47:31 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-Version:Date:Subject:To:From:in-reply-to: references; bh=xXq7B9e0NHmuJylimEr/jCrTbn8vOOP5ukm8oi664TM=; b=WpjFN3zNXGorqe PKD3cSbfxh6cuCur6pH4QeUacg4TxX9LIOCWJ82LM3Lrzk1EMfRh5Tk9Eh2eqhMywGPd+cIE4sWUc OSA2mHSauR8WvIMNv6wqFba/L44J8y2RfCS4eMzIKVHSGjp2AXgUprzLewMsKRXHmlMNNGpiF0i1h Dlj8XwzGu6+9uy1j8pIc+QJysfbBfp64CpaJT1POtmmmCd6Tl0+qmirBKKEbuEP1noA71WuREmE/d xbD1esbYAy2BHgW0Cgn8XCXpfwf+6sTfDg65a1XyRl7QILVD7oIkoYIgeB36r8s3bJtlf7qXqdAb1 CJS6k8iV4UqwyjTidD8A==; Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201] helo=gnu.org) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pDOiZ-0002zU-82; Thu, 05 Jan 2023 06:47:31 -0500 From: =?UTF-8?q?Ludovic=20Court=C3=A8s?= To: guix-patches@gnu.org Subject: [PATCH] doc: cookbook: Add "Installing Guix on a Cluster" chapter. Date: Thu, 5 Jan 2023 12:47:23 +0100 Message-Id: <20230105114723.14928-1-ludo@gnu.org> X-Mailer: git-send-email 2.38.1 MIME-Version: 1.0 X-Debbugs-Cc: Ricardo Wurmus Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit Cc: =?UTF-8?q?Ludovic=20Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) From: Ludovic Courtès This is derived from the article at , with clarifications and updates. * doc/guix-cookbook.texi (Installing Guix on a Cluster): New chapter. --- doc/guix-cookbook.texi | 433 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 414 insertions(+), 19 deletions(-) Hello! This turns into a living document that we can keep up-to-date. Comments? Suggestions? Ludo'. diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi index 795e7d3b25..5e7209c9c5 100644 --- a/doc/guix-cookbook.texi +++ b/doc/guix-cookbook.texi @@ -21,7 +21,8 @@ Copyright @copyright{} 2020 Brice Waegeneire@* Copyright @copyright{} 2020 André Batista@* Copyright @copyright{} 2020 Christine Lemmer-Webber@* Copyright @copyright{} 2021 Joshua Branson@* -Copyright @copyright{} 2022 Maxim Cournoyer* +Copyright @copyright{} 2022 Maxim Cournoyer@* +Copyright @copyright{} 2023 Ludovic Courtès Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or @@ -73,8 +74,9 @@ Weblate} (@pxref{Translating Guix,,, guix, GNU Guix reference manual}). * Packaging:: Packaging tutorials * System Configuration:: Customizing the GNU System * Containers:: Isolated environments and nested systems -* Advanced package management:: Power to the users! +* Advanced package management:: Power to the users! * Environment management:: Control environment +* Installing Guix on a Cluster:: High-performance computing. * Acknowledgments:: Thanks! * GNU Free Documentation License:: The license of this document. @@ -83,28 +85,45 @@ Weblate} (@pxref{Translating Guix,,, guix, GNU Guix reference manual}). @detailmenu --- The Detailed Node Listing --- -Scheme tutorials - -* A Scheme Crash Course:: Learn the basics of Scheme - Packaging -* Packaging Tutorial:: Let's add a package to Guix! +* Packaging Tutorial:: A tutorial on how to add packages to Guix. System Configuration -* Auto-Login to a Specific TTY:: Automatically Login a User to a Specific TTY -* Customizing the Kernel:: Creating and using a custom Linux kernel on Guix System. -* Guix System Image API:: Customizing images to target specific platforms. -* Using security keys:: How to use security keys with Guix System. -* Connecting to Wireguard VPN:: Connecting to a Wireguard VPN. -* Customizing a Window Manager:: Handle customization of a Window manager on Guix System. -* Running Guix on a Linode Server:: Running Guix on a Linode Server. Running Guix on a Linode Server -* Setting up a bind mount:: Setting up a bind mount in the file-systems definition. -* Getting substitutes from Tor:: Configuring Guix daemon to get substitutes through Tor. -* Setting up NGINX with Lua:: Configuring NGINX web-server to load Lua modules. +* Auto-Login to a Specific TTY:: Automatically Login a User to a Specific TTY +* Customizing the Kernel:: Creating and using a custom Linux kernel on Guix System. +* Guix System Image API:: Customizing images to target specific platforms. +* Using security keys:: How to use security keys with Guix System. +* Connecting to Wireguard VPN:: Connecting to a Wireguard VPN. +* Customizing a Window Manager:: Handle customization of a Window manager on Guix System. +* Running Guix on a Linode Server:: Running Guix on a Linode Server +* Setting up a bind mount:: Setting up a bind mount in the file-systems definition. +* Getting substitutes from Tor:: Configuring Guix daemon to get substitutes through Tor. +* Setting up NGINX with Lua:: Configuring NGINX web-server to load Lua modules. * Music Server with Bluetooth Audio:: Headless music player with Bluetooth output. +Containers + +* Guix Containers:: Perfectly isolated environments +* Guix System Containers:: A system inside your system + +Advanced package management + +* Guix Profiles in Practice:: Strategies for multiple profiles and manifests. + +Environment management + +* Guix environment via direnv:: Setup Guix environment with direnv + +Installing Guix on a Cluster + +* Setting Up a Head Node:: The node that runs the daemon. +* Setting Up Compute Nodes:: Client nodes. +* Cluster Network Access:: Dealing with network access restrictions. +* Cluster Disk Usage:: Disk usage considerations. +* Cluster Security Considerations:: Keeping the cluster secure. + @end detailmenu @end menu @@ -3637,6 +3656,380 @@ will have predefined environment variables and procedures. Run @command{direnv allow} to setup the environment for the first time. + +@c ********************************************************************* +@node Installing Guix on a Cluster +@chapter Installing Guix on a Cluster + +@cindex cluster installation +@cindex high-performance computing, HPC +@cindex HPC, high-performance computing +Guix is appealing to scientists and @acronym{HPC, high-performance +computing} practitioners: it makes it easy to deploy potentially complex +software stacks, and it lets you do so in a reproducible fashion---you +can redeploy the exact same software on different machines and at +different points in time. + +In this chapter we look at how a cluster sysadmin can install Guix for +system-wide use, such that it can be used on all the cluster nodes, and +discuss the various tradeoffs@footnote{This chapter is adapted from a +@uref{https://hpc.guix.info/blog/2017/11/installing-guix-on-a-cluster/, +blog post published on the Guix-HPC web site in 2017}.}. + +@quotation Note +Here we assume that the cluster is running a GNU/Linux distro other than +Guix System and that we are going to install Guix on top of it. +@end quotation + +@menu +* Setting Up a Head Node:: The node that runs the daemon. +* Setting Up Compute Nodes:: Client nodes. +* Cluster Network Access:: Dealing with network access restrictions. +* Cluster Disk Usage:: Disk usage considerations. +* Cluster Security Considerations:: Keeping the cluster secure. +@end menu + +@node Setting Up a Head Node +@section Setting Up a Head Node + +The recommended approach is to set up one @emph{head node} running +@command{guix-daemon} and exporting @file{/gnu/store} over NFS to +compute nodes. + +Remember that @command{guix-daemon} is responsible for spawning build +processes and downloads on behalf of clients (@pxref{Invoking +guix-daemon,,, guix, GNU Guix Reference Manual}), and more generally +accessing @file{/gnu/store}, which contains all the package binaries +built by all the users (@pxref{The Store,,, guix, GNU Guix Reference +Manual}). ``Client'' here refers to all the Guix commands that users +see, such as @code{guix install}. On a cluster, these commands may be +running on the compute nodes and we'll want them to talk to the head +node's @code{guix-daemon} instance. + +To begin with, the head node can be installed following the usual binary +installation instructions (@pxref{Binary Installation,,, guix, GNU Guix +Reference Manual}). Thanks to the installation script, this should be +quick. Once installation is complete, we need to make some adjustments. + +Since we want @code{guix-daemon} to be reachable not just from the head +node but also from the compute nodes, we need to arrange so that it +listens for connections over TCP/IP. To do that, we'll edit the systemd +startup file for @command{guix-daemon}, +@file{/etc/systemd/system/guix-daemon.service}, and add a +@code{--listen} argument to the @code{ExecStart} line so that it looks +something like this: + +@example +ExecStart=/var/guix/profiles/per-user/root/current-guix/bin/guix-daemon --build-users-group=guixbuild --listen=/var/guix/daemon-socket/socket --listen=0.0.0.0 +@end example + +For these changes to take effect, the service needs to be restarted: + +@example +systemctl daemon-reload +systemctl restart guix-daemon +@end example + +@quotation Note +The @code{--listen=0.0.0.0} bit means that @code{guix-daemon} will +process @emph{all} incoming TCP connections on port 44146 +(@pxref{Invoking guix-daemon,,, guix, GNU Guix Reference Manual}). This +is usually fine in a cluster setup where the head node is reachable +exclusively from the cluster's local area network---you don't want that +to be exposed to the Internet! +@end quotation + +The next step is to define our NFS exports in +@uref{https://linux.die.net/man/5/exports,@file{/etc/exports}} by adding +something along these lines: + +@example +/gnu/store *(ro) +/var/guix *(rw, async) +/var/log/guix *(ro) +@end example + +The @file{/gnu/store} directory can be exported read-only since only +@command{guix-daemon} on the master node will ever modify it. +@file{/var/guix} contains @emph{user profiles} as managed by @code{guix +package}; thus, to allow users to install packages with @code{guix +package}, this must be read-write. + +Users can create as many profiles as they like in addition to the +default profile, @file{~/.guix-profile}. For instance, @code{guix +package -p ~/dev/python-dev -i python} installs Python in a profile +reachable from the @code{~/dev/python-dev} symlink. To make sure that +this profile is protected from garbage collection---i.e., that Python +will not be removed from @file{/gnu/store} while this profile exists---, +@emph{home directories should be mounted on the head node} as well so +that @code{guix-daemon} knows about these non-standard profiles and +avoids collecting software they refer to. + +It may be a good idea to periodically remove unused bits from +@file{/gnu/store} by running @command{guix gc} (@pxref{Invoking guix +gc,,, guix, GNU Guix Reference Manual}). This can be done by adding a +crontab entry on the head node: + +@example +root@@master# crontab -e +@end example + +@noindent +... with something like this: + +@example +# Every day at 5AM, run the garbage collector to make sure +# at least 10 GB are free on /gnu/store. +0 5 * * 1 /usr/local/bin/guix gc -F10G +@end example + +We're done with the head node! Let's look at compute nodes now. + +@node Setting Up Compute Nodes +@section Setting Up Compute Nodes + +First of all, we need compute nodes to mount those NFS directories that +the head node exports. This can be done by adding the following lines +to @uref{https://linux.die.net/man/5/fstab,@file{/etc/fstab}}: + +@example +@var{head-node}:/gnu/store /gnu/store nfs defaults,_netdev,vers=3 0 0 +@var{head-node}:/var/guix /var/guix nfs defaults,_netdev,vers=3 0 0 +@var{head-node}:/var/log/guix /var/log/guix nfs defaults,_netdev,vers=3 0 0 +@end example + +@noindent +... where @var{head-node} is the name or IP address of your head node. +From there on, assuming the mount points exist, you should be able to +mount each of these on the compute nodes. + +Next, we need to provide a default @command{guix} command that users can +run when they first connect to the cluster (eventually they will invoke +@command{guix pull}, which will provide them with their ``own'' +@command{guix} command). Similar to what the binary installation script +did on the head node, we'll store that in @file{/usr/local/bin}: + +@example +mkdir -p /usr/local/bin +ln -s /var/guix/profiles/per-user/root/current-guix/bin/guix \ + /usr/local/bin/guix +@end example + +We then need to tell @code{guix} to talk to the daemon running on our +master node, by adding these lines to @code{/etc/profile}: + +@example +GUIX_DAEMON_SOCKET="guix://@var{head-node}" +export GUIX_DAEMON_SOCKET +@end example + +To avoid warnings and make sure @code{guix} uses the right locale, we +need to tell it to use locale data provided by Guix (@pxref{Application +Setup,,, guix, GNU Guix Reference Manual}): + +@example +GUIX_LOCPATH=/var/guix/profiles/per-user/root/guix-profile/lib/locale +export GUIX_LOCPATH + +# Here we must use a valid locale name. Try "ls $GUIX_LOCPATH/*" +# to see what names can be used. +LC_ALL=fr_FR.utf8 +export LC_ALL +@end example + +For convenience, @code{guix package} automatically generates +@file{~/.guix-profile/etc/profile}, which defines all the environment +variables necessary to use the packages---@code{PATH}, +@code{C_INCLUDE_PATH}, @code{PYTHONPATH}, etc. Thus it's a good idea to +source it from @code{/etc/profile}: + +@example +GUIX_PROFILE="$HOME/.guix-profile" +if [ -f "$GUIX_PROFILE/etc/profile" ]; then + . "$GUIX_PROFILE/etc/profile" +fi +@end example + +Last but not least, Guix provides command-line completion notably for +Bash and zsh. In @code{/etc/bashrc}, consider adding this line: + +@verbatim +. /var/guix/profiles/per-user/root/current-guix/etc/bash_completion.d/guix +@end verbatim + +Voilà! + +You can check that everything's in place by logging in on a compute node +and running: + +@example +guix install hello +@end example + +The daemon on the head node should download pre-built binaries on your +behalf and unpack them in @file{/gnu/store}, and @command{guix install} +should create @file{~/.guix-profile} containing the +@file{~/.guix-profile/bin/hello} command. + +@node Cluster Network Access +@section Network Access + +Guix requires network access to download source code and pre-built +binaries. The good news is that only the head node needs that since +compute nodes simply delegate to it. + +It is customary for cluster nodes to have access at best to a +@emph{white list} of hosts. Our head node needs at least +@code{ci.guix.gnu.org} in this white list since this is where it gets +pre-built binaries from by default, for all the packages that are in +Guix proper. + +Incidentally, @code{ci.guix.gnu.org} also serves as a +@emph{content-addressed mirror} of the source code of those packages. +Consequently, it is sufficient to have @emph{only} +@code{ci.guix.gnu.org} in that white list. + +Software packages maintained in a separate repository such as one of the +various @uref{https://hpc.guix.info/channels, HPC channels} are of +course unavailable from @code{ci.guix.gnu.org}. For these packages, you +may want to extend the white list such that source and pre-built +binaries (assuming this-party servers provide binaries for these +packages) can be downloaded. As a last resort, users can always +download source on their workstation and add it to the cluster's +@file{/gnu/store}, like this: + +@verbatim +GUIX_DAEMON_SOCKET=ssh://compute-node.example.org \ + guix download http://starpu.gforge.inria.fr/files/starpu-1.2.3/starpu-1.2.3.tar.gz +@end verbatim + +The above command downloads @code{starpu-1.2.3.tar.gz} @emph{and} sends +it to the cluster's @code{guix-daemon} instance over SSH. + +Air-gapped clusters require more work. At the moment, our suggestion +would be to download all the necessary source code on a workstation +running Guix. For instance, using the @option{--sources} option of +@command{guix build} (@pxref{Invoking guix build,,, guix, GNU Guix +Reference Manual}), the example below downloads all the source code the +@code{openmpi} package depends on: + +@example +$ guix build --sources=transitive openmpi + +@dots{} + +/gnu/store/xc17sm60fb8nxadc4qy0c7rqph499z8s-openmpi-1.10.7.tar.bz2 +/gnu/store/s67jx92lpipy2nfj5cz818xv430n4b7w-gcc-5.4.0.tar.xz +/gnu/store/npw9qh8a46lrxiwh9xwk0wpi3jlzmjnh-gmp-6.0.0a.tar.xz +/gnu/store/hcz0f4wkdbsvsdky3c0vdvcawhdkyldb-mpfr-3.1.5.tar.xz +/gnu/store/y9akh452n3p4w2v631nj0injx7y0d68x-mpc-1.0.3.tar.gz +/gnu/store/6g5c35q8avfnzs3v14dzl54cmrvddjm2-glibc-2.25.tar.xz +/gnu/store/p9k48dk3dvvk7gads7fk30xc2pxsd66z-hwloc-1.11.8.tar.bz2 +/gnu/store/cry9lqidwfrfmgl0x389cs3syr15p13q-gcc-5.4.0.tar.xz +/gnu/store/7ak0v3rzpqm2c5q1mp3v7cj0rxz0qakf-libfabric-1.4.1.tar.bz2 +/gnu/store/vh8syjrsilnbfcf582qhmvpg1v3rampf-rdma-core-14.tar.gz +… +@end example + +(In case you're wondering, that's more than 320@ MiB of +@emph{compressed} source code.) + +We can then make a big archive containing all of this (@pxref{Invoking +guix archive,,, guix, GNU Guix Reference Manual}): + +@verbatim +$ guix archive --export \ + `guix build --sources=transitive openmpi` \ + > openmpi-source-code.nar +@end verbatim + +@dots{} and we can eventually transfer that archive to the cluster on +removable storage and unpack it there: + +@verbatim +$ guix archive --import < openmpi-source-code.nar +@end verbatim + +This process has to be repeated every time new source code needs to be +brought to the cluster. + +As we write this, the research institutes involved in Guix-HPC do not +have air-gapped clusters though. If you have experience with such +setups, we would like to hear feedback and suggestions. + +@node Cluster Disk Usage +@section Disk Usage + +@cindex disk usage, on a cluster +A common concern of sysadmins' is whether this is all going to eat a lot +of disk space. If anything, if something is going to exhaust disk +space, it's going to be scientific data sets rather than compiled +software---that's our experience with almost ten years of Guix usage on +HPC clusters. Nevertheless, it's worth taking a look at how Guix +contributes to disk usage. + +First, having several versions or variants of a given package in +@file{/gnu/store} does not necessarily cost much, because +@command{guix-daemon} implements deduplication of identical files, and +package variants are likely to have a number of common files. + +As mentioned above, we recommend having a cron job to run @code{guix gc} +periodically, which removes @emph{unused} software from +@file{/gnu/store}. However, there's always a possibility that users will +keep lots of software in their profiles, or lots of old generations of +their profiles, which is ``live'' and cannot be deleted from the +viewpoint of @command{guix gc}. + +The solution to this is for users to regularly remove old generations of +their profile. For instance, the following command removes generations +that are more than two-month old: + +@example +guix package --delete-generations=2m +@end example + +Likewise, it's a good idea to invite users to regularly upgrade their +profile, which can reduce the number of variants of a given piece of +software stored in @file{/gnu/store}: + +@example +guix pull +guix upgrade +@end example + +As a last resort, it is always possible for sysadmins to do some of this +on behalf of their users. Nevertheless, one of the strengths of Guix is +the freedom and control users get on their software environment, so we +strongly recommend leaving users in control. + +@node Cluster Security Considerations +@section Security Considerations + +@cindex security, on a cluster +On an HPC cluster, Guix is typically used to manage scientific software. +Security-critical software such as the operating system kernel and +system services such as @code{sshd} and the batch scheduler remain under +control of sysadmins. + +The Guix project has a good track record delivering security updates in +a timely fashion (@pxref{Security Updates,,, guix, GNU Guix Reference +Manual}). To get security updates, users have to run @code{guix pull && +guix upgrade}. + +Because Guix uniquely identifies software variants, it is easy to see if +a vulnerable piece of software is in use. For instance, to check whether +the glibc@ 2.25 variant without the mitigation patch against +``@uref{https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt,Stack +Clash}'', one can check whether user profiles refer to it at all: + +@example +guix gc --referrers /gnu/store/…-glibc-2.25 +@end example + +This will report whether profiles exist that refer to this specific +glibc variant. + + @c ********************************************************************* @node Acknowledgments @chapter Acknowledgments @@ -3658,8 +4051,10 @@ information on these fine people. The @file{THANKS} file lists people who have helped by reporting bugs, taking care of the infrastructure, providing artwork and themes, making suggestions, and more---thank you! -This document includes adapted sections from articles that have previously -been published on the Guix blog at @uref{https://guix.gnu.org/blog}. +This document includes adapted sections from articles that have +previously been published on the Guix blog at +@uref{https://guix.gnu.org/blog} and on the Guix-HPC blog at +@uref{https://hpc.guix.info/blog}. @c ********************************************************************* base-commit: 473692b812b4ab4267d9bddad0fb27787d2112ff -- 2.38.1 ------------=_1673284202-13692-1-- From unknown Sun Aug 17 01:49:36 2025 X-Loop: help-debbugs@gnu.org Subject: [bug#60573] [PATCH] doc: cookbook: Add "Installing Guix on a Cluster" chapter. Resent-From: Simon Tournier Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 12 Jan 2023 17:48:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 60573 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Ludovic =?UTF-8?Q?Court=C3=A8s?= , 60573@debbugs.gnu.org Cc: Ricardo Wurmus , Ludovic =?UTF-8?Q?Court=C3=A8s?= Received: via spool by 60573-submit@debbugs.gnu.org id=B60573.167354563210696 (code B ref 60573); Thu, 12 Jan 2023 17:48:02 +0000 Received: (at 60573) by debbugs.gnu.org; 12 Jan 2023 17:47:12 +0000 Received: from localhost ([127.0.0.1]:48526 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pG1fQ-0002mO-Fz for submit@debbugs.gnu.org; Thu, 12 Jan 2023 12:47:11 -0500 Received: from mail-wr1-f49.google.com ([209.85.221.49]:41967) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pG1fI-0002l4-HZ for 60573@debbugs.gnu.org; Thu, 12 Jan 2023 12:47:01 -0500 Received: by mail-wr1-f49.google.com with SMTP id w1so18855272wrt.8 for <60573@debbugs.gnu.org>; Thu, 12 Jan 2023 09:47:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:from:to:cc:subject:date:message-id :reply-to; bh=GkfNdLx9jdfqZKNfqdw31r7iIv94ppFLd0Fte9+fnUM=; b=BEcssfttIUi+kBstSTTxqdG8evv6vrMSjSXZSTgkMFBEAZ3WtIgzal8G0csLvaVidb /wJ6owgvJ4i0coxQcCasnJ4hghLpa+fAmbRjRtFt0Y/7j14RxIezAD0WSBN9va5QZejq jiwi5UlY6bvgi0fYfSvgUpXUdFPoaQmy0bUFE8k4KGx0kUhodILKF+8HL27a6a6NIzNW 8bgZWczlKwv3fh8mWN+/uun7IkZik3Nqalr/wLkTk18g+ZuFwP3XBSa32QYJxUE1Qdwy Rt74pOgecMjPihIqYh0qL8DTF0cWbk37pHGSceZbDqsu+R9uksgS5eK2WfOddokqHBE5 eKig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:references :in-reply-to:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GkfNdLx9jdfqZKNfqdw31r7iIv94ppFLd0Fte9+fnUM=; b=W+B320sQPkAZcxYzvQLxkOU7g8a67ke2Nfs4cBuXwd+DhPQo4DC4EwO3SLDACAGrcX pWYfq9VTcfJn6CL73kHJbki2q/3GnuNNh2XYVF1xF5WRiGMtmknwu5T4ke+VyoaPnCLy BVVrXIbYVvQ7FWIoXDr9i1KnpMMsE65kzVXwJyrw8HlvWIyAzWUFu1JEIbKbzb2wxUZf L1bE1q6yMxV88h/SU6JdZieFDsfJkirN9RGIBoexVlktPQQ6RHAI3zYN9WbwNDemCfMD NwTRdlwfyAq504cKmqZUPu02ba/vjbG2I/yGmiRhpf6djifrxU9BvdtvW8MOwXdJY+fa IPhA== X-Gm-Message-State: AFqh2koqwj+E/gE3EFldC5I0UpUuOUwlYUHQWvbZ1Clxcqju95VZHuIE kJQMInaSVD4jzyb+FEbozhs= X-Google-Smtp-Source: AMrXdXt60IJDzWsvQU6YS/F1XcbHHh7l1Z4M+DiBVpquC9gcniphbZ4WPgcm4NyWszXXroPWj6J9Pw== X-Received: by 2002:a5d:688d:0:b0:28d:9b93:e31f with SMTP id h13-20020a5d688d000000b0028d9b93e31fmr6235913wru.4.1673545614851; Thu, 12 Jan 2023 09:46:54 -0800 (PST) Received: from pfiuh07 ([193.48.40.241]) by smtp.gmail.com with ESMTPSA id z6-20020a05600c0a0600b003d96efd09b7sm31935891wmp.19.2023.01.12.09.46.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Jan 2023 09:46:54 -0800 (PST) From: Simon Tournier In-Reply-To: <20230105114723.14928-1-ludo@gnu.org> References: <20230105114723.14928-1-ludo@gnu.org> Date: Thu, 12 Jan 2023 15:10:24 +0100 Message-ID: <875ydbyhbj.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 1.1 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hi Ludo, On jeu., 05 janv. 2023 at 12:47, Ludovic =?UTF-8?Q?Court=C3=A8s?= wrote: > From: Ludovic =?UTF-8?Q?Court=C3=A8s?= > > This is derived from the article at > , with [...] Content analysis details: (1.1 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (zimon.toutoune[at]gmail.com) 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [209.85.221.49 listed in wl.mailspike.net] -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [209.85.221.49 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 1.1 DATE_IN_PAST_03_06 Date: is 3 to 6 hours before Received: date X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.1 (/) Hi Ludo, On jeu., 05 janv. 2023 at 12:47, Ludovic Court=C3=A8s wrote: > From: Ludovic Court=C3=A8s > > This is derived from the article at > , with > clarifications and updates. > > * doc/guix-cookbook.texi (Installing Guix on a Cluster): New chapter. You did more than just adding a new chapter. ;-) Cool! I was doing something similar when preparing a v2 for Guix extension. > -Scheme tutorials > - > -* A Scheme Crash Course:: Learn the basics of Scheme > - > Packaging >=20=20 > -* Packaging Tutorial:: Let's add a package to Guix! > +* Packaging Tutorial:: A tutorial on how to add packages to Guix. >=20=20 > System Configuration >=20=20 > -* Auto-Login to a Specific TTY:: Automatically Login a User to a Spec= ific TTY > -* Customizing the Kernel:: Creating and using a custom Linux ke= rnel on Guix System. > -* Guix System Image API:: Customizing images to target specifi= c platforms. > -* Using security keys:: How to use security keys with Guix S= ystem. > -* Connecting to Wireguard VPN:: Connecting to a Wireguard VPN. > -* Customizing a Window Manager:: Handle customization of a Window man= ager on Guix System. > -* Running Guix on a Linode Server:: Running Guix on a Linode Server. Ru= nning Guix on a Linode Server > -* Setting up a bind mount:: Setting up a bind mount in the file-= systems definition. > -* Getting substitutes from Tor:: Configuring Guix daemon to get subst= itutes through Tor. > -* Setting up NGINX with Lua:: Configuring NGINX web-server to load= Lua modules. > +* Auto-Login to a Specific TTY:: Automatically Login a User to a Specifi= c TTY > +* Customizing the Kernel:: Creating and using a custom Linux kerne= l on Guix System. > +* Guix System Image API:: Customizing images to target specific p= latforms. > +* Using security keys:: How to use security keys with Guix Syst= em. > +* Connecting to Wireguard VPN:: Connecting to a Wireguard VPN. > +* Customizing a Window Manager:: Handle customization of a Window manage= r on Guix System. > +* Running Guix on a Linode Server:: Running Guix on a Linode Server > +* Setting up a bind mount:: Setting up a bind mount in the file-systems = definition. > +* Getting substitutes from Tor:: Configuring Guix daemon to get substitu= tes through Tor. > +* Setting up NGINX with Lua:: Configuring NGINX web-server to load Lua m= odules. > * Music Server with Bluetooth Audio:: Headless music player with Bluetoo= th output. >=20=20 > +Containers > + > +* Guix Containers:: Perfectly isolated environments > +* Guix System Containers:: A system inside your system > + > +Advanced package management > + > +* Guix Profiles in Practice:: Strategies for multiple profiles and m= anifests. > + > +Environment management > + > +* Guix environment via direnv:: Setup Guix environment with direnv > + > +Installing Guix on a Cluster > + > +* Setting Up a Head Node:: The node that runs the daemon. > +* Setting Up Compute Nodes:: Client nodes. > +* Cluster Network Access:: Dealing with network access restrictions. > +* Cluster Disk Usage:: Disk usage considerations. > +* Cluster Security Considerations:: Keeping the cluster secure. Cheers, simon