GNU bug report logs -
#76501
[PATCH 0/1] Fix deployment of smaller Hetzner instances
Previous Next
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 76501 in the body.
You can then email your comments to 76501 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
ludo <at> gnu.org, maxim.cournoyer <at> gmail.com, guix-patches <at> gnu.org
:
bug#76501
; Package
guix-patches
.
(Sun, 23 Feb 2025 12:16:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Roman Scherer <roman <at> burningswell.com>
:
New bug report received and forwarded. Copy sent to
ludo <at> gnu.org, maxim.cournoyer <at> gmail.com, guix-patches <at> gnu.org
.
(Sun, 23 Feb 2025 12:16:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Hello Guix,
Fabio Natali reached out to me that there is an "out of disk space" issue when
deploying smaller instances with the hetzner-environment-type.
I thought I tried it on smaller instances. But either I didn't try it on the
smaller instances, or another theory: I tried it back in the day, but Guix got
larger in the meantime?
Looking closer I discovered that the size of the root partition of the rescue
system depends on the instance type, and is a lot smaller on those smaller ones.
On a cax11 instance, booted into the rescue system from which a minimal Guix
system is installed, "df -h" shows:
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
[2a01:4ff:ff00::b007:1]:/nfs 1.2T 999G 151G 87% /root/.oldroot/nfs
overlay 1.9G 1.9G 0 100% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 768M 864K 767M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda1 38G 44K 36G 1% /mnt
/dev/sda15 244M 152K 244M 1% /mnt/boot/efi
tmpfs 384M 0 384M 0% /run/user/0
The 1.9G of / on the the rescue system seem to be not enough to host Guix
installed as a foreign distro on the rescue system, plus the packages needed
to install the new Guix bootstrap system.
To fix this I came up with the following solution:
- before installing Guix on the rescue system, I make sure that /gnu/store has
enough space.
- this is done by bind mounting /mnt/tmp/gnu/store (here /mnt is the root of
the new Guix system having more disk space) to /gnu/store.
- then Guix is installed with apt-get on the rescue system using the store
that points into the tmp directory of the new Guix system.
- A minimal Guix system is installed onto /mnt, rebooted and from there the
final operating system config is applied. When the minimal Guix system
boots, the /tmp/gnu/store is gone and not used anymore.
This seems to work. I have tried it and Fabio also reported that it works for
him.
What do you think of this strategy? Is there a better one?
I attached a patch for this and also changed the instance type that are used
in the tests to smaller ones, so this is covered for the future. Could you
please review it?
Fabio also asked me why I choose to use medium sized instances as the default
instead of the smallest. My thinking was so people trying this for the first
time have a good experience and not have to deal with an under-powered
instance. I would leave it that way. If you think we should default to smaller
ones, please let me know.
Unfortunatly the above solution did not work with the smallest CPX11 instance,
with 2 VCPUs, 2 GB of RAM and 40 GB disk space.
The rescue system only 970M:
root <at> rescue /usr/lib # df -h
Filesystem Size Used Avail Use% Mounted on
udev 961M 0 961M 0% /dev
[2a01:4ff:ff00::b007:1]:/nfs 1.2T 999G 151G 87% /root/.oldroot/nfs
overlay 970M 821M 150M 85% /
tmpfs 970M 0 970M 0% /dev/shm
tmpfs 388M 668K 388M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda1 38G 1.3G 34G 4% /mnt
/dev/sda15 241M 142K 241M 1% /mnt/boot/efi
tmpfs 194M 0 194M 0% /run/user/0
Installing Guix via apt-get works. But when installing the minimal bootstrap
Guix system it fails with:
...
downloading from https://ci.guix.gnu.org/nar/lzip/lclbcq0jds63zal1p55g6v0mwz90s44y-guile-git-0.5.2 ...
downloading from https://ci.guix.gnu.org/nar/gzip/g2ajyl8xk9aarxrgjbng2hkj3qm2v0z2-tar-1.34 ...
downloading from https://ci.guix.gnu.org/nar/gzip/v06gnr579r0jmr36aha3wkbd1y27ccg7-disarchive-0.4.0 ...
downloading from https://ci.guix.gnu.org/nar/lzip/9nvx97hr8kkr26gzwni2fblfn0yq0xjw-guix-1.4.0rc2 ...
error (ignored): aborting transaction: cannot rollback - no transaction is active
guix system: error: committing transaction: database or disk is full
Not sure what to do about that. I added a note to the manual that CPX11
instances are not supported at the moment.
Thanks Roman.
Roman Scherer (1):
gnu: machine: hetzner: Fix deployment on smaller instances.
doc/guix.texi | 4 +++-
gnu/machine/hetzner.scm | 9 ++++++++-
tests/machine/hetzner.scm | 4 ++--
3 files changed, 13 insertions(+), 4 deletions(-)
base-commit: 5f4c785fc3caa0fd960ebcf9c1ea6ab396b96f25
--
2.48.1
Information forwarded
to
ludo <at> gnu.org, maxim.cournoyer <at> gmail.com, guix-patches <at> gnu.org
:
bug#76501
; Package
guix-patches
.
(Sun, 23 Feb 2025 12:23:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 76501 <at> debbugs.gnu.org (full text, mbox):
* gnu/machine/hetzner.scm (hetzner-machine-rescue-install-os): Avoid out of
disk space error by bind mounting /mnt/tmp/gnu/store to /gnu/store.
* tests/machine/hetzner.scm: Test with smaller instances.
* doc/guix.texi: Mention unsupported instance.
Change-Id: If8bfb6733de493b51813b3e82e255849192f7cba
---
doc/guix.texi | 4 +++-
gnu/machine/hetzner.scm | 9 ++++++++-
tests/machine/hetzner.scm | 4 ++--
3 files changed, 13 insertions(+), 4 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index 83ba0f3292..51db3f9c77 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -45361,7 +45361,9 @@ Invoking guix deploy
@code{"cx42"} corresponds to a x86_64 server that has 8 VCPUs, 16 GB of
memory and 160 GB of storage, while @code{"cax31"} to the AArch64
equivalent. Other server types and their current prices can be found
-@uref{https://www.hetzner.com/cloud/#pricing, here}.
+@uref{https://www.hetzner.com/cloud/#pricing, here}. The @code{"cpx11"}
+server type is currently not supported, since its rescue system is too
+small to bootstrap a Guix system from.
@item @code{ssh-key}
The file name of the SSH private key to use to authenticate with the
diff --git a/gnu/machine/hetzner.scm b/gnu/machine/hetzner.scm
index 1d9a3f4385..e87149a06a 100644
--- a/gnu/machine/hetzner.scm
+++ b/gnu/machine/hetzner.scm
@@ -551,6 +551,13 @@ (define (hetzner-machine-rescue-install-os machine ssh-session server)
cp /root/.ssh/authorized_keys /mnt/root/.ssh/authorized_keys
chmod 600 /mnt/root/.ssh/authorized_keys
+# Small instance don't have much disk space. Bind mount the store of the
+# resuce system to the tmp directory of the new Guix system.
+mkdir -p /mnt/tmp/gnu/store
+mkdir -p /gnu/store
+mount --bind /mnt/tmp/gnu/store /gnu/store
+
+apt-get install guix --assume-yes
cat > /tmp/guix/deploy/hetzner-os.scm << EOF
(use-modules (gnu) (guix utils))
(use-package-modules ssh)
@@ -605,7 +612,7 @@ (define (hetzner-machine-rescue-install-packages machine ssh-session)
(format #f "#!/usr/bin/env bash
set -eo pipefail
apt-get update
-apt-get install guix cloud-initramfs-growroot --assume-yes"))
+apt-get install cloud-initramfs-growroot --assume-yes"))
(format #t "successfully installed rescue system packages on '~a'\n" name)))
(define (hetzner-machine-delete machine server)
diff --git a/tests/machine/hetzner.scm b/tests/machine/hetzner.scm
index 39eac4a4d5..1552bcb9a0 100644
--- a/tests/machine/hetzner.scm
+++ b/tests/machine/hetzner.scm
@@ -64,7 +64,7 @@ (define %arm-machine
(environment hetzner-environment-type)
(configuration (hetzner-configuration
(labels %labels)
- (server-type "cax41")
+ (server-type "cax11")
(ssh-key %ssh-key-file)))))
(define %x86-machine
@@ -76,7 +76,7 @@ (define %x86-machine
(environment hetzner-environment-type)
(configuration (hetzner-configuration
(labels %labels)
- (server-type "cpx51")
+ (server-type "cx22")
(ssh-key %ssh-key-file)))))
(define (cleanup machine)
--
2.48.1
Information forwarded
to
guix-patches <at> gnu.org
:
bug#76501
; Package
guix-patches
.
(Mon, 24 Feb 2025 12:07:01 GMT)
Full text and
rfc822 format available.
Message #11 received at 76501 <at> debbugs.gnu.org (full text, mbox):
On 2025-02-23, 13:22 +0100, Roman Scherer <roman <at> burningswell.com> wrote:
> * gnu/machine/hetzner.scm (hetzner-machine-rescue-install-os): Avoid out of
> disk space error by bind mounting /mnt/tmp/gnu/store to /gnu/store.
> * tests/machine/hetzner.scm: Test with smaller instances.
> * doc/guix.texi: Mention unsupported instance.
Hi All.
Thanks Roman for your work on 'hetzner-environment-type', it's really
cool that 'guix deploy' gets expanded with other integrations and
features. Also thanks for getting back to me so quickly when I reached
out the other day.
I confirm that the patch works well for me. I was able to provision
various cx22 (and above) Hetzner instances with it.
Just a cosmetic thing, here it should be "rescue":
> +# Small instance don't have much disk space. Bind mount the store of the
> +# resuce system to the tmp directory of the new Guix system.
Thanks, best wishes, Fabio.
--
Fabio Natali
https://fabionatali.com
Information forwarded
to
guix-patches <at> gnu.org
:
bug#76501
; Package
guix-patches
.
(Tue, 25 Feb 2025 10:28:03 GMT)
Full text and
rfc822 format available.
Message #14 received at 76501 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
I also opened a support ticket at Hetzner with the request to have more
disk space on the rescue system of smaller instances. They said they
handed the request over to their customers team and will prioritize it,
depending on demand.
@Fabio / anyone listening: If you want to see this happening, maybe also
send them an email about this ;)
Fabio Natali <me <at> fabionatali.com> writes:
> On 2025-02-23, 13:22 +0100, Roman Scherer <roman <at> burningswell.com> wrote:
>> * gnu/machine/hetzner.scm (hetzner-machine-rescue-install-os): Avoid out of
>> disk space error by bind mounting /mnt/tmp/gnu/store to /gnu/store.
>> * tests/machine/hetzner.scm: Test with smaller instances.
>> * doc/guix.texi: Mention unsupported instance.
>
> Hi All.
>
> Thanks Roman for your work on 'hetzner-environment-type', it's really
> cool that 'guix deploy' gets expanded with other integrations and
> features. Also thanks for getting back to me so quickly when I reached
> out the other day.
>
> I confirm that the patch works well for me. I was able to provision
> various cx22 (and above) Hetzner instances with it.
>
> Just a cosmetic thing, here it should be "rescue":
>
>> +# Small instance don't have much disk space. Bind mount the store of the
>> +# resuce system to the tmp directory of the new Guix system.
>
> Thanks, best wishes, Fabio.
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
guix-patches <at> gnu.org
:
bug#76501
; Package
guix-patches
.
(Tue, 25 Feb 2025 13:22:01 GMT)
Full text and
rfc822 format available.
Message #17 received at 76501 <at> debbugs.gnu.org (full text, mbox):
On 2025-02-25, 11:27 +0100, Roman Scherer <roman <at> burningswell.com> wrote:
> I also opened a support ticket at Hetzner with the request to have more
> disk space on the rescue system of smaller instances. They said they
> handed the request over to their customers team and will prioritize it,
> depending on demand.
>
> @Fabio / anyone listening: If you want to see this happening, maybe also
> send them an email about this ;)
Hi Roman,
Absolutely, good idea and glad to do that. I think the Guix Foundation
has a Hetzner account too, we could think of sending an email from that
account too. I'll try and reach out to someone at the Guix Foundation
(Tanguy? Chris?) to suggest this.
Shall we generally indicate that we've run into issues with the size of
the rescue system - or do we have any number that we can attach to our
request? E.g. anything above X GB on all instances (including the
smallest ones)?
This is orthogonal and probably worth a separate thread but I wonder how
this works with the other guix deploy backend, the DigitalOcean one, and
if there's any similar limitation there.
Thanks, best wishes, Fabio.
--
Fabio Natali
https://fabionatali.com
Information forwarded
to
guix-patches <at> gnu.org
:
bug#76501
; Package
guix-patches
.
(Wed, 26 Feb 2025 08:53:01 GMT)
Full text and
rfc822 format available.
Message #20 received at 76501 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi Fabio,
Fabio Natali <me <at> fabionatali.com> writes:
> On 2025-02-25, 11:27 +0100, Roman Scherer <roman <at> burningswell.com> wrote:
>> I also opened a support ticket at Hetzner with the request to have more
>> disk space on the rescue system of smaller instances. They said they
>> handed the request over to their customers team and will prioritize it,
>> depending on demand.
>>
>> @Fabio / anyone listening: If you want to see this happening, maybe also
>> send them an email about this ;)
>
> Hi Roman,
>
> Absolutely, good idea and glad to do that. I think the Guix Foundation
> has a Hetzner account too, we could think of sending an email from that
> account too. I'll try and reach out to someone at the Guix Foundation
> (Tanguy? Chris?) to suggest this.
>
> Shall we generally indicate that we've run into issues with the size of
> the rescue system - or do we have any number that we can attach to our
> request? E.g. anything above X GB on all instances (including the
> smallest ones)?
The current approach works with the smallest ARM instance, cax11, and it
has 1.9GB free space when booted into the rescue system. Maybe ask for a
bit more than this, just to be safe.
> This is orthogonal and probably worth a separate thread but I wonder how
> this works with the other guix deploy backend, the DigitalOcean one, and
> if there's any similar limitation there.
On Digital Ocean guix deploy does not use a resuce system, but instead
boots into a Debian system that then gets "infected" with a Guix system,
by installing it on top of the Debian system, and moving directories
like /etc around. I tried this initially, but run into issues with the
network not resolving hosts anymore when /etc got moved. It looked like
the infect script used for Digitial Ocean is tied to a specific Debian
version, that wasn't available on Hetzner.
> Thanks, best wishes, Fabio.
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
ludo <at> gnu.org, maxim.cournoyer <at> gmail.com, guix-patches <at> gnu.org
:
bug#76501
; Package
guix-patches
.
(Thu, 27 Feb 2025 18:26:02 GMT)
Full text and
rfc822 format available.
Message #23 received at 76501 <at> debbugs.gnu.org (full text, mbox):
* gnu/machine/hetzner.scm (hetzner-machine-rescue-install-os): Avoid out of
disk space error by bind mounting /mnt/tmp/gnu/store to /gnu/store.
* tests/machine/hetzner.scm: Test with smaller instances.
* doc/guix.texi: Mention unsupported instance.
Change-Id: If8bfb6733de493b51813b3e82e255849192f7cba
---
doc/guix.texi | 4 +++-
gnu/machine/hetzner.scm | 9 ++++++++-
tests/machine/hetzner.scm | 4 ++--
3 files changed, 13 insertions(+), 4 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index a036c85c31..f0891da924 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -45380,7 +45380,9 @@ Invoking guix deploy
@code{"cx42"} corresponds to a x86_64 server that has 8 VCPUs, 16 GB of
memory and 160 GB of storage, while @code{"cax31"} to the AArch64
equivalent. Other server types and their current prices can be found
-@uref{https://www.hetzner.com/cloud/#pricing, here}.
+@uref{https://www.hetzner.com/cloud/#pricing, here}. The @code{"cpx11"}
+server type is currently not supported, since its rescue system is too
+small to bootstrap a Guix system from.
@item @code{ssh-key}
The file name of the SSH private key to use to authenticate with the
diff --git a/gnu/machine/hetzner.scm b/gnu/machine/hetzner.scm
index 1d9a3f4385..bc8d2efbd3 100644
--- a/gnu/machine/hetzner.scm
+++ b/gnu/machine/hetzner.scm
@@ -551,6 +551,13 @@ (define (hetzner-machine-rescue-install-os machine ssh-session server)
cp /root/.ssh/authorized_keys /mnt/root/.ssh/authorized_keys
chmod 600 /mnt/root/.ssh/authorized_keys
+# Small instance don't have much disk space. Bind mount the store of the
+# rescue system to the tmp directory of the new Guix system.
+mkdir -p /mnt/tmp/gnu/store
+mkdir -p /gnu/store
+mount --bind /mnt/tmp/gnu/store /gnu/store
+
+apt-get install guix --assume-yes
cat > /tmp/guix/deploy/hetzner-os.scm << EOF
(use-modules (gnu) (guix utils))
(use-package-modules ssh)
@@ -605,7 +612,7 @@ (define (hetzner-machine-rescue-install-packages machine ssh-session)
(format #f "#!/usr/bin/env bash
set -eo pipefail
apt-get update
-apt-get install guix cloud-initramfs-growroot --assume-yes"))
+apt-get install cloud-initramfs-growroot --assume-yes"))
(format #t "successfully installed rescue system packages on '~a'\n" name)))
(define (hetzner-machine-delete machine server)
diff --git a/tests/machine/hetzner.scm b/tests/machine/hetzner.scm
index 39eac4a4d5..1552bcb9a0 100644
--- a/tests/machine/hetzner.scm
+++ b/tests/machine/hetzner.scm
@@ -64,7 +64,7 @@ (define %arm-machine
(environment hetzner-environment-type)
(configuration (hetzner-configuration
(labels %labels)
- (server-type "cax41")
+ (server-type "cax11")
(ssh-key %ssh-key-file)))))
(define %x86-machine
@@ -76,7 +76,7 @@ (define %x86-machine
(environment hetzner-environment-type)
(configuration (hetzner-configuration
(labels %labels)
- (server-type "cpx51")
+ (server-type "cx22")
(ssh-key %ssh-key-file)))))
(define (cleanup machine)
base-commit: f1810c84f72090b02544ac41d0582c4785728ec4
--
2.48.1
Information forwarded
to
guix-patches <at> gnu.org
:
bug#76501
; Package
guix-patches
.
(Thu, 27 Feb 2025 18:27:02 GMT)
Full text and
rfc822 format available.
Message #26 received at 76501 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi,
I sent v2 that fixes the spelling.
Thanks, Roman.
Fabio Natali <me <at> fabionatali.com> writes:
> On 2025-02-23, 13:22 +0100, Roman Scherer <roman <at> burningswell.com> wrote:
>> * gnu/machine/hetzner.scm (hetzner-machine-rescue-install-os): Avoid out of
>> disk space error by bind mounting /mnt/tmp/gnu/store to /gnu/store.
>> * tests/machine/hetzner.scm: Test with smaller instances.
>> * doc/guix.texi: Mention unsupported instance.
>
> Hi All.
>
> Thanks Roman for your work on 'hetzner-environment-type', it's really
> cool that 'guix deploy' gets expanded with other integrations and
> features. Also thanks for getting back to me so quickly when I reached
> out the other day.
>
> I confirm that the patch works well for me. I was able to provision
> various cx22 (and above) Hetzner instances with it.
>
> Just a cosmetic thing, here it should be "rescue":
>
>> +# Small instance don't have much disk space. Bind mount the store of the
>> +# resuce system to the tmp directory of the new Guix system.
>
> Thanks, best wishes, Fabio.
[signature.asc (application/pgp-signature, inline)]
Reply sent
to
Ludovic Courtès <ludo <at> gnu.org>
:
You have taken responsibility.
(Fri, 28 Feb 2025 21:56:03 GMT)
Full text and
rfc822 format available.
Notification sent
to
Roman Scherer <roman <at> burningswell.com>
:
bug acknowledged by developer.
(Fri, 28 Feb 2025 21:56:03 GMT)
Full text and
rfc822 format available.
Message #31 received at 76501-done <at> debbugs.gnu.org (full text, mbox):
Roman Scherer <roman <at> burningswell.com> skribis:
> * gnu/machine/hetzner.scm (hetzner-machine-rescue-install-os): Avoid out of
> disk space error by bind mounting /mnt/tmp/gnu/store to /gnu/store.
> * tests/machine/hetzner.scm: Test with smaller instances.
> * doc/guix.texi: Mention unsupported instance.
>
> Change-Id: If8bfb6733de493b51813b3e82e255849192f7cba
Applied, thanks!
Information forwarded
to
guix-patches <at> gnu.org
:
bug#76501
; Package
guix-patches
.
(Sat, 01 Mar 2025 09:13:02 GMT)
Full text and
rfc822 format available.
Message #34 received at 76501-done <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Thank you Ludo!
Ludovic Courtès <ludo <at> gnu.org> writes:
> Roman Scherer <roman <at> burningswell.com> skribis:
>
>> * gnu/machine/hetzner.scm (hetzner-machine-rescue-install-os): Avoid out of
>> disk space error by bind mounting /mnt/tmp/gnu/store to /gnu/store.
>> * tests/machine/hetzner.scm: Test with smaller instances.
>> * doc/guix.texi: Mention unsupported instance.
>>
>> Change-Id: If8bfb6733de493b51813b3e82e255849192f7cba
>
> Applied, thanks!
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
guix-patches <at> gnu.org
:
bug#76501
; Package
guix-patches
.
(Sat, 22 Mar 2025 22:15:01 GMT)
Full text and
rfc822 format available.
Message #37 received at 76501 <at> debbugs.gnu.org (full text, mbox):
Hello,
I am coming a bit late for this issue, but I have found a technique that
might be usefull for related problems.
The problem here is we want to bootstrap guix on a small vps. When set
in "rescue mode" the vps root file system is really small and guix is too
big to be installed here before being moved to the "disk filesystem"
usually in /dev/sda
The workaround I found is to build a guix image locally, and stream the
result directly over ssh to the /dev/sda disk.
Once the vps is in rescue mode, this is done with such a command :
cat $(guix system image minimal_os.scm) | ssh -C <IP> dd of=/dev/sda status=progress
Depending on the vps, --image-type=mbr-raw should be passed to guix
system image. This is the case for ovh.
Note that I also needed to add virtio driver for the minimal_os to work
(initrd-modules (cons "virtio_scsi" %base-initrd-modules))
This technique worked so far with the smallest vps from ovh and scaleway.
Hovewer, I guess that as it should work for any vps where rescue mode
over ssh is available.
Hope this can help some of you
Benjamin
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Sun, 20 Apr 2025 11:24:07 GMT)
Full text and
rfc822 format available.
This bug report was last modified 57 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.