GNU bug report logs - #27401
[PATCH] doc: Suggest a QEMU image size large enough for the system examples.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 27401 in the body.
You can then email your comments to 27401 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: guix-patches <at> gnu.org
Subject: [PATCH] doc: Suggest a QEMU image size large enough for the system
 examples.
Date: Sat, 17 Jun 2017 01:14:54 -0400

* doc/guix.texi (Installing GuixSD in a Virtual Machine): Increase suggested
image size from 5 GB to 50 GB.
---
 doc/guix.texi | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index db0e2fbd7..4933a98dd 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -7796,11 +7796,11 @@ Create a disk image that will hold the installed system.  To make a
 qcow2-formatted disk image, use the @command{qemu-img} command:
 
 @example
-qemu-img create -f qcow2 guixsd.img 5G
+qemu-img create -f qcow2 guixsd.img 50G
 @end example
 
-The resulting file will be much smaller than 5GB, but it will grow as
-the virtualized storage device is filled up.
+The resulting file will be much smaller than 50 GB (typically less than
+1 MB), but it will grow as the virtualized storage device is filled up.
 
 @item
 Boot the USB installation image in an VM:
-- 
2.13.1

Message #8 received at 27401 <at> debbugs.gnu.org (full text, mbox):

From: Marius Bakke <mbakke <at> fastmail.com>
To: Leo Famulari <leo <at> famulari.name>, 27401 <at> debbugs.gnu.org
Subject: Re: [bug#27401] [PATCH] doc: Suggest a QEMU image size large enough
 for	the system examples.
Date: Sat, 17 Jun 2017 14:20:47 +0200

[Message part 1 (text/plain, inline)]

Leo Famulari <leo <at> famulari.name> writes:

> * doc/guix.texi (Installing GuixSD in a Virtual Machine): Increase suggested
> image size from 5 GB to 50 GB.

LGTM.

[signature.asc (application/pgp-signature, inline)]

Message #13 received at 27401 <at> debbugs.gnu.org (full text, mbox):

From: Jan Nieuwenhuizen <janneke <at> gnu.org>
To: Leo Famulari <leo <at> famulari.name>
Cc: 27401 <at> debbugs.gnu.org
Subject: Re: [bug#27401] [PATCH] doc: Suggest a QEMU image size large enough
 for the system examples.
Date: Mon, 19 Jun 2017 19:55:03 +0200

Leo Famulari writes:

> * doc/guix.texi (Installing GuixSD in a Virtual Machine): Increase suggested
> image size from 5 GB to 50 GB.

LGTM, however -- What about something like this patch too?  I seems that
the VMs we create (although they are of the ballooning cow2 kind), are
2GiB -- i.e., almost unusable for Real Stuff (TM)?

$ git diff
diff --git a/Makefile.am b/Makefile.am
index a3b3e557a..ff2ae120b 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -592,7 +592,7 @@ GUIXSD_VM_IMAGE_BASE = guixsd-vm-image-$(PACKAGE_VERSION)
 GUIXSD_INSTALLATION_IMAGE_SIZE ?= 950MiB
 
 # Size of the VM image (for x86_64 typically).
-GUIXSD_VM_IMAGE_SIZE ?= 2GiB
+GUIXSD_VM_IMAGE_SIZE ?= 50GiB
 
 # The release process works in several phases:
 #

-- 
Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Message #16 received at 27401 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Jan Nieuwenhuizen <janneke <at> gnu.org>
Cc: 27401 <at> debbugs.gnu.org
Subject: Re: [bug#27401] [PATCH] doc: Suggest a QEMU image size large enough
 for the system examples.
Date: Mon, 19 Jun 2017 17:48:20 -0400

[Message part 1 (text/plain, inline)]

On Mon, Jun 19, 2017 at 07:55:03PM +0200, Jan Nieuwenhuizen wrote:
> Leo Famulari writes:
> 
> > * doc/guix.texi (Installing GuixSD in a Virtual Machine): Increase suggested
> > image size from 5 GB to 50 GB.
> 
> LGTM, however -- What about something like this patch too?  I seems that
> the VMs we create (although they are of the ballooning cow2 kind), are
> 2GiB -- i.e., almost unusable for Real Stuff (TM)?

I chose 2GiB for those images because it's pretty easy to grow a
partition and ext4 filesystem from a running system. On the other hand,
I couldn't find an easy way to shrink them.

We can't predict the size of the virtual block device storage provided
by a VPS hoster, so I chose this small size with these expectations:
1) 2GiB will never be too big
2) users will resize after booting

I didn't come up with a better solution for the problem of not knowing
the storage size in advance. What do you think?

[signature.asc (application/pgp-signature, inline)]

Message #19 received at 27401 <at> debbugs.gnu.org (full text, mbox):

From: Jan Nieuwenhuizen <janneke <at> gnu.org>
To: Leo Famulari <leo <at> famulari.name>
Cc: 27401 <at> debbugs.gnu.org
Subject: Re: [bug#27401] [PATCH] doc: Suggest a QEMU image size large enough
 for the system examples.
Date: Tue, 20 Jun 2017 20:03:17 +0200

Leo Famulari writes:

>> LGTM, however -- What about something like this patch too?  I seems that
>> the VMs we create (although they are of the ballooning cow2 kind), are
>> 2GiB -- i.e., almost unusable for Real Stuff (TM)?
>
> I chose 2GiB for those images because it's pretty easy to grow a
> partition and ext4 filesystem from a running system. On the other hand,
> I couldn't find an easy way to shrink them.
>
> We can't predict the size of the virtual block device storage provided
> by a VPS hoster,

Ah, sure.  I missed that aspect.

> so I chose this small size with these expectations:
> 1) 2GiB will never be too big

Yes.

> 2) users will resize after booting

When I was asked to help with this, I was confronted with a 2GiB sda1
and right behind that an EFI sda2.  That was running in vmware (don't
ask ;-).  We hesitated, removed the sda2 and did the resize thing.

Sadly I have no further information, instead of testing the reboot we
decided to go for qemu.

> I didn't come up with a better solution for the problem of not knowing
> the storage size in advance. What do you think?

Yes, I think you made the right choice.

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke <at> gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Message #22 received at 27401 <at> debbugs.gnu.org (full text, mbox):

From: Leo Famulari <leo <at> famulari.name>
To: Jan Nieuwenhuizen <janneke <at> gnu.org>
Cc: 27401 <at> debbugs.gnu.org
Subject: Re: [bug#27401] [PATCH] doc: Suggest a QEMU image size large enough
 for the system examples.
Date: Fri, 23 Jun 2017 17:13:28 -0400

[Message part 1 (text/plain, inline)]

On Tue, Jun 20, 2017 at 08:03:17PM +0200, Jan Nieuwenhuizen wrote:
> Leo Famulari writes:
> > 2) users will resize after booting
> 
> When I was asked to help with this, I was confronted with a 2GiB sda1
> and right behind that an EFI sda2.  That was running in vmware (don't
> ask ;-).  We hesitated, removed the sda2 and did the resize thing.

Ah, I didn't plan for that sort of disk layout.

> Sadly I have no further information, instead of testing the reboot we
> decided to go for qemu.
> 
> > I didn't come up with a better solution for the problem of not knowing
> > the storage size in advance. What do you think?
> 
> Yes, I think you made the right choice.

Okay. I'd like to clarify and say that I think we could do something
more sophisticated, but I'm not sure what we would do :)

[signature.asc (application/pgp-signature, inline)]

Message #25 received at 27401 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Leo Famulari <leo <at> famulari.name>
Cc: 27401 <at> debbugs.gnu.org, Jan Nieuwenhuizen <janneke <at> gnu.org>
Subject: Guessing the disk image size in 'guix system'
Date: Mon, 26 Jun 2017 13:51:13 +0200

[Message part 1 (text/plain, inline)]

Hello!

Leo Famulari <leo <at> famulari.name> skribis:

> On Tue, Jun 20, 2017 at 08:03:17PM +0200, Jan Nieuwenhuizen wrote:

[...]

>> > I didn't come up with a better solution for the problem of not knowing
>> > the storage size in advance. What do you think?
>> 
>> Yes, I think you made the right choice.
>
> Okay. I'd like to clarify and say that I think we could do something
> more sophisticated, but I'm not sure what we would do :)

We can do “guix size $(guix system build foo.scm)” and add 10% or so to
that.

Maybe something like this mostly-untested patch would work?

Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/gnu/system/vm.scm b/gnu/system/vm.scm
index 392737d07..15cc0d993 100644
--- a/gnu/system/vm.scm
+++ b/gnu/system/vm.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2013, 2014, 2015, 2016 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2013, 2014, 2015, 2016, 2017 Ludovic Courtès <ludo <at> gnu.org>
 ;;; Copyright © 2016 Christopher Allan Webber <cwebber <at> dustycloud.org>
 ;;; Copyright © 2016 Leo Famulari <leo <at> famulari.name>
 ;;; Copyright © 2017 Mathieu Othacehe <m.othacehe <at> gmail.com>
@@ -108,8 +108,7 @@
                                              (references-graphs #f)
                                              (memory-size 256)
                                              (disk-image-format "qcow2")
-                                             (disk-image-size
-                                              (* 100 (expt 2 20))))
+                                             (disk-image-size 'guess))
   "Evaluate EXP in a QEMU virtual machine running LINUX with INITRD (a
 derivation).  In the virtual machine, EXP has access to all its inputs from the
 store; it should put its output files in the `/xchg' directory, which is
@@ -138,19 +137,24 @@ made available under the /xchg CIFS share."
     (define builder
       ;; Code that launches the VM that evaluates EXP.
       (with-imported-modules (source-module-closure '((guix build utils)
+                                                      (guix build store-copy)
                                                       (gnu build vm)))
         #~(begin
             (use-modules (guix build utils)
+                         (guix build store-copy)
                          (gnu build vm))
 
-            (let ((inputs  '#$(list qemu coreutils))
-                  (linux   (string-append #$linux "/"
-                                          #$(system-linux-image-file-name)))
-                  (initrd  (string-append #$initrd "/initrd"))
-                  (loader  #$loader)
-                  (graphs  '#$(match references-graphs
-                                (((graph-files . _) ...) graph-files)
-                                (_ #f))))
+            (let* ((inputs  '#$(list qemu coreutils))
+                   (linux   (string-append #$linux "/"
+                                           #$(system-linux-image-file-name)))
+                   (initrd  (string-append #$initrd "/initrd"))
+                   (loader  #$loader)
+                   (graphs  '#$(match references-graphs
+                                 (((graph-files . _) ...) graph-files)
+                                 (_ #f)))
+                   (size    #$(if (eq? 'guess disk-image-size)
+                                  #~(round (* 1.1 (closure-size graphs)))
+                                  disk-image-size)))
 
               (set-path-environment-variable "PATH" '("bin") inputs)
 
@@ -160,7 +164,7 @@ made available under the /xchg CIFS share."
                                 #:memory-size #$memory-size
                                 #:make-disk-image? #$make-disk-image?
                                 #:disk-image-format #$disk-image-format
-                                #:disk-image-size #$disk-image-size
+                                #:disk-image-size size
                                 #:references-graphs graphs)))))
 
     (gexp->derivation name builder
@@ -198,10 +202,13 @@ the image."
   (expression->derivation-in-linux-vm
    name
    (with-imported-modules (source-module-closure '((gnu build vm)
+                                                   (guix build store-copy)
                                                    (guix build utils)))
      #~(begin
          (use-modules (gnu build vm)
-                      (guix build utils))
+                      (guix build store-copy)
+                      (guix build utils)
+                      (srfi srfi-26))
 
          (let ((inputs
                 '#$(append (list qemu parted e2fsprogs dosfstools)
@@ -227,9 +234,14 @@ the image."
                                #:copy-closures? #$copy-inputs?
                                #:register-closures? #$register-closures?
                                #:system-directory #$os-drv))
+                  (root-size  #$(if (eq? 'guess disk-image-size)
+                                    #~(closure-size
+                                       (map (cut string-append "/xchg/" <>)
+                                            graphs))
+                                    (- disk-image-size
+                                       (* 50 (expt 2 20)))))
                   (partitions (list (partition
-                                     (size #$(- disk-image-size
-                                                (* 50 (expt 2 20))))
+                                     (size root-size)
                                      (label #$file-system-label)
                                      (file-system #$file-system-type)
                                      (flags '(boot))
diff --git a/guix/build/store-copy.scm b/guix/build/store-copy.scm
index a296bdf78..fe2eb6f69 100644
--- a/guix/build/store-copy.scm
+++ b/guix/build/store-copy.scm
@@ -1,5 +1,5 @@
 ;;; GNU Guix --- Functional package management for GNU
-;;; Copyright © 2013, 2014 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2013, 2014, 2017 Ludovic Courtès <ludo <at> gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -20,7 +20,9 @@
   #:use-module (guix build utils)
   #:use-module (srfi srfi-1)
   #:use-module (ice-9 rdelim)
+  #:use-module (ice-9 ftw)
   #:export (read-reference-graph
+            closure-size
             populate-store))
 
 ;;; Commentary:
@@ -46,6 +48,37 @@ The data at PORT is the format produced by #:references-graphs."
            (loop (read-line port)
                  result)))))
 
+(define (file-size file)
+  "Return the size of bytes of FILE, entering it if FILE is a directory."
+  (file-system-fold (const #t)
+                    (lambda (file stat result)    ;leaf
+                      (+ (stat:size stat) result))
+                    (lambda (directory stat result) ;down
+                      (+ (stat:size stat) result))
+                    (lambda (directory stat result) ;up
+                      result)
+                    (lambda (file stat result)    ;skip
+                      result)
+                    (lambda (file stat errno result)
+                      (format (current-error-port)
+                              "file-size: ~a: ~a~%" file
+                              (strerror errno))
+                      result)
+                    0
+                    file
+                    lstat))
+
+(define (closure-size reference-graphs)
+  "Return an estimate of the size of the closure described by
+REFERENCE-GRAPHS, a list of reference-graph files."
+  (define (graph-from-file file)
+    (call-with-input-file file read-reference-graph))
+
+  (define items
+    (delete-duplicates (append-map graph-from-file reference-graphs)))
+
+  (reduce + 0 (map file-size items)))
+
 (define* (populate-store reference-graphs target)
   "Populate the store under directory TARGET with the items specified in
 REFERENCE-GRAPHS, a list of reference-graph files."
diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm
index 35675cc01..7e20b10da 100644
--- a/guix/scripts/system.scm
+++ b/guix/scripts/system.scm
@@ -854,7 +854,7 @@ Some ACTIONS support additional ARGS.\n"))
     (build-hook? . #t)
     (max-silent-time . 3600)
     (verbosity . 0)
-    (image-size . ,(* 900 (expt 2 20)))
+    (image-size . guess)
     (install-bootloader? . #t)))
 
 

Message #28 received at 27401 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Leo Famulari <leo <at> famulari.name>
Cc: 27401 <at> debbugs.gnu.org
Subject: Re: [bug#27401] Guessing the disk image size in 'guix system'
Date: Fri, 30 Jun 2017 00:30:21 +0200

ludo <at> gnu.org (Ludovic Courtès) skribis:

> We can do “guix size $(guix system build foo.scm)” and add 10% or so to
> that.
>
> Maybe something like this mostly-untested patch would work?

I’ve committed a variant of this patch as
a8ac4f081a9a679498ea42ccfe001f218bba3043.

I’ve tested it with several OS configs and it worked, except for
“lightweight-desktop.tmpl” where it failed with a confusing GRUB error
that might have been caused by ENOSPC.

Feedback and suggestion to improve the estimate welcome!

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.