GNU bug report logs -
#78210
[PATCH] guix: include store parent dirs in docker layer
Previous Next
To reply to this bug, email your comments to 78210 AT debbugs.gnu.org.
There is no need to reopen the bug first.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
guix-patches <at> gnu.org:
bug#78210; Package
guix-patches.
(Fri, 02 May 2025 11:46:03 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Ray Miller <ray <at> 1729.org.uk>:
New bug report received and forwarded. Copy sent to
guix-patches <at> gnu.org.
(Fri, 02 May 2025 11:46:03 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
This patch adds the /gnu and /gnu/store directories to the Docker layer
created by `guix pack -f docker ...` which enables the Docker image to be
used to create an AWS Lambda function. Without the patch, creating the AWS
Lambda function fails with this error:
"MissingParentDirectory: Parent directory does not exist for file:
gnu/store/zic27jikg36d6wjj4cz8hyriyfl3ygiz-info-dir/"
My first attempt to fix this was just to add the /gnu and /gnu/store
directories
to `directives` but the Docker image failed in AWS Lambda with the same
error. These
directories need to appear in the tarball for the layer *before* the
packages,
so the change to the order of the tar arguments is also needed.
* guix/scripts/pack.scm: add /gnu and /gnu/store directories to the
docker layer.
* guix/docker.scm: change order of arguments to tar so parent
directories are added before their contents.
Change-Id: I2b103c59981e828c965564ccc5d2415b00a7e52e
---
guix/docker.scm | 4 ++--
guix/scripts/pack.scm | 6 ++++--
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/guix/docker.scm b/guix/docker.scm
index 60ce13cbde..9911bb84bb 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -365,10 +365,10 @@ (define* (build-docker-image image paths prefix
(apply invoke "tar" "-cf" "../layer.tar"
`(,@transformation-options
,@(tar-base-options)
- ,@(if max-layers '() paths)
,@(scandir "."
(lambda (file)
- (not (member file '("." ".."))))))))
+ (not (member file '("." "..")))))
+ ,@(if max-layers '() paths))))
(delete-file-recursively "extra")))
;; It is possible for "/" to show up in the archive, especially
when
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 7ab2c0d447..5cb9cd0b48 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -580,9 +580,11 @@ (define* (docker-image name profile
(,source -> ,target))))))
(define directives
- ;; Create a /tmp directory, as some programs expect it, and
- ;; create SYMLINKS.
+ ;; Create /tmp, /gnu, and /gnu/store directories, as some
+ ;; programs expect them, and create SYMLINKS.
`((directory "/tmp" ,(getuid) ,(getgid) #o1777)
+ (directory "/gnu" ,(getuid) ,(getgid) #o755)
+ (directory "/gnu/store" ,(getuid) ,(getgid) #o755)
,@(append-map symlink->directives '#$symlinks)))
(define (form-entry-point prefix entry-point
entry-point-argument)
base-commit: 4fe4cf9fdd959126d3c53c3df4504d851e7b736a
--
2.47.1
[OpenPGP_0x043F786C4CD681B8.asc (application/pgp-keys, attachment)]
[OpenPGP_signature.asc (application/pgp-signature, attachment)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#78210; Package
guix-patches.
(Sun, 18 May 2025 21:04:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 78210 <at> debbugs.gnu.org (full text, mbox):
Hi Ray,
Ray Miller <ray <at> 1729.org.uk> writes:
> This patch adds the /gnu and /gnu/store directories to the Docker layer
> created by `guix pack -f docker ...` which enables the Docker image to be
> used to create an AWS Lambda function. Without the patch, creating the AWS
> Lambda function fails with this error:
>
> "MissingParentDirectory: Parent directory does not exist for file:
> gnu/store/zic27jikg36d6wjj4cz8hyriyfl3ygiz-info-dir/"
>
> My first attempt to fix this was just to add the /gnu and /gnu/store
> directories
> to `directives` but the Docker image failed in AWS Lambda with the same
> error. These
> directories need to appear in the tarball for the layer *before* the
> packages,
> so the change to the order of the tar arguments is also needed.
>
> * guix/scripts/pack.scm: add /gnu and /gnu/store directories to the
> docker layer.
> * guix/docker.scm: change order of arguments to tar so parent
> directories are added before their contents.
>
> Change-Id: I2b103c59981e828c965564ccc5d2415b00a7e52e
Neat!
Could you include in the commit log a line like:
Fixes <https://issues.guix.gnu.org/XYZ>.
… so we can keep track of where this was reported and discussed?
> +++ b/guix/scripts/pack.scm
> @@ -580,9 +580,11 @@ (define* (docker-image name profile
> (,source -> ,target))))))
>
> (define directives
> - ;; Create a /tmp directory, as some programs expect it, and
> - ;; create SYMLINKS.
> + ;; Create /tmp, /gnu, and /gnu/store directories, as some
> + ;; programs expect them, and create SYMLINKS.
> `((directory "/tmp" ,(getuid) ,(getgid) #o1777)
> + (directory "/gnu" ,(getuid) ,(getgid) #o755)
> + (directory "/gnu/store" ,(getuid) ,(getgid) #o755)
It’s a bit trickier, because “/gnu/store” is not hardcoded.
Instead, you need to recurse over the components of (%store-prefix), so
something like:
`((directory "/tmp" …)
#$@(map (lambda (component)
#~(directory #$component …))
(string-tokenize (%store-prefix)
(char-set-complement (char-set #\/)))))
But perhaps this should actually be done in (guix docker) so that ‘guix
system image -t docker’ also benefits from it?
Thanks,
Ludo’.
Information forwarded
to
guix-patches <at> gnu.org:
bug#78210; Package
guix-patches.
(Sat, 24 May 2025 17:28:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 78210 <at> debbugs.gnu.org (full text, mbox):
Hi Ludo,
Thank you for the feedback and suggestions.
On 18/05/2025 21:34, Ludovic Courtès wrote:
> Neat!
>
> Could you include in the commit log a line like:
>
> Fixes <https://issues.guix.gnu.org/XYZ>.
>
> … so we can keep track of where this was reported and discussed?
>
Done.
>> +++ b/guix/scripts/pack.scm
>> @@ -580,9 +580,11 @@ (define* (docker-image name profile
>> (,source -> ,target))))))
>>
>> (define directives
>> - ;; Create a /tmp directory, as some programs expect it, and
>> - ;; create SYMLINKS.
>> + ;; Create /tmp, /gnu, and /gnu/store directories, as some
>> + ;; programs expect them, and create SYMLINKS.
>> `((directory "/tmp" ,(getuid) ,(getgid) #o1777)
>> + (directory "/gnu" ,(getuid) ,(getgid) #o755)
>> + (directory "/gnu/store" ,(getuid) ,(getgid) #o755)
> It’s a bit trickier, because “/gnu/store” is not hardcoded.
Ah, OK. Good catch!
> Instead, you need to recurse over the components of (%store-prefix), so
> something like:
>
> `((directory "/tmp" …)
> #$@(map (lambda (component)
> #~(directory #$component …))
> (string-tokenize (%store-prefix)
> (char-set-complement (char-set #\/)))))
It turns out to be a bit simpler because these directives are processed by
evaluate-populate-directive which calls mkdir-p. (I discovered this after
implementing a recursive solution then thinking "there must be a better
way" and reading some source code...)
> But perhaps this should actually be done in (guix docker) so that ‘guix
> system image -t docker’ also benefits from it?
I tested this by running 'guix systemimage -t docker' and inspecting the
layer it generates. It turns out not to suffer from the same problem: the
store directory is present in the layer.
Updated patch to follow.
Ray.
Information forwarded
to
guix-patches <at> gnu.org:
bug#78210; Package
guix-patches.
(Sat, 24 May 2025 17:28:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 78210 <at> debbugs.gnu.org (full text, mbox):
* guix/scripts/pack.scm: add store directory to the docker
layer.
* guix/docker.scm: change order of arguments to tar so parent
directories are added before their contents.
Fixes <https://issues.guix.gnu.org/78210>
Change-Id: I2b103c59981e828c965564ccc5d2415b00a7e52e
---
guix/docker.scm | 4 ++--
guix/scripts/pack.scm | 8 +++++---
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/guix/docker.scm b/guix/docker.scm
index 60ce13cbde..9911bb84bb 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -365,10 +365,10 @@ (define* (build-docker-image image paths prefix
(apply invoke "tar" "-cf" "../layer.tar"
`(,@transformation-options
,@(tar-base-options)
- ,@(if max-layers '() paths)
,@(scandir "."
(lambda (file)
- (not (member file '("." ".."))))))))
+ (not (member file '("." "..")))))
+ ,@(if max-layers '() paths))))
(delete-file-recursively "extra")))
;; It is possible for "/" to show up in the archive, especially
when
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 7ab2c0d447..9c14b7bd81 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -551,7 +551,8 @@ (define* (docker-image name profile
(guix build store-copy)
(guix build utils) ;for
%xz-parallel-args
(guix profiles)
- (guix search-paths))
+ (guix search-paths)
+ (guix store))
#:select? not-config?))
#~(begin
(use-modules (guix docker) (guix build store-copy)
@@ -580,9 +581,10 @@ (define* (docker-image name profile
(,source -> ,target))))))
(define directives
- ;; Create a /tmp directory, as some programs expect it, and
- ;; create SYMLINKS.
+ ;; Create /tmp and %store-prefix directories, as some
+ ;; programs expect them, and create SYMLINKS.
`((directory "/tmp" ,(getuid) ,(getgid) #o1777)
+ (directory #$(%store-prefix) ,(getuid) ,(getgid) #o755)
,@(append-map symlink->directives '#$symlinks)))
(define (form-entry-point prefix entry-point
entry-point-argument)
base-commit: 096dedd0bb13523002c814b001429c2f65b6f10d
--
2.49.0
Reply sent
to
Ludovic Courtès <ludo <at> gnu.org>:
You have taken responsibility.
(Wed, 04 Jun 2025 08:52:04 GMT)
Full text and
rfc822 format available.
Notification sent
to
Ray Miller <ray <at> 1729.org.uk>:
bug acknowledged by developer.
(Wed, 04 Jun 2025 08:52:04 GMT)
Full text and
rfc822 format available.
Message #19 received at 78210-done <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi Ray,
Ray Miller <ray <at> 1729.org.uk> writes:
> * guix/scripts/pack.scm: add store directory to the docker
> layer.
> * guix/docker.scm: change order of arguments to tar so parent
> directories are added before their contents.
>
> Fixes <https://issues.guix.gnu.org/78210>
>
> Change-Id: I2b103c59981e828c965564ccc5d2415b00a7e52e
I tweaked the commit log and applied it with the changes below.
Thanks!
Ludo’.
[Message part 2 (text/x-patch, inline)]
diff --git a/guix/docker.scm b/guix/docker.scm
index 9911bb84bb5..bb6d896a635 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -365,6 +365,7 @@ (define* (build-docker-image image paths prefix
(apply invoke "tar" "-cf" "../layer.tar"
`(,@transformation-options
,@(tar-base-options)
+ ;; Add parent directories before their contents.
,@(scandir "."
(lambda (file)
(not (member file '("." "..")))))
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index e69316089d8..857cb46af37 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -580,7 +580,7 @@ (define* (docker-image name profile
(,source -> ,target))))))
(define directives
- ;; Create /tmp and %store-prefix directories, as some
+ ;; Create the /tmp and %store-prefix directories, as some
;; programs expect them, and create SYMLINKS.
`((directory "/tmp" ,(getuid) ,(getgid) #o1777)
(directory #$(%store-prefix) ,(getuid) ,(getgid) #o755)
This bug report was last modified 14 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.