GNU bug report logs -
#75426
[PATCH] docker: Build tarballs reproducibly.
Previous Next
Reported by: Ludovic Courtès <ludo <at> gnu.org>
Date: Tue, 7 Jan 2025 22:57:01 UTC
Severity: normal
Tags: patch
Done: Ludovic Courtès <ludo <at> gnu.org>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 75426 in the body.
You can then email your comments to 75426 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
guix <at> cbaines.net, dev <at> jpoiret.xyz, ludo <at> gnu.org, othacehe <at> gnu.org, zimon.toutoune <at> gmail.com, me <at> tobias.gr, guix-patches <at> gnu.org:
bug#75426; Package
guix-patches.
(Tue, 07 Jan 2025 22:57:01 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Ludovic Courtès <ludo <at> gnu.org>:
New bug report received and forwarded. Copy sent to
guix <at> cbaines.net, dev <at> jpoiret.xyz, ludo <at> gnu.org, othacehe <at> gnu.org, zimon.toutoune <at> gmail.com, me <at> tobias.gr, guix-patches <at> gnu.org.
(Tue, 07 Jan 2025 22:57:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
Fixes <https://issues.guix.gnu.org/75090>.
* guix/docker.scm (tar): New procedure.
(create-empty-tar, build-docker-image): Use it instead of calling
‘invoke’ directly.
Reported-by: Simon Josefsson <simon <at> josefsson.org>
Change-Id: Ia899c43ed6a3809ff845de0953e3d38cccf24609
---
guix/docker.scm | 25 ++++++++++++++++---------
1 file changed, 16 insertions(+), 9 deletions(-)
diff --git a/guix/docker.scm b/guix/docker.scm
index b33c5824dd..d9764f61fb 100644
--- a/guix/docker.scm
+++ b/guix/docker.scm
@@ -1,6 +1,6 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2017 Ricardo Wurmus <rekado <at> elephly.net>
-;;; Copyright © 2017, 2018, 2019, 2021 Ludovic Courtès <ludo <at> gnu.org>
+;;; Copyright © 2017-2019, 2021, 2025 Ludovic Courtès <ludo <at> gnu.org>
;;; Copyright © 2018 Chris Marusich <cmmarusich <at> gmail.com>
;;; Copyright © 2021 Maxim Cournoyer <maxim.cournoyer <at> gmail.com>
;;; Copyright © 2023 Oleg Pykhalov <go.wigust <at> gmail.com>
@@ -170,8 +170,15 @@ (define (size-sorted-store-items items max-layers)
(1- items-length)))))
(list head tail)))
+(define (tar . arguments)
+ "Invoke 'tar' with the given ARGUMENTS together with options to build
+tarballs in a reproducible fashion."
+ (apply invoke "tar" "--mtime=@1"
+ "--owner=0" "--group=0" "--numeric-owner"
+ "--sort=name" "--mode=go+u,go-w" arguments))
+
(define (create-empty-tar file)
- (invoke "tar" "-cf" file "--files-from" "/dev/null"))
+ (tar "-cf" file "--files-from" "/dev/null"))
(define* (build-docker-image image paths prefix
#:key
@@ -255,7 +262,7 @@ (define* (build-docker-image image paths prefix
(file-name (string-append file-hash "/layer.tar")))
(mkdir file-hash)
(rename-file "layer.tar" file-name)
- (invoke "tar" "-rf" "image.tar" file-name)
+ (tar "-rf" "image.tar" file-name)
(delete-file file-name)
file-hash))
(define layers-hashes
@@ -268,20 +275,20 @@ (define* (build-docker-image image paths prefix
(let* ((head-layers
(map
(lambda (file)
- (invoke "tar" "cf" "layer.tar" file)
+ (tar "cf" "layer.tar" file)
(seal-layer))
head))
(tail-layer
(begin
(create-empty-tar "layer.tar")
(for-each (lambda (file)
- (invoke "tar" "-rf" "layer.tar" file))
+ (tar "-rf" "layer.tar" file))
tail)
(let* ((file-hash (layer-diff-id "layer.tar"))
(file-name (string-append file-hash "/layer.tar")))
(mkdir file-hash)
(rename-file "layer.tar" file-name)
- (invoke "tar" "-rf" "image.tar" file-name)
+ (tar "-rf" "image.tar" file-name)
(delete-file file-name)
file-hash)))
(customization-layer
@@ -290,7 +297,7 @@ (define* (build-docker-image image paths prefix
(file-name (string-append file-hash "/layer.tar")))
(mkdir file-hash)
(rename-file file-id file-name)
- (invoke "tar" "-rf" "image.tar" file-name)
+ (tar "-rf" "image.tar" file-name)
file-hash))
(all-layers
(append head-layers (list tail-layer customization-layer))))
@@ -300,7 +307,7 @@ (define* (build-docker-image image paths prefix
(map (cut string-append <> "/layer.tar")
all-layers)
repository))))
- (invoke "tar" "-rf" "image.tar" "manifest.json")
+ (tar "-rf" "image.tar" "manifest.json")
all-layers))))
(let* ((directory "/tmp/docker-image") ;temporary working directory
(id (docker-id prefix))
@@ -388,7 +395,7 @@ (define* (build-docker-image image paths prefix
#:entry-point entry-point))))
(if max-layers
(begin
- (invoke "tar" "-rf" "image.tar" "config.json")
+ (tar "-rf" "image.tar" "config.json")
(if compressor
(begin
(apply invoke `(,@compressor "image.tar"))
base-commit: eeb019eb595bbb29f83389deb2fc823ed6402dd5
--
2.47.1
Reply sent
to
Ludovic Courtès <ludo <at> gnu.org>:
You have taken responsibility.
(Fri, 24 Jan 2025 23:08:03 GMT)
Full text and
rfc822 format available.
Notification sent
to
Ludovic Courtès <ludo <at> gnu.org>:
bug acknowledged by developer.
(Fri, 24 Jan 2025 23:08:03 GMT)
Full text and
rfc822 format available.
Message #10 received at 75426-done <at> debbugs.gnu.org (full text, mbox):
Ludovic Courtès <ludo <at> gnu.org> skribis:
> Fixes <https://issues.guix.gnu.org/75090>.
>
> * guix/docker.scm (tar): New procedure.
> (create-empty-tar, build-docker-image): Use it instead of calling
> ‘invoke’ directly.
>
> Reported-by: Simon Josefsson <simon <at> josefsson.org>
> Change-Id: Ia899c43ed6a3809ff845de0953e3d38cccf24609
Pushed as 646202bf73f90de4f9b7cc66248b8f8e6e381014.
Ludo’.
Information forwarded
to
guix-patches <at> gnu.org:
bug#75426; Package
guix-patches.
(Wed, 29 Jan 2025 19:03:02 GMT)
Full text and
rfc822 format available.
Message #13 received at 75426 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi! I suspect something went wrong with this patch, now 'guix pack'
fail and give the error below. Maybe the 'cf' has to come first?
https://gitlab.com/debdistutils/guix/container/-/jobs/8988707317
/Simon
tar: You must specify one of the '-Acdtrux', '--delete' or '--test-
label' options
Try 'tar --help' or 'tar --usage' for more information.
Backtrace:
7 (primitive-load "/gnu/store/hyx3flr5r251fc3x0z0l6r36159?")
In guix/docker.scm:
387:6 6 (build-docker-image "/gnu/store/vwia06dwxrsmf152spa6n2?"
?)
In ice-9/ports.scm:
433:17 5 (call-with-output-file _ _ #:binary _ #:encoding _)
476:4 4 (_ _)
In guix/docker.scm:
277:15 3 (_)
In srfi/srfi-1.scm:
586:17 2 (map1 ("/gnu/store/dn7ya77a3za7jqrihdql0hcxc0i32mmf-?" ?))
In guix/docker.scm:
279:18 1 (_ "/gnu/store/dn7ya77a3za7jqrihdql0hcxc0i32mmf-guix-1.?")
In guix/build/utils.scm:
822:6 0 (invoke "tar" "--mtime=@1" "--owner=0" "--group=0" "--?"
?)
guix/build/utils.scm:822:6: In procedure invoke:
ERROR:
1. &invoke-error:
program: "tar"
arguments: ("--mtime=@1" "--owner=0" "--group=0" "--numeric-
owner" "--sort=name" "--mode=go+u,go-w" "cf" "layer.tar"
"/gnu/store/dn7ya77a3za7jqrihdql0hcxc0i32mmf-guix-1.4.0-31.121e96d")
exit-status: 2
term-signal: #f
stop-signal: #f
lör 2025-01-25 klockan 00:07 +0100 skrev Ludovic Courtès:
> Ludovic Courtès <ludo <at> gnu.org> skribis:
>
> > Fixes <https://issues.guix.gnu.org/75090>.
> >
> > * guix/docker.scm (tar): New procedure.
> > (create-empty-tar, build-docker-image): Use it instead of calling
> > ‘invoke’ directly.
> >
> > Reported-by: Simon Josefsson <simon <at> josefsson.org>
> > Change-Id: Ia899c43ed6a3809ff845de0953e3d38cccf24609
>
> Pushed as 646202bf73f90de4f9b7cc66248b8f8e6e381014.
>
> Ludo’.
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#75426; Package
guix-patches.
(Fri, 31 Jan 2025 16:14:01 GMT)
Full text and
rfc822 format available.
Message #16 received at 75426 <at> debbugs.gnu.org (full text, mbox):
Hi,
Simon Josefsson <simon <at> josefsson.org> skribis:
> Hi! I suspect something went wrong with this patch, now 'guix pack'
> fail and give the error below. Maybe the 'cf' has to come first?
>
> https://gitlab.com/debdistutils/guix/container/-/jobs/8988707317
> tar: You must specify one of the '-Acdtrux', '--delete' or '--test-label' options
> Try 'tar --help' or 'tar --usage' for more information.
Hmm I don’t see this message in the GitLab log above, and I cannot
reproduce the problem:
--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix pack -f docker sed
/gnu/store/h2hdp469v3014b82qsvz5fkw00sfxdgh-sed-docker-pack.tar.gz
$ git log |head -3
commit 97fb1887ad10000c067168176c504274e29e4430
Author: Ashish SHUKLA <ashish.is <at> lostca.se>
Date: Mon Jan 20 21:46:10 2025 +0000
--8<---------------cut here---------------end--------------->8---
Could you try to come up with a command and commit that reproduces it?
Thanks,
Ludo’.
Information forwarded
to
guix-patches <at> gnu.org:
bug#75426; Package
guix-patches.
(Fri, 31 Jan 2025 22:45:02 GMT)
Full text and
rfc822 format available.
Message #19 received at 75426 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
fre 2025-01-31 klockan 17:12 +0100 skrev Ludovic Courtès:
> Hi,
>
> Simon Josefsson <simon <at> josefsson.org> skribis:
>
> > Hi! I suspect something went wrong with this patch, now 'guix
> > pack'
> > fail and give the error below. Maybe the 'cf' has to come first?
> >
> > https://gitlab.com/debdistutils/guix/container/-/jobs/8988707317
>
> > tar: You must specify one of the '-Acdtrux', '--delete' or '--test-
> > label' options
> > Try 'tar --help' or 'tar --usage' for more information.
>
> Hmm I don’t see this message in the GitLab log above, and I cannot
> reproduce the problem:
>
> --8<---------------cut here---------------start------------->8---
> $ ./pre-inst-env guix pack -f docker sed
> /gnu/store/h2hdp469v3014b82qsvz5fkw00sfxdgh-sed-docker-pack.tar.gz
> $ git log |head -3
> commit 97fb1887ad10000c067168176c504274e29e4430
> Author: Ashish SHUKLA <ashish.is <at> lostca.se>
> Date: Mon Jan 20 21:46:10 2025 +0000
> --8<---------------cut here---------------end--------------->8---
>
> Could you try to come up with a command and commit that reproduces
> it?
Running the commands in the log like below on my local trisquel machine
with guix triggers the same problem. Any ideas?
(FWIW, the message is hidden inside the log, but the error and filename
of the log is printed in the gitlab log above, same as below; it is
possible to click on 'Job artifacts' to find the log files.)
/Simon
jas <at> kaka:~$ guix describe
Generation 26 31 jan 2025 23:37:27 (aktuell)
guix d48da2d
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: d48da2d21610f9cf5f76cd846703b12beedb1fd5
jas <at> kaka:~$ guix pack coreutils --save-provenance -S /bin=bin -S
/share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2
net-base
The following derivation will be built:
/gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-base-
docker-pack.tar.gz.drv
bygger /gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-base-
docker-pack.tar.gz.drv…
|builder for `/gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-
net-base-docker-pack.tar.gz.drv' failed with exit code 1
bygge av /gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-
base-docker-pack.tar.gz.drv misslyckades
Granska bygglogg vid
”/var/log/guix/drvs/qk/z6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-
base-docker-pack.tar.gz.drv.bz2”.
guix pack: fel: build of `/gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-
coreutils-net-base-docker-pack.tar.gz.drv' failed
jas <at> kaka:~$ bzcat /var/log/guix/drvs/qk/z6wc1qq23ah1xk387givjvk9qlgwcn-
coreutils-net-base-docker-pack.tar.gz.drv.bz2 | tail -30
tar: You must specify one of the '-Acdtrux', '--delete' or '--test-
label' options
Try 'tar --help' or 'tar --usage' for more information.
Backtrace:
7 (primitive-load "/gnu/store/5z5322v39y5mwninj36m877zgsx?")
In guix/docker.scm:
387:6 6 (build-docker-image "/gnu/store/8ac4lljjiqp3a7zydh6l9v?"
?)
In ice-9/ports.scm:
433:17 5 (call-with-output-file _ _ #:binary _ #:encoding _)
476:4 4 (_ _)
In guix/docker.scm:
277:15 3 (_)
In srfi/srfi-1.scm:
586:17 2 (map1 ("/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-?" ?))
In guix/docker.scm:
279:18 1 (_ "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2?")
In guix/build/utils.scm:
822:6 0 (invoke "tar" "--mtime=@1" "--owner=0" "--group=0" "--?"
?)
guix/build/utils.scm:822:6: In procedure invoke:
ERROR:
1. &invoke-error:
program: "tar"
arguments: ("--mtime=@1" "--owner=0" "--group=0" "--numeric-
owner" "--sort=name" "--mode=go+u,go-w" "cf" "layer.tar"
"/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39")
exit-status: 2
term-signal: #f
stop-signal: #f
jas <at> kaka:~$
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
guix-patches <at> gnu.org:
bug#75426; Package
guix-patches.
(Tue, 04 Feb 2025 16:24:02 GMT)
Full text and
rfc822 format available.
Message #22 received at 75426 <at> debbugs.gnu.org (full text, mbox):
Hello Simon,
Simon Josefsson <simon <at> josefsson.org> skribis:
> jas <at> kaka:~$ guix pack coreutils --save-provenance -S /bin=bin -S
> /share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2
> net-base
[...]
> guix/build/utils.scm:822:6: In procedure invoke:
> ERROR:
> 1. &invoke-error:
> program: "tar"
> arguments: ("--mtime=@1" "--owner=0" "--group=0" "--numeric-
> owner" "--sort=name" "--mode=go+u,go-w" "cf" "layer.tar"
> "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39")
Fixed in 285a1cb449f60798dc83f7f1016700b4ab2374a8.
Ludo’.
Information forwarded
to
guix-patches <at> gnu.org:
bug#75426; Package
guix-patches.
(Tue, 04 Feb 2025 21:57:02 GMT)
Full text and
rfc822 format available.
Message #25 received at 75426 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Ludovic Courtès <ludo <at> gnu.org> writes:
> Fixed in 285a1cb449f60798dc83f7f1016700b4ab2374a8.
Hi! Wonderful, confirmed working. These two pipelines produce
bit-by-bit identical docker pack images for stage1 of my guix gitlab
container images now:
https://gitlab.com/debdistutils/guix/container/-/jobs/9042454862
https://gitlab.com/debdistutils/guix/container/-/jobs/9042230873
58b98623ac2b75de521c8db6a904e60f4faad17dc08c2ccc6eab445a9f77cfdd
/Simon
[signature.asc (application/pgp-signature, inline)]
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Wed, 05 Mar 2025 12:24:08 GMT)
Full text and
rfc822 format available.
This bug report was last modified 159 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.