GNU bug report logs -
#75090
Make 'guix pack -f docker' tarballs reproducible?
Previous Next
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 75090 in the body.
You can then email your comments to 75090 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
bug-guix <at> gnu.org:
bug#75090; Package
guix.
(Wed, 25 Dec 2024 17:13:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Simon Josefsson <simon <at> josefsson.org>:
New bug report received and forwarded. Copy sent to
bug-guix <at> gnu.org.
(Wed, 25 Dec 2024 17:13:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi
I am creating docker archives using:
guix pack guix bash-minimal coreutils-minimal net-base --save-provenance -S /bin=bin -S /share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2
To my surprise the output was not reproducible between re-runs.
The reason is because of the timestamp and ownership information in the
outer tarball. The internals are identical and reproducible. See
diffoscope output below.
I tried to work around it by wrapping either the 'guix pack' or
'guix-daemon' commands with this environment variable, which I suggest
for inspiration as additional parameters to tar:
TAR_OPTIONS="--owner=0 --group=0 --numeric-owner --sort=name --mode=go+u,go-w --mtime=@0"
I would prefer 'guix pack' produced reproducible archives by default.
Alternatively, provide a way to allow me as user to specify some
parameters for 'guix pack' to make that happen.
/Simon
jas <at> kaka:~/src/guix-container$ diffoscope stage1-docker-pack.tar.gz-1 stage1-docker-pack.tar.gz-2
--- stage1-docker-pack.tar.gz-1
+++ stage1-docker-pack.tar.gz-2
│ --- stage1-docker-pack.tar.gz-1-content
├── +++ stage1-docker-pack.tar.gz-2-content
│ ├── file list
│ │ @@ -1,10 +1,10 @@
│ │ --rw-r--r-- 0 nixbld (997) nixbld (999) 421457920 2024-12-25 16:31:15.000000 sha256:e69812bf459ea0fba42d1d6fd518410a4e588ddd4e4c007ddb4dd48c9c04293a/layer.tar
│ │ --rw-r--r-- 0 nixbld (997) nixbld (999) 56330240 2024-12-25 16:31:16.000000 sha256:45e67bf9fcad2f255f20dc614224b9e4260da1b63f2a361c2479e1ed64a9210a/layer.tar
│ │ --rw-r--r-- 0 nixbld (997) nixbld (999) 37632000 2024-12-25 16:31:16.000000 sha256:a8d1b46be57ba5a41051dedcf2d8d7bb2f13a9d58078729a962d04f5178274ba/layer.tar
│ │ --rw-r--r-- 0 nixbld (997) nixbld (999) 41523200 2024-12-25 16:31:16.000000 sha256:0756f500c123ba4f34cda21e5232932799fd36c15243f7fcb1ef38ff6ec7533d/layer.tar
│ │ --rw-r--r-- 0 nixbld (997) nixbld (999) 37806080 2024-12-25 16:31:17.000000 sha256:bf18d11d88b81af3f6fb49b7d4b092d479b7967ac8dc4980cc381170997c6ccf/layer.tar
│ │ --rw-r--r-- 0 nixbld (997) nixbld (999) 17582080 2024-12-25 16:31:17.000000 sha256:9263a9904763737f9e8bdf08ca52cede34c2fa9e99abe7f9ef273111752cb2ca/layer.tar
│ │ --rw-r--r-- 0 nixbld (997) nixbld (999) 147763200 2024-12-25 16:31:20.000000 sha256:3d9a70bc298db46d9fdd95badacd3ec5586f3965110bb85b748be6bcfc57b171/layer.tar
│ │ --rw-r--r-- 0 nixbld (997) nixbld (999) 10240 2024-12-25 16:31:14.000000 sha256:3fb6718bc797283e8283fe1b843596ace2e62db47d5b38d228a64a6bbb7c3564/layer.tar
│ │ --rw-r--r-- 0 nixbld (997) nixbld (999) 736 2024-12-25 16:31:21.000000 manifest.json
│ │ --rw-r--r-- 0 nixbld (997) nixbld (999) 842 2024-12-25 16:31:21.000000 config.json
│ │ +-rw-r--r-- 0 nixbld (997) nixbld (999) 421457920 2024-12-25 16:41:20.000000 sha256:e69812bf459ea0fba42d1d6fd518410a4e588ddd4e4c007ddb4dd48c9c04293a/layer.tar
│ │ +-rw-r--r-- 0 nixbld (997) nixbld (999) 56330240 2024-12-25 16:41:21.000000 sha256:45e67bf9fcad2f255f20dc614224b9e4260da1b63f2a361c2479e1ed64a9210a/layer.tar
│ │ +-rw-r--r-- 0 nixbld (997) nixbld (999) 37632000 2024-12-25 16:41:22.000000 sha256:a8d1b46be57ba5a41051dedcf2d8d7bb2f13a9d58078729a962d04f5178274ba/layer.tar
│ │ +-rw-r--r-- 0 nixbld (997) nixbld (999) 41523200 2024-12-25 16:41:22.000000 sha256:0756f500c123ba4f34cda21e5232932799fd36c15243f7fcb1ef38ff6ec7533d/layer.tar
│ │ +-rw-r--r-- 0 nixbld (997) nixbld (999) 37806080 2024-12-25 16:41:22.000000 sha256:bf18d11d88b81af3f6fb49b7d4b092d479b7967ac8dc4980cc381170997c6ccf/layer.tar
│ │ +-rw-r--r-- 0 nixbld (997) nixbld (999) 17582080 2024-12-25 16:41:23.000000 sha256:9263a9904763737f9e8bdf08ca52cede34c2fa9e99abe7f9ef273111752cb2ca/layer.tar
│ │ +-rw-r--r-- 0 nixbld (997) nixbld (999) 147763200 2024-12-25 16:41:25.000000 sha256:3d9a70bc298db46d9fdd95badacd3ec5586f3965110bb85b748be6bcfc57b171/layer.tar
│ │ +-rw-r--r-- 0 nixbld (997) nixbld (999) 10240 2024-12-25 16:41:19.000000 sha256:3fb6718bc797283e8283fe1b843596ace2e62db47d5b38d228a64a6bbb7c3564/layer.tar
│ │ +-rw-r--r-- 0 nixbld (997) nixbld (999) 736 2024-12-25 16:41:26.000000 manifest.json
│ │ +-rw-r--r-- 0 nixbld (997) nixbld (999) 842 2024-12-25 16:41:26.000000 config.json
jas <at> kaka:~/src/guix-container$
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org:
bug#75090; Package
guix.
(Tue, 07 Jan 2025 22:59:02 GMT)
Full text and
rfc822 format available.
Message #8 received at 75090 <at> debbugs.gnu.org (full text, mbox):
Hi Simon,
Simon Josefsson <simon <at> josefsson.org> skribis:
> I am creating docker archives using:
>
> guix pack guix bash-minimal coreutils-minimal net-base --save-provenance -S /bin=bin -S /share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2
>
> To my surprise the output was not reproducible between re-runs.
>
> The reason is because of the timestamp and ownership information in the
> outer tarball. The internals are identical and reproducible. See
> diffoscope output below.
>
> I tried to work around it by wrapping either the 'guix pack' or
> 'guix-daemon' commands with this environment variable, which I suggest
> for inspiration as additional parameters to tar:
>
> TAR_OPTIONS="--owner=0 --group=0 --numeric-owner --sort=name --mode=go+u,go-w --mtime=@0"
>
> I would prefer 'guix pack' produced reproducible archives by default.
Indeed. I sent a fix based on your suggestion:
<https://issues.guix.gnu.org/75426>.
Thanks,
Ludo’.
Reply sent
to
Ludovic Courtès <ludo <at> gnu.org>:
You have taken responsibility.
(Fri, 24 Jan 2025 23:08:02 GMT)
Full text and
rfc822 format available.
Notification sent
to
Simon Josefsson <simon <at> josefsson.org>:
bug acknowledged by developer.
(Fri, 24 Jan 2025 23:08:02 GMT)
Full text and
rfc822 format available.
Message #13 received at 75090-done <at> debbugs.gnu.org (full text, mbox):
Ludovic Courtès <ludo <at> gnu.org> skribis:
> Fixes <https://issues.guix.gnu.org/75090>.
>
> * guix/docker.scm (tar): New procedure.
> (create-empty-tar, build-docker-image): Use it instead of calling
> ‘invoke’ directly.
>
> Reported-by: Simon Josefsson <simon <at> josefsson.org>
> Change-Id: Ia899c43ed6a3809ff845de0953e3d38cccf24609
Pushed as 646202bf73f90de4f9b7cc66248b8f8e6e381014.
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org:
bug#75090; Package
guix.
(Wed, 29 Jan 2025 19:03:02 GMT)
Full text and
rfc822 format available.
Message #16 received at 75090 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi! I suspect something went wrong with this patch, now 'guix pack'
fail and give the error below. Maybe the 'cf' has to come first?
https://gitlab.com/debdistutils/guix/container/-/jobs/8988707317
/Simon
tar: You must specify one of the '-Acdtrux', '--delete' or '--test-
label' options
Try 'tar --help' or 'tar --usage' for more information.
Backtrace:
7 (primitive-load "/gnu/store/hyx3flr5r251fc3x0z0l6r36159?")
In guix/docker.scm:
387:6 6 (build-docker-image "/gnu/store/vwia06dwxrsmf152spa6n2?"
?)
In ice-9/ports.scm:
433:17 5 (call-with-output-file _ _ #:binary _ #:encoding _)
476:4 4 (_ _)
In guix/docker.scm:
277:15 3 (_)
In srfi/srfi-1.scm:
586:17 2 (map1 ("/gnu/store/dn7ya77a3za7jqrihdql0hcxc0i32mmf-?" ?))
In guix/docker.scm:
279:18 1 (_ "/gnu/store/dn7ya77a3za7jqrihdql0hcxc0i32mmf-guix-1.?")
In guix/build/utils.scm:
822:6 0 (invoke "tar" "--mtime=@1" "--owner=0" "--group=0" "--?"
?)
guix/build/utils.scm:822:6: In procedure invoke:
ERROR:
1. &invoke-error:
program: "tar"
arguments: ("--mtime=@1" "--owner=0" "--group=0" "--numeric-
owner" "--sort=name" "--mode=go+u,go-w" "cf" "layer.tar"
"/gnu/store/dn7ya77a3za7jqrihdql0hcxc0i32mmf-guix-1.4.0-31.121e96d")
exit-status: 2
term-signal: #f
stop-signal: #f
lör 2025-01-25 klockan 00:07 +0100 skrev Ludovic Courtès:
> Ludovic Courtès <ludo <at> gnu.org> skribis:
>
> > Fixes <https://issues.guix.gnu.org/75090>.
> >
> > * guix/docker.scm (tar): New procedure.
> > (create-empty-tar, build-docker-image): Use it instead of calling
> > ‘invoke’ directly.
> >
> > Reported-by: Simon Josefsson <simon <at> josefsson.org>
> > Change-Id: Ia899c43ed6a3809ff845de0953e3d38cccf24609
>
> Pushed as 646202bf73f90de4f9b7cc66248b8f8e6e381014.
>
> Ludo’.
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org:
bug#75090; Package
guix.
(Fri, 31 Jan 2025 16:13:02 GMT)
Full text and
rfc822 format available.
Message #19 received at 75090 <at> debbugs.gnu.org (full text, mbox):
Hi,
Simon Josefsson <simon <at> josefsson.org> skribis:
> Hi! I suspect something went wrong with this patch, now 'guix pack'
> fail and give the error below. Maybe the 'cf' has to come first?
>
> https://gitlab.com/debdistutils/guix/container/-/jobs/8988707317
> tar: You must specify one of the '-Acdtrux', '--delete' or '--test-label' options
> Try 'tar --help' or 'tar --usage' for more information.
Hmm I don’t see this message in the GitLab log above, and I cannot
reproduce the problem:
--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix pack -f docker sed
/gnu/store/h2hdp469v3014b82qsvz5fkw00sfxdgh-sed-docker-pack.tar.gz
$ git log |head -3
commit 97fb1887ad10000c067168176c504274e29e4430
Author: Ashish SHUKLA <ashish.is <at> lostca.se>
Date: Mon Jan 20 21:46:10 2025 +0000
--8<---------------cut here---------------end--------------->8---
Could you try to come up with a command and commit that reproduces it?
Thanks,
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org:
bug#75090; Package
guix.
(Fri, 31 Jan 2025 22:45:02 GMT)
Full text and
rfc822 format available.
Message #22 received at 75090 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
fre 2025-01-31 klockan 17:12 +0100 skrev Ludovic Courtès:
> Hi,
>
> Simon Josefsson <simon <at> josefsson.org> skribis:
>
> > Hi! I suspect something went wrong with this patch, now 'guix
> > pack'
> > fail and give the error below. Maybe the 'cf' has to come first?
> >
> > https://gitlab.com/debdistutils/guix/container/-/jobs/8988707317
>
> > tar: You must specify one of the '-Acdtrux', '--delete' or '--test-
> > label' options
> > Try 'tar --help' or 'tar --usage' for more information.
>
> Hmm I don’t see this message in the GitLab log above, and I cannot
> reproduce the problem:
>
> --8<---------------cut here---------------start------------->8---
> $ ./pre-inst-env guix pack -f docker sed
> /gnu/store/h2hdp469v3014b82qsvz5fkw00sfxdgh-sed-docker-pack.tar.gz
> $ git log |head -3
> commit 97fb1887ad10000c067168176c504274e29e4430
> Author: Ashish SHUKLA <ashish.is <at> lostca.se>
> Date: Mon Jan 20 21:46:10 2025 +0000
> --8<---------------cut here---------------end--------------->8---
>
> Could you try to come up with a command and commit that reproduces
> it?
Running the commands in the log like below on my local trisquel machine
with guix triggers the same problem. Any ideas?
(FWIW, the message is hidden inside the log, but the error and filename
of the log is printed in the gitlab log above, same as below; it is
possible to click on 'Job artifacts' to find the log files.)
/Simon
jas <at> kaka:~$ guix describe
Generation 26 31 jan 2025 23:37:27 (aktuell)
guix d48da2d
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: d48da2d21610f9cf5f76cd846703b12beedb1fd5
jas <at> kaka:~$ guix pack coreutils --save-provenance -S /bin=bin -S
/share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2
net-base
The following derivation will be built:
/gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-base-
docker-pack.tar.gz.drv
bygger /gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-base-
docker-pack.tar.gz.drv…
|builder for `/gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-
net-base-docker-pack.tar.gz.drv' failed with exit code 1
bygge av /gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-
base-docker-pack.tar.gz.drv misslyckades
Granska bygglogg vid
”/var/log/guix/drvs/qk/z6wc1qq23ah1xk387givjvk9qlgwcn-coreutils-net-
base-docker-pack.tar.gz.drv.bz2”.
guix pack: fel: build of `/gnu/store/qkz6wc1qq23ah1xk387givjvk9qlgwcn-
coreutils-net-base-docker-pack.tar.gz.drv' failed
jas <at> kaka:~$ bzcat /var/log/guix/drvs/qk/z6wc1qq23ah1xk387givjvk9qlgwcn-
coreutils-net-base-docker-pack.tar.gz.drv.bz2 | tail -30
tar: You must specify one of the '-Acdtrux', '--delete' or '--test-
label' options
Try 'tar --help' or 'tar --usage' for more information.
Backtrace:
7 (primitive-load "/gnu/store/5z5322v39y5mwninj36m877zgsx?")
In guix/docker.scm:
387:6 6 (build-docker-image "/gnu/store/8ac4lljjiqp3a7zydh6l9v?"
?)
In ice-9/ports.scm:
433:17 5 (call-with-output-file _ _ #:binary _ #:encoding _)
476:4 4 (_ _)
In guix/docker.scm:
277:15 3 (_)
In srfi/srfi-1.scm:
586:17 2 (map1 ("/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-?" ?))
In guix/docker.scm:
279:18 1 (_ "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2?")
In guix/build/utils.scm:
822:6 0 (invoke "tar" "--mtime=@1" "--owner=0" "--group=0" "--?"
?)
guix/build/utils.scm:822:6: In procedure invoke:
ERROR:
1. &invoke-error:
program: "tar"
arguments: ("--mtime=@1" "--owner=0" "--group=0" "--numeric-
owner" "--sort=name" "--mode=go+u,go-w" "cf" "layer.tar"
"/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39")
exit-status: 2
term-signal: #f
stop-signal: #f
jas <at> kaka:~$
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
bug-guix <at> gnu.org:
bug#75090; Package
guix.
(Tue, 04 Feb 2025 16:24:02 GMT)
Full text and
rfc822 format available.
Message #25 received at 75090 <at> debbugs.gnu.org (full text, mbox):
Hello Simon,
Simon Josefsson <simon <at> josefsson.org> skribis:
> jas <at> kaka:~$ guix pack coreutils --save-provenance -S /bin=bin -S
> /share=share -f docker --image-tag=guix --max-layers=8 --verbosity=2
> net-base
[...]
> guix/build/utils.scm:822:6: In procedure invoke:
> ERROR:
> 1. &invoke-error:
> program: "tar"
> arguments: ("--mtime=@1" "--owner=0" "--group=0" "--numeric-
> owner" "--sort=name" "--mode=go+u,go-w" "cf" "layer.tar"
> "/gnu/store/hw6g2kjayxnqi8rwpnmpraalxi0djkxc-glibc-2.39")
Fixed in 285a1cb449f60798dc83f7f1016700b4ab2374a8.
Ludo’.
Information forwarded
to
bug-guix <at> gnu.org:
bug#75090; Package
guix.
(Tue, 04 Feb 2025 21:57:02 GMT)
Full text and
rfc822 format available.
Message #28 received at 75090 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Ludovic Courtès <ludo <at> gnu.org> writes:
> Fixed in 285a1cb449f60798dc83f7f1016700b4ab2374a8.
Hi! Wonderful, confirmed working. These two pipelines produce
bit-by-bit identical docker pack images for stage1 of my guix gitlab
container images now:
https://gitlab.com/debdistutils/guix/container/-/jobs/9042454862
https://gitlab.com/debdistutils/guix/container/-/jobs/9042230873
58b98623ac2b75de521c8db6a904e60f4faad17dc08c2ccc6eab445a9f77cfdd
/Simon
[signature.asc (application/pgp-signature, inline)]
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org.
(Wed, 05 Mar 2025 12:24:18 GMT)
Full text and
rfc822 format available.
This bug report was last modified 101 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.