Package: guix;
Reported by: Denis 'GNUtoo' Carikli <GNUtoo <at> cyberdimension.org>
Date: Sun, 1 Aug 2021 00:22:01 UTC
Severity: normal
To reply to this bug, email your comments to 49801 AT debbugs.gnu.org.
Toggle the display of automated, internal messages from the tracker.
View this report as an mbox folder, status mbox, maintainer mbox
bug-guix <at> gnu.org:bug#49801; Package guix.
(Sun, 01 Aug 2021 00:22:02 GMT) Full text and rfc822 format available.Denis 'GNUtoo' Carikli <GNUtoo <at> cyberdimension.org>:bug-guix <at> gnu.org.
(Sun, 01 Aug 2021 00:22:02 GMT) Full text and rfc822 format available.Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
From: Denis 'GNUtoo' Carikli <GNUtoo <at> cyberdimension.org> To: bug-guix <at> gnu.org Subject: Guix time machine provenance/manifest reproducibility issue? Date: Sun, 1 Aug 2021 02:21:42 +0200
[Message part 1 (text/plain, inline)]
Hi,
I've been trying to reproduce a tarball
(sz1lkq3ryr5iv6amy6f3d2pziks27g28-tarball-pack.tar.xz) that I generated
with guix pack on guix master the 28 January 2021.
To build it, in January, I used the following commands:
> guix pull
> guix pack \
> --compression=xz \
> --save-provenance \
> -RR \
> --symlink=/usr/local/bin/repo=bin/repo \
> --symlink=/usr/local/bin/repo-env.sh=etc/profile \
> git-repo le-certs nss-certs git python-certifi
That tarball is publicly available in the Replicant ftp server[1].
The extracted provenance file (named manifest) has the following
content:
> ;; This file was automatically generated and is for internal use only.
> ;; It cannot be passed to the '--manifest' option.
>
> (manifest
> (version 3)
> (packages
> (("git-repo"
> "2.4.1"
> "out"
> "/gnu/store/d4frkcdq15a7gyfjdggwg44ryi46fa2d-git-repo-2.4.1R"
> (propagated-inputs ())
> (search-paths ())
> (properties
> (provenance
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> (branch "master")
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> (introduction
> (channel-introduction
> (version 0)
> (commit
> "9edb3f66fd807b096b48283debdcddccfea34bad")
> (signer
> "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A
> 54FA"))))))) ("le-certs"
> "0"
> "out"
> "/gnu/store/x004p4hnyy0ickg2f5msvrpszhy9hzpl-le-certs-0R"
> (propagated-inputs ())
> (search-paths ())
> (properties
> (provenance
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> (branch "master")
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> (introduction
> (channel-introduction
> (version 0)
> (commit
> "9edb3f66fd807b096b48283debdcddccfea34bad")
> (signer
> "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A
> 54FA"))))))) ("nss-certs"
> "3.57"
> "out"
> "/gnu/store/shc8qpw1y2k7q668rx4gl6aff0wp1n6v-nss-certs-3.57R"
> (propagated-inputs ())
> (search-paths ())
> (properties
> (provenance
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> (branch "master")
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> (introduction
> (channel-introduction
> (version 0)
> (commit
> "9edb3f66fd807b096b48283debdcddccfea34bad")
> (signer
> "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A
> 54FA"))))))) ("git"
> "2.30.0"
> "out"
> "/gnu/store/378nlw54nxy991jcilnnbrxasnfvv9wl-git-2.30.0R"
> (propagated-inputs ())
> (search-paths
> (("GIT_SSL_CAINFO"
> ("etc/ssl/certs/ca-certificates.crt")
> #f
> regular
> #f)
> ("GIT_EXEC_PATH"
> ("libexec/git-core")
> #f
> directory
> #f)))
> (properties
> (provenance
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> (branch "master")
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> (introduction
> (channel-introduction
> (version 0)
> (commit
> "9edb3f66fd807b096b48283debdcddccfea34bad")
> (signer
> "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A
> 54FA"))))))) ("python-certifi"
> "2020.11.8"
> "out"
> "/gnu/store/hmp6ab9kw1z3hjns9h1fm3afsq4g6j7x-python-certifi-2020.11.8R"
> (propagated-inputs ())
> (search-paths ())
> (properties
> (provenance
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> (branch "master")
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> (introduction
> (channel-introduction
> (version 0)
> (commit
> "9edb3f66fd807b096b48283debdcddccfea34bad")
> (signer
> "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A
> 54FA"))))))))))
So I tried to reproduce it with the following command:
> guix time-machine \
> --commit=f9bd4621dd92a9415276706b476b9bd2973411fa -- \
> pack \
> --compression=xz \
> --save-provenance \
> -RR \
> --symlink=/usr/local/bin/repo=bin/repo \
> --symlink=/usr/local/bin/repo-env.sh=etc/profile \
> git-repo le-certs nss-certs git python-certifi
But the new tarball filename was different.
vivien in #guix helped me a lot by trying to build that tarball too and
me and viven have the same filename with guix-time-machine:
bfxvk59q0m034iyq5zkk841zkisayyjl-tarball-pack.tar.xz
We then managed to get to the root cause of the difference.
All the binaries were the sames. All the differences comes from the
fact that the provenance file (named 'manifest') is different.
That difference then produces a different profile name and also affects
/usr/bin as that references the profile.
Diffing the two provenance files gives that:
> +++
> bfxvk59q0m034iyq5zkk841zkisayyjl-tarball-pack/gnu/store/216jiimdyw7zyx8s9b3fz67aw69ydkvw-profile/manifest
> 1970-01-01 01:00:01.000000000 +0100 @@ -15,9 +15,10 @@ (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> - (branch "master")
> + (branch #f)
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> + (name guix)
> (introduction
> (channel-introduction
> (version 0)
> @@ -36,9 +37,10 @@
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> - (branch "master")
> + (branch #f)
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> + (name guix)
> (introduction
> (channel-introduction
> (version 0)
> @@ -57,9 +59,10 @@
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> - (branch "master")
> + (branch #f)
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> + (name guix)
> (introduction
> (channel-introduction
> (version 0)
> @@ -88,9 +91,10 @@
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> - (branch "master")
> + (branch #f)
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> + (name guix)
> (introduction
> (channel-introduction
> (version 0)
> @@ -109,9 +113,10 @@
> (repository
> (version 0)
> (url "https://git.savannah.gnu.org/git/guix.git")
> - (branch "master")
> + (branch #f)
> (commit
> "f9bd4621dd92a9415276706b476b9bd2973411fa")
> + (name guix)
> (introduction
> (channel-introduction
I've tried to add --branch=master to guix time-machine and used guix
gc -D to remove the older tarball as it didn't rebuild it even with
--rounds=2, and at the end I still got the exact same
bfxvk59q0m034iyq5zkk841zkisayyjl-tarball-pack.tar.xz tarball (I've
compared both with cmp).
Am I doing something wrong, or is there an issue that needs to be fixed
somehow?
References:
-----------
[1]https://ftp.osuosl.org/pub/replicant/build-tools/repo/28-01-2021/
Denis.
[Message part 2 (application/pgp-signature, inline)]
bug-guix <at> gnu.org:bug#49801; Package guix.
(Tue, 17 Aug 2021 17:49:01 GMT) Full text and rfc822 format available.Message #8 received at 49801 <at> debbugs.gnu.org (full text, mbox):
From: zimoun <zimon.toutoune <at> gmail.com> To: Denis 'GNUtoo' Carikli <GNUtoo <at> cyberdimension.org>, 49801 <at> debbugs.gnu.org Subject: Re: bug#49801: Guix time machine provenance/manifest reproducibility issue? Date: Tue, 17 Aug 2021 14:11:32 +0200
Hi,
Thanks for the report.
On Sun, 01 Aug 2021 at 02:21, Denis 'GNUtoo' Carikli <GNUtoo <at> cyberdimension.org> wrote:
> Diffing the two provenance files gives that:
>> +++
>> bfxvk59q0m034iyq5zkk841zkisayyjl-tarball-pack/gnu/store/216jiimdyw7zyx8s9b3fz67aw69ydkvw-profile/manifest
>> 1970-01-01 01:00:01.000000000 +0100 @@ -15,9 +15,10 @@ (repository
>> (version 0)
>> (url "https://git.savannah.gnu.org/git/guix.git")
>> - (branch "master")
>> + (branch #f)
>> (commit
>> "f9bd4621dd92a9415276706b476b9bd2973411fa")
>> + (name guix)
>> (introduction
>> (channel-introduction
>> (version 0)
Well, I think it comes from ’channel-list’ in the ’time-machine’.
Specifically, it reads in guix/scripts/pull.scm:
--8<---------------cut here---------------start------------->8---
(channel (inherit guix)
(url url) (commit commit) (branch #f)))
--8<---------------cut here---------------end--------------->8---
other said, the name of the branch is “lost”. Hum, I do not know if
this is done on purpose or not. Maybe this change
--8<---------------cut here---------------start------------->8---
(cons (match ref
(('commit . commit)
(channel (inherit guix)
(url url) (commit commit))
(('branch . branch)
(channel (inherit guix)
(url url) (commit #f) (branch branch)))
(#f
(channel (inherit guix) (url url))))
(remove guix-channel? channels))
--8<---------------cut here---------------end--------------->8---
is enough. But, I do not know what would happens for:
guix pull --commit=<hash>
where <hash> is not a commit from the branch master.
All the best,
simon
bug-guix <at> gnu.org:bug#49801; Package guix.
(Wed, 01 Sep 2021 22:28:01 GMT) Full text and rfc822 format available.Message #11 received at 49801 <at> debbugs.gnu.org (full text, mbox):
From: Denis 'GNUtoo' Carikli <GNUtoo <at> cyberdimension.org> To: zimoun <zimon.toutoune <at> gmail.com> Cc: 49801 <at> debbugs.gnu.org Subject: Re: bug#49801: Guix time machine provenance/manifest reproducibility issue? Date: Thu, 2 Sep 2021 00:27:42 +0200
[Message part 1 (text/plain, inline)]
Hi again.
With and without this patch:
> diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
> index fb8ce50fa7..af1cf77f07 100644
> --- a/guix/scripts/pull.scm
> +++ b/guix/scripts/pull.scm
> @@ -739,7 +739,7 @@ Use '~/.config/guix/channels.scm' instead."))
> (cons (match ref
> (('commit . commit)
> (channel (inherit guix)
> - (url url) (commit commit) (branch
> #f)))
> + (url url) (commit commit)))
> (('branch . branch)
> (channel (inherit guix)
> (url url) (commit #f) (branch
> branch)))
on top of 95c29d2746943733cbe8df7013854d45bb0df413 ("gnu: electron-cash:
Update to 4.2.5." which is today's master HEAD), I get the same diff
with and without time-machine.
I made and used this Makefile to build two hello tarball in both cases:
> COMMIT ?= 95c29d2746943733cbe8df7013854d45bb0df413
>
> all: \
> hello-guix-$(COMMIT).tar.xz \
> hello-time-machine-$(COMMIT).tar.xz \
>
> hello-guix-$(COMMIT).tar.xz:
> install -m 644 \
> `../pre-inst-env \
> guix pack \
> --compression=xz --save-provenance hello` \
> $@
>
> hello-time-machine-$(COMMIT).tar.xz:
> install -m 644 \
> `../pre-inst-env guix time-machine \
> --branch=master \
> --commit=$(COMMIT) \
> -- \
> pack --compression=xz --save-provenance hello` \
> $@
And once the file named manifest is extracted from both tarballs I get
this diff (with and without your slightly modified patch):
> --- ./hello-guix-95c29d2746943733cbe8df7013854d45bb0df413/gnu/store/lw9x5aimyqcq5iazj786fv7q5l3h0syk-profile/manifest 1970-01-01 01:00:01.000000000 +0100
> +++ ./hello-time-machine-95c29d2746943733cbe8df7013854d45bb0df413/gnu/store/30pf6ppiqpjsjaaiw35kc5lp6dcixpf1-profile/manifest 1970-01-01 01:00:01.000000000 +0100
> @@ -12,4 +12,19 @@
> "/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10"
> (propagated-inputs ())
> (search-paths ())
> - (properties)))))
> + (properties
> + (provenance
> + (repository
> + (version 0)
> + (url "https://git.savannah.gnu.org/git/guix.git")
> + (branch #f)
> + (commit
> + "95c29d2746943733cbe8df7013854d45bb0df413")
> + (name guix)
> + (introduction
> + (channel-introduction
> + (version 0)
> + (commit
> + "9edb3f66fd807b096b48283debdcddccfea34bad")
> + (signer
> + "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))))))))
PS: In the diff at the top there is a slight difference with the patch
that you suggested: I only removed (branch #f) so I end up with one
more parenthesis at the end.
Denis.
[Message part 2 (application/pgp-signature, inline)]
bug-guix <at> gnu.org:bug#49801; Package guix.
(Thu, 02 Sep 2021 08:11:02 GMT) Full text and rfc822 format available.Message #14 received at 49801 <at> debbugs.gnu.org (full text, mbox):
From: zimoun <zimon.toutoune <at> gmail.com> To: "Denis 'GNUtoo' Carikli" <GNUtoo <at> cyberdimension.org> Cc: 49801 <at> debbugs.gnu.org Subject: Re: bug#49801: Guix time machine provenance/manifest reproducibility issue? Date: Thu, 2 Sep 2021 10:10:22 +0200
Hi Denis,
Thanks for the investigation and the attempt.
Well, I miss if it works or not...
On Thu, 2 Sept 2021 at 00:27, Denis 'GNUtoo' Carikli
<GNUtoo <at> cyberdimension.org> wrote:
> With and without this patch:
> > diff --git a/guix/scripts/pull.scm b/guix/scripts/pull.scm
> > index fb8ce50fa7..af1cf77f07 100644
> > --- a/guix/scripts/pull.scm
> > +++ b/guix/scripts/pull.scm
> > @@ -739,7 +739,7 @@ Use '~/.config/guix/channels.scm' instead."))
> > (cons (match ref
> > (('commit . commit)
> > (channel (inherit guix)
> > - (url url) (commit commit) (branch
> > #f)))
> > + (url url) (commit commit)))
> > (('branch . branch)
> > (channel (inherit guix)
> > (url url) (commit #f) (branch
> > branch)))
>
> on top of 95c29d2746943733cbe8df7013854d45bb0df413 ("gnu: electron-cash:
> Update to 4.2.5." which is today's master HEAD), I get the same diff
> with and without time-machine.
...here I understand the patch fixes the issue...
> I made and used this Makefile to build two hello tarball in both cases:
> > COMMIT ?= 95c29d2746943733cbe8df7013854d45bb0df413
> >
> > all: \
> > hello-guix-$(COMMIT).tar.xz \
> > hello-time-machine-$(COMMIT).tar.xz \
> >
> > hello-guix-$(COMMIT).tar.xz:
> > install -m 644 \
> > `../pre-inst-env \
> > guix pack \
> > --compression=xz --save-provenance hello` \
> > $@
> >
> > hello-time-machine-$(COMMIT).tar.xz:
> > install -m 644 \
> > `../pre-inst-env guix time-machine \
> > --branch=master \
> > --commit=$(COMMIT) \
> > -- \
> > pack --compression=xz --save-provenance hello` \
> > $@
>
> And once the file named manifest is extracted from both tarballs I get
> this diff (with and without your slightly modified patch):
> > --- ./hello-guix-95c29d2746943733cbe8df7013854d45bb0df413/gnu/store/lw9x5aimyqcq5iazj786fv7q5l3h0syk-profile/manifest 1970-01-01 01:00:01.000000000 +0100
> > +++ ./hello-time-machine-95c29d2746943733cbe8df7013854d45bb0df413/gnu/store/30pf6ppiqpjsjaaiw35kc5lp6dcixpf1-profile/manifest 1970-01-01 01:00:01.000000000 +0100
> > @@ -12,4 +12,19 @@
> > "/gnu/store/a462kby1q51ndvxdv3b6p0rsixxrgx1h-hello-2.10"
> > (propagated-inputs ())
> > (search-paths ())
> > - (properties)))))
> > + (properties
> > + (provenance
> > + (repository
> > + (version 0)
> > + (url "https://git.savannah.gnu.org/git/guix.git")
> > + (branch #f)
> > + (commit
> > + "95c29d2746943733cbe8df7013854d45bb0df413")
> > + (name guix)
> > + (introduction
> > + (channel-introduction
> > + (version 0)
> > + (commit
> > + "9edb3f66fd807b096b48283debdcddccfea34bad")
> > + (signer
> > + "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))))))))
...but then here I see it does not fix it.
However, because you run "./pre-inst-env guix pack --save-provenance",
it seems expected that the 'properties' is empty. From my
understanding, '(find guix-channels? channels)' does not return the
'guix' channel because it is the current Git checkout. It is not the
case with "guix time-machine" because it creates an inferior using the
'guix' channel.
Moreover, if you want to try the patch, you need to run:
./pre-inst-env guix pull -p /tmp/new
./tmp/new/bin/guix describe # return commit 12345
./tmp/new/bin/guix pack --save-provenance
./tmp/new/bin/guix time-machine --commit=12345 -- pack --save-provenance
and be careful with the '--localstatedir' and '--sysconfdir' variables
at './configure' time.
Well, from my point of view, the Guix way would be:
guix describe -f channels > channels.scm
guix pack --save-provenance
then later or elsewehere
guix time-machine -C channels.scm -- pack --save-provenance
Although, it will not fix the bug you are exposing. :-)
WDYT?
Last, I have not carefully checked and maybe I am wrong, the both
options "--commit=1234 --branch=master" are exclusive I guess; i.e.,
the argument 'master' passed to '--branch' is not used in this case,
IIUC.
Cheers,
simon
bug-guix <at> gnu.org:bug#49801; Package guix.
(Thu, 02 Sep 2021 14:13:02 GMT) Full text and rfc822 format available.Message #17 received at 49801 <at> debbugs.gnu.org (full text, mbox):
From: Denis 'GNUtoo' Carikli <GNUtoo <at> cyberdimension.org> To: zimoun <zimon.toutoune <at> gmail.com> Cc: 49801 <at> debbugs.gnu.org Subject: Re: bug#49801: Guix time machine provenance/manifest reproducibility issue? Date: Thu, 2 Sep 2021 16:12:19 +0200
[Message part 1 (text/plain, inline)]
On Thu, 2 Sep 2021 10:10:22 +0200 zimoun <zimon.toutoune <at> gmail.com> wrote: > Hi Denis, > > Thanks for the investigation and the attempt. > > Well, I miss if it works or not... It doesn't work. The issue was that if you build a tarball with guix pack, without guix time-machine, you can't reproduce it with guix-time-machine. Between the two tarballs, everything is the same but the provenance file. So here the idea is to make sure that the provenance file is the same between tarballs made with and without guix time-machine. Here I get a diff between tarballs made with and without guix time-machine, with or without your patch, so the patch doesn't fix it yet for guix master of yesterday. Between when I reported the bug and the test I did yesterday, the HEAD of guix master changed though. Denis.
[Message part 2 (application/pgp-signature, inline)]
bug-guix <at> gnu.org:bug#49801; Package guix.
(Thu, 02 Sep 2021 19:45:01 GMT) Full text and rfc822 format available.Message #20 received at 49801 <at> debbugs.gnu.org (full text, mbox):
From: zimoun <zimon.toutoune <at> gmail.com> To: Denis 'GNUtoo' Carikli <GNUtoo <at> cyberdimension.org> Cc: 49801 <at> debbugs.gnu.org Subject: Re: bug#49801: Guix time machine provenance/manifest reproducibility issue? Date: Thu, 02 Sep 2021 21:30:46 +0200
Hi Denis,
On Thu, 02 Sep 2021 at 16:12, Denis 'GNUtoo' Carikli <GNUtoo <at> cyberdimension.org> wrote:
> Between the two tarballs, everything is the same but the provenance
> file.
>
> So here the idea is to make sure that the provenance file is the same
> between tarballs made with and without guix time-machine.
Yes, for sure. IMHO, the Guix way would be:
guix describe -f channels > channels.scm
guix pack --save-provenance
then later or elsewehere
guix time-machine -C channels.scm -- pack --save-provenance
It is a workaround of the bug your reported. ;-)
Does it work?
> Here I get a diff between tarballs made with and without guix
> time-machine, with or without your patch, so the patch doesn't fix it
> yet for guix master of yesterday.
I think your tests about the patch are not correct. As I wrote, this is
what you should try, IIUC:
--8<---------------cut here---------------start------------->8---
Moreover, if you want to try the patch, you need to run:
./pre-inst-env guix pull -p /tmp/new
./tmp/new/bin/guix describe # return commit 12345
./tmp/new/bin/guix pack --save-provenance
./tmp/new/bin/guix time-machine --commit=12345 -- pack --save-provenance
and be careful with the '--localstatedir' and '--sysconfdir' variables
at './configure' time.
--8<---------------cut here---------------end--------------->8---
All the best,
simon
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.