GNU bug report logs - #22423
git-fetch does not update checked out tree when commit hash changes

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 22423 in the body.
You can then email your comments to 22423 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Pjotr Prins <pjotr.public12 <at> thebird.nl>
To: bug-guix <at> gnu.org
Subject: git-fetch does not update checked out tree when commit hash changes
Date: Thu, 21 Jan 2016 07:54:03 +0100

I can reliably reproduce this using a recent version of GNU Guix. 

When updating the commit hash to a different commit the git-fetch
derivation *does* change (I checked in guile), but the checked out git
tree in the store does not change - it gets shared between the
commits. I am not sure why the tree gets shared, but the effect is
that the same package gets installed using the same
/gnu/store/xxx-git-checkout.

Removing the git-checkout dir and updating the Hash gives a missing
dir error (as expected when they use the same).

Message #10 received at 22423-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Pjotr Prins <pjotr.public12 <at> thebird.nl>
Cc: 22423-done <at> debbugs.gnu.org
Subject: Re: bug#22423: git-fetch does not update checked out tree when commit
 hash changes
Date: Thu, 21 Jan 2016 09:50:18 +0100

Pjotr Prins <pjotr.public12 <at> thebird.nl> skribis:

> When updating the commit hash to a different commit the git-fetch
> derivation *does* change (I checked in guile), but the checked out git
> tree in the store does not change - it gets shared between the
> commits. I am not sure why the tree gets shared, but the effect is
> that the same package gets installed using the same
> /gnu/store/xxx-git-checkout.

This is expected: origins are fixed-output derivations, meaning that it
does not matter how we perform them (using Git, over HTTP, or thanks to
an avian carrier), as long as the result has the specified sha256.

Thus, when you change, say, the Git commit ID or origin ‘method’ without
changing the ‘sha256’ field, nothing happens: the daemon says “OK, I
already have a store item with that ‘sha256’, so I don’t do anything.”

Clearly, one has to be cautious with this, it’s easy to mistakenly use
the old source.

Hope this clarifies things!

Ludo’.

Message #13 received at 22423-done <at> debbugs.gnu.org (full text, mbox):

From: Pjotr Prins <pjotr.public12 <at> thebird.nl>
To: 22423-done <at> debbugs.gnu.org
Subject: Re: bug#22423: git-fetch does not update checked out tree when
 commit hash changes
Date: Thu, 21 Jan 2016 10:08:53 +0100

On Thu, Jan 21, 2016 at 09:50:18AM +0100, Ludovic Courtès wrote:
> This is expected: origins are fixed-output derivations, meaning that it
> does not matter how we perform them (using Git, over HTTP, or thanks to
> an avian carrier), as long as the result has the specified sha256.
> 
> Thus, when you change, say, the Git commit ID or origin ‘method’ without
> changing the ‘sha256’ field, nothing happens: the daemon says “OK, I
> already have a store item with that ‘sha256’, so I don’t do anything.”
> 
> Clearly, one has to be cautious with this, it’s easy to mistakenly use
> the old source.

Hmmm. I thought the sha256 was calculated over the derivation +
sources, so any relevant change would trigger a build. Apparently it
is triggered by the sha256 field only. Good to know.

Pj.

Message #16 received at 22423-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Pjotr Prins <pjotr.public12 <at> thebird.nl>
Cc: 22423-done <at> debbugs.gnu.org
Subject: Re: bug#22423: git-fetch does not update checked out tree when commit
 hash changes
Date: Thu, 21 Jan 2016 11:10:14 +0100

Pjotr Prins <pjotr.public12 <at> thebird.nl> skribis:

> On Thu, Jan 21, 2016 at 09:50:18AM +0100, Ludovic Courtès wrote:
>> This is expected: origins are fixed-output derivations, meaning that it
>> does not matter how we perform them (using Git, over HTTP, or thanks to
>> an avian carrier), as long as the result has the specified sha256.
>> 
>> Thus, when you change, say, the Git commit ID or origin ‘method’ without
>> changing the ‘sha256’ field, nothing happens: the daemon says “OK, I
>> already have a store item with that ‘sha256’, so I don’t do anything.”
>> 
>> Clearly, one has to be cautious with this, it’s easy to mistakenly use
>> the old source.
>
> Hmmm. I thought the sha256 was calculated over the derivation +
> sources

What you’re saying is true of the hash that appears in /gnu/store file
name, but I was referring to the ‘sha256’ field of origins, which is a
different thing.

Ludo’.

Message #19 received at 22423-done <at> debbugs.gnu.org (full text, mbox):

From: Jookia <166291 <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Pjotr Prins <pjotr.public12 <at> thebird.nl>, 22423-done <at> debbugs.gnu.org
Subject: Re: bug#22423: git-fetch does not update checked out tree when
 commit hash changes
Date: Thu, 21 Jan 2016 22:50:02 +1100

On Thu, Jan 21, 2016 at 11:10:14AM +0100, Ludovic Courtès wrote:
> Pjotr Prins <pjotr.public12 <at> thebird.nl> skribis:
> 
> > On Thu, Jan 21, 2016 at 09:50:18AM +0100, Ludovic Courtès wrote:
> >> This is expected: origins are fixed-output derivations, meaning that it
> >> does not matter how we perform them (using Git, over HTTP, or thanks to
> >> an avian carrier), as long as the result has the specified sha256.
> >> 
> >> Thus, when you change, say, the Git commit ID or origin ‘method’ without
> >> changing the ‘sha256’ field, nothing happens: the daemon says “OK, I
> >> already have a store item with that ‘sha256’, so I don’t do anything.”
> >> 
> >> Clearly, one has to be cautious with this, it’s easy to mistakenly use
> >> the old source.
> >
> > Hmmm. I thought the sha256 was calculated over the derivation +
> > sources
> 
> What you’re saying is true of the hash that appears in /gnu/store file
> name, but I was referring to the ‘sha256’ field of origins, which is a
> different thing.
> 
> Ludo’.

I think this is a bit of a problem even if it's expected: Often times we can't
calculate the hash until it's downloaded and get a hash mismatch. The other day
I rebuilt NixOS almost entirely on my machine and changed the revision on
Firefox to a new branch but didn't change the hash since I expected a mismatch.

Needles to say I realized what happened when I checked Firefox's version. I
think it'd be great to have a 'INVALID' hash we can use for development that
just prints a mismatch and errors out like usual. Maybe this is possible in
Guix, but it didn't seem documented and it's not possible in NixOS.

Cheers,
Jookia.

Message #22 received at 22423-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Jookia <166291 <at> gmail.com>
Cc: Pjotr Prins <pjotr.public12 <at> thebird.nl>, 22423-done <at> debbugs.gnu.org
Subject: Re: bug#22423: git-fetch does not update checked out tree when commit
 hash changes
Date: Thu, 21 Jan 2016 17:25:40 +0100

Jookia <166291 <at> gmail.com> skribis:

> On Thu, Jan 21, 2016 at 11:10:14AM +0100, Ludovic Courtès wrote:
>> Pjotr Prins <pjotr.public12 <at> thebird.nl> skribis:
>> 
>> > On Thu, Jan 21, 2016 at 09:50:18AM +0100, Ludovic Courtès wrote:
>> >> This is expected: origins are fixed-output derivations, meaning that it
>> >> does not matter how we perform them (using Git, over HTTP, or thanks to
>> >> an avian carrier), as long as the result has the specified sha256.
>> >> 
>> >> Thus, when you change, say, the Git commit ID or origin ‘method’ without
>> >> changing the ‘sha256’ field, nothing happens: the daemon says “OK, I
>> >> already have a store item with that ‘sha256’, so I don’t do anything.”
>> >> 
>> >> Clearly, one has to be cautious with this, it’s easy to mistakenly use
>> >> the old source.
>> >
>> > Hmmm. I thought the sha256 was calculated over the derivation +
>> > sources
>> 
>> What you’re saying is true of the hash that appears in /gnu/store file
>> name, but I was referring to the ‘sha256’ field of origins, which is a
>> different thing.
>> 
>> Ludo’.
>
> I think this is a bit of a problem even if it's expected: Often times we can't
> calculate the hash until it's downloaded and get a hash mismatch. The other day
> I rebuilt NixOS almost entirely on my machine and changed the revision on
> Firefox to a new branch but didn't change the hash since I expected a mismatch.
>
> Needles to say I realized what happened when I checked Firefox's version. I
> think it'd be great to have a 'INVALID' hash we can use for development that
> just prints a mismatch and errors out like usual. Maybe this is possible in
> Guix, but it didn't seem documented and it's not possible in NixOS.

The problem is that Guix cannot guess that the ‘sha256’ the developer
provided in the ‘origin’ form is actually wrong.  Only the developer
knows.

This is mitigated in Guix by the fact that we usually include the
version string or Git commit ID in the origin’s file name, with patterns
like:

  (define foo
    (let ((commit "deadbeef"))
      (package
        (name "foo")
        (version commit)
        (source (origin
                  (method git-fetch)
                  (uri (git-reference (commit commit) …))
                  (file-name (string-append name "-" commit)) ;<– !
                  (sha256 …)
                  …))
        …)))

The good thing with this pattern is that, when you modify the value of
‘commit’, you also end up modifying ‘file-name’, thus triggering a
re-download.  So if you had forgotten to update ‘sha256’, you
immediately get a hash mismatch error.

In practice, I’ve found this to be rather helpful.

Additionally, ‘guix lint’ emits warnings for packages that do not follow
this pattern, hopefully making the problem that Pjotr describes less
likely.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.