GNU bug report logs - #70339
Constructing hg-fetch fixed-output derivation requires Mercurial

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 70339 in the body.
You can then email your comments to 70339 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: bug-guix <at> gnu.org
Subject: Constructing hg-fetch fixed-output derivation requires Mercurial
Date: Wed, 10 Apr 2024 18:18:14 +0200

Hi,

For instance,

--8<---------------cut here---------------start------------->8---
$ guix build -S -d hg-commitsigs
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
3,7 MB will be downloaded:
  /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
 mercurial-6.2.2  3.5MiB                                                   529KiB/s 00:07 ▕██████████████████▏ 100.0%

/gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
--8<---------------cut here---------------end--------------->8---

and it is unexpected.  The construction of the fixed-output derivation
does not need to download stuff; it only needs to compose stuff
detailing how to do.  Any download (or build) must happen when running
the derivation itself.

The issue: later – say 1 or 2 years from now – the command:

   guix time-machone --commit=929ddec -- build -S -d hg-commitsigs

will start to build all what Mercurial needs (python etc.).  If for some
reasons*, only one of Mercurial dependencies fails then we are doomed.

    ( Context: I am working on a fixed-output translator; rely on
      current strategies for downloading and do not require all the past
      stack just for downloading source code. )

I think it comes from this part:

--8<---------------cut here---------------start------------->8---
                     (hg-fetch '#$(hg-reference-url ref)
                               '#$(hg-reference-changeset ref)
                               #$output
                               #:hg-command (string-append #+hg "/bin/hg")))
--8<---------------cut here---------------end--------------->8---

from ’hg-fetch’ in (guix hg-download).  Here the #+hg is not required
because just before there is:

            (set-path-environment-variable "PATH" '("bin")
                                           (match '#+inputs
                                             (((names dirs outputs ...) ...)
                                              dirs)))

So relying on string "hg" should be enough; as it is done in
’git-fetch/in-band*’ for one example.

Do I miss something?


Cheers,
simon

*reasons of failures: See
 https://guix.gnu.org/en/blog/2024/adventures-on-the-quest-for-long-term-reproducible-deployment

Message #8 received at 70339 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: Simon Tournier <zimon.toutoune <at> gmail.com>
Cc: 70339 <at> debbugs.gnu.org
Subject: Re: bug#70339: Constructing hg-fetch fixed-output derivation
 requires Mercurial
Date: Fri, 12 Apr 2024 11:30:00 +0200

Hello!

Simon Tournier <zimon.toutoune <at> gmail.com> skribis:

> $ guix build -S -d hg-commitsigs
> substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> 3,7 MB will be downloaded:
>   /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
> substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
> downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
>  mercurial-6.2.2  3.5MiB                                                   529KiB/s 00:07 ▕██████████████████▏ 100.0%
>
> /gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
>
>
> and it is unexpected.

That running ‘hg clone’ requires Mercurial isn’t totally unexpected to
me.  :-)

> I think it comes from this part:
>
>                      (hg-fetch '#$(hg-reference-url ref)
>                                '#$(hg-reference-changeset ref)
>                                #$output
>                                #:hg-command (string-append #+hg "/bin/hg")))
>
> from ’hg-fetch’ in (guix hg-download).  Here the #+hg is not required
> because just before there is:
>
>             (set-path-environment-variable "PATH" '("bin")
>                                            (match '#+inputs
>                                              (((names dirs outputs ...) ...)
>                                               dirs)))

Maybe, but one way or another, Mercurial is necessary.

Now, the ‘guix recover’ tool (or whatever you call it) you’re working on
could create a different fixed-output derivation producing the same
result but without using Mercurial; typically, the builder of that
derivation would download from SWH.

Does that make sense?

HTH,
Ludo’.

Message #11 received at 70339 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Cc: 70339 <at> debbugs.gnu.org
Subject: Re: bug#70339: Constructing hg-fetch fixed-output derivation requires
 Mercurial
Date: Fri, 12 Apr 2024 14:42:35 +0200

Hi Ludo,

On Fri, 12 Apr 2024 at 11:30, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:

> > $ guix build -S -d hg-commitsigs
> > substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
> > 3,7 MB will be downloaded:
> >   /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
> > substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
> > downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
> >  mercurial-6.2.2  3.5MiB                                                   529KiB/s 00:07 ▕██████████████████▏ 100.0%
> >
> > /gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
> >
> > and it is unexpected.
>
> That running ‘hg clone’ requires Mercurial isn’t totally unexpected to
> me.  :-)

There is a misunderstanding, I guess.

Running 'hg clone' requires to have a local copy of Mercurial, yes for sure. :-)

However, just ask what it will run (please note the dash d in guix
build -S -d hg-commitsigs) must not require to have a local copy of
Mercurial (binary).  If you still think yes, why is it not the case
for fixed-output derivations relying on the old Git builder?

> > I think it comes from this part:
> >
> >                      (hg-fetch '#$(hg-reference-url ref)
> >                                '#$(hg-reference-changeset ref)
> >                                #$output
> >                                #:hg-command (string-append #+hg "/bin/hg")))
> >
> > from ’hg-fetch’ in (guix hg-download).  Here the #+hg is not required
> > because just before there is:
[...]
> Maybe, but one way or another, Mercurial is necessary.

Again, I think it is a bug from #+hg instead of plain "hg".  Having a
local copy of Mercurial (binary) must not be required to just display
the fixed-output derivation.  For running this fixed-output
derivation, yes for sure.

> Now, the ‘guix recover’ tool (or whatever you call it) you’re working on
> could create a different fixed-output derivation producing the same
> result but without using Mercurial; typically, the builder of that
> derivation would download from SWH.
>
> Does that make sense?

Yes, it makes sense; see my very first attempt in [1] :-).

But you cannot apply this strategy for fixed-output derivations
relying on Mercurial.  You need first to build Mercurial (and thus all
the Python stack) just to display the fixed-output derivation.  Then,
yes once you have this fixed-output derivation, it is possible to
manipulate it for getting another one.

This report is about #+hg that needs to be fixed for the future.
And because of that, the strategy above for fixed-output derivations
relying on Mercurial is doomed for the past, IMHO.  Except if you have
an idea. ;-)

Cheers,
simon


1: https://gitlab.com/zimoun/guix-drv

Message #14 received at 70339 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludovic.courtes <at> inria.fr>
To: Simon Tournier <zimon.toutoune <at> gmail.com>
Cc: 70339 <at> debbugs.gnu.org
Subject: Re: bug#70339: Constructing hg-fetch fixed-output derivation
 requires Mercurial
Date: Fri, 12 Apr 2024 16:05:23 +0200

Simon Tournier <zimon.toutoune <at> gmail.com> skribis:

> Hi Ludo,
>
> On Fri, 12 Apr 2024 at 11:30, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:
>
>> > $ guix build -S -d hg-commitsigs
>> > substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
>> > 3,7 MB will be downloaded:
>> >   /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2
>> > substituting /gnu/store/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2...
>> > downloading from https://ci.guix.gnu.org/nar/lzip/6fya762sz5hjdj04vdn5g3v6zii6f11d-mercurial-6.2.2 ...
>> >  mercurial-6.2.2  3.5MiB                                                   529KiB/s 00:07 ▕██████████████████▏ 100.0%
>> >
>> > /gnu/store/pkb6zd9xfmxx6rsh4p7w3glh7xqg5sqy-hg-commitsigs-0.1.0-0.b53eb68-checkout.drv
>> >
>> > and it is unexpected.
>>
>> That running ‘hg clone’ requires Mercurial isn’t totally unexpected to
>> me.  :-)
>
> There is a misunderstanding, I guess.
>
> Running 'hg clone' requires to have a local copy of Mercurial, yes for sure. :-)
>
> However, just ask what it will run (please note the dash d in guix
> build -S -d hg-commitsigs) must not require to have a local copy of
> Mercurial (binary).  If you still think yes, why is it not the case
> for fixed-output derivations relying on the old Git builder?

Oh sorry, I had missed the ‘-d’ bit.

In this case, what’s happening is grafts: Guix downloads (or builds)
Mercurial so it can compute its grafted derivation.

>> Now, the ‘guix recover’ tool (or whatever you call it) you’re working on
>> could create a different fixed-output derivation producing the same
>> result but without using Mercurial; typically, the builder of that
>> derivation would download from SWH.
>>
>> Does that make sense?
>
> Yes, it makes sense; see my very first attempt in [1] :-).

[...]

> 1: https://gitlab.com/zimoun/guix-drv

Nice!

Thanks,
Ludo’.

Message #19 received at 70339-done <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludovic.courtes <at> inria.fr>
Cc: 70339-done <at> debbugs.gnu.org
Subject: Re: bug#70339: Constructing hg-fetch fixed-output derivation
 requires Mercurial
Date: Mon, 22 Apr 2024 02:22:00 +0200

Hi,

On ven., 12 avril 2024 at 16:05, Ludovic Courtès <ludovic.courtes <at> inria.fr> wrote:

> In this case, what’s happening is grafts: Guix downloads (or builds)
> Mercurial so it can compute its grafted derivation.

Ah indeed!  Damned. :-)

Closing.

Cheers,
simon

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.