GNU bug report logs - #74015
'guix shell --export-manifest' fails for some transformations

Previous Next

To reply to this bug, email your comments to 74015 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: bug-guix <at> gnu.org
Subject: 'guix shell --export-manifest' fails for some transformations
Date: Fri, 25 Oct 2024 19:08:17 +0200

Hi,

I notice this:

--8<---------------cut here---------------start------------->8---
$ cat manifest.scm
(use-modules (guix transformations) (guix packages)
             (guix download) (guix build-system r))

(define r-knitr
  (specification->package "r-knitr"))

(define r-knitr-other
  (package
    (inherit r-knitr)
    (version "1.47")
    (source (origin
              (method url-fetch)
              (uri (cran-uri "knitr" version))
              (sha256
               (base32
                "1kx6g8ddbigwx3bmq771gwi3din3fxjn59l801904kjaz6dq9pgs"))))))

(define r-ggplot2
  (package
    (inherit
     ((package-input-rewriting `((,r-knitr . ,r-knitr-other)))
      (specification->package "r-ggplot2")))
    (name "my-ggplot")))

(define transform
  (options->transformation
    '((with-c-toolchain . "r-minimal=gcc-toolchain <at> 12")
      (with-c-toolchain . "r=gcc-toolchain <at> 12"))))

(packages->manifest
 (append
  (map specification->package
       (list "python" "python-numpy"))
  (map transform
       (list (specification->package "r")
             r-ggplot2))))

$ guix shell -m manifest.scm --export-manifest
guix shell: warning: transformation 'with-c-toolchain' had no effect on my-ggplot <at> 3.5.1.
;; What follows is a "manifest" equivalent to the command line you gave.
;; You can store it in a file that you may then pass to any 'guix' command
;; that accepts a '--manifest' (or '-m') option.

(use-modules (guix transformations))

(define transform1
  (options->transformation
    '((with-c-toolchain . "r-minimal=gcc-toolchain <at> 12")
      (with-c-toolchain . "r=gcc-toolchain <at> 12"))))

(packages->manifest
  (list (specification->package "python")
        (specification->package "python-numpy")
        (transform1 (specification->package "r"))
        (transform1 (specification->package "my-ggplot"))))
--8<---------------cut here---------------end--------------->8---

Well, this is incorrect.  While I understand the approximation when
dealing with profiles built over time in the imperative way, here I am
missing why the transformation of ’r-ggplot2’ is skipped.  And even the
export manifest is broken.

It’s skipped because:

--8<---------------cut here---------------start------------->8---
$ guix shell -m manifest.scm
$ grep ggplot $GUIX_ENVIRONMENT/manifest
     ("my-ggplot"
      "/gnu/store/x0jbmvg0nbl7lyk8zd938rq2c7c9qkn4-my-ggplot-3.5.1"
--8<---------------cut here---------------end--------------->8---

Please note that if the symbol ’r-ggplot2’ is not named “my-ggplot” but
kept as ’r-ggplot2“:

        (define r-ggplot2
          ((package-input-rewriting `((,r-knitr . ,r-knitr-other)))
              (specification->package "r-ggplot2")))

then the bug is maybe worse because the exported manifest “works” but
without the transformation… which can be misleading and hard to detect.

Well, I understand it’s hard to capture this kind of transformation.
Still, it appears to me as an half-baked feature. :-)

That’s annoying in some context as “guix pack --save-provenance” [1].
Even, it defeats the idea of a self-contained reproducible binary
container. ;-)

Maybe, aside the profile ’manifest’ file, we could store all the
manifests provided by the command line.  Something as:

    /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
    /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
    /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2
    …

where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
and ’bar.scm’ from:

    -m foo.scm -m bar.scm

This way, the option --export-manifest could be improved.

WDYT?

Cheers,
simon

1: https://hpc.guix.info/blog/2021/10/when-docker-images-become-fixed-point

Message #8 received at 74015 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Simon Tournier <zimon.toutoune <at> gmail.com>
Cc: 74015 <at> debbugs.gnu.org
Subject: Re: bug#74015: 'guix shell --export-manifest' fails for some
 transformations
Date: Sun, 10 Nov 2024 12:38:50 +0100

Hi,

Simon Tournier <zimon.toutoune <at> gmail.com> skribis:

> $ guix shell -m manifest.scm --export-manifest

‘--export-manifest’ is meant to “translate” a command line, which it can
do faithfully; there’s no way it could possibly “recreate” what
‘manifest.scm’, which may contain arbitrary Scheme code (and someone who
already has a ‘manifest.scm’ file probably doesn’t need
‘--export-manifest’).

It’s necessarily a lossy process.  Some of the transformations
‘manifest.scm’ uses may be recorded as metadata in manifest entries, but
others, such as uses of ‘package-input-rewriting’ here, inevitably get
lost.

I would close it as not-a-bug; WDYT?

Ludo’.

Message #11 received at 74015 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 74015 <at> debbugs.gnu.org
Subject: Re: bug#74015: 'guix shell --export-manifest' fails for some
 transformations
Date: Sun, 10 Nov 2024 14:06:16 +0100

Hi,

On Sun, 10 Nov 2024 at 12:38, Ludovic Courtès <ludo <at> gnu.org> wrote:

>> $ guix shell -m manifest.scm --export-manifest
>
> ‘--export-manifest’ is meant to “translate” a command line, which it can
> do faithfully; there’s no way it could possibly “recreate” what
> ‘manifest.scm’, which may contain arbitrary Scheme code (and someone who
> already has a ‘manifest.scm’ file probably doesn’t need
> ‘--export-manifest’).

Yes, one might need. :-)

As I explained: « That’s annoying in some context as “guix pack
--save-provenance” [1].  Even, it defeats the idea of a self-contained
reproducible binary container. »

For sure, I understand it’s hard to capture this kind of transformation.
Still, an improvement appears to me possible (see below).

1: https://hpc.guix.info/blog/2021/10/when-docker-images-become-fixed-point


> It’s necessarily a lossy process.

This is where I disagree. :-) I mean, yes I agree that building
profile/manifest is somehow a lossy process because some Scheme is
potentially evaluated on the road.  However, I am proposing: Aside the
profile ’manifest’ file (lossy process), we could store all the
manifests provided by the command line.  Something as:

    /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
    /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
    /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2
    …

where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
and ’bar.scm’ from:

    -m foo.scm -m bar.scm

This way, the option --export-manifest could be improved, eventually.

In other word, I think that saving as-is the manifests costs almost
nothing and it paves the way to more robust self-contained binary packs.

Well, maybe this extra work could be only done when there is an option
’--save-provenance’ applied.

> I would close it as not-a-bug; WDYT?

I still think it’s a feature that it’s possible to improve.

Cheers,
simon

Message #14 received at 74015 <at> debbugs.gnu.org (full text, mbox):

From: Suhail Singh <suhailsingh247 <at> gmail.com>
To: Simon Tournier <zimon.toutoune <at> gmail.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 74015 <at> debbugs.gnu.org
Subject: Re: bug#74015: 'guix shell --export-manifest' fails for some
 transformations
Date: Tue, 12 Nov 2024 10:33:59 -0500

Simon Tournier <zimon.toutoune <at> gmail.com> writes:

> However, I am proposing: Aside the profile ’manifest’ file (lossy
> process), we could store all the manifests provided by the command
> line.  Something as:
>
>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2
>     …
>
> where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
> and ’bar.scm’ from:
>
>     -m foo.scm -m bar.scm
>
> ...
>
> Well, maybe this extra work could be only done when there is an option
> ’--save-provenance’ applied.

For --save-provenance, is your proposal that in addition to saving the
additional manifests, the commandline that was used would also be saved?

-- 
Suhail

Message #17 received at 74015 <at> debbugs.gnu.org (full text, mbox):

From: Simon Tournier <zimon.toutoune <at> gmail.com>
To: Suhail Singh <suhailsingh247 <at> gmail.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 74015 <at> debbugs.gnu.org
Subject: Re: bug#74015: 'guix shell --export-manifest' fails for some
 transformations
Date: Sat, 16 Nov 2024 08:20:49 +0100

Hi,

On Tue, 12 Nov 2024 at 10:33, Suhail Singh <suhailsingh247 <at> gmail.com> wrote:
> Simon Tournier <zimon.toutoune <at> gmail.com> writes:
>
>> However, I am proposing: Aside the profile ’manifest’ file (lossy
>> process), we could store all the manifests provided by the command
>> line.  Something as:
>>
>>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
>>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
>>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2
>>     …
>>
>> where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
>> and ’bar.scm’ from:
>>
>>     -m foo.scm -m bar.scm
>>
>> ...
>>
>> Well, maybe this extra work could be only done when there is an option
>> ’--save-provenance’ applied.
>
> For --save-provenance, is your proposal that in addition to saving the
> additional manifests, the commandline that was used would also be saved?

We could save the command line too using another file. :-)

Cheers,
simon

Message #20 received at 74015 <at> debbugs.gnu.org (full text, mbox):

From: Suhail Singh <suhailsingh247 <at> gmail.com>
To: Simon Tournier <zimon.toutoune <at> gmail.com>
Cc: Ludovic Courtès <ludo <at> gnu.org>,
 Suhail Singh <suhailsingh247 <at> gmail.com>, 74015 <at> debbugs.gnu.org
Subject: Re: bug#74015: 'guix shell --export-manifest' fails for some
 transformations
Date: Sun, 17 Nov 2024 14:05:38 -0500

Simon Tournier <zimon.toutoune <at> gmail.com> writes:

>> For --save-provenance, is your proposal that in addition to saving the
>> additional manifests, the commandline that was used would also be saved?
>
> We could save the command line too using another file. :-)

That would be helpful :)

Given an image generated with --save-provenance enabled, making it
easier for it to be reconstructed by another would certainly be useful.

-- 
Suhail

Message #23 received at 74015 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Simon Tournier <zimon.toutoune <at> gmail.com>
Cc: 74015 <at> debbugs.gnu.org
Subject: Re: bug#74015: 'guix shell --export-manifest' fails for some
 transformations
Date: Wed, 20 Nov 2024 11:29:11 +0100

Hi,

Simon Tournier <zimon.toutoune <at> gmail.com> skribis:

> On Sun, 10 Nov 2024 at 12:38, Ludovic Courtès <ludo <at> gnu.org> wrote:
>
>>> $ guix shell -m manifest.scm --export-manifest
>>
>> ‘--export-manifest’ is meant to “translate” a command line, which it can
>> do faithfully; there’s no way it could possibly “recreate” what
>> ‘manifest.scm’, which may contain arbitrary Scheme code (and someone who
>> already has a ‘manifest.scm’ file probably doesn’t need
>> ‘--export-manifest’).
>
> Yes, one might need. :-)
>
> As I explained: « That’s annoying in some context as “guix pack
> --save-provenance” [1].  Even, it defeats the idea of a self-contained
> reproducible binary container. »

I see, but if you have the source ‘manifest.scm’, best practice is to
publish it.

>> It’s necessarily a lossy process.
>
> This is where I disagree. :-)

You are free to disagree but it’s a fact.

> I mean, yes I agree that building profile/manifest is somehow a lossy
> process because some Scheme is potentially evaluated on the road.
> However, I am proposing: Aside the profile ’manifest’ file (lossy
> process), we could store all the manifests provided by the command
> line.  Something as:
>
>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest
>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig1
>     /gnu/store/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-profile/manifest.orig2
>     …
>
> where manifest.orig1 and manifest.orig2 are the copy of files ’foo.scm’
> and ’bar.scm’ from:
>
>     -m foo.scm -m bar.scm

Hmm yes, maybe ‘guix pack -m manifest.scm --save-provenance’ could
arrange to put ‘manifest.scm’ inside the pack, similar to how ‘guix
system reconfigure’ inserts ‘configuration.scm’ inside the system.

I’m all for it!

Thanks,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.