Subject: Re: bug#52684: [BUG] Multiple Packages Failing to Build

From: Maxime Devos <[email-masked]>
Date: Wed, 22 Dec 2021 17:36:14 +0000

Hi,

Christopher Rodriguez schreef op wo 22-12-2021 om 12:07 [-0500]:
> So, I took some time to do some digging this morning, and I now have a 
> few results and a few more questions.
> 
> First, I tried `guix shell --pure python beets beets-bandcamp` to ensure 
> that the plugin would be detected once `GUIX_PYTHONPATH` was set as You 
> had mentioned. That did work, though the environment did not have 
> `python-isodate`, which the plugin complained was missing. If this is 
> the case, should python-isodate then be a progagated input, since it is 
> apparently required at runtime?
> 

Yes, but not for that reason. If it's required at runtime, it needs to
be in 'inputs' or 'propagated-inputs'. Ideally, 'inputs' would be
sufficient, but python doesn't have RUNPATH so we have to do with
propagation.

It's probably possible to modify the plugins to adjust the load path
when the plugin is loaded, avoiding the need of propagation, though
that might be fragile and complicated. (I don't have any examples of
packages currently doing this.)

>  Or is there a difference w/r/t the
> `--pure` environment that won't be present on an actual install?

The --pure just makes sure you don't reuse old environment variables,
which is useful to make sure there aren't any missing propagated-inputs
(or anything that needs to be substitute*-ed, but no substitution
appears to be required in beets).

It is largely equivalent to removing all packages in your user profile
and system profile (including things like 'bash' and 'coreutils'!) and
then installing python, beets and beets-bandcamp, except that
"guix shell --pure" is way more convenient.

> Second, I ran `cat $(which beet)` on my currently installed `beets` 
> package for the `PYTHON_PATH` that is currently being set. It has a lot 
> of entries, all like the following: 
> `/gnu/store/8df74df68i14lfjsny07x7cq6ffn0fs5-beets-1.5.0/lib/python3.8/site-packages:/gnu/store/nc3rprg62sigx8s9ps02wb8zbaz8qzl3-python-flask-1.1.2/lib/python3.8/site-packages`. 

> I'm currently diving each of these packages to see how I might add code 
> to set `beets-bandcamp` to add to this list, though I wonder if that is 
> actually possible, since the plugin will usually be installed after the 
> root program. Is this path pulled from a global location, or is it 
> determined at install time?

You cannot retroactively modify store items.
I.e., the build code of 'beets-bandcamp' cannot modify the 'beet'
binary. If that were possible, it would open a can of reproducibility
and security worms (think e.g. of multi-user systems -- they are rare,
but they do exist).

The path in a wrapper of a package P is determined when the package P
is built.

> Third, I did try pulling
> 
> ```scheme
> (native-search-paths
>    (list (guix-pythonpath-search-path)))
> ```
> 
> into the `beets` definition, but that brought a *lot* of other 
> dependencies in. Same with using

Not sure what you mean here. A lot more dependencies in GUIX_PYTHONPATH
in the environment variables of the profile, or in the wrapper, or ...?

> ```scheme
> (native-inputs 
>    `(("sitecustomize.py" ,(local-file (search-auxiliary-file 
>         "python/sitecustomize.py")))))
> ```

I don't know how sitecustomize.py works.

> Sorry if I am missing something obvious, here. I will keep looking, just 
> wanted to log what I've tried so far.
> 
> Once I figure out how search paths actually work inside of `guix`, I'm 
> sure the solution will be fairly simple. But I haven't quite gotten 
> there yet.
> 
> It seems like the solution has to do with `wrap-package`, but looking at 
> the `beets` package definition, I only see the following lines related 
> to wrapping the binary:
> 
> ```scheme
>           ;; Wrap the executable, so it can find python-gi (aka 
>           ;; pygobject) and gstreamer plugins. 
>           (add-after 'wrap 'wrap-typelib
>             (lambda* (#:key outputs #:allow-other-keys)
>               (let ((prog (string-append (assoc-ref outputs "out")
>                                          "/bin/beet"))
>                     (plugins (getenv "GST_PLUGIN_SYSTEM_PATH"))
>                     (types (getenv "GI_TYPELIB_PATH")))
>                 (wrap-program prog
>                   `("GST_PLUGIN_SYSTEM_PATH" ":" prefix (,plugins))
>                   `("GI_TYPELIB_PATH" ":" prefix (,types)))
>                 #t))))))

Look in python-build-system.scm, the build system also does some
wrapping (you can wrap a program multiple times!).

> ```
> It doesn't seem like PYTHONPATH is being interacted with here. Is that 
> something that might be defined elsewhere?
> 

python-build-system sets GUIX_PYTHONPATH. I don't know the difference
between PYTHONPATH and GUIX_PYTHONPATH. Some compatibility mechanism
between foreign distros and Guix I presume.

>  I'm tempted to merge in the 
> following, which I pulled from another package (solfege), to explicitly 
> set PYTHONPATH… But I don't want to change something if it is set elsewhere.
> 
> 
> ```scheme
>           (add-after 'install 'wrap-program
>             (lambda* (#:key inputs outputs #:allow-other-keys)
>               ;; Make sure 'solfege' runs with the correct PYTHONPATH. 
>               (let* ((out (assoc-ref outputs "out"))
>                      (path (getenv "GUIX_PYTHONPATH")))
>                 (wrap-program (string-append out "/bin/solfege")
>                   `("GUIX_PYTHONPATH" ":" prefix (,path))))
>               #t)))))
> ```

See python-build-system and some replies above ;-.
> Any input or clarification will be appreciated! I'm learning a lot as I 
> (slowly) work through this! Thank You for Your time!

Greetings,
Maxime.