GNU bug report logs - #53406
"grafts fail with different architecture inputs?"

Previous Next

To reply to this bug, email your comments to 53406 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: John Kehayias <john.kehayias <at> protonmail.com>
To: "bug-guix <at> gnu.org" <bug-guix <at> gnu.org>
Subject: union-build incorrectly handles grafts
Date: Fri, 21 Jan 2022 06:48:38 +0000

[Message part 1 (text/plain, inline)]

Hello Guix,

Stumbled upon a mysterious problem today with grafts. Let me report what I see and how to reproduce it:

I'm attaching a minimal working example of the problem, namely using union-build creates the wrong symlink when a package has a propagated input that has a replacement. At least that seems to be the key ingredients. (Note the code originally came from another channel, so I left the header intact. The union-build looks the same as in e.g. python-pyqt+qscintilla though.)

It looks like a link is made to a wrong (i.e. previous) version library. It is pointing to the replacement package in the store, but trying to use the previous name.

If I run the attached file with guix build -f graft-test.scm and look at libexpat in the created package with, e.g.

ls -la $(guix build -f graft-test.scm)/lib/libexpat*

I get

lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/a3ixqfx1cfjgbz5gb925fv7dwy2h55gs-fhs-union-0.0/lib/libexpat.la -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.la*
lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/a3ixqfx1cfjgbz5gb925fv7dwy2h55gs-fhs-union-0.0/lib/libexpat.so -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so*
lrwxrwxrwx 1 root root 73 Dec 31  1969 /gnu/store/a3ixqfx1cfjgbz5gb925fv7dwy2h55gs-fhs-union-0.0/lib/libexpat.so.1 -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1*
lrwxrwxrwx 1 root root 77 Dec 31  1969 /gnu/store/a3ixqfx1cfjgbz5gb925fv7dwy2h55gs-fhs-union-0.0/lib/libexpat.so.1.8.1 -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1.8.1

Note that the last link is broken: looking at this expat library (2.4.3, the replacement) it has

-r-xr-xr-x 1 root root  961 Dec 31  1969 /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.la*
lrwxrwxrwx 1 root root   17 Dec 31  1969 /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so -> libexpat.so.1.8.3*
lrwxrwxrwx 1 root root   17 Dec 31  1969 /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1 -> libexpat.so.1.8.3*
-r-xr-xr-x 2 root root 191K Dec 31  1969 /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1.8.3*

This is why the last link from the union-build package is broken, it tries to point to libexpat.so.1.8.1 (the previous version, but in the newer version's directory).

There are no error or warnings that I saw in building the package or in a profile (e.g. guix shell). I put in an output in the builder to see what directories were passed to the union-build, giving me

directories: (/gnu/store/d0xskl9xjb0z6miaf87xszi8iq6h8sqs-apr-util-1.6.1 /gnu/store/iwcw80p8lkqsqbvchjvypvl06qlbjc3d-expat-2.4.1)

So the original expat propagated-input store directory is passed to the builder (as expected), but in the end it does use the replacement version. Just with a version number gone wrong in a library link.

I came across this actually when combining both x86_64 and i686 into a single profile. There it would error on a conflict between file and directory, but in the lib/cmake of expat, again something with different version numbers appearing but between x86_64 and i686. I don't have an example of that handy right now, but it sent me in the direction of seeing why expat was coming up, when I came across this (I'm guessing leading to the problem I saw).

In case it is relevant, I did notice other packages have had explicit replacements used in propagated inputs previously, like in https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c5df560fd3762c0dbe99562f52223c73d445e597 expat is propagated by fontconfig and apr-util, possibly other places? A bit tricky to search with the expat license and all.

This was on commit: e415a73d2c11e43f4db3b797ed73ab6ebcc336e0 My original problem (conflict in x86_64 and i686 union) happened after the expat replacement commit, 2045852b096131a714409aa0cc4fe17938f60b15 which was my first clue.

I hope that is helpful in reproducing and finding out what is happening. Something to do with union-build I guess, but it is very late here right now so I hope this all made sense. Happy to help investigate and debug.

John

[graft-test.scm (text/x-scheme, attachment)]

Message #8 received at 53406 <at> debbugs.gnu.org (full text, mbox):

From: John Kehayias <john.kehayias <at> protonmail.com>
To: "53406 <at> debbugs.gnu.org" <53406 <at> debbugs.gnu.org>
Subject: Re: union-build incorrectly handles grafts
Date: Fri, 21 Jan 2022 06:54:57 +0000

Oh, and in the share/doc directory of the union:

lrwxrwxrwx 1 root root 71 Dec 31  1969 expat -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/share/doc/expat/
lrwxrwxrwx 1 root root 77 Dec 31  1969 expat-2.4.1 -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/share/doc/expat-2.4.1

Second link broken, first link correct, both to the replacement (2.4.3 version) store directory.

Message #11 received at 53406 <at> debbugs.gnu.org (full text, mbox):

From: John Kehayias <john.kehayias <at> protonmail.com>
To: "53406 <at> debbugs.gnu.org" <53406 <at> debbugs.gnu.org>
Subject: Re: union-build incorrectly handles grafts
Date: Sat, 22 Jan 2022 16:56:31 +0000

I haven't had the chance to see what is happening inside union-build yet, but did confirm that propagating expat/fixed instead of expat does not have the same problem.

I tried with

(define fontconfig-fixed
  (package
    (inherit fontconfig)
    (propagated-inputs
     (modify-inputs (package-propagated-inputs fontconfig)
       (replace "expat" (@@ (gnu packages xml) expat/fixed))))))

and then (fhs-union (list fontconfig-fixed)) did not have any broken links trying to go to the wrong version. Using this also works around my original problem of a multi-arch union build having collisions (so that seems to stem from whatever is happening here).

Message #14 received at 53406 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: John Kehayias <john.kehayias <at> protonmail.com>
Cc: 53406 <at> debbugs.gnu.org
Subject: Re: bug#53406: union-build incorrectly handles grafts
Date: Mon, 24 Jan 2022 15:18:42 +0100

Hi,

John Kehayias <john.kehayias <at> protonmail.com> skribis:

> If I run the attached file with guix build -f graft-test.scm and look at libexpat in the created package with, e.g.
>
> ls -la $(guix build -f graft-test.scm)/lib/libexpat*
>
> I get
>
> lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/a3ixqfx1cfjgbz5gb925fv7dwy2h55gs-fhs-union-0.0/lib/libexpat.la -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.la*
> lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/a3ixqfx1cfjgbz5gb925fv7dwy2h55gs-fhs-union-0.0/lib/libexpat.so -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so*
> lrwxrwxrwx 1 root root 73 Dec 31  1969 /gnu/store/a3ixqfx1cfjgbz5gb925fv7dwy2h55gs-fhs-union-0.0/lib/libexpat.so.1 -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1*
> lrwxrwxrwx 1 root root 77 Dec 31  1969 /gnu/store/a3ixqfx1cfjgbz5gb925fv7dwy2h55gs-fhs-union-0.0/lib/libexpat.so.1.8.1 -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1.8.1
>
> Note that the last link is broken: looking at this expat library (2.4.3, the replacement) it has
>
> -r-xr-xr-x 1 root root  961 Dec 31  1969 /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.la*
> lrwxrwxrwx 1 root root   17 Dec 31  1969 /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so -> libexpat.so.1.8.3*
> lrwxrwxrwx 1 root root   17 Dec 31  1969 /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1 -> libexpat.so.1.8.3*
> -r-xr-xr-x 2 root root 191K Dec 31  1969 /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1.8.3*
>
> This is why the last link from the union-build package is broken, it tries to point to libexpat.so.1.8.1 (the previous version, but in the newer version's directory).

Grafting is a pretty basic process: in this case it replaces occurrences
of /gnu/store/…-expat-2.4.1 with /gnu/store/…-expat-2.4.3, nothing more.
It cannot guess that libexpat.so.1.8.1 was renamed to libexpat.so.1.8.3
or anything like that.

Is it a problem?  Normally no, because users of shared libraries don’t
refer to libraries by their fully-qualified name:

--8<---------------cut here---------------start------------->8---
$ objdump -x $(guix build dbus-glib)/bin/dbus-binding-tool|grep NEED.*expat
  NEEDED               libexpat.so.1
$ objdump -x  $(guix build dbus-glib)/bin/dbus-binding-tool|grep RUNPATH
  RUNPATH              /gnu/store/wwmxxlmlhwljn39z0gsj6iai3zk67a2g-dbus-glib-0.110/lib:/gnu/store/5s6iz5f777rh23q4kv8gvqrsyy61cbjh-dbus-1.12.20/lib:/gnu/store/s0w7szfsajdy6cnrz2w7z4h5spyl4aaj-expat-2.4.1/lib:/gnu/store/2fk1gz2s7ppdicynscra9b19byrrr866-glibc-2.33/lib:/gnu/store/90lbavffg0csrf208nw0ayj1bz5knl47-gcc-10.3.0-lib/lib:/gnu/store/qqs98rxwjrji6aaf6dqwp7q4m545g2sn-glib-2.70.0/lib:/gnu/store/90lbavffg0csrf208nw0ayj1bz5knl47-gcc-10.3.0-lib/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../..
--8<---------------cut here---------------end--------------->8---

Likewise, ‘etc/ld.so.cache’ contains a reference to ‘libexpat.so.1’, not
to ‘libexpat.so.1.8.1’.

Does that make sense?  Or am I overlooking something?

Thanks,
Ludo’.

Message #17 received at 53406 <at> debbugs.gnu.org (full text, mbox):

From: John Kehayias <john.kehayias <at> protonmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 53406 <at> debbugs.gnu.org
Subject: Re: bug#53406: union-build incorrectly handles grafts
Date: Tue, 25 Jan 2022 03:22:44 +0000

[Message part 1 (text/plain, inline)]

Hi Ludo’,

Thanks for explaining!

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Monday, January 24th, 2022 at 9:18 AM, Ludovic Courtès wrote:

>
> Grafting is a pretty basic process: in this case it replaces occurrences
> of /gnu/store/…-expat-2.4.1 with /gnu/store/…-expat-2.4.3, nothing more.
> It cannot guess that libexpat.so.1.8.1 was renamed to libexpat.so.1.8.3
> or anything like that.
>
> Is it a problem? Normally no, because users of shared libraries don’t
> refer to libraries by their fully-qualified name:
>
> --8<---------------cut here---------------start------------->8---
> $ objdump -x $(guix build dbus-glib)/bin/dbus-binding-tool|grep NEED.*expat
> NEEDED libexpat.so.1
> $ objdump -x $(guix build dbus-glib)/bin/dbus-binding-tool|grep RUNPATH
> RUNPATH /gnu/store/wwmxxlmlhwljn39z0gsj6iai3zk67a2g-dbus-glib-0.110/lib:/gnu/store/5s6iz5f777rh23q4kv8gvqrsyy61cbjh-dbus-1.12.20/lib:/gnu/store/s0w7szfsajdy6cnrz2w7z4h5spyl4aaj-expat-2.4.1/lib:/gnu/store/2fk1gz2s7ppdicynscra9b19byrrr866-glibc-2.33/lib:/gnu/store/90lbavffg0csrf208nw0ayj1bz5knl47-gcc-10.3.0-lib/lib:/gnu/store/qqs98rxwjrji6aaf6dqwp7q4m545g2sn-glib-2.70.0/lib:/gnu/store/90lbavffg0csrf208nw0ayj1bz5knl47-gcc-10.3.0-lib/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../..
> --8<---------------cut here---------------end--------------->8---
>
> Likewise, ‘etc/ld.so.cache’ contains a reference to ‘libexpat.so.1’, not
> to ‘libexpat.so.1.8.1’.
>

(Your example output was not of a grafted libexpat in the RUNPATH, but point taken; I see the newer expat version on my (grafted) dbus-glib).

> Does that make sense? Or am I overlooking something?
>

Yes, that makes sense, thank you for clarifying. So this is the currently expected behavior. Ideally grafting would be smarter to maybe avoid this (missing changes in e.g. version number)? But I would guess this is not something that would be expected to cause a problem for the vast majority of cases, as you explain, and adds complexity to the process.

I'm glad to hear it all works! But...

Perhaps I was too hasty in noting this "problem" which like I said was not the error I originally encountered. I was using a package that constructs both the 64- and 32-bit libraries to put in a container (say, a /lib32 and /lib64 or something similar to an FHS environment). A collision was happening between a file and directory, one being a good symlink and the other broken, rather than a "real" mismatch in file vs directory. Anyway, going back to that what I see is that one link is broken for the above reasons, but the good one is good because it is to the *ungrafted* library store path. I don't know now if these 2 things are connected other than one led me to the other, but I turn now to what demonstrates my original problem.

I don't know why this happens or if it is something in this building process that is not correct, but I did come up with a minimal example (attached). The code is a bit odd in its stripped down form, though hopefully is clear in what way this would be used to do something useful (again, like an FHS environment or other container). Apologies for the old style and lack of gexps which I'm finally getting used to. The example package just tries to make a dummy package that has, for illustration, a "/lib64" and "/lib32" which link to the respective union-build inputs (of a single library for simplicity). I don't think the actual package being made matters so much, or how it is constructed, but that two inputs are union-builds of the same library (x86_64 and i686) which should have a graft of expat. Just my guess though.

Doing:

ls -la $(guix build -f graft-test.scm)/lib64/lib/libexpat*
lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.la -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.la*
lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.so -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so*
lrwxrwxrwx 1 root root 73 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.so.1 -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1*
lrwxrwxrwx 1 root root 77 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.so.1.8.1 -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1.8.1

is what we saw already: libexpat is the newer (replacement, 2.4.3) version, with the full version symlink broken since the version number is wrong. Likewise in other pieces that have the version number, like share/doc. Okay, that's expected. But now, in the i686-linux union-build input:

ls -la $(guix build -f graft-test.scm)/lib32/lib/libexpat*
lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.la -> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.la*
lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.so -> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.so*
lrwxrwxrwx 1 root root 73 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.so.1 -> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.so.1*
lrwxrwxrwx 1 root root 77 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.so.1.8.1 -> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.so.1.8.1*

all the links are good and to the original (version 2.4.1) expat. In other words, the constructed union64 and union32 inputs (in the sample code) do not both get grafted, even though doing just the fhs-union command on it's own (not building both for another package) does graft for either architecture. At least that seems like the most obvious difference between the earlier example and this new one.

Why? Does the grafting just happen "once" somehow and misses the "same" input again (but built for different system)? Is this expected or just a weird/wrong way to do this kind of build which is causing this? I'm not sure if this is just with union-build or if it would happen just with inputs of the same library but different architectures. I didn't know how to do that quickly off hand, so I haven't tried it yet.

Thanks for taking the time to look and explain, much appreciated!

John

[graft-test.scm (text/x-scheme, attachment)]

Message #20 received at 53406 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: John Kehayias <john.kehayias <at> protonmail.com>
Cc: 53406 <at> debbugs.gnu.org
Subject: Re: bug#53406: union-build incorrectly handles grafts
Date: Tue, 25 Jan 2022 14:47:41 +0100

Hi John,

John Kehayias <john.kehayias <at> protonmail.com> skribis:

> Yes, that makes sense, thank you for clarifying. So this is the currently expected behavior. Ideally grafting would be smarter to maybe avoid this (missing changes in e.g. version number)? But I would guess this is not something that would be expected to cause a problem for the vast majority of cases, as you explain, and adds complexity to the process.

I don’t think it can be smarter; grafting is just text substitution, it
knows nothing about the files it’s fiddling with (with the exception of
debugging sections in ELF files).

> Perhaps I was too hasty in noting this "problem" which like I said was not the error I originally encountered. I was using a package that constructs both the 64- and 32-bit libraries to put in a container (say, a /lib32 and /lib64 or something similar to an FHS environment). A collision was happening between a file and directory, one being a good symlink and the other broken, rather than a "real" mismatch in file vs directory. Anyway, going back to that what I see is that one link is broken for the above reasons, but the good one is good because it is to the *ungrafted* library store path. I don't know now if these 2 things are connected other than one led me to the other, but I turn now to what demonstrates my original problem.
>
> I don't know why this happens or if it is something in this building process that is not correct, but I did come up with a minimal example (attached). The code is a bit odd in its stripped down form, though hopefully is clear in what way this would be used to do something useful (again, like an FHS environment or other container). Apologies for the old style and lack of gexps which I'm finally getting used to. The example package just tries to make a dummy package that has, for illustration, a "/lib64" and "/lib32" which link to the respective union-build inputs (of a single library for simplicity). I don't think the actual package being made matters so much, or how it is constructed, but that two inputs are union-builds of the same library (x86_64 and i686) which should have a graft of expat. Just my guess though.
>
> Doing:
>
> ls -la $(guix build -f graft-test.scm)/lib64/lib/libexpat*
> lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.la -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.la*
> lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.so -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so*
> lrwxrwxrwx 1 root root 73 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.so.1 -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1*
> lrwxrwxrwx 1 root root 77 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.so.1.8.1 -> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1.8.1
>
> is what we saw already: libexpat is the newer (replacement, 2.4.3) version, with the full version symlink broken since the version number is wrong. Likewise in other pieces that have the version number, like share/doc. Okay, that's expected. But now, in the i686-linux union-build input:
>
> ls -la $(guix build -f graft-test.scm)/lib32/lib/libexpat*
> lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.la -> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.la*
> lrwxrwxrwx 1 root root 71 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.so -> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.so*
> lrwxrwxrwx 1 root root 73 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.so.1 -> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.so.1*
> lrwxrwxrwx 1 root root 77 Dec 31  1969 /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.so.1.8.1 -> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.so.1.8.1*
>
> all the links are good and to the original (version 2.4.1) expat. In other words, the constructed union64 and union32 inputs (in the sample code) do not both get grafted, even though doing just the fhs-union command on it's own (not building both for another package) does graft for either architecture. At least that seems like the most obvious difference between the earlier example and this new one.
>
> Why? Does the grafting just happen "once" somehow and misses the "same" input again (but built for different system)? Is this expected or just a weird/wrong way to do this kind of build which is causing this? I'm not sure if this is just with union-build or if it would happen just with inputs of the same library but different architectures. I didn't know how to do that quickly off hand, so I haven't tried it yet.

Woow, it’s a sophisticated example.  :-)

I don’t actually have the Expat replacement we’re talking about so I
can’t easily test.

At first sight I’d say it should work (both lib32 and lib64 should refer
to the Expat replacement), but it could be that something somewhere
assumes all the packages are built for the system.  That would need more
investigation work in (guix packages) in (guix grafts).

Thanks,
Ludo’.

Message #25 received at 53406 <at> debbugs.gnu.org (full text, mbox):

From: John Kehayias <john.kehayias <at> protonmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 53406 <at> debbugs.gnu.org
Subject: Re: bug#53406: union-build incorrectly handles grafts
Date: Sun, 15 Oct 2023 20:13:19 +0000

Hi Ludo’,

Long time since I've thought about this bug, but with all the recent
grafts I thought to return to it. I'll have to make a new example to
dig again, but wanted to think through where we might look to see
what's happening first.

On Tue, Jan 25, 2022 at 02:47 PM, Ludovic Courtès wrote:

> Hi John,
>
> John Kehayias <john.kehayias <at> protonmail.com> skribis:
>
[...]
>> Perhaps I was too hasty in noting this "problem" which like I said
>> was not the error I originally encountered. I was using a package
>> that constructs both the 64- and 32-bit libraries to put in a
>> container (say, a /lib32 and /lib64 or something similar to an FHS
>> environment). A collision was happening between a file and
>> directory, one being a good symlink and the other broken, rather
>> than a "real" mismatch in file vs directory. Anyway, going back to
>> that what I see is that one link is broken for the above reasons,
>> but the good one is good because it is to the *ungrafted* library
>> store path. I don't know now if these 2 things are connected other
>> than one led me to the other, but I turn now to what demonstrates my
>> original problem.
>>
>> I don't know why this happens or if it is something in this building
>> process that is not correct, but I did come up with a minimal
>> example (attached). The code is a bit odd in its stripped down form,
>> though hopefully is clear in what way this would be used to do
>> something useful (again, like an FHS environment or other
>> container). Apologies for the old style and lack of gexps which I'm
>> finally getting used to. The example package just tries to make a
>> dummy package that has, for illustration, a "/lib64" and "/lib32"
>> which link to the respective union-build inputs (of a single library
>> for simplicity). I don't think the actual package being made matters
>> so much, or how it is constructed, but that two inputs are
>> union-builds of the same library (x86_64 and i686) which should have
>> a graft of expat. Just my guess though.
>>
>> Doing:
>>
>> ls -la $(guix build -f graft-test.scm)/lib64/lib/libexpat*
>> lrwxrwxrwx 1 root root 71 Dec 31 1969
>> /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.la
>> ->
>> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.la*
>> lrwxrwxrwx 1 root root 71 Dec 31 1969
>> /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.so
>> ->
>> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so*
>> lrwxrwxrwx 1 root root 73 Dec 31 1969
>> /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.so.1
>> ->
>> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1*
>> lrwxrwxrwx 1 root root 77 Dec 31 1969
>> /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib64/lib/libexpat.so.1.8.1
>> ->
>> /gnu/store/2q8wwhd3prib0swky68rbx9hl0xxs6hf-expat-2.4.3/lib/libexpat.so.1.8.1
>>
>> is what we saw already: libexpat is the newer (replacement, 2.4.3)
>> version, with the full version symlink broken since the version
>> number is wrong. Likewise in other pieces that have the version
>> number, like share/doc. Okay, that's expected. But now, in the
>> i686-linux union-build input:
>>
>> ls -la $(guix build -f graft-test.scm)/lib32/lib/libexpat*
>> lrwxrwxrwx 1 root root 71 Dec 31 1969
>> /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.la
>> ->
>> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.la*
>> lrwxrwxrwx 1 root root 71 Dec 31 1969
>> /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.so
>> ->
>> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.so*
>> lrwxrwxrwx 1 root root 73 Dec 31 1969
>> /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.so.1
>> ->
>> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.so.1*
>> lrwxrwxrwx 1 root root 77 Dec 31 1969
>> /gnu/store/qbh16hfdv8pnfw01k6izbs3jkji6i978-test-pkg-0.0/lib32/lib/libexpat.so.1.8.1
>> ->
>> /gnu/store/b0jns3vzhhpna7lim8bc3dr0payzx5yy-expat-2.4.1/lib/libexpat.so.1.8.1*
>>
>> all the links are good and to the original (version 2.4.1) expat. In
>> other words, the constructed union64 and union32 inputs (in the
>> sample code) do not both get grafted, even though doing just the
>> fhs-union command on it's own (not building both for another
>> package) does graft for either architecture. At least that seems
>> like the most obvious difference between the earlier example and
>> this new one.
>>
>> Why? Does the grafting just happen "once" somehow and misses the
>> "same" input again (but built for different system)? Is this
>> expected or just a weird/wrong way to do this kind of build which is
>> causing this? I'm not sure if this is just with union-build or if it
>> would happen just with inputs of the same library but different
>> architectures. I didn't know how to do that quickly off hand, so I
>> haven't tried it yet.
>
> Woow, it’s a sophisticated example.  :-)
>

This is potentially relevant again since I wanted to see about adding
multi-arch support to guix shell --emulate-fhs. Likely there will be a
collision in the profile created for this same reason. (These days
from say libx11.)

> I don’t actually have the Expat replacement we’re talking about so I
> can’t easily test.
>
> At first sight I’d say it should work (both lib32 and lib64 should refer
> to the Expat replacement), but it could be that something somewhere
> assumes all the packages are built for the system.  That would need more
> investigation work in (guix packages) in (guix grafts).
>

Taking a quick look, and not knowing much about graft internals, is
the place to look more in (guix packages) bag-grafts, input-graft,
package->derivation maybe?

From what you said, and my guess as well, is that mixed #:system in
the inputs is somehow "missed" by grafting. Maybe just using the same
#:system as the package? If that's the case, grafting needs to check
the system of each input package rather than assuming it follows from
the package, would that make sense? Maybe some clues in how
cross-compiling works, or some package we have that does mix systems
(like dxvk)?

I'll reconstruct an example or maybe see if dxvk sheds some light with
grafts, though looks like it is currently broken first...

Thanks!
John

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.