GNU bug report logs - #64309
Python dlopen()s musl libc

Previous Next

To reply to this bug, email your comments to 64309 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Athena Martin <secure <at> alm.website>
To: bug-guix <at> gnu.org
Subject: Python dlopen()s musl libc
Date: Mon, 26 Jun 2023 20:42:30 -0400

[Message part 1 (text/plain, inline)]

I've had experiences now with multiple Guix packages, including gajim
(bug 60235) and now python-neovim-remote, which have an issue where
Python tries to dlopen() libc, but finds the system libc instead of
Guix's, resulting on Alpine Linux hosts in a crash with this message:

ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

There are a variety of tracebacks that lead up to this, depending on
the package in question.

[Message part 2 (application/pgp-signature, inline)]

Message #8 received at 64309 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Athena Martin <secure <at> alm.website>
Cc: 64309 <at> debbugs.gnu.org
Subject: Re: bug#64309: Python dlopen()s musl libc
Date: Fri, 07 Jul 2023 16:00:12 +0200

Hi,

Athena Martin <secure <at> alm.website> skribis:

> I've had experiences now with multiple Guix packages, including gajim
> (bug 60235) and now python-neovim-remote, which have an issue where
> Python tries to dlopen() libc, but finds the system libc instead of
> Guix's, resulting on Alpine Linux hosts in a crash with this message:
>
> ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory
>
> There are a variety of tracebacks that lead up to this, depending on
> the package in question.

Could you provide a command to reproduce this?

Thanks in advance,
Ludo’.

Message #13 received at 64309 <at> debbugs.gnu.org (full text, mbox):

From: Athena Martin <secure <at> alm.website>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 64309 <at> debbugs.gnu.org
Subject: Re: bug#64309: Python dlopen()s musl libc
Date: Sat, 8 Jul 2023 16:16:58 -0400

[Message part 1 (text/plain, inline)]

> > ImportError: libc.musl-x86_64.so.1: cannot open shared object file:
> > No such file or directory
> 
> Could you provide a command to reproduce this?

On an Alpine Linux Edge host, I reproduce with:

$ guix install python-neovim-remote
$ nvr

It's possible that there's some quirk of my specific environment.

(I'm currently in the process of switching to Guix System so I may not
be able to reproduce this forever.)

[Message part 2 (application/pgp-signature, inline)]

Message #16 received at 64309 <at> debbugs.gnu.org (full text, mbox):

From: Josselin Poiret <dev <at> jpoiret.xyz>
To: Athena Martin <secure <at> alm.website>, Ludovic Courtès
 <ludo <at> gnu.org>
Cc: 64309 <at> debbugs.gnu.org
Subject: Re: bug#64309: Python dlopen()s musl libc
Date: Sun, 09 Jul 2023 10:35:03 +0200

[Message part 1 (text/plain, inline)]

Hi Athena,

Athena Martin via Bug reports for GNU Guix <bug-guix <at> gnu.org> writes:

> On an Alpine Linux Edge host, I reproduce with:
>
> $ guix install python-neovim-remote
> $ nvr
>
> It's possible that there's some quirk of my specific environment.
>
> (I'm currently in the process of switching to Guix System so I may not
> be able to reproduce this forever.)

Can you do `LD_DEBUG=libs nvr` so that we get a log of what ld's trying
to load?

Best,
-- 
Josselin Poiret

[signature.asc (application/pgp-signature, inline)]

Message #19 received at 64309 <at> debbugs.gnu.org (full text, mbox):

From: Athena Martin <secure <at> alm.website>
To: Josselin Poiret <dev <at> jpoiret.xyz>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 64309 <at> debbugs.gnu.org
Subject: Re: bug#64309: Python dlopen()s musl libc
Date: Sun, 9 Jul 2023 16:22:21 -0400

[Message part 1 (text/plain, inline)]

> Can you do `LD_DEBUG=libs nvr` so that we get a log of what ld's
> trying to load?

I've attached the full log. The first mention of musl is on line 200 and
everything seems to happen pretty fast, here's the most relevant
portion:

     16405:	find library=libc.musl-x86_64.so.1 [0]; searching
     16405:	 search cache=/gnu/store/kj6wzba6p192baizq99b489rs8bynpn7-python-3.10.7/etc/ld.so.cache
     16405:	 search path=/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib		(system search path)
     16405:	  trying file=/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.musl-x86_64.so.1
     16405:	
Traceback (most recent call last):
  File "/gnu/store/dsgxdqs620pp284bfm1drbsjqpb36i4n-python-neovim-remote-2.5.1/bin/.nvr-real", line 4, in <module>
    import nvr.nvr as mod
  File "/home/alm/.local/lib/python3.10/site-packages/nvr/__init__.py", line 1, in <module>
    from .nvr import main
  File "/home/alm/.local/lib/python3.10/site-packages/nvr/nvr.py", line 34, in <module>
    import psutil
  File "/home/alm/.local/lib/python3.10/site-packages/psutil/__init__.py", line 102, in <module>
    from . import _pslinux as _psplatform
  File "/home/alm/.local/lib/python3.10/site-packages/psutil/_pslinux.py", line 26, in <module>
    from . import _psutil_linux as cext
ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

After that it starts calling fini()s and terminates.

[stderr.log (text/plain, attachment)]

[Message part 3 (application/pgp-signature, inline)]

Message #22 received at 64309 <at> debbugs.gnu.org (full text, mbox):

From: Josselin Poiret <dev <at> jpoiret.xyz>
To: Athena Martin <secure <at> alm.website>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 64309 <at> debbugs.gnu.org
Subject: Re: bug#64309: Python dlopen()s musl libc
Date: Mon, 10 Jul 2023 09:13:32 +0200

[Message part 1 (text/plain, inline)]

Hi Athena,

So it's not the LD_DEBUG output that hold a clue, but rather the Python
traceback.

Athena Martin <secure <at> alm.website> writes:

>   File "/gnu/store/dsgxdqs620pp284bfm1drbsjqpb36i4n-python-neovim-remote-2.5.1/bin/.nvr-real", line 4, in <module>
>     import nvr.nvr as mod
>   File "/home/alm/.local/lib/python3.10/site-packages/nvr/__init__.py", line 1, in <module>
>     from .nvr import main
>   File "/home/alm/.local/lib/python3.10/site-packages/nvr/nvr.py", line 34, in <module>
>     import psutil
>   File "/home/alm/.local/lib/python3.10/site-packages/psutil/__init__.py", line 102, in <module>
>     from . import _pslinux as _psplatform
>   File "/home/alm/.local/lib/python3.10/site-packages/psutil/_pslinux.py", line 26, in <module>
>     from . import _psutil_linux as cext
> ImportError: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory

The nvr package in ~/.local seems to be used instead of a Guix package.
That locally installed nvr package expects to use the host's libc, but
since the python interpreter being used has a fixed RPATH and system
search path it won't find it.

.nvr-real should definitely be using the Python code inside the store, I
wonder why that isn't being done.  Maybe our sitecustomize.py is
misbehaving?  Can you do `guix shell python-neovim-remote python --
python3` then type `import sys.path; sys.path`?

Best,
-- 
Josselin Poiret

[signature.asc (application/pgp-signature, inline)]

Message #25 received at 64309 <at> debbugs.gnu.org (full text, mbox):

From: Athena Martin <secure <at> alm.website>
To: Josselin Poiret <dev <at> jpoiret.xyz>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 64309 <at> debbugs.gnu.org
Subject: Re: bug#64309: Python dlopen()s musl libc
Date: Mon, 10 Jul 2023 15:47:34 -0400

[Message part 1 (text/plain, inline)]

> So it's not the LD_DEBUG output that hold a clue, but rather the
> Python traceback.

> The nvr package in ~/.local seems to be used instead of a Guix
> package. That locally installed nvr package expects to use the host's
> libc, but since the python interpreter being used has a fixed RPATH
> and system search path it won't find it.

Ah, I see.

> .nvr-real should definitely be using the Python code inside the
> store, I wonder why that isn't being done.  Maybe our
> sitecustomize.py is misbehaving?  Can you do `guix shell
> python-neovim-remote python -- python3` then type `import sys.path;
> sys.path`?

$ guix shell python-neovim-remote python -- python3
>>> sys.path
['', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload', '/home/alm/.local/lib/python3.10/site-packages', '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']

(It's not from the environment:)

$ guix shell python-neovim-remote python --pure -- python3
>>> sys.path
['', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload', '/home/alm/.local/lib/python3.10/site-packages', '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']
>>> os.environ
environ({'HOME': '/home/alm', 'LOGNAME': 'alm', 'PAGER': 'less', 'DISPLAY': ':0', 'USER': 'alm', 'TERM': 'xterm-256color', 'PATH': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/bin', 'GUIX_PYTHONPATH': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', 'GUIX_ENVIRONMENT': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile'})

[Message part 2 (application/pgp-signature, inline)]

Message #28 received at 64309 <at> debbugs.gnu.org (full text, mbox):

From: Athena Martin <secure <at> alm.website>
To: Josselin Poiret <dev <at> jpoiret.xyz>
Cc: Ludovic Courtès <ludo <at> gnu.org>, 64309 <at> debbugs.gnu.org
Subject: Re: bug#64309: Python dlopen()s musl libc
Date: Mon, 10 Jul 2023 16:21:22 -0400

[Message part 1 (text/plain, inline)]

> The nvr package in ~/.local seems to be used instead of a Guix
> package. That locally installed nvr package expects to use the
> host's libc, but since the python interpreter being used has a
> fixed RPATH and system search path it won't find it.  

I've just checked and temporarily removing .local/lib/python3.10 makes
Guix's nvr work, and takes the .local site-packages off sys.path.

[Message part 2 (application/pgp-signature, inline)]

Message #31 received at 64309 <at> debbugs.gnu.org (full text, mbox):

From: Josselin Poiret <dev <at> jpoiret.xyz>
To: Athena Martin <secure <at> alm.website>
Cc: jgart <at> dismail.de, Ludovic Courtès <ludo <at> gnu.org>,
 lars <at> 6xq.net, 64309 <at> debbugs.gnu.org
Subject: Re: bug#64309: Python dlopen()s musl libc
Date: Tue, 11 Jul 2023 10:34:06 +0200

[Message part 1 (text/plain, inline)]

Hi Athena,

Athena Martin <secure <at> alm.website> writes:

> $ guix shell python-neovim-remote python -- python3
>>>> sys.path
> ['', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload', '/home/alm/.local/lib/python3.10/site-packages', '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']
>
> (It's not from the environment:)
>
> $ guix shell python-neovim-remote python --pure -- python3
>>>> sys.path
> ['', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python310.zip', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/lib-dynload', '/home/alm/.local/lib/python3.10/site-packages', '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', '/gnu/store/dy3xh053ahkhrp2jamggq8cpsyvp8mg0-python-3.10.7/lib/python3.10/site-packages']
>>>> os.environ
> environ({'HOME': '/home/alm', 'LOGNAME': 'alm', 'PAGER': 'less', 'DISPLAY': ':0', 'USER': 'alm', 'TERM': 'xterm-256color', 'PATH': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/bin', 'GUIX_PYTHONPATH': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile/lib/python3.10/site-packages', 'GUIX_ENVIRONMENT': '/gnu/store/ll75wx2cvm1dbbxjr095lcs1653q2zz1-profile'})

So, looks like the default Python behavior is to load the usercustomize
after the sitecustomize [1], which leads to exactly the behavior you're
experiencing.  I don't know what we should do here, maybe pass `-s` to
the shebang line of Python to disable loading the usercustomize?  That
would probably be a world-rebuild though.  CC'ing the Python team to see
what they think.

[1] https://docs.python.org/3/library/site.html

Best,
-- 
Josselin Poiret

[signature.asc (application/pgp-signature, inline)]

Message #34 received at 64309 <at> debbugs.gnu.org (full text, mbox):

From: Lars-Dominik Braun <lars <at> 6xq.net>
To: Josselin Poiret <dev <at> jpoiret.xyz>
Cc: jgart <at> dismail.de, Ludovic Courtès <ludo <at> gnu.org>,
 Athena Martin <secure <at> alm.website>, 64309 <at> debbugs.gnu.org
Subject: Re: bug#64309: Python dlopen()s musl libc
Date: Tue, 11 Jul 2023 14:43:36 +0200

[Message part 1 (text/plain, inline)]

Hi,

there’s a similar issue #63912 and a thread on the guix-devel
mailinglist at
https://lists.gnu.org/archive/html/guix-devel/2023-06/msg00066.html

> So, looks like the default Python behavior is to load the usercustomize
> after the sitecustomize [1], which leads to exactly the behavior you're
> experiencing.  I don't know what we should do here, maybe pass `-s` to
> the shebang line of Python to disable loading the usercustomize?  That
> would probably be a world-rebuild though.  CC'ing the Python team to see
> what they think.

I think the problem is bigger than usercustomize. Any custom PYTHONPATH
also slips through and causes this issue, as well as any custom
GUIX_PYTHONPATH, because the executable wrapper appends it (think nested
`guix shell` invokations with different versions of a library for
an example where this could go wrong).

Guix-managed Python packages (libraries nor applications) should
generally not pick up dependencies from random paths – only those
from their package description, so we can keep Guix’ promise of being
self-contained.

I have experimented with customizing Python’s importing mechanism
through a custom MetaPathFinder. It works by adding a __guix_pythonpath__
variable to every Python package’s __init__.py file and modifying the
module loader’s search path accordingly if such a variable exists. It
would provide exactly that guarantee, but it’s just a PoC at this
point – see attached file.

Apart from that I don’t see a good short-term solution right now. It’s
just how Python works.

Cheers,
Lars

[mysitecustomize.py (text/plain, attachment)]

Message #37 received at 64309 <at> debbugs.gnu.org (full text, mbox):

From: Josselin Poiret <dev <at> jpoiret.xyz>
To: Lars-Dominik Braun <lars <at> 6xq.net>
Cc: jgart <at> dismail.de, Ludovic Courtès <ludo <at> gnu.org>,
 Athena Martin <secure <at> alm.website>, 64309 <at> debbugs.gnu.org
Subject: Re: bug#64309: Python dlopen()s musl libc
Date: Wed, 12 Jul 2023 10:50:40 +0200

[Message part 1 (text/plain, inline)]

Hi Lars,

Lars-Dominik Braun <lars <at> 6xq.net> writes:

> I think the problem is bigger than usercustomize. Any custom PYTHONPATH
> also slips through and causes this issue, as well as any custom
> GUIX_PYTHONPATH, because the executable wrapper appends it (think nested
> `guix shell` invokations with different versions of a library for
> an example where this could go wrong).
>
> Guix-managed Python packages (libraries nor applications) should
> generally not pick up dependencies from random paths – only those
> from their package description, so we can keep Guix’ promise of being
> self-contained.
>
> I have experimented with customizing Python’s importing mechanism
> through a custom MetaPathFinder. It works by adding a __guix_pythonpath__
> variable to every Python package’s __init__.py file and modifying the
> module loader’s search path accordingly if such a variable exists. It
> would provide exactly that guarantee, but it’s just a PoC at this
> point – see attached file.

Woah, looks like a neat solution.  Do you think it would scale for all
our Python packages without manual intervention?  If so, this would
definitely be the way forward.

> Apart from that I don’t see a good short-term solution right now. It’s
> just how Python works.

I mostly agree with you, but for this rather common case of having also
a usercustomize it would be nice to circumvent it.  In general, I don't
think we ever want a Guix-produced Python script to load the
usercustomize, hence my suggestion.  The other case of PYTHONPATH is
also annoying but can be tamed by modifying the env variable
temporarily.

Best,
-- 
Josselin Poiret

[signature.asc (application/pgp-signature, inline)]

Message #40 received at 64309 <at> debbugs.gnu.org (full text, mbox):

From: Lars-Dominik Braun <lars <at> 6xq.net>
To: Josselin Poiret <dev <at> jpoiret.xyz>
Cc: jgart <at> dismail.de, Ludovic Courtès <ludo <at> gnu.org>,
 Athena Martin <secure <at> alm.website>, 64309 <at> debbugs.gnu.org
Subject: Re: bug#64309: Python dlopen()s musl libc
Date: Sun, 16 Jul 2023 10:35:48 +0200

Hi Josselin,

> Woah, looks like a neat solution.  Do you think it would scale for all
> our Python packages without manual intervention?  If so, this would
> definitely be the way forward.

I hope so, but haven’t tried it yet.

> I mostly agree with you, but for this rather common case of having also
> a usercustomize it would be nice to circumvent it.  In general, I don't
> think we ever want a Guix-produced Python script to load the
> usercustomize, hence my suggestion.  The other case of PYTHONPATH is
> also annoying but can be tamed by modifying the env variable
> temporarily.

True, PYTHONPATH can be unset more easily than moving the user site
dir. I’ll have a look at #64573, which should work around this issue.

Cheers,
Lars

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.