GNU bug report logs - #30229
Python modules installed by pip in virtualenv can't find shared objects.

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: zimoun <zimon.toutoune <at> gmail.com>
Cc: tracker <at> debbugs.gnu.org
Subject: bug#30229: closed (Python modules installed by pip in virtualenv
 can't find shared objects.)
Date: Thu, 13 Jan 2022 15:03:02 +0000

[Message part 1 (text/plain, inline)]

Your message dated Thu, 13 Jan 2022 15:59:42 +0100
with message-id <864k67k7fl.fsf <at> gmail.com>
and subject line Re: bug#30229: Python modules installed by pip in virtualenv can't find shared objects.
has caused the debbugs.gnu.org bug report #30229,
regarding Python modules installed by pip in virtualenv can't find shared objects.
to be marked as done.

(If you believe you have received this mail in error, please contact
help-debbugs <at> gnu.org.)


-- 
30229: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=30229
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Fis Trivial <ybbs.daans <at> hotmail.com>
To: "bug-guix <at> gnu.org" <bug-guix <at> gnu.org>
Subject: Python modules installed by pip in virtualenv can't find shared
 objects.
Date: Tue, 23 Jan 2018 11:49:59 +0000

When a python module needs to load a dynamic shared object, it looks in the
path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this environment
variable to export the needed path for python.

* Backtrace
In my case, it's lightgbm (installed by pip), needs libgomp.so from gcc:lib for
openmp support. Here is the backtrace:

--8<---------------cut here---------------start------------->8---
Traceback (most recent call last):
  File "ex1.py", line 5, in <module>
    import lightgbm as lgb
  File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/__init__.py", line 8, in <module>
    from .basic import Booster, Dataset
  File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 32, in <module>
    _LIB = _load_lib()
  File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 27, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 425, in LoadLibrary
    return self._dlltype(name)
  File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 347, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libgomp.so.1: cannot open shared object file: No such file or directory
--8<---------------cut here---------------end--------------->8---

* Reproduce
To install lightgbm, simply use `pip install lightgbm`(in virtualenv).
Then in python shell:
import lightgbm as lgb

* Ad-hoc
export LD_LIBRARY_PATH=~/.guix-profile/lib:$LD_LIBRARY_PATH


[1]: https://stackoverflow.com/a/1100016

[Message part 3 (message/rfc822, inline)]

From: zimoun <zimon.toutoune <at> gmail.com>
To: Ricardo Wurmus <rekado <at> elephly.net>
Cc: Fis Trivial <ybbs.daans <at> hotmail.com>, 30229-done <at> debbugs.gnu.org
Subject: Re: bug#30229: Python modules installed by pip in virtualenv can't
 find shared objects.
Date: Thu, 13 Jan 2022 15:59:42 +0100

Hi,

On Thu, 25 Nov 2021 at 00:39, zimoun <zimon.toutoune <at> gmail.com> wrote:
> On Tue, 23 Jan 2018 at 13:00, Ricardo Wurmus <rekado <at> elephly.net> wrote:
>
>>> When a python module needs to load a dynamic shared object, it looks in the
>>> path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this
>>> environment
>>> variable to export the needed path for python.
>>
>> We cannot set this environment variable by default lest we break other
>> packages that may be installed.  LD_LIBRARY_PATH is dangerous as it
>> tells the runtime linker to prefer libraries in the specified
>> directories.
>>
>> For Guix packages we use different means to embed store paths in
>> binaries, which are looked up at runtime.  For binaries that’s achieved
>> with RUNPATH; for others we patch the sources to ensure that libraries
>> are not looked up merely by name but by absolute path.
>>
>> In your particular case (installing packages without Guix) I think the
>> best way is to manually set LD_LIBRARY_PATH on demand, or to set
>> LD_PRELOAD to the specific libraries that are required.
>>
>> In general, though, I recommend using Guix for package management and
>> development instead of virtualenv and pip.
>
> Regarding the improvements of ’guix import pypi’ since 2018, and because
> tweaking LD_LIBRARY_PATH is dangerous, I do not see what could be the
> next action to solve this.
>
> Therefore, I propose to close it.  WDYT?

After 7 weeks of delay, I am closing.


Cheers,
simon

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.