Subject: bug#30229: closed (Re: bug#30229: Python modules installed by pip in virtualenv can't find shared objects.)

From: [email-masked] (GNU bug Tracking System)
Date: Thu, 13 Jan 2022 15:03:02 +0000

Your bug report

#30229: Python modules installed by pip in virtualenv can't find shared objects.

which was filed against the guix package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
30229: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=30229
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#30229: Python modules installed by pip in virtualenv can't find shared objects.
From: zimoun <[email-masked]>
To: Ricardo Wurmus <[email-masked]>
Cc: Fis Trivial <[email-masked]>, [email-masked]
Date: Thu, 13 Jan 2022 15:59:42 +0100
List-Id: <debbugs-submit.debbugs.gnu.org>

Hi,

On Thu, 25 Nov 2021 at 00:39, zimoun <[email-masked]> wrote:
> On Tue, 23 Jan 2018 at 13:00, Ricardo Wurmus <[email-masked]> wrote:
>
>>> When a python module needs to load a dynamic shared object, it looks in the
>>> path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this
>>> environment
>>> variable to export the needed path for python.
>>
>> We cannot set this environment variable by default lest we break other
>> packages that may be installed.  LD_LIBRARY_PATH is dangerous as it
>> tells the runtime linker to prefer libraries in the specified
>> directories.
>>
>> For Guix packages we use different means to embed store paths in
>> binaries, which are looked up at runtime.  For binaries that’s achieved
>> with RUNPATH; for others we patch the sources to ensure that libraries
>> are not looked up merely by name but by absolute path.
>>
>> In your particular case (installing packages without Guix) I think the
>> best way is to manually set LD_LIBRARY_PATH on demand, or to set
>> LD_PRELOAD to the specific libraries that are required.
>>
>> In general, though, I recommend using Guix for package management and
>> development instead of virtualenv and pip.
>
> Regarding the improvements of ’guix import pypi’ since 2018, and because
> tweaking LD_LIBRARY_PATH is dangerous, I do not see what could be the
> next action to solve this.
>
> Therefore, I propose to close it.  WDYT?

After 7 weeks of delay, I am closing.

Cheers,
simon

Subject: Python modules installed by pip in virtualenv can't find shared objects.
From: Fis Trivial <[email-masked]>
To: "[email-masked]" <[email-masked]>
Date: Tue, 23 Jan 2018 11:49:59 +0000
List-Id: <debbugs-submit.debbugs.gnu.org>

When a python module needs to load a dynamic shared object, it looks in the
path provided by *LD_LIBRARY_PATH*(1), but guix doesn't modify this environment
variable to export the needed path for python.

* Backtrace
In my case, it's lightgbm (installed by pip), needs libgomp.so from gcc:lib for
openmp support. Here is the backtrace:

--8<---------------cut here---------------start------------->8---
Traceback (most recent call last):
  File "ex1.py", line 5, in <module>
    import lightgbm as lgb
  File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/__init__.py", line 8, in <module>
    from .basic import Booster, Dataset
  File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 32, in <module>
    _LIB = _load_lib()
  File "/home/fis/Workspace/tianchi/medical_treatment/lib/python3.5/site-packages/lightgbm/basic.py", line 27, in _load_lib
    lib = ctypes.cdll.LoadLibrary(lib_path[0])
  File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 425, in LoadLibrary
    return self._dlltype(name)
  File "/gnu/store/jb3n0bsdpkhvyb8y70jyr8fcx8fqssr9-python-3.5.3/lib/python3.5/ctypes/__init__.py", line 347, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libgomp.so.1: cannot open shared object file: No such file or directory
--8<---------------cut here---------------end--------------->8---

* Reproduce
To install lightgbm, simply use `pip install lightgbm`(in virtualenv).
Then in python shell:
import lightgbm as lgb

* Ad-hoc
export LD_LIBRARY_PATH=~/.guix-profile/lib:$LD_LIBRARY_PATH

[1]: https://stackoverflow.com/a/1100016