Subject: Julia keeps build directory in the output

From: zimoun <[email-masked]>
Date: Wed, 05 May 2021 15:22:58 +0200

Hi,

Packaging Julia stuff, I note something as a bug, I guess.  Julia seems
to keep references to the build directory in the output.

--8<---------------cut here---------------start------------->8---
$ find $(guix build julia --no-grafts) -type f \
       -exec grep '/tmp/guix-build-julia-1.5.3.drv-0' {} \;
Binary file /gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/lib/libjulia.so.1.5 matches
Binary file /gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/lib/julia/sys.so matches
Binary file /gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/lib/julia/libccalltest.so.debug matches
Binary file /gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/lib/julia/libllvmcalltest.so matches
Binary file /gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/bin/.julia-real matches
Binary file /gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/share/julia/base.cache matches
Binary file /gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/share/julia/test/depot/compiled/v1.5/Bar/HXSAn_w3IH9.ji matches
Binary file /gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/share/julia/test/depot/compiled/v1.5/Foo/MYb1d_w3IH9.ji matches
Binary file /gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/share/julia/test/depot/compiled/v1.5/Foo/TeeT6_w3IH9.ji matches
Binary file /gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/share/julia/test/depot/compiled/v1.5/Qux/YFfiR_w3IH9.ji matches
Binary file /gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/share/julia/test/depot/compiled/v1.5/Baz/rONVA_w3IH9.ji matches
--8<---------------cut here---------------end--------------->8---

The consequence is ’dlopen’ is broken and some Julia packages cannot be
pre-compiled.  Here, an example using the Julia package manager:

--8<---------------cut here---------------start------------->8---
$ rm -fr ~/.julia # for the sake of the illustration ;-)
$ cat /tmp/foo.jl
using Pkg
Pkg.add("GZip")
println("# => Now let pre-compile")
using GZip
exit()

$ guix environment --ad-hoc julia -- julia --load /tmp/foo.jl
 Installing known registries into `~/.julia`
######################################################################### 100.0%
      Added registry `General` to `~/.julia/registries/General`
  Resolving package versions...
  Installed GZip ─ v0.5.1
Updating `~/.julia/environments/v1.5/Project.toml`
  [92fee26a] + GZip v0.5.1
Updating `~/.julia/environments/v1.5/Manifest.toml`
  [92fee26a] + GZip v0.5.1
  [8f399da3] + Libdl
# => Now let pre-compile
ERROR: LoadError: LoadError: could not load library "libz"
libz.so: cannot open shared object file: No such file or directory
Stacktrace:

[...]

in expression starting at /home/simon/.julia/packages/GZip/JNmGn/src/zlib_h.jl:13
in expression starting at /home/simon/.julia/packages/GZip/JNmGn/src/GZip.jl:73
ERROR: LoadError: Failed to precompile GZip [92fee26a-97fe-5a0c-ad85-20a5f3185b63] to /home/simon/.julia/compiled/v1.5/GZip/s2LKY_jQTtL.ji.
…
--8<---------------cut here---------------end--------------->8---

Let tweak the GZip ’ccall’ and pre-compile again to show that the
build-directory is contained in the output, and the Julia internal trace
raises it.

--8<---------------cut here---------------start------------->8---
$ diff ~/.julia/packages/GZip/JNmGn/src/GZip.jl{,.orig} 
75c75
< const GZLIB_VERSION = unsafe_string(ccall(:zlibVersion, Ptr{UInt8}, ()))
---
> const GZLIB_VERSION = unsafe_string(ccall((:zlibVersion, GZip._zlib), Ptr{UInt8}, ()))

$ diff ~/.julia/packages/GZip/JNmGn/src/zlib_h.jl{,.orig} 
13c13
< zlib_version =  unsafe_string(ccall(:zlibVersion, Ptr{UInt8}, ()))
---
> zlib_version =  unsafe_string(ccall((:zlibVersion, _zlib), Ptr{UInt8}, ()))
84c84
< const zlib_compile_flags = ccall(:zlibCompileFlags, UInt, ())
---
> const zlib_compile_flags = ccall((:zlibCompileFlags, _zlib), UInt, ())

$ guix environment --ad-hoc julia -- julia -e 'using GZip'
ERROR: LoadError: could not load library "libz"
libz.so: cannot open shared object file: No such file or directory
Stacktrace:
 [1] dlopen(::String, ::UInt32; throw_error::Bool) at /tmp/guix-build-julia-1.5.3.drv-0/julia-1.5.3/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
 [2] dlopen at /tmp/guix-build-julia-1.5.3.drv-0/julia-1.5.3/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)

[...]

in expression starting at /home/simon/.julia/packages/GZip/JNmGn/src/GZip.jl:206
…
--8<---------------cut here---------------end--------------->8---

I have tried to tweak LD_LIBRARY_PATH in the julia definition and
rebuild it.  Still the same error.  Note building the julia package
takes ~2h on my machine, so the trial-error takes ages. ;-)

Another illustration:

--8<---------------cut here---------------start------------->8---
$ cat /tmp/bar/pkg.scm
(define-module (pkg)
  #:use-module (guix packages)
  #:use-module (guix git-download)
  #:use-module (guix build-system julia)
  #:use-module ((guix licenses) #:prefix license:))

(define-public julia-gzip
  (package
    (name "julia-gzip")
    (version "0.5.1")
    (source
     (origin
       (method git-fetch)
       (uri (git-reference
             (url "https://github.com/JuliaIO/GZip.jl")
             (commit (string-append "v" version))))
       (file-name (git-file-name name version))
       (sha256
        (base32 "06bx3l1nsvdswgl344347cjh6ffd2r1hw3lz4h0aicayc566717f"))))
    (build-system julia-build-system)
    (home-page "https://github.com/JuliaIO/GZip.jl")
    (synopsis "TODO")
    (description "TODO")
    (license license:expat)))

$ guix build -L /tmp/bar/ julia-gzip
The following derivation will be built:
   /gnu/store/2wiyhvqrhq8bbiw34gyxclv6zzf98vy3-julia-gzip-0.5.1.drv
building /gnu/store/2wiyhvqrhq8bbiw34gyxclv6zzf98vy3-julia-gzip-0.5.1.drv...
starting phase `set-SOURCE-DATE-EPOCH'
phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds
starting phase `set-paths'
environment variable `PATH' set to `/gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/bin:/gnu/store/v6f44zccwh9z5zk3pjlywjybbi8n2hjh-tar-1.32/bin:/gnu/store/ncydgq2znms5n1d2k5yqshhf58nsixwv-gzip-1.10/bin:/gnu/store/i8h2pcxqdq07ijm3ibkka8f4smn1w48v-bzip2-1.0.8/bin:/gnu/store/9860f1abqj8wjjnwl8a9v54pdcc3bhgf-xz-5.2.4/bin:/gnu/store/60g7r3l01fd7c58yjbm6krgcwj1jkpwg-file-5.38/bin:/gnu/store/n4n560pfvvw50a9369axw5vj5rrqfj1n-diffutils-3.7/bin:/gnu/store/cd5qf3kcnlq35p9k392pjdpdzpsnds70-patch-2.7.6/bin:/gnu/store/hic7snhayfl7m6cpfqqr73nmm19bpqkg-findutils-4.7.0/bin:/gnu/store/swqdvwri9dbv6zssg6v0by7l05hd6wxp-gawk-5.0.1/bin:/gnu/store/ishk7fswcs4gkwcp8mh788z4mvvl9bxh-sed-4.8/bin:/gnu/store/bhs4rj58v8j1narb2454raan2ps38xd8-grep-3.4/bin:/gnu/store/57xj5gcy1jbl9ai2lnrqnpr0dald9i65-coreutils-8.32/bin:/gnu/store/hm40bxnv8jxmbc1lpb7zfimii4xm9m81-make-4.3/bin:/gnu/store/pwcp239kjf7lnj5i4lkdzcfcxwcfyk72-bash-minimal-5.0.16/bin:/gnu/store/mpa04aq8lblbcviyxywxcsb1zbi0mf39-ld-wrapper-0/bin:/gnu/store/m1z7cdbqsqyp9xnjw5cvlb4a7gkcg3m4-binutils-2.34/bin:/gnu/store/rn75fm7adgx3pw5j8pg3bczfqq1y17lk-gcc-7.5.0/bin:/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/bin:/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/sbin'
environment variable `JULIA_LOAD_PATH' unset
environment variable `JULIA_DEPOT_PATH' set to `/gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/share/julia/'
environment variable `BASH_LOADABLES_PATH' unset
environment variable `C_INCLUDE_PATH' set to `/gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/include:/gnu/store/i8h2pcxqdq07ijm3ibkka8f4smn1w48v-bzip2-1.0.8/include:/gnu/store/9860f1abqj8wjjnwl8a9v54pdcc3bhgf-xz-5.2.4/include:/gnu/store/60g7r3l01fd7c58yjbm6krgcwj1jkpwg-file-5.38/include:/gnu/store/swqdvwri9dbv6zssg6v0by7l05hd6wxp-gawk-5.0.1/include:/gnu/store/hm40bxnv8jxmbc1lpb7zfimii4xm9m81-make-4.3/include:/gnu/store/m1z7cdbqsqyp9xnjw5cvlb4a7gkcg3m4-binutils-2.34/include:/gnu/store/rn75fm7adgx3pw5j8pg3bczfqq1y17lk-gcc-7.5.0/include:/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/include:/gnu/store/gfapkk5c6hvl1d94m4sqnhn7f9l5gqyh-linux-libre-headers-5.4.20/include'
environment variable `CPLUS_INCLUDE_PATH' set to `/gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/include:/gnu/store/i8h2pcxqdq07ijm3ibkka8f4smn1w48v-bzip2-1.0.8/include:/gnu/store/9860f1abqj8wjjnwl8a9v54pdcc3bhgf-xz-5.2.4/include:/gnu/store/60g7r3l01fd7c58yjbm6krgcwj1jkpwg-file-5.38/include:/gnu/store/swqdvwri9dbv6zssg6v0by7l05hd6wxp-gawk-5.0.1/include:/gnu/store/hm40bxnv8jxmbc1lpb7zfimii4xm9m81-make-4.3/include:/gnu/store/m1z7cdbqsqyp9xnjw5cvlb4a7gkcg3m4-binutils-2.34/include:/gnu/store/rn75fm7adgx3pw5j8pg3bczfqq1y17lk-gcc-7.5.0/include/c++:/gnu/store/rn75fm7adgx3pw5j8pg3bczfqq1y17lk-gcc-7.5.0/include:/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/include:/gnu/store/gfapkk5c6hvl1d94m4sqnhn7f9l5gqyh-linux-libre-headers-5.4.20/include'
environment variable `LIBRARY_PATH' set to `/gnu/store/i1dgzqfjvkkjlfhpcwc33lz17vslq50y-julia-1.5.3/lib:/gnu/store/i8h2pcxqdq07ijm3ibkka8f4smn1w48v-bzip2-1.0.8/lib:/gnu/store/9860f1abqj8wjjnwl8a9v54pdcc3bhgf-xz-5.2.4/lib:/gnu/store/60g7r3l01fd7c58yjbm6krgcwj1jkpwg-file-5.38/lib:/gnu/store/swqdvwri9dbv6zssg6v0by7l05hd6wxp-gawk-5.0.1/lib:/gnu/store/m1z7cdbqsqyp9xnjw5cvlb4a7gkcg3m4-binutils-2.34/lib:/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib:/gnu/store/s3dcqzwqaakv1yx37by9chksdbkgih17-glibc-2.31-static/lib:/gnu/store/hwcky7446s952w0mwchhmm211ll07zrq-glibc-utf8-locales-2.31/lib'
environment variable `GUIX_LOCPATH' set to `/gnu/store/hwcky7446s952w0mwchhmm211ll07zrq-glibc-utf8-locales-2.31/lib/locale'
phase `set-paths' succeeded after 0.0 seconds
starting phase `install-locale'
using 'en_US.utf8' locale for category "LC_ALL"
phase `install-locale' succeeded after 0.0 seconds
starting phase `unpack'

[...]

phase `unpack' succeeded after 0.0 seconds
starting phase `patch-source-shebangs'
phase `patch-source-shebangs' succeeded after 0.0 seconds
starting phase `patch-generated-file-shebangs'
phase `patch-generated-file-shebangs' succeeded after 0.0 seconds
starting phase `install'

[...]

phase `install' succeeded after 0.0 seconds
starting phase `check'
ERROR: LoadError: LoadError: could not load library "libz"
libz.so: cannot open shared object file: No such file or directory
Stacktrace:

[...]

in expression starting at /gnu/store/dak0gnj5kqdma55xy279345qzsal9zdq-julia-gzip-0.5.1/share/julia/packages/GZip/src/zlib_h.jl:13
in expression starting at /gnu/store/dak0gnj5kqdma55xy279345qzsal9zdq-julia-gzip-0.5.1/share/julia/packages/GZip/src/GZip.jl:73
ERROR: LoadError: Failed to precompile GZip [92fee26a-97fe-5a0c-ad85-20a5f3185b63] to /tmp/.julia/compiled/v1.5/GZip/s2LKY_y94HW.ji.
Stacktrace:

[...]

in expression starting at /gnu/store/dak0gnj5kqdma55xy279345qzsal9zdq-julia-gzip-0.5.1/share/julia/packages/GZip/test/runtests.jl:1
command "julia" "--depwarn=yes" "/gnu/store/dak0gnj5kqdma55xy279345qzsal9zdq-julia-gzip-0.5.1/share/julia/packages/GZip/test/runtests.jl" failed with status 1
builder for `/gnu/store/2wiyhvqrhq8bbiw34gyxclv6zzf98vy3-julia-gzip-0.5.1.drv' failed with exit code 1
build of /gnu/store/2wiyhvqrhq8bbiw34gyxclv6zzf98vy3-julia-gzip-0.5.1.drv failed
View build log at '/var/log/guix/drvs/2w/iyhvqrhq8bbiw34gyxclv6zzf98vy3-julia-gzip-0.5.1.drv.bz2'.
guix build: error: build of `/gnu/store/2wiyhvqrhq8bbiw34gyxclv6zzf98vy3-julia-gzip-0.5.1.drv' failed
--8<---------------cut here---------------end--------------->8---

Cheers,
simon