Subject: [bug#27438] [PATCH] guix: build: ruby-build-system: Install to the vendor directory

From: Christopher Baines <[email-masked]>
Date: Sun, 17 Dec 2017 14:19:37 +0000

Hey,

It's been a while since I looked at this, but I think it's still
relevant.

As a reminder, currently the search paths for the Ruby packages in Guix
are a bit off [1], which makes it difficult to use Ruby gems with
different versions of Ruby.

Rubygems by default installs gems to directories specific to the version
of Ruby, and this can be helpful when managing multiple versions of Ruby
and different sets of gem packages in a shared directory, but with Guix,
this isn't necessary, as the store can be used.

Ruby/Rubygems supports installing gems in to a vendor directory, which
is for gems installed by the package system [2]. While Guix blurs the
lines between the "package system" and the "user", using this vendor
directory might come in useful.

I've attached an updated patch, and successfully built every Ruby
package [3], as well as all of the packages that were changed in the
patch. The only failure was ruby-pg, which failed to build before this
patch.

Ideally when using specific versions of ruby, any gems that need to be
build with that version of ruby, would be. While this change doesn't do
that, it does simplify the search paths of ruby packages and the
packages for ruby gems.

Reading the bug report again, I think there was nearly consensus about
merging this, but I wanted to check if anyone has any comments or
experience with Ruby/Rubygems before merging this?

Thanks,

Chris

1:
scheme@(guile-user)> (use-modules (gnu packages) (guix packages) (guix search-paths) (ice-9 match))
scheme@(guile-user)> (map (lambda (ruby) (cons (package-version ruby) (match (package-native-search-paths ruby) (() #f) ((($ <search-path-specification> _ files)) files)))) (find-packages-by-name "ruby"))
$1 = (("2.4.3" "lib/ruby/gems/2.4.0") ("2.4.2" "lib/ruby/gems/2.4.0") ("2.3.6" "lib/ruby/gems/2.4.0") ("2.2.9" "lib/ruby/gems/2.4.0") ("2.1.10" "lib/ruby/gems/2.1.0") ("1.8.7-p374" . #f))

2: http://svn.ruby-lang.org/repos/ruby/tags/v1_8_7/NEWS

3: guix build --keep-going $(guix package -s ruby- | recsel -P name | tr '\n' ' ')

From 2df7d9916fdcdafdbf3e8010ba275fd39ecbd308 Mon Sep 17 00:00:00 2001
From: Christopher Baines <[email-masked]>
Date: Sun, 17 Dec 2017 08:25:44 +0000
Subject: [PATCH] guix: build: ruby-build-system: Install to the vendor
 directory

* guix/build/ruby-build-system.scm (install): Install gems to the vendor
  directory, rather than the GEM_HOME. The vendor directory does not include
  the version of ruby used to install the gem in the path, which makes it
  easier to add it to the GEM_PATH for all versions of ruby to use.
  (gem-home): Remove procedure.
* gnu/packages/ruby.scm (ruby, ruby-2.1)[native-search-paths]: Switch to
  lib/ruby/vendor_ruby.
  (ruby-1.8)[native-search-paths]: Remove native-search-paths.
  (gem-directory): Remove procedure.
  (ruby-ansi, ruby-ae)[arguments]: Remove use of gem-directory.
  (ruby-metaclass, ruby-instantiator, ruby-introspection, ruby-mocha,
  ruby-nokogiri, ruby-minitest-tu-shim, ruby-redcloth)[arguments]: Remove use
  of gem-home.
  (ruby-git, ruby-httpclient)[arguments]: Remove use of GEM_HOME.
* gnu/packages/databases.scm (es-dump-restore)[arguments]: Remove use of
  GEM_HOME.
---
 gnu/packages/databases.scm       |  8 ++---
 gnu/packages/ruby.scm            | 78 +++++++++++++++-------------------------
 guix/build/ruby-build-system.scm | 51 +++++++++++++-------------
 3 files changed, 56 insertions(+), 81 deletions(-)

diff --git a/gnu/packages/databases.scm b/gnu/packages/databases.scm
index a6a7c3178..8f16b46c5 100644
--- a/gnu/packages/databases.scm
+++ b/gnu/packages/databases.scm
@@ -297,10 +297,10 @@ SQL, Key/Value, XML/XQuery or Java Object storage for their data model.")
            (lambda* (#:key outputs #:allow-other-keys)
              (wrap-program (string-append (assoc-ref outputs "out")
                                           "/bin/es_dump_restore")
-               `("GEM_PATH" ":" prefix (,(string-append
-                                          (getenv "GEM_PATH")
-                                          ":"
-                                          (getenv "GEM_HOME")))))
+               `("GEM_PATH" ":" prefix (,(getenv "GEM_PATH")
+                                        ,(string-append
+                                          (assoc-ref outputs "out")
+                                          "/lib/ruby/vendor_ruby"))))
              #t)))))
     (propagated-inputs
      `(("ruby-httpclient" ,ruby-httpclient)
diff --git a/gnu/packages/ruby.scm b/gnu/packages/ruby.scm
index 34387e575..b98ab2877 100644
--- a/gnu/packages/ruby.scm
+++ b/gnu/packages/ruby.scm
@@ -94,9 +94,7 @@
     (native-search-paths
      (list (search-path-specification
             (variable "GEM_PATH")
-            (files (list (string-append "lib/ruby/gems/"
-                                        (version-major+minor version)
-                                        ".0"))))))
+            (files (list (string-append "lib/ruby/vendor_ruby"))))))
     (synopsis "Programming language interpreter")
     (description "Ruby is a dynamic object-oriented programming language with
 a focus on simplicity and productivity.")
@@ -180,13 +178,7 @@ a focus on simplicity and productivity.")
                             "lib/mkmf.rb"
                             "process.c")
                (("/bin/sh") (which "sh")))
-             #t)))))
-    (native-search-paths
-     (list (search-path-specification
-            (variable "GEM_PATH")
-            (files (list (string-append "lib/ruby/gems/"
-                                        (version-major+minor version)
-                                        ".0"))))))))
+             #t)))))))
 
 (define-public ruby-1.8
   (package (inherit ruby)
@@ -216,11 +208,6 @@ a focus on simplicity and productivity.")
                (("/bin/sh") (which "sh")))
              #t)))))))
 
-(define (gem-directory ruby-version)
-  "Return the relative gem install directory for RUBY-VERSION."
-  (string-append "/lib/ruby/gems/" (version-major+minor ruby-version)
-                 ".0/gems"))
-
 (define-public ruby-highline
   (package
     (name "ruby-highline")
@@ -1320,13 +1307,11 @@ It allows writing tests, checking results and automated testing in Ruby.")
        (modify-phases %standard-phases
          (add-after 'unpack 'add-test-unit-to-search-path
            (lambda* (#:key inputs #:allow-other-keys)
-             (let* ((test-unit (assoc-ref inputs "ruby-test-unit"))
-                    (test-unit-home (gem-home test-unit
-                                              ,(package-version ruby))))
+             (let* ((test-unit (assoc-ref inputs "ruby-test-unit")))
                (substitute* "Rakefile"
                  (("t\\.libs << \"test\"" line)
                   (string-append line "; t.libs << \""
-                                 test-unit-home
+                                 test-unit "/lib/ruby/vendor_ruby"
                                  "/gems/test-unit-"
                                  ,(package-version ruby-test-unit)
                                  "/lib\""))))
@@ -1385,13 +1370,11 @@ as a base class when writing classes that depend upon
        (modify-phases %standard-phases
          (add-after 'unpack 'add-test-unit-to-search-path
            (lambda* (#:key inputs #:allow-other-keys)
-             (let* ((test-unit (assoc-ref inputs "ruby-test-unit"))
-                    (test-unit-home (gem-home test-unit ,(package-version
-                                                          ruby))))
+             (let* ((test-unit (assoc-ref inputs "ruby-test-unit")))
                (substitute* "Rakefile"
                  (("t\\.libs << \"test\"" line)
                   (string-append line "; t.libs << \""
-                                 test-unit-home
+                                 test-unit "/lib/ruby/vendor_ruby"
                                  "/gems/test-unit-"
                                  ,(package-version ruby-test-unit)
                                  "/lib\""))))
@@ -1424,13 +1407,11 @@ knowing anything about the constructor.")
        (modify-phases %standard-phases
          (add-after 'unpack 'add-test-unit-to-search-path
           (lambda* (#:key inputs #:allow-other-keys)
-            (let* ((test-unit (assoc-ref inputs "ruby-test-unit"))
-                   (test-unit-home (gem-home test-unit ,(package-version
-                                                         ruby))))
+            (let* ((test-unit (assoc-ref inputs "ruby-test-unit")))
               (substitute* "Rakefile"
                 (("t\\.libs << \"test\"" line)
                  (string-append line "; t.libs << \""
-                                test-unit-home
+                                test-unit "/lib/ruby/vendor_ruby"
                                 "/gems/test-unit-"
                                 ,(package-version ruby-test-unit)
                                 "/lib\""))))
@@ -1497,13 +1478,11 @@ conversion to (X)HTML.")
        (modify-phases %standard-phases
          (add-after 'unpack 'add-test-unit-to-search-path
           (lambda* (#:key inputs #:allow-other-keys)
-            (let* ((test-unit (assoc-ref inputs "ruby-test-unit"))
-                   (test-unit-home (gem-home test-unit
-                                             ,(package-version ruby))))
+            (let* ((test-unit (assoc-ref inputs "ruby-test-unit")))
               (substitute* "Rakefile"
                 (("t\\.libs << 'test'" line)
                  (string-append line "; t.libs << \""
-                                test-unit-home
+                                test-unit "/lib/ruby/vendor_ruby"
                                 "/gems/test-unit-"
                                 ,(package-version ruby-test-unit)
                                 "/lib\""))))
@@ -1868,9 +1847,10 @@ run as a daemon and to be controlled by simple start/stop/restart commands.")
                       ;; store.
                       (let ((git    (string-append (assoc-ref inputs "git")
                                                    "/bin/git"))
-                            (config (string-append (getenv "GEM_HOME")
-                                                   "/gems/git-" ,version
-                                                   "/lib/git/config.rb")))
+                            (config (string-append
+                                     (assoc-ref outputs "out")
+                                     "/lib/ruby/vendor_ruby/gems/git-"
+                                     ,version "/lib/git/config.rb")))
                         (substitute* (list config)
                           (("'git'")
                            (string-append "'" git "'")))
@@ -2055,13 +2035,11 @@ to reproduce user environments.")
            ;; 'pkg-config' is not included in the GEM_PATH during
            ;; installation, so we add it directly to the load path.
            (lambda* (#:key inputs #:allow-other-keys)
-             (let* ((pkg-config (assoc-ref inputs "ruby-pkg-config"))
-                    (pkg-config-home (gem-home pkg-config
-                                               ,(package-version ruby))))
+             (let* ((pkg-config (assoc-ref inputs "ruby-pkg-config")))
                (substitute* "ext/nokogiri/extconf.rb"
                  (("gem 'pkg-config'.*")
                   (string-append "$:.unshift '"
-                                 pkg-config-home
+                                 pkg-config "/lib/ruby/vendor_ruby"
                                  "/gems/pkg-config-"
                                  ,(package-version ruby-pkg-config)
                                  "/lib'\n"))))
@@ -2468,13 +2446,11 @@ development of Ruby gems.")
        (modify-phases %standard-phases
          (add-after 'unpack 'fix-test-include-path
           (lambda* (#:key inputs #:allow-other-keys)
-             (let* ((minitest (assoc-ref inputs "ruby-minitest-4"))
-                    (minitest-home (gem-home minitest
-                                             ,(package-version ruby))))
+             (let* ((minitest (assoc-ref inputs "ruby-minitest-4")))
                (substitute* "Rakefile"
                  (("Hoe\\.add_include_dirs .*")
                   (string-append "Hoe.add_include_dirs \""
-                                 minitest-home
+                                 minitest "/lib/ruby/vendor_ruby"
                                  "/gems/minitest-"
                                  ,(package-version ruby-minitest-4)
                                  "/lib" "\""))))
@@ -2880,7 +2856,9 @@ alternative to Marshal for Object serialization. ")
           (lambda* (#:key outputs #:allow-other-keys)
             (wrap-program (string-append (assoc-ref outputs "out")
                                          "/bin/redcloth")
-              `("GEM_HOME" ":" prefix (,(getenv "GEM_HOME"))))
+              `("GEM_PATH" ":" prefix (,(string-append
+                                         (assoc-ref outputs "out")
+                                         "/lib/ruby/vendor_ruby"))))
             #t)))))
     (native-inputs
      `(("bundler" ,bundler)
@@ -3734,7 +3712,7 @@ It has built-in support for the legacy @code{cookies.txt} and
            (lambda* (#:key outputs #:allow-other-keys)
              (wrap-program (string-append (assoc-ref outputs "out")
                                           "/bin/httpclient")
-               `("GEM_HOME" ":" prefix (,(getenv "GEM_HOME"))))
+               `("GEM_PATH" ":" prefix (,(getenv "GEM_PATH"))))
              #t)))))
     (native-inputs
      `(("ruby-rack" ,ruby-rack)))
@@ -3777,9 +3755,9 @@ requests either using arguments or with an interactive prompt.")
          (add-before 'validate-runpath 'replace-broken-symlink
            (lambda* (#:key outputs #:allow-other-keys)
              (let* ((out (assoc-ref outputs "out"))
-                    (file (string-append out
-                                         ,(gem-directory (package-version ruby))
-                                         "/ansi-" ,version "/lib/ansi.yml")))
+                    (file (string-append
+                           out "/lib/ruby/vendor_ruby/gems/ansi-"
+                           ,version "/lib/ansi.yml")))
                ;; XXX: This symlink is broken since ruby 2.4.
                ;; https://lists.gnu.org/archive/html/guix-devel/2017-06/msg00034.html
                (delete-file file)
@@ -3977,9 +3955,9 @@ requirement specifications systems like Cucumber.")
          (add-before 'validate-runpath 'replace-broken-symlink
            (lambda* (#:key outputs #:allow-other-keys)
              (let* ((out (assoc-ref outputs "out"))
-                    (file (string-append out
-                                         ,(gem-directory (package-version ruby))
-                                         "/ae-" ,version "/lib/ae.yml")))
+                    (file (string-append
+                           out "/lib/ruby/vendor_ruby/gems/ae-"
+                           ,version "/lib/ae.yml")))
                ;; XXX: This symlink is broken since ruby 2.4.
                ;; https://lists.gnu.org/archive/html/guix-devel/2017-06/msg00034.html
                (delete-file file)
diff --git a/guix/build/ruby-build-system.scm b/guix/build/ruby-build-system.scm
index c2d276627..e3a811151 100644
--- a/guix/build/ruby-build-system.scm
+++ b/guix/build/ruby-build-system.scm
@@ -27,8 +27,7 @@
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-26)
   #:export (%standard-phases
-            ruby-build
-            gem-home))
+            ruby-build))
 
 ;; Commentary:
 ;;
@@ -129,40 +128,48 @@ GEM-FLAGS are passed to the 'gem' invokation, if present."
                                          (assoc-ref inputs "ruby"))
                            1))
          (out (assoc-ref outputs "out"))
-         (gem-home (string-append out "/lib/ruby/gems/" ruby-version ".0"))
+         (vendor-dir (string-append out "/lib/ruby/vendor_ruby"))
          (gem-file (first-matching-file "\\.gem$"))
          (gem-file-basename (basename gem-file))
          (gem-name (substring gem-file-basename
                               0
-                              (- (string-length gem-file-basename) 4)))
-         (gem-directory (string-append gem-home "/gems/" gem-name)))
-    (setenv "GEM_HOME" gem-home)
-    (mkdir-p gem-home)
-    (and (apply system* "gem" "install" gem-file
-                "--local" "--ignore-dependencies"
-                ;; Executables should go into /bin, not /lib/ruby/gems.
-                "--bindir" (string-append out "/bin")
-                gem-flags)
+                              (- (string-length gem-file-basename) 4))))
+    (setenv "GEM_VENDOR" vendor-dir)
+    (and (let ((install-succeeded?
+                (zero?
+                 (apply system* "gem" "install" gem-file
+                        "--local" "--ignore-dependencies" "--vendor"
+                        ;; Executables should go into /bin, not
+                        ;; /lib/ruby/gems.
+                        "--bindir" (string-append out "/bin")
+                        gem-flags))))
+           (or install-succeeded?
+               (begin
+                 (simple-format #t "installation failed\n")
+                 (let ((failed-output-dir (string-append (getcwd) "/out")))
+                   (mkdir failed-output-dir)
+                   (copy-recursively out failed-output-dir))
+                 #f)))
          (begin
            ;; Remove the cached gem file as this is unnecessary and contains
            ;; timestamped files rendering builds not reproducible.
-           (let ((cached-gem (string-append gem-home "/cache/" gem-file)))
+           (let ((cached-gem (string-append vendor-dir "/cache/" gem-file)))
              (log-file-deletion cached-gem)
              (delete-file cached-gem))
            ;; For gems with native extensions, several Makefile-related files
            ;; are created that contain timestamps or other elements making
            ;; them not reproducible.  They are unnecessary so we remove them.
-           (if (file-exists? (string-append gem-directory "/ext"))
+           (if (file-exists? (string-append vendor-dir "/ext"))
                (begin
                  (for-each (lambda (file)
                              (log-file-deletion file)
                              (delete-file file))
                            (append
-                            (find-files (string-append gem-home "/doc")
+                            (find-files (string-append vendor-dir "/doc")
                                         "page-Makefile.ri")
-                            (find-files (string-append gem-home "/extensions")
+                            (find-files (string-append vendor-dir "/extensions")
                                         "gem_make.out")
-                            (find-files (string-append gem-directory "/ext")
+                            (find-files (string-append vendor-dir "/ext")
                                         "Makefile")))))
            #t))))
 
@@ -182,13 +189,3 @@ GEM-FLAGS are passed to the 'gem' invokation, if present."
 (define* (ruby-build #:key inputs (phases %standard-phases)
                      #:allow-other-keys #:rest args)
   (apply gnu:gnu-build #:inputs inputs #:phases phases args))
-
-(define (gem-home store-path ruby-version)
-  "Return a string to the gem home directory in the store given a STORE-PATH
-and the RUBY-VERSION used to build that ruby package"
-  (string-append
-   store-path
-   "/lib/ruby/gems/"
-   (regexp-substitute #f
-                      (string-match "^[0-9]+\\.[0-9]+" ruby-version)
-                      0 ".0")))
-- 
2.15.1