GNU bug report logs - #32046
[PATCH] import: gem: Add recursive import.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 32046 in the body.
You can then email your comments to 32046 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Oleg Pykhalov <go.wigust <at> gmail.com>
To: guix-patches <at> gnu.org
Cc: Oleg Pykhalov <go.wigust <at> gmail.com>
Subject: [PATCH] import: gem: Add recursive import.
Date: Tue,  3 Jul 2018 23:33:27 +0300

* doc/guix.texi (Invoking guix import): Document gem recursive import.
* guix/import/gem.scm (gem->guix-package): Return package and dependencies
values.
(gem-recursive-import): New procedure.
* guix/scripts/import/gem.scm (show-help, %options): Add recursive option.
(guix-import-gem): Use 'gem-recursive-import'.
---
 doc/guix.texi               |  8 ++++++
 guix/import/gem.scm         | 49 ++++++++++++++++++++++---------------
 guix/scripts/import/gem.scm | 26 ++++++++++++++++----
 3 files changed, 58 insertions(+), 25 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 841bc2a34..fa24bcf28 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -6370,6 +6370,14 @@ The command below imports metadata for the @code{rails} Ruby package:
 guix import gem rails
 @end example
 
+@table @code
+@item --recursive
+@itemx -r
+Traverse the dependency graph of the given upstream package recursively
+and generate package expressions for all those packages that are not yet
+in Guix.
+@end table
+
 @item cpan
 @cindex CPAN
 Import metadata from @uref{https://www.metacpan.org/, MetaCPAN}@footnote{This
diff --git a/guix/import/gem.scm b/guix/import/gem.scm
index 646163fb7..9e6d2e72e 100644
--- a/guix/import/gem.scm
+++ b/guix/import/gem.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 David Thompson <davet <at> gnu.org>
 ;;; Copyright © 2016 Ben Woodcroft <donttrustben <at> gmail.com>
+;;; Copyright © 2018 Oleg Pykhalov <go.wigust <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -20,6 +21,7 @@
 (define-module (guix import gem)
   #:use-module (ice-9 match)
   #:use-module (ice-9 pretty-print)
+  #:use-module (ice-9 receive)
   #:use-module (srfi srfi-1)
   #:use-module (rnrs bytevectors)
   #:use-module (json)
@@ -33,7 +35,8 @@
   #:use-module (guix base32)
   #:use-module (guix build-system ruby)
   #:export (gem->guix-package
-            %gem-updater))
+            %gem-updater
+            gem-recursive-import))
 
 (define (rubygems-fetch name)
   "Return an alist representation of the RubyGems metadata for the package NAME,
@@ -115,29 +118,30 @@ VERSION, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES."
                  ((license) (license->symbol license))
                  (_ `(list ,@(map license->symbol licenses)))))))
 
-(define* (gem->guix-package package-name #:optional version)
+(define* (gem->guix-package package-name #:optional (repo 'rubygems) version)
   "Fetch the metadata for PACKAGE-NAME from rubygems.org, and return the
 `package' s-expression corresponding to that package, or #f on failure."
   (let ((package (rubygems-fetch package-name)))
     (and package
-         (let ((name         (assoc-ref package "name"))
-               (version      (assoc-ref package "version"))
-               (hash         (assoc-ref package "sha"))
-               (synopsis     (assoc-ref package "info")) ; nothing better to use
-               (description  (beautify-description
-                              (assoc-ref package "info")))
-               (home-page    (assoc-ref package "homepage_uri"))
-               (dependencies (map (lambda (dep)
-                                    (let ((name (assoc-ref dep "name")))
-                                      (if (string=? name "bundler")
-                                          "bundler" ; special case, no prefix
-                                          (ruby-package-name name))))
-                                  (assoc-ref* package "dependencies"
-                                              "runtime")))
-               (licenses     (map string->license
-                                  (assoc-ref package "licenses"))))
-           (make-gem-sexp name version hash home-page synopsis
-                          description dependencies licenses)))))
+         (let* ((name         (assoc-ref package "name"))
+                (version      (assoc-ref package "version"))
+                (hash         (assoc-ref package "sha"))
+                (synopsis     (assoc-ref package "info")) ; nothing better to use
+                (description  (beautify-description
+                               (assoc-ref package "info")))
+                (home-page    (assoc-ref package "homepage_uri"))
+                (dependencies-names (map (lambda (dep) (assoc-ref dep "name"))
+                                         (assoc-ref* package "dependencies" "runtime")))
+                (dependencies (map (lambda (dep)
+                                     (if (string=? dep "bundler")
+                                         "bundler" ; special case, no prefix
+                                         (ruby-package-name dep)))
+                                   dependencies-names))
+                (licenses     (map string->license
+                                   (assoc-ref package "licenses"))))
+           (values (make-gem-sexp name version hash home-page synopsis
+                                  description dependencies licenses)
+                   dependencies-names)))))
 
 (define (guix-package->gem-name package)
   "Given a PACKAGE built from rubygems.org, return the name of the
@@ -192,3 +196,8 @@ package on RubyGems."
    (description "Updater for RubyGem packages")
    (pred gem-package?)
    (latest latest-release)))
+
+(define* (gem-recursive-import package-name #:optional version)
+  (recursive-import package-name '()
+                    #:repo->guix-package gem->guix-package
+                    #:guix-name ruby-package-name))
diff --git a/guix/scripts/import/gem.scm b/guix/scripts/import/gem.scm
index 349a0a072..64afbd464 100644
--- a/guix/scripts/import/gem.scm
+++ b/guix/scripts/import/gem.scm
@@ -1,5 +1,6 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2015 David Thompson <davet <at> gnu.org>
+;;; Copyright © 2018 Oleg Pykhalov <go.wigust <at> gmail.com>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -25,6 +26,7 @@
   #:use-module (srfi srfi-1)
   #:use-module (srfi srfi-11)
   #:use-module (srfi srfi-37)
+  #:use-module (srfi srfi-41)
   #:use-module (ice-9 match)
   #:use-module (ice-9 format)
   #:export (guix-import-gem))
@@ -44,6 +46,8 @@ Import and convert the RubyGems package for PACKAGE-NAME.\n"))
   -h, --help             display this help and exit"))
   (display (G_ "
   -V, --version          display version information and exit"))
+  (display (G_ "
+  -r, --recursive        generate package expressions for all Gem packages that are not yet in Guix"))
   (newline)
   (show-bug-report-information))
 
@@ -56,6 +60,9 @@ Import and convert the RubyGems package for PACKAGE-NAME.\n"))
          (option '(#\V "version") #f #f
                  (lambda args
                    (show-version-and-exit "guix import pypi")))
+         (option '(#\r "recursive") #f #f
+                 (lambda (opt name arg result)
+                   (alist-cons 'recursive #t result)))
          %standard-import-options))
 
 
@@ -81,11 +88,20 @@ Import and convert the RubyGems package for PACKAGE-NAME.\n"))
                            (reverse opts))))
     (match args
       ((package-name)
-       (let ((sexp (gem->guix-package package-name)))
-         (unless sexp
-           (leave (G_ "failed to download meta-data for package '~a'~%")
-                  package-name))
-         sexp))
+       (if (assoc-ref opts 'recursive)
+           (map (match-lambda
+                  ((and ('package ('name name) . rest) pkg)
+                   `(define-public ,(string->symbol name)
+                      ,pkg))
+                  (_ #f))
+                (reverse
+                 (stream->list
+                  (gem-recursive-import package-name 'rubygems))))
+           (let ((sexp (gem->guix-package package-name)))
+             (unless sexp
+               (leave (G_ "failed to download meta-data for package '~a'~%")
+                      package-name))
+             sexp)))
       (()
        (leave (G_ "too few arguments~%")))
       ((many ...)
-- 
2.18.0

Message #8 received at 32046 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Oleg Pykhalov <go.wigust <at> gmail.com>
Cc: 32046 <at> debbugs.gnu.org
Subject: Re: [bug#32046] [PATCH] import: gem: Add recursive import.
Date: Mon, 09 Jul 2018 14:33:15 +0200

Hi Oleg,

Oleg Pykhalov <go.wigust <at> gmail.com> skribis:

> * doc/guix.texi (Invoking guix import): Document gem recursive import.
> * guix/import/gem.scm (gem->guix-package): Return package and dependencies
> values.
> (gem-recursive-import): New procedure.
> * guix/scripts/import/gem.scm (show-help, %options): Add recursive option.
> (guix-import-gem): Use 'gem-recursive-import'.

Nice!

Could you add a test in tests/gem.scm?

>  (define-module (guix import gem)
>    #:use-module (ice-9 match)
>    #:use-module (ice-9 pretty-print)
> +  #:use-module (ice-9 receive)

I don’t think this is needed.

> @@ -44,6 +46,8 @@ Import and convert the RubyGems package for PACKAGE-NAME.\n"))
>    -h, --help             display this help and exit"))
>    (display (G_ "
>    -V, --version          display version information and exit"))
> +  (display (G_ "
> +  -r, --recursive        generate package expressions for all Gem packages that are not yet in Guix"))

Please keep lines below 80-char wide.

Apart form that LGTM.  Could you send an  updated patch?

Thank you!

Ludo’.

Message #11 received at 32046 <at> debbugs.gnu.org (full text, mbox):

From: Oleg Pykhalov <go.wigust <at> gmail.com>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 32046 <at> debbugs.gnu.org
Subject: Re: [bug#32046] [PATCH] import: gem: Add recursive import.
Date: Tue, 10 Jul 2018 16:31:59 +0300

[Message part 1 (text/plain, inline)]

Hello Ludovic,

Thank you for review.

ludo <at> gnu.org (Ludovic Courtès) writes:

> Hi Oleg,
>
> Oleg Pykhalov <go.wigust <at> gmail.com> skribis:
>
>> * doc/guix.texi (Invoking guix import): Document gem recursive import.
>> * guix/import/gem.scm (gem->guix-package): Return package and dependencies
>> values.
>> (gem-recursive-import): New procedure.
>> * guix/scripts/import/gem.scm (show-help, %options): Add recursive option.
>> (guix-import-gem): Use 'gem-recursive-import'.
>
> Nice!
>
> Could you add a test in tests/gem.scm?
>
>>  (define-module (guix import gem)
>>    #:use-module (ice-9 match)
>>    #:use-module (ice-9 pretty-print)
>> +  #:use-module (ice-9 receive)
>
> I don’t think this is needed.
>
>> @@ -44,6 +46,8 @@ Import and convert the RubyGems package for PACKAGE-NAME.\n"))
>>    -h, --help             display this help and exit"))
>>    (display (G_ "
>>    -V, --version          display version information and exit"))
>> +  (display (G_ "
>> + -r, --recursive generate package expressions for all Gem packages
>> that are not yet in Guix"))
>
> Please keep lines below 80-char wide.
>
> Apart form that LGTM.  Could you send an  updated patch?

I applied all your suggestions.  Here is a new patch.

<#secure method=pgpmime mode=sign>

[0001-import-gem-Add-recursive-import.patch (text/x-patch, attachment)]

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 32046 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Oleg Pykhalov <go.wigust <at> gmail.com>
Cc: 32046 <at> debbugs.gnu.org
Subject: Re: [bug#32046] [PATCH] import: gem: Add recursive import.
Date: Tue, 10 Jul 2018 17:50:19 +0200

Hello,

Oleg Pykhalov <go.wigust <at> gmail.com> skribis:

> From ff11b55aac10d9012690586c670b171bce1f955b Mon Sep 17 00:00:00 2001
> From: Oleg Pykhalov <go.wigust <at> gmail.com>
> Date: Tue, 3 Jul 2018 23:28:42 +0300
> Subject: [PATCH] import: gem: Add recursive import.
>
> * doc/guix.texi (Invoking guix import): Document gem recursive import.
> * guix/import/gem.scm (gem->guix-package): Return package and dependencies
> values.
> (gem-recursive-import): New procedure.
> * guix/scripts/import/gem.scm (show-help, %options): Add recursive option.
> (guix-import-gem): Use 'gem-recursive-import'.
> * tests/gem.scm (test-json): Rename to 'test-foo-json'.
> ("gem->guix-package"): Use 'test-foo-json'.
> (test-bar-json, test-bundler-json): New variables.
> ("gem-recursive-import"): New test.

Awesome, it LGTM.  Thank you!

Ludo’.

Message #17 received at 32046 <at> debbugs.gnu.org (full text, mbox):

From: Oleg Pykhalov <go.wigust <at> gmail.com>
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: 32046-done <at> debbugs.gnu.org, 32046 <at> debbugs.gnu.org
Subject: Re: [bug#32046] [PATCH] import: gem: Add recursive import.
Date: Wed, 11 Jul 2018 07:13:49 +0300

[Message part 1 (text/plain, inline)]

ludo <at> gnu.org (Ludovic Courtès) writes:

> Oleg Pykhalov <go.wigust <at> gmail.com> skribis:
>
>> From ff11b55aac10d9012690586c670b171bce1f955b Mon Sep 17 00:00:00 2001
>> From: Oleg Pykhalov <go.wigust <at> gmail.com>
>> Date: Tue, 3 Jul 2018 23:28:42 +0300
>> Subject: [PATCH] import: gem: Add recursive import.
>>
>> * doc/guix.texi (Invoking guix import): Document gem recursive import.
>> * guix/import/gem.scm (gem->guix-package): Return package and dependencies
>> values.
>> (gem-recursive-import): New procedure.
>> * guix/scripts/import/gem.scm (show-help, %options): Add recursive option.
>> (guix-import-gem): Use 'gem-recursive-import'.
>> * tests/gem.scm (test-json): Rename to 'test-foo-json'.
>> ("gem->guix-package"): Use 'test-foo-json'.
>> (test-bar-json, test-bundler-json): New variables.
>> ("gem-recursive-import"): New test.
>
> Awesome, it LGTM.  Thank you!

Pushed as 88388766f778d344699e7a8a0a4d970c403007e3

Thanks,
Oleg.

[signature.asc (application/pgp-signature, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.