GNU bug report logs -
#35813
[PATCH] import: crate: add recursive option
Previous Next
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 35813 in the body.
You can then email your comments to 35813 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
guix-patches <at> gnu.org
:
bug#35813
; Package
guix-patches
.
(Mon, 20 May 2019 19:43:02 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Karl Meakin <karl.w.f.meakin <at> gmail.com>
:
New bug report received and forwarded. Copy sent to
guix-patches <at> gnu.org
.
(Mon, 20 May 2019 19:43:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
* guix/script/import/crate.scm: Add recursive option.
* guix/import/crate.scm (crate-recursive-import): New variable.
---
doc/guix.texi | 8 ++++++++
guix/import/crate.scm | 24 ++++++++++++++++--------
guix/scripts/import/crate.scm | 27 ++++++++++++++++++++++-----
3 files changed, 46 insertions(+), 13 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index ae9ad0739e..636bb7521d 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -8787,6 +8787,14 @@ in Guix.
Import metadata from the crates.io Rust package repository
@uref{https://crates.io, crates.io}.
+@table @code
+@item --recursive
+@itemx -r
+Traverse the dependency graph of the given upstream package recursively
+and generate package expressions for all those packages that are not yet
+in Guix.
+@end table
+
@item opam
@cindex OPAM
@cindex OCaml
diff --git a/guix/import/crate.scm b/guix/import/crate.scm
index e0b400d054..d8554b0e7a 100644
--- a/guix/import/crate.scm
+++ b/guix/import/crate.scm
@@ -37,6 +37,7 @@
#:use-module (srfi srfi-26)
#:export (crate->guix-package
guix-package->crate-name
+ crate-recursive-import
%crate-updater))
(define (crate-fetch crate-name callback)
@@ -86,8 +87,8 @@
VERSION, INPUTS, NATIVE-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
(let* ((port (http-fetch (crate-uri name version)))
(guix-name (crate-name->package-name name))
- (inputs (map crate-name->package-name inputs))
- (native-inputs (map crate-name->package-name native-inputs))
+ (input-packages (map crate-name->package-name inputs))
+ (native-input-packages (map crate-name->package-name native-inputs))
(pkg `(package
(name ,guix-name)
(version ,version)
@@ -99,8 +100,8 @@ VERSION, INPUTS, NATIVE-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
(base32
,(bytevector->nix-base32-string (port-sha256 port))))))
(build-system cargo-build-system)
- ,@(maybe-native-inputs native-inputs "src")
- ,@(maybe-inputs inputs "src")
+ ,@(maybe-native-inputs native-input-packages "src")
+ ,@(maybe-inputs input-packages "src")
(home-page ,(match home-page
(() "")
(_ home-page)))
@@ -111,12 +112,14 @@ VERSION, INPUTS, NATIVE-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
((license) license)
(_ `(list ,@license)))))))
(close-port port)
- pkg))
+ (values pkg inputs)))
-(define (crate->guix-package crate-name)
- "Fetch the metadata for CRATE-NAME from crates.io, and return the
+(define crate->guix-package
+ (memoize
+ (lambda* (crate-name _)
+ "Fetch the metadata for CRATE-NAME from crates.io, and return the
`package' s-expression corresponding to that package, or #f on failure."
- (crate-fetch crate-name make-crate-sexp))
+ (crate-fetch crate-name make-crate-sexp))))
(define (guix-package->crate-name package)
"Return the crate name of PACKAGE."
@@ -158,6 +161,11 @@ VERSION, INPUTS, NATIVE-INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, and LICENSE."
(version version)
(urls (list url)))))
+(define* (crate-recursive-import package-name)
+ (recursive-import package-name #f
+ #:repo->guix-package crate->guix-package
+ #:guix-name crate-name->package-name))
+
(define %crate-updater
(upstream-updater
(name 'crates)
diff --git a/guix/scripts/import/crate.scm b/guix/scripts/import/crate.scm
index cab9a4397b..8fadcdd57c 100644
--- a/guix/scripts/import/crate.scm
+++ b/guix/scripts/import/crate.scm
@@ -27,6 +27,7 @@
#:use-module (srfi srfi-1)
#:use-module (srfi srfi-11)
#:use-module (srfi srfi-37)
+ #:use-module (srfi srfi-41)
#:use-module (ice-9 match)
#:use-module (ice-9 format)
#:export (guix-import-crate))
@@ -45,6 +46,8 @@ Import and convert the crate.io package for PACKAGE-NAME.\n"))
(display (G_ "
-h, --help display this help and exit"))
(display (G_ "
+ -r, --recursive import packages recursively"))
+ (display (G_ "
-V, --version display version information and exit"))
(newline)
(show-bug-report-information))
@@ -58,6 +61,9 @@ Import and convert the crate.io package for PACKAGE-NAME.\n"))
(option '(#\V "version") #f #f
(lambda args
(show-version-and-exit "guix import crate")))
+ (option '(#\r "recursive") #f #f
+ (lambda (opt name arg result)
+ (alist-cons 'recursive #t result)))
%standard-import-options))
@@ -83,11 +89,22 @@ Import and convert the crate.io package for PACKAGE-NAME.\n"))
(reverse opts))))
(match args
((package-name)
- (let ((sexp (crate->guix-package package-name)))
- (unless sexp
- (leave (G_ "failed to download meta-data for package '~a'~%")
- package-name))
- sexp))
+ (if (assoc-ref opts 'recursive)
+ ;; Recursive import
+ (map (match-lambda
+ ((and ('package ('name name) . rest) pkg)
+ `(define-public ,(string->symbol name)
+ ,pkg))
+ (_ #f))
+ (reverse
+ (stream->list
+ (crate-recursive-import package-name))))
+ ;; Single import
+ (let ((sexp (crate->guix-package package-name #f)))
+ (unless sexp
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-name))
+ sexp)))
(()
(leave (G_ "too few arguments~%")))
((many ...)
--
2.21.0
Information forwarded
to
guix-patches <at> gnu.org
:
bug#35813
; Package
guix-patches
.
(Fri, 24 May 2019 15:43:01 GMT)
Full text and
rfc822 format available.
Message #8 received at 35813 <at> debbugs.gnu.org (full text, mbox):
Hi Karl,
Karl Meakin <karl.w.f.meakin <at> gmail.com> skribis:
> * guix/script/import/crate.scm: Add recursive option.
> * guix/import/crate.scm (crate-recursive-import): New variable.
Thanks for working on this! Please also mention the guix.texi change
here.
I prefer to let Ivan and Danny comment on this. My only question is:
would it be possible to augment ‘tests/crate.scm’ with tests for
recursive imports? That would be great.
Thanks,
Ludo’.
Information forwarded
to
guix-patches <at> gnu.org
:
bug#35813
; Package
guix-patches
.
(Sat, 25 May 2019 19:39:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 35813 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi Karl,
Thanks for the patch, overall the implementation looks good to me!
> On May 24, 2019, at 8:42 AM, Ludovic Courtès <ludo <at> gnu.org> wrote:
>
> would it be possible to augment ‘tests/crate.scm’ with tests for
> recursive imports? That would be great.
I second Ludo’s comment here, having some tests around this would be great!
Also, have you had a chance to test this by importing a Rust application?
I tried to import hexyl (which has a small dependency closure compared to other
applications), but the final result had some missing crate input definitions
(e.g. Maybe a crate depends on rust-foo, but rust-foo wasn't defined in the
output either). Maybe I did something wrong, but it would be good to confirm!
Also, don't worry about actually trying to build any imported packages as of
yet, you might hit a cycle issue between proc-macro2 and quote. I've got a
patch open[0] for fixing the cargo-build-system to handle this!
--Ivan
[0]: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35318 <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=35318>
[Message part 2 (text/html, inline)]
Information forwarded
to
guix-patches <at> gnu.org
:
bug#35813
; Package
guix-patches
.
(Sat, 20 Jul 2019 18:31:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 35813 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
What's the status on this? I was about to start working on a recursive
importer for Rust crates until I noticed this patch.
[Message part 2 (text/html, inline)]
Information forwarded
to
guix-patches <at> gnu.org
:
bug#35813
; Package
guix-patches
.
(Sat, 20 Jul 2019 21:44:01 GMT)
Full text and
rfc822 format available.
Message #17 received at 35813 <at> debbugs.gnu.org (full text, mbox):
Hi Brian,
> On Jul 20, 2019, at 11:30 AM, Brian Leung <bkleung89 <at> gmail.com> wrote:
>
> What's the status on this? I was about to start working on a recursive importer for Rust crates until I noticed this patch.
On the patch side a couple of updates are still needed:
* a rebase on the latest master
* an update to also include the native-inputs/dev-dependency inputs in the loop for importing additional crates
* some tests to verify the behavior
Not sure if Karl has a had a chance to update this yet
—Ivan
Information forwarded
to
guix-patches <at> gnu.org
:
bug#35813
; Package
guix-patches
.
(Mon, 05 Aug 2019 17:52:02 GMT)
Full text and
rfc822 format available.
Message #20 received at 35813 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
I took Karl's changes and updated them accordingly. I've also added a small
test. The patch containing his importer, my changes, and my test is
attached (the commit was made using my name--not sure if I should instead
apply Karl's patch).
[Message part 2 (text/html, inline)]
[0001-gnu-Add-crate-recursive-import.patch (text/x-patch, attachment)]
Information forwarded
to
guix-patches <at> gnu.org
:
bug#35813
; Package
guix-patches
.
(Tue, 06 Aug 2019 03:44:01 GMT)
Full text and
rfc822 format available.
Message #23 received at 35813 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
OK, I updated to remove print statements I missed.
On Mon, Aug 5, 2019 at 7:50 PM Brian Leung <bkleung89 <at> gmail.com> wrote:
> I took Karl's changes and updated them accordingly. I've also added a
> small test. The patch containing his importer, my changes, and my test is
> attached (the commit was made using my name--not sure if I should instead
> apply Karl's patch).
>
[Message part 2 (text/html, inline)]
[0001-gnu-Add-crate-recursive-import.patch (text/x-patch, attachment)]
Information forwarded
to
guix-patches <at> gnu.org
:
bug#35813
; Package
guix-patches
.
(Tue, 06 Aug 2019 16:05:01 GMT)
Full text and
rfc822 format available.
Message #26 received at 35813 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Should have sent this to you too, Ivan.
On Tue, Aug 6, 2019 at 5:42 AM Brian Leung <bkleung89 <at> gmail.com> wrote:
> OK, I updated to remove print statements I missed.
>
> On Mon, Aug 5, 2019 at 7:50 PM Brian Leung <bkleung89 <at> gmail.com> wrote:
>
>> I took Karl's changes and updated them accordingly. I've also added a
>> small test. The patch containing his importer, my changes, and my test is
>> attached (the commit was made using my name--not sure if I should instead
>> apply Karl's patch).
>>
>
[Message part 2 (text/html, inline)]
Information forwarded
to
guix-patches <at> gnu.org
:
bug#35813
; Package
guix-patches
.
(Thu, 08 Aug 2019 10:41:02 GMT)
Full text and
rfc822 format available.
Message #29 received at 35813 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
On Tue, Aug 06, 2019 at 06:03:23PM +0200, Brian Leung wrote:
> Should have sent this to you too, Ivan.
>
> On Tue, Aug 6, 2019 at 5:42 AM Brian Leung <bkleung89 <at> gmail.com> wrote:
>
> > OK, I updated to remove print statements I missed.
> >
> > On Mon, Aug 5, 2019 at 7:50 PM Brian Leung <bkleung89 <at> gmail.com> wrote:
> >
> >> I took Karl's changes and updated them accordingly. I've also added a
> >> small test. The patch containing his importer, my changes, and my test is
> >> attached (the commit was made using my name--not sure if I should instead
> >> apply Karl's patch).
> >>
> >
I ran 'guix import crate -r afl' on a machine where I had a bunch of
crates pre-packaged the (very shorted output looked like this:
(define-public rust-xdg
...
(define-public rust-xdg
...
(define-public rust-afl
(package
(name "rust-afl")
(version "0.4.4")
(source
(origin
(method url-fetch)
(uri (crate-uri "afl" version))
(file-name
(string-append name "-" version ".tar.gz"))
(sha256
(base32
"14k6hnwzqn7rrs0hs87vcfqj4334k9wff38d15378frlxpviaard"))))
(build-system cargo-build-system)
(arguments
`(#:cargo-inputs
(("rust-cc" ,rust-cc)
("rust-clap" ,rust-clap)
("rust-rustc-version" ,rust-rustc-version)
("rust-xdg" ,rust-xdg))
#:cargo-development-inputs
(("rust-rustc-version" ,rust-rustc-version)
("rust-xdg" ,rust-xdg))))
(home-page "https://github.com/rust-fuzz/afl.rs")
(synopsis
"Fuzzing Rust code with american-fuzzy-lop")
(description
"Fuzzing Rust code with american-fuzzy-lop")
(license #f)))
I know rust-xdg is there twice, but IMO it should only be printed once.
also 'guix import crate -r rusty-fork' gives me #f
'guix import crate rusty-fork' gives me:
guix import: error: failed to download meta-data for package 'rusty-fork'
--
Efraim Flashner <efraim <at> flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to
guix-patches <at> gnu.org
:
bug#35813
; Package
guix-patches
.
(Sat, 07 Sep 2019 21:51:01 GMT)
Full text and
rfc822 format available.
Message #32 received at 35813 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Hi Efraim,
An updated patch is attached.
I was and still am confused why my previous patch was yielding duplicates.
I'm not getting duplicates right now, even with packages that previously
yielded duplicates (I tried this on ripgrep); please test and let me know
if there are any issues. Maybe I fixed the issue in the course of rebasing?
And "guix import crate -r asfeusnoetuhesont" should now print "failed to
download meta-data for...", though I don't know how to do this more
idiomatically without using error or leave (which would end the recursive
import earlier than desired). And the double quotes actually appear, which
is not ideal. I'd appreciate any advice on how to clean this up.
Best,
Brian
On Thu, Aug 8, 2019 at 12:39 PM Efraim Flashner <efraim <at> flashner.co.il>
wrote:
> On Tue, Aug 06, 2019 at 06:03:23PM +0200, Brian Leung wrote:
> > Should have sent this to you too, Ivan.
> >
> > On Tue, Aug 6, 2019 at 5:42 AM Brian Leung <bkleung89 <at> gmail.com> wrote:
> >
> > > OK, I updated to remove print statements I missed.
> > >
> > > On Mon, Aug 5, 2019 at 7:50 PM Brian Leung <bkleung89 <at> gmail.com>
> wrote:
> > >
> > >> I took Karl's changes and updated them accordingly. I've also added a
> > >> small test. The patch containing his importer, my changes, and my
> test is
> > >> attached (the commit was made using my name--not sure if I should
> instead
> > >> apply Karl's patch).
> > >>
> > >
>
> I ran 'guix import crate -r afl' on a machine where I had a bunch of
> crates pre-packaged the (very shorted output looked like this:
>
> (define-public rust-xdg
> ...
>
> (define-public rust-xdg
> ...
>
> (define-public rust-afl
> (package
> (name "rust-afl")
> (version "0.4.4")
> (source
> (origin
> (method url-fetch)
> (uri (crate-uri "afl" version))
> (file-name
> (string-append name "-" version ".tar.gz"))
> (sha256
> (base32
> "14k6hnwzqn7rrs0hs87vcfqj4334k9wff38d15378frlxpviaard"))))
> (build-system cargo-build-system)
> (arguments
> `(#:cargo-inputs
> (("rust-cc" ,rust-cc)
> ("rust-clap" ,rust-clap)
> ("rust-rustc-version" ,rust-rustc-version)
> ("rust-xdg" ,rust-xdg))
> #:cargo-development-inputs
> (("rust-rustc-version" ,rust-rustc-version)
> ("rust-xdg" ,rust-xdg))))
> (home-page "https://github.com/rust-fuzz/afl.rs")
> (synopsis
> "Fuzzing Rust code with american-fuzzy-lop")
> (description
> "Fuzzing Rust code with american-fuzzy-lop")
> (license #f)))
>
> I know rust-xdg is there twice, but IMO it should only be printed once.
>
> also 'guix import crate -r rusty-fork' gives me #f
> 'guix import crate rusty-fork' gives me:
> guix import: error: failed to download meta-data for package 'rusty-fork'
>
>
> --
> Efraim Flashner <efraim <at> flashner.co.il> אפרים פלשנר
> GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
>
[Message part 2 (text/html, inline)]
[0001-gnu-Add-crate-recursive-import.patch (text/x-patch, attachment)]
Information forwarded
to
guix-patches <at> gnu.org
:
bug#35813
; Package
guix-patches
.
(Sun, 08 Sep 2019 07:58:01 GMT)
Full text and
rfc822 format available.
Message #35 received at 35813 <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
As a simple test I ran 'guix import crate encoding -r' and it gave me
the 6 packages I expected. 'guix import crate winapi -r' only gave me
rust-winapi, as the dependent crates are already packaged. When I tried
'guix import crate rand -r' it found the updated version and started
importing all the new dependencies also.
It looks good. I'm tempted to leave it running with 'guix import crate
serde -r' just to see if we're ever going to make it there.
I see that it imports A then B then C then D, and prints out D then C
then B then A. For the one I tested with is still rust-encoding. I'll
try my hand at ascii art:
encoding
|
-------------------------------------------------------
| | | | |
japanese korean simpchinese singlebyte tradchinese
| | | | |
-------------------------------------------------------
|
encoding-tests
import went encoding, japanese, tests, korean, simpchinese, singlebyte,
tradchinese
I think the only thing I would wish for would be to do tests, then the
languages and then encoding (best for upstreaming one at a time), or to
do them alphabetically (plop them in alphabetically all at once). This
I'm happy to live without I think.
The other thing was I ran 'guix import crate security-framework -r' and
after ~40 crates it crashed on me with:
web/http.scm:1186:15: In procedure read-response-line:
Bad Read-Header-Line header: #<eof>
and I would prefer to have the ~40 crates it did grab first to be
printed out and not lost. Between these two I would like most to not
lose the imported crates than worrying over the printed order.
Great job! From what I've tested I think it's ready as-is and any
changes would just be gravy.
--
Efraim Flashner <efraim <at> flashner.co.il> אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
[signature.asc (application/pgp-signature, inline)]
Reply sent
to
Brian Leung <bkleung89 <at> gmail.com>
:
You have taken responsibility.
(Sun, 13 Oct 2019 07:44:02 GMT)
Full text and
rfc822 format available.
Notification sent
to
Karl Meakin <karl.w.f.meakin <at> gmail.com>
:
bug acknowledged by developer.
(Sun, 13 Oct 2019 07:44:02 GMT)
Full text and
rfc822 format available.
Message #40 received at 35813-done <at> debbugs.gnu.org (full text, mbox):
[Message part 1 (text/plain, inline)]
Closing this since someone else already merged their own crate recursive
importer.
On Sun, Sep 8, 2019 at 12:57 AM Efraim Flashner <efraim <at> flashner.co.il>
wrote:
> As a simple test I ran 'guix import crate encoding -r' and it gave me
> the 6 packages I expected. 'guix import crate winapi -r' only gave me
> rust-winapi, as the dependent crates are already packaged. When I tried
> 'guix import crate rand -r' it found the updated version and started
> importing all the new dependencies also.
>
> It looks good. I'm tempted to leave it running with 'guix import crate
> serde -r' just to see if we're ever going to make it there.
>
> I see that it imports A then B then C then D, and prints out D then C
> then B then A. For the one I tested with is still rust-encoding. I'll
> try my hand at ascii art:
>
> encoding
> |
> -------------------------------------------------------
> | | | | |
> japanese korean simpchinese singlebyte tradchinese
> | | | | |
> -------------------------------------------------------
> |
> encoding-tests
>
> import went encoding, japanese, tests, korean, simpchinese, singlebyte,
> tradchinese
>
> I think the only thing I would wish for would be to do tests, then the
> languages and then encoding (best for upstreaming one at a time), or to
> do them alphabetically (plop them in alphabetically all at once). This
> I'm happy to live without I think.
>
> The other thing was I ran 'guix import crate security-framework -r' and
> after ~40 crates it crashed on me with:
> web/http.scm:1186:15: In procedure read-response-line:
> Bad Read-Header-Line header: #<eof>
> and I would prefer to have the ~40 crates it did grab first to be
> printed out and not lost. Between these two I would like most to not
> lose the imported crates than worrying over the printed order.
>
> Great job! From what I've tested I think it's ready as-is and any
> changes would just be gravy.
>
>
> --
> Efraim Flashner <efraim <at> flashner.co.il> אפרים פלשנר
> GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted
>
[Message part 2 (text/html, inline)]
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Sun, 10 Nov 2019 12:24:07 GMT)
Full text and
rfc822 format available.
This bug report was last modified 5 years and 279 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.