GNU bug report logs -
#26373
Various improvements to the CRAN importer
Previous Next
Reported by: Ricardo Wurmus <rekado <at> elephly.net>
Date: Wed, 5 Apr 2017 16:41:01 UTC
Severity: normal
Done: Ricardo Wurmus <rekado <at> elephly.net>
Bug is archived. No further changes may be made.
To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 26373 in the body.
You can then email your comments to 26373 AT debbugs.gnu.org in the normal way.
Toggle the display of automated, internal messages from the tracker.
Report forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Wed, 05 Apr 2017 16:41:01 GMT)
Full text and
rfc822 format available.
Acknowledgement sent
to
Ricardo Wurmus <rekado <at> elephly.net>
:
New bug report received and forwarded. Copy sent to
guix-patches <at> gnu.org
.
(Wed, 05 Apr 2017 16:41:02 GMT)
Full text and
rfc822 format available.
Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):
The following patch series improves the CRAN importer.
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Wed, 05 Apr 2017 16:43:01 GMT)
Full text and
rfc822 format available.
Message #8 received at 26373 <at> debbugs.gnu.org (full text, mbox):
* guix/import/cran.scm (bioconductor-package?): Exclude experiment packages,
because they cannot be updated with the default bioconductor updater.
---
guix/import/cran.scm | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/guix/import/cran.scm b/guix/import/cran.scm
index 8e24f6e17..f63d23972 100644
--- a/guix/import/cran.scm
+++ b/guix/import/cran.scm
@@ -429,7 +429,9 @@ dependencies."
;; the Github mirror, so we have to exclude them
;; from the set of bioconductor packages that can be
;; updated automatically.
- (not (string-contains uri "/data/annotation/"))))))
+ (not (string-contains uri "/data/annotation/"))
+ ;; Experiment packages are in a separate repository.
+ (not (string-contains uri "/data/experiment/"))))))
(and (string-prefix? "r-" (package-name package))
(match (and=> (package-source package) origin-uri)
((? string? uri)
--
2.12.2
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Wed, 05 Apr 2017 16:43:02 GMT)
Full text and
rfc822 format available.
Message #11 received at 26373 <at> debbugs.gnu.org (full text, mbox):
* guix/import/cran.scm (bioconductor-experiment-package?): New variable.
---
guix/import/cran.scm | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/guix/import/cran.scm b/guix/import/cran.scm
index f63d23972..48ab7355d 100644
--- a/guix/import/cran.scm
+++ b/guix/import/cran.scm
@@ -453,6 +453,19 @@ dependencies."
(any predicate uris))
(_ #f)))))
+(define (bioconductor-experiment-package? package)
+ "Return true if PACKAGE is an R experiment package from Bioconductor."
+ (let ((predicate (lambda (uri)
+ (and (string-prefix? "http://bioconductor.org" uri)
+ (string-contains uri "/data/experiment/")))))
+ (and (string-prefix? "r-" (package-name package))
+ (match (and=> (package-source package) origin-uri)
+ ((? string? uri)
+ (predicate uri))
+ ((? list? uris)
+ (any predicate uris))
+ (_ #f)))))
+
(define %cran-updater
(upstream-updater
(name 'cran)
--
2.12.2
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Wed, 05 Apr 2017 16:43:02 GMT)
Full text and
rfc822 format available.
Message #14 received at 26373 <at> debbugs.gnu.org (full text, mbox):
* guix/import/cran.scm (tarball-files-match-pattern?): New procedure.
(needs-zlib?): Implement in terms of "tarball-files-match-pattern?".
---
guix/import/cran.scm | 25 +++++++++++++++----------
1 file changed, 15 insertions(+), 10 deletions(-)
diff --git a/guix/import/cran.scm b/guix/import/cran.scm
index 48ab7355d..be3b678cd 100644
--- a/guix/import/cran.scm
+++ b/guix/import/cran.scm
@@ -201,17 +201,16 @@ empty list when the FIELD cannot be found."
(check "*.f95")
(check "*.f")))
-(define (needs-zlib? tarball)
- "Return #T if any of the Makevars files in the src directory of the TARBALL
-contain a zlib linker flag."
+(define (tarball-files-match-pattern? tarball regexp . file-patterns)
+ "Return #T if any of the files represented by FILE-PATTERNS in the TARBALL
+match the given REGEXP."
(call-with-temporary-directory
(lambda (dir)
- (let ((pattern (make-regexp "-lz")))
+ (let ((pattern (make-regexp regexp)))
(parameterize ((current-error-port (%make-void-port "rw+")))
- (system* "tar"
- "xf" tarball "-C" dir
- "--wildcards"
- "*/src/Makevars*" "*/src/configure*" "*/configure*"))
+ (apply system* "tar"
+ "xf" tarball "-C" dir
+ `("--wildcards" ,@file-patterns)))
(any (lambda (file)
(call-with-input-file file
(lambda (port)
@@ -220,10 +219,16 @@ contain a zlib linker flag."
(cond
((eof-object? line) #f)
((regexp-exec pattern line) #t)
- (else (loop)))))))
- #t)
+ (else (loop))))))))
(find-files dir))))))
+(define (needs-zlib? tarball)
+ "Return #T if any of the Makevars files in the src directory of the TARBALL
+contain a zlib linker flag."
+ (tarball-files-match-pattern?
+ tarball "-lz"
+ "*/src/Makevars*" "*/src/configure*" "*/configure*"))
+
(define (description->package repository meta)
"Return the `package' s-expression for an R package published on REPOSITORY
from the alist META, which was derived from the R package's DESCRIPTION file."
--
2.12.2
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Wed, 05 Apr 2017 16:43:03 GMT)
Full text and
rfc822 format available.
Message #17 received at 26373 <at> debbugs.gnu.org (full text, mbox):
* guix/import/cran.scm (needs-pkg-config?): New procedure.
(description->package): Use it.
---
guix/import/cran.scm | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/guix/import/cran.scm b/guix/import/cran.scm
index be3b678cd..423835637 100644
--- a/guix/import/cran.scm
+++ b/guix/import/cran.scm
@@ -229,6 +229,13 @@ contain a zlib linker flag."
tarball "-lz"
"*/src/Makevars*" "*/src/configure*" "*/configure*"))
+(define (needs-pkg-config? tarball)
+ "Return #T if any of the Makevars files in the src directory of the TARBALL
+reference the pkg-config tool."
+ (tarball-files-match-pattern?
+ tarball "pkg-config"
+ "*/src/Makevars*" "*/src/configure*" "*/configure*"))
+
(define (description->package repository meta)
"Return the `package' s-expression for an R package published on REPOSITORY
from the alist META, which was derived from the R package's DESCRIPTION file."
@@ -278,11 +285,12 @@ from the alist META, which was derived from the R package's DESCRIPTION file."
(build-system r-build-system)
,@(maybe-inputs sysdepends)
,@(maybe-inputs (map guix-name propagate) 'propagated-inputs)
- ,@(if (needs-fortran? tarball)
- `((native-inputs (,'quasiquote
- ,(list "gfortran"
- (list 'unquote 'gfortran)))))
- '())
+ ,@(maybe-inputs
+ `(,@(if (needs-fortran? tarball)
+ '("gfortran") '())
+ ,@(if (needs-pkg-config? tarball)
+ '("pkg-config") '()))
+ 'native-inputs)
(home-page ,(if (string-null? home-page)
(string-append base-url name)
home-page))
--
2.12.2
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Wed, 05 Apr 2017 16:43:03 GMT)
Full text and
rfc822 format available.
Message #20 received at 26373 <at> debbugs.gnu.org (full text, mbox):
* guix/import/cran.scm (package->upstream-name): Check that "start" and "end"
are valid before using them as substring indices.
---
guix/import/cran.scm | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/guix/import/cran.scm b/guix/import/cran.scm
index 423835637..557d694ad 100644
--- a/guix/import/cran.scm
+++ b/guix/import/cran.scm
@@ -384,9 +384,10 @@ dependencies."
((or (? string? url) (url _ ...))
(let ((end (string-rindex url #\_))
(start (string-rindex url #\/)))
- ;; The URL ends on
- ;; (string-append "/" name "_" version ".tar.gz")
- (substring url (+ start 1) end)))
+ (and start end
+ ;; The URL ends on
+ ;; (string-append "/" name "_" version ".tar.gz")
+ (substring url (+ start 1) end))))
(_ #f)))
(_ #f)))))
--
2.12.2
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Wed, 05 Apr 2017 16:43:04 GMT)
Full text and
rfc822 format available.
Message #23 received at 26373 <at> debbugs.gnu.org (full text, mbox):
* gnu/packages/bioinformatics.scm (latest-cran-release,
latest-bioconductor-release): Abort early when meta data cannot be downloaded.
---
guix/import/cran.scm | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/guix/import/cran.scm b/guix/import/cran.scm
index 557d694ad..fc7a1ed84 100644
--- a/guix/import/cran.scm
+++ b/guix/import/cran.scm
@@ -398,7 +398,8 @@ dependencies."
(package->upstream-name package))
(define meta
- (fetch-description 'cran upstream-name))
+ (false-if-exception
+ (fetch-description 'cran upstream-name)))
(and meta
(let ((version (assoc-ref meta "Version")))
@@ -415,7 +416,8 @@ dependencies."
(package->upstream-name package))
(define meta
- (fetch-description 'bioconductor upstream-name))
+ (false-if-exception
+ (fetch-description 'bioconductor upstream-name)))
(and meta
(let ((version (assoc-ref meta "Version")))
--
2.12.2
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Mon, 10 Apr 2017 09:51:02 GMT)
Full text and
rfc822 format available.
Message #26 received at 26373 <at> debbugs.gnu.org (full text, mbox):
Ricardo Wurmus <rekado <at> elephly.net> skribis:
> * guix/import/cran.scm (bioconductor-package?): Exclude experiment packages,
> because they cannot be updated with the default bioconductor updater.
OK.
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Mon, 10 Apr 2017 09:52:02 GMT)
Full text and
rfc822 format available.
Message #29 received at 26373 <at> debbugs.gnu.org (full text, mbox):
Ricardo Wurmus <rekado <at> elephly.net> skribis:
> * guix/import/cran.scm (bioconductor-experiment-package?): New variable.
It seems to be a private and unused procedure. Is it really needed?
:-)
Thanks,
Ludo’.
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Mon, 10 Apr 2017 09:53:01 GMT)
Full text and
rfc822 format available.
Message #32 received at 26373 <at> debbugs.gnu.org (full text, mbox):
Ricardo Wurmus <rekado <at> elephly.net> skribis:
> * guix/import/cran.scm (tarball-files-match-pattern?): New procedure.
> (needs-zlib?): Implement in terms of "tarball-files-match-pattern?".
LGTM!
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Mon, 10 Apr 2017 09:53:02 GMT)
Full text and
rfc822 format available.
Message #35 received at 26373 <at> debbugs.gnu.org (full text, mbox):
Ricardo Wurmus <rekado <at> elephly.net> skribis:
> * guix/import/cran.scm (needs-pkg-config?): New procedure.
> (description->package): Use it.
LGTM!
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Mon, 10 Apr 2017 09:54:01 GMT)
Full text and
rfc822 format available.
Message #38 received at 26373 <at> debbugs.gnu.org (full text, mbox):
Ricardo Wurmus <rekado <at> elephly.net> skribis:
> * guix/import/cran.scm (package->upstream-name): Check that "start" and "end"
> are valid before using them as substring indices.
LGTM!
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Mon, 10 Apr 2017 09:57:02 GMT)
Full text and
rfc822 format available.
Message #41 received at 26373 <at> debbugs.gnu.org (full text, mbox):
Ricardo Wurmus <rekado <at> elephly.net> skribis:
> * gnu/packages/bioinformatics.scm (latest-cran-release,
> latest-bioconductor-release): Abort early when meta data cannot be downloaded.
> ---
> guix/import/cran.scm | 6 ++++--
> 1 file changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/guix/import/cran.scm b/guix/import/cran.scm
> index 557d694ad..fc7a1ed84 100644
> --- a/guix/import/cran.scm
> +++ b/guix/import/cran.scm
> @@ -398,7 +398,8 @@ dependencies."
> (package->upstream-name package))
>
> (define meta
> - (fetch-description 'cran upstream-name))
> + (false-if-exception
> + (fetch-description 'cran upstream-name)))
I would prefer catching only the relevant exception. So I suppose
something like:
(guard (c ((http-get-error? c)
(if (= 404 (http-get-error-code c))
#f
(raise c))))
(fetch-description 'cran upstream-name))
However I see that ‘fetch-description’ already does that, so what
exceptions are we protecting against?
Thanks,
Ludo’.
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Mon, 10 Apr 2017 13:51:01 GMT)
Full text and
rfc822 format available.
Message #44 received at 26373 <at> debbugs.gnu.org (full text, mbox):
Ludovic Courtès <ludo <at> gnu.org> writes:
> Ricardo Wurmus <rekado <at> elephly.net> skribis:
>
>> * guix/import/cran.scm (bioconductor-experiment-package?): New variable.
>
> It seems to be a private and unused procedure. Is it really needed?
> :-)
Oh, it wasn’t supposed to be private. There are three kinds of
bioconductor packages: the normal ones, experiment packages, and
annotation packages.
The latter two cannot be updated the usual way, so they are excluded
from “bioconductor-package?”, but I thought that one may still want to
be able to distinguish between them. It’s true, though, that there’s
nothing that can use “bioconductor-experiment-package?” yet, but it’s
possible to add an updater for just the experiment packages. (They use
a different SVN repository.)
--
Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
https://elephly.net
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Mon, 10 Apr 2017 13:56:02 GMT)
Full text and
rfc822 format available.
Message #47 received at 26373 <at> debbugs.gnu.org (full text, mbox):
Ludovic Courtès <ludo <at> gnu.org> writes:
> Ricardo Wurmus <rekado <at> elephly.net> skribis:
>
>> * gnu/packages/bioinformatics.scm (latest-cran-release,
>> latest-bioconductor-release): Abort early when meta data cannot be downloaded.
>> ---
>> guix/import/cran.scm | 6 ++++--
>> 1 file changed, 4 insertions(+), 2 deletions(-)
>>
>> diff --git a/guix/import/cran.scm b/guix/import/cran.scm
>> index 557d694ad..fc7a1ed84 100644
>> --- a/guix/import/cran.scm
>> +++ b/guix/import/cran.scm
>> @@ -398,7 +398,8 @@ dependencies."
>> (package->upstream-name package))
>>
>> (define meta
>> - (fetch-description 'cran upstream-name))
>> + (false-if-exception
>> + (fetch-description 'cran upstream-name)))
>
> I would prefer catching only the relevant exception. So I suppose
> something like:
>
> (guard (c ((http-get-error? c)
> (if (= 404 (http-get-error-code c))
> #f
> (raise c))))
> (fetch-description 'cran upstream-name))
>
> However I see that ‘fetch-description’ already does that, so what
> exceptions are we protecting against?
I don’t know what the desired behaviour here is. When updating
packages, I think it’s good to keep going. If there’s an error
downloading the package meta data I want “meta” to be “#f”, which leads
to skipping the update.
Without turning errors to “#f” I wasn’t able to just update all packages
with “guix refresh -t cran,bioconductor -u”. That said, I don’t like
this. I feel that it would be better to add a “--keep-going” option for
“refresh” and implement it in the shared updater code rather than here.
--
Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
https://elephly.net
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Mon, 10 Apr 2017 21:36:01 GMT)
Full text and
rfc822 format available.
Message #50 received at 26373 <at> debbugs.gnu.org (full text, mbox):
Ricardo Wurmus <rekado <at> elephly.net> skribis:
> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> Ricardo Wurmus <rekado <at> elephly.net> skribis:
>>
>>> * guix/import/cran.scm (bioconductor-experiment-package?): New variable.
>>
>> It seems to be a private and unused procedure. Is it really needed?
>> :-)
>
> Oh, it wasn’t supposed to be private. There are three kinds of
> bioconductor packages: the normal ones, experiment packages, and
> annotation packages.
>
> The latter two cannot be updated the usual way, so they are excluded
> from “bioconductor-package?”, but I thought that one may still want to
> be able to distinguish between them. It’s true, though, that there’s
> nothing that can use “bioconductor-experiment-package?” yet, but it’s
> possible to add an updater for just the experiment packages. (They use
> a different SVN repository.)
OK, I see. Make sure to export it, then! :-)
Thanks for the explanation,
Ludo’.
Information forwarded
to
guix-patches <at> gnu.org
:
bug#26373
; Package
guix-patches
.
(Mon, 10 Apr 2017 21:38:02 GMT)
Full text and
rfc822 format available.
Message #53 received at 26373 <at> debbugs.gnu.org (full text, mbox):
Ricardo Wurmus <rekado <at> elephly.net> skribis:
> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>> Ricardo Wurmus <rekado <at> elephly.net> skribis:
>>
>>> * gnu/packages/bioinformatics.scm (latest-cran-release,
>>> latest-bioconductor-release): Abort early when meta data cannot be downloaded.
>>> ---
>>> guix/import/cran.scm | 6 ++++--
>>> 1 file changed, 4 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/guix/import/cran.scm b/guix/import/cran.scm
>>> index 557d694ad..fc7a1ed84 100644
>>> --- a/guix/import/cran.scm
>>> +++ b/guix/import/cran.scm
>>> @@ -398,7 +398,8 @@ dependencies."
>>> (package->upstream-name package))
>>>
>>> (define meta
>>> - (fetch-description 'cran upstream-name))
>>> + (false-if-exception
>>> + (fetch-description 'cran upstream-name)))
>>
>> I would prefer catching only the relevant exception. So I suppose
>> something like:
>>
>> (guard (c ((http-get-error? c)
>> (if (= 404 (http-get-error-code c))
>> #f
>> (raise c))))
>> (fetch-description 'cran upstream-name))
>>
>> However I see that ‘fetch-description’ already does that, so what
>> exceptions are we protecting against?
>
> I don’t know what the desired behaviour here is. When updating
> packages, I think it’s good to keep going. If there’s an error
> downloading the package meta data I want “meta” to be “#f”, which leads
> to skipping the update.
>
> Without turning errors to “#f” I wasn’t able to just update all packages
> with “guix refresh -t cran,bioconductor -u”.
What was the exception?
I think a good approach is to catch precisely the kind of error that we
don’t want to see. ‘false-if-exception’ catches everything and could
thus hide genuine errors/bugs (including unbound variables and similar),
which sounds undesirable.
WDYT?
Ludo’.
Reply sent
to
Ricardo Wurmus <rekado <at> elephly.net>
:
You have taken responsibility.
(Tue, 16 May 2017 19:49:01 GMT)
Full text and
rfc822 format available.
Notification sent
to
Ricardo Wurmus <rekado <at> elephly.net>
:
bug acknowledged by developer.
(Tue, 16 May 2017 19:49:01 GMT)
Full text and
rfc822 format available.
Message #58 received at 26373-done <at> debbugs.gnu.org (full text, mbox):
Ludovic Courtès <ludo <at> gnu.org> writes:
> Ricardo Wurmus <rekado <at> elephly.net> skribis:
>
>> Ludovic Courtès <ludo <at> gnu.org> writes:
>>
>>> Ricardo Wurmus <rekado <at> elephly.net> skribis:
>>>
>>>> * gnu/packages/bioinformatics.scm (latest-cran-release,
>>>> latest-bioconductor-release): Abort early when meta data cannot be downloaded.
>>>> ---
>>>> guix/import/cran.scm | 6 ++++--
>>>> 1 file changed, 4 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/guix/import/cran.scm b/guix/import/cran.scm
>>>> index 557d694ad..fc7a1ed84 100644
>>>> --- a/guix/import/cran.scm
>>>> +++ b/guix/import/cran.scm
>>>> @@ -398,7 +398,8 @@ dependencies."
>>>> (package->upstream-name package))
>>>>
>>>> (define meta
>>>> - (fetch-description 'cran upstream-name))
>>>> + (false-if-exception
>>>> + (fetch-description 'cran upstream-name)))
>>>
>>> I would prefer catching only the relevant exception. So I suppose
>>> something like:
>>>
>>> (guard (c ((http-get-error? c)
>>> (if (= 404 (http-get-error-code c))
>>> #f
>>> (raise c))))
>>> (fetch-description 'cran upstream-name))
>>>
>>> However I see that ‘fetch-description’ already does that, so what
>>> exceptions are we protecting against?
>>
>> I don’t know what the desired behaviour here is. When updating
>> packages, I think it’s good to keep going. If there’s an error
>> downloading the package meta data I want “meta” to be “#f”, which leads
>> to skipping the update.
>>
>> Without turning errors to “#f” I wasn’t able to just update all packages
>> with “guix refresh -t cran,bioconductor -u”.
>
> What was the exception?
>
> I think a good approach is to catch precisely the kind of error that we
> don’t want to see. ‘false-if-exception’ catches everything and could
> thus hide genuine errors/bugs (including unbound variables and similar),
> which sounds undesirable.
>
> WDYT?
I don’t remember why I wanted this, so I didn’t push this patch. I
agree that it’s much better to catch errors precisely.
Thanks for the review! I pushed all the other patches (with the
exception of one that Mathieu had already implemented independently) to
master.
--
Ricardo
GPG: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC
https://elephly.net
bug archived.
Request was from
Debbugs Internal Request <help-debbugs <at> gnu.org>
to
internal_control <at> debbugs.gnu.org
.
(Wed, 14 Jun 2017 11:24:04 GMT)
Full text and
rfc822 format available.
This bug report was last modified 8 years and 64 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.