From debbugs-submit-bounces@debbugs.gnu.org Sun Aug 04 06:26:09 2019 Received: (at submit) by debbugs.gnu.org; 4 Aug 2019 10:26:09 +0000 Received: from localhost ([127.0.0.1]:60463 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1huDiL-0006JY-Cb for submit@debbugs.gnu.org; Sun, 04 Aug 2019 06:26:09 -0400 Received: from lists.gnu.org ([209.51.188.17]:37671) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1huDiI-0006JP-2R for submit@debbugs.gnu.org; Sun, 04 Aug 2019 06:26:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57947) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1huDiH-0006NF-2V for guix-patches@gnu.org; Sun, 04 Aug 2019 06:26:06 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.7 required=5.0 tests=BAYES_40,RCVD_IN_DNSWL_LOW, URIBL_BLOCKED autolearn=disabled version=3.3.2 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1huDiG-0004kN-5B for guix-patches@gnu.org; Sun, 04 Aug 2019 06:26:05 -0400 Received: from mail-out.m-online.net ([212.18.0.10]:47793) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1huDiF-0004ih-Vg for guix-patches@gnu.org; Sun, 04 Aug 2019 06:26:04 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 461cVw6DGqz1rGRh for ; Sun, 4 Aug 2019 12:26:00 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 461cVw5M88z1qqkk for ; Sun, 4 Aug 2019 12:26:00 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id BbXtngaXReSv for ; Sun, 4 Aug 2019 12:26:00 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-55-210.dynamic.mnet-online.de [188.174.55.210]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS for ; Sun, 4 Aug 2019 12:26:00 +0200 (CEST) Received: from thisbe.fritz.box (unknown [192.168.110.24]) by hermia.goebel-consult.de (Postfix) with ESMTP id 21ADC60052 for ; Sun, 4 Aug 2019 12:26:45 +0200 (CEST) From: Hartmut Goebel To: guix-patches@gnu.org Subject: [PATCH 0/2] Make the KDE updater find packaes in subdirectories Date: Sun, 4 Aug 2019 12:25:59 +0200 Message-Id: <20190804102559.32296-1-h.goebel@crazy-compilers.com> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 212.18.0.10 X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Als make the KDE updater no longer rely on FTP access. Formerly packages living in a path like /stable/frameworks/5.60/portingAids/kross-5.60.0.tar.xz have not been found due to the additional directory level. These patches change the KDE updater to download the "ls -lR" file from downloader.kde.org and use a pattern for searching the relevant files. Sine I'm sitt a beginner in Guile, I apprechiate any enhancement suggesti= on to the code. I assume esp. the loop in download.kde.org-files and canonicalize-path could be written more scheme-ish. Hartmut Goebel (2): gnu-maintenance: KDE updater no longer relies on FTP access. gnu-maintenance: KDE updater finds packages even in sub-directory. guix/gnu-maintenance.scm | 99 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 92 insertions(+), 7 deletions(-) --=20 2.21.0 From debbugs-submit-bounces@debbugs.gnu.org Sun Aug 04 06:29:01 2019 Received: (at 36919) by debbugs.gnu.org; 4 Aug 2019 10:29:01 +0000 Received: from localhost ([127.0.0.1]:60469 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1huDl7-0006Nd-2s for submit@debbugs.gnu.org; Sun, 04 Aug 2019 06:29:01 -0400 Received: from mail-out.m-online.net ([212.18.0.9]:48899) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1huDl4-0006NO-Hl for 36919@debbugs.gnu.org; Sun, 04 Aug 2019 06:28:59 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 461cZK567Dz1rHDX for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:28:57 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 461cZK3kHhz1qqkk for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:28:57 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id y6F1Nvk9O4mK for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:28:56 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-55-210.dynamic.mnet-online.de [188.174.55.210]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:28:56 +0200 (CEST) Received: from thisbe.fritz.box (unknown [192.168.110.24]) by hermia.goebel-consult.de (Postfix) with ESMTP id 2535860052 for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:29:42 +0200 (CEST) From: Hartmut Goebel To: 36919@debbugs.gnu.org Subject: [PATCH 0/2] Make the KDE updater find packaes in subdirectories Date: Sun, 4 Aug 2019 12:28:54 +0200 Message-Id: <20190804102856.32609-1-h.goebel@crazy-compilers.com> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 36919 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Als make the KDE updater no longer rely on FTP access. Formerly packages living in a path like /stable/frameworks/5.60/portingAids/kross-5.60.0.tar.xz have not been found due to the additional directory level. These patches change the KDE updater to download the "ls -lR" file from downloader.kde.org and use a pattern for searching the relevant files. Sine I'm sitt a beginner in Guile, I apprechiate any enhancement suggestion to the code. I assume esp. the loop in download.kde.org-files and canonicalize-path could be written more scheme-ish. Hartmut Goebel (2): gnu-maintenance: KDE updater no longer relies on FTP access. gnu-maintenance: KDE updater finds packages even in sub-directory. guix/gnu-maintenance.scm | 99 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 92 insertions(+), 7 deletions(-) -- 2.21.0 From debbugs-submit-bounces@debbugs.gnu.org Sun Aug 04 06:29:07 2019 Received: (at 36919) by debbugs.gnu.org; 4 Aug 2019 10:29:07 +0000 Received: from localhost ([127.0.0.1]:60473 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1huDl8-0006O0-Bv for submit@debbugs.gnu.org; Sun, 04 Aug 2019 06:29:07 -0400 Received: from mail-out.m-online.net ([212.18.0.9]:40910) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1huDl5-0006NR-TP for 36919@debbugs.gnu.org; Sun, 04 Aug 2019 06:29:00 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 461cZM3FYVz1rJ78 for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:28:59 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 461cZM39Hhz1qqkk for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:28:59 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id 5jTlVdkBMeYS for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:28:58 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-55-210.dynamic.mnet-online.de [188.174.55.210]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:28:58 +0200 (CEST) Received: from thisbe.fritz.box (unknown [192.168.110.24]) by hermia.goebel-consult.de (Postfix) with ESMTP id 482C4600A3 for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:29:42 +0200 (CEST) From: Hartmut Goebel To: 36919@debbugs.gnu.org Subject: [PATCH 1/2] gnu-maintenance: KDE updater no longer relies on FTP access. Date: Sun, 4 Aug 2019 12:28:55 +0200 Message-Id: <20190804102856.32609-2-h.goebel@crazy-compilers.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190804102856.32609-1-h.goebel@crazy-compilers.com> References: <20190804102856.32609-1-h.goebel@crazy-compilers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 36919 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * guix/gnu-maintenance.scm (%kde-file-list-uri): New variable. (download.kde.org-files): New procedure. (latest-kde-release): Change to use DOWNLOAD.KDE.ORG-FILES and search for files in this list. --- guix/gnu-maintenance.scm | 77 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 70 insertions(+), 7 deletions(-) diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm index d63d44f629..730e2519ee 100644 --- a/guix/gnu-maintenance.scm +++ b/guix/gnu-maintenance.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Ludovic Courtès ;;; Copyright © 2012, 2013 Nikita Karetnikov +;;; Copyright © 2019 Hartmut Goebel ;;; ;;; This file is part of GNU Guix. ;;; @@ -24,6 +25,7 @@ #:use-module (sxml simple) #:use-module (ice-9 regex) #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) @@ -615,15 +617,76 @@ releases are on gnu.org." (define gnu-hosted? (url-prefix-predicate "mirror://gnu/")) +(define %kde-file-list-uri + ;; URI of the file list (ls -lR format) for download.kde.org. + (string->uri "https://download.kde.org/ls-lR.bz2")) + +(define download.kde.org-files + (mlambda () + "Return the list of files available at download.kde.org." + ;; XXX: Memoize the whole procedure to work around the fact that + ;; 'http-fetch/cached' caches the bzip2-compressed version. + + (define (canonicalize-path path) + (if (string-prefix? "/srv/archives/ftp/" path) + (set! path (string-drop path 17))) + (if (string-suffix? ":" path) + (set! path (string-drop-right path 1))) + (if (not (string-suffix? "/" path)) + (set! path (string-append path "/"))) + path) + + (define (ls-lR-line->filename path line) + ;; remove mode, blocks, user, group, size, date, time and one space + (regexp-substitute + #f (string-match "^(\\S+\\s+){6}\\S+\\s" line) path 'post)) + + (let ((entries `()) + (port (decompressed-port + 'bzip2 + (http-fetch/cached %kde-file-list-uri #:ttl 3600)))) + (do ((path (read-line port) (read-line port))) + ((or (eof-object? path) (string= path ""))) + (set! path (canonicalize-path path)) + (do ((line (read-line port) (read-line port))) + ((or (eof-object? line) (string= line ""))) + (if (string-prefix? "-" line) + ;; regular file + (set! entries + (cons (ls-lR-line->filename path line) + entries))))) + entries))) + (define (latest-kde-release package) "Return the latest release of PACKAGE, the name of an KDE.org package." - (let ((uri (string->uri (origin-uri (package-source package))))) - (false-if-ftp-error - (latest-ftp-release - (package-upstream-name package) - #:server "ftp.mirrorservice.org" - #:directory (string-append "/sites/ftp.kde.org/pub/kde/" - (dirname (dirname (uri-path uri)))))))) + (let* ((uri (string->uri (origin-uri (package-source package)))) + (directory (dirname (dirname (uri-path uri)))) + (name (package-upstream-name package)) + (files (download.kde.org-files)) + (relevant (filter (lambda (file) + (and (string-prefix? directory file) + (release-file? name (basename file)) + )) + files))) + (match (sort relevant (lambda (file1 file2) + (version>? (sans-extension (basename file1)) + (sans-extension (basename file2))))) + ((and tarballs (reference _ ...)) + (let* ((version (tarball->version reference)) + (tarballs (filter (lambda (file) + (string=? (sans-extension + (basename file)) + (sans-extension + (basename reference)))) + tarballs))) + (upstream-source + (package name) + (version version) + (urls (map (lambda (file) + (string-append "mirror://kde/" file)) + tarballs))))) + (() + #f)))) (define (latest-xorg-release package) "Return the latest release of PACKAGE, the name of an X.org package." -- 2.21.0 From debbugs-submit-bounces@debbugs.gnu.org Sun Aug 04 06:29:08 2019 Received: (at 36919) by debbugs.gnu.org; 4 Aug 2019 10:29:08 +0000 Received: from localhost ([127.0.0.1]:60475 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1huDlD-0006OM-Ss for submit@debbugs.gnu.org; Sun, 04 Aug 2019 06:29:08 -0400 Received: from mail-out.m-online.net ([212.18.0.10]:46317) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1huDl7-0006Np-VP for 36919@debbugs.gnu.org; Sun, 04 Aug 2019 06:29:02 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 461cZP3fVKz1r7hh for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:29:01 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 461cZP0vrBz1qqkk for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:29:01 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id AIFtA_O-W634 for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:28:59 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-55-210.dynamic.mnet-online.de [188.174.55.210]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:28:59 +0200 (CEST) Received: from thisbe.fritz.box (unknown [192.168.110.24]) by hermia.goebel-consult.de (Postfix) with ESMTP id 5BD72601D6 for <36919@debbugs.gnu.org>; Sun, 4 Aug 2019 12:29:42 +0200 (CEST) From: Hartmut Goebel To: 36919@debbugs.gnu.org Subject: [PATCH 2/2] gnu-maintenance: KDE updater finds packages even in sub-directory. Date: Sun, 4 Aug 2019 12:28:56 +0200 Message-Id: <20190804102856.32609-3-h.goebel@crazy-compilers.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190804102856.32609-1-h.goebel@crazy-compilers.com> References: <20190804102856.32609-1-h.goebel@crazy-compilers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 36919 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Fixes and finally fixes . Formerly packages living in a path like /stable/frameworks/5.60/portingAids/kross-5.60.0.tar.xz have not been found. * guix/gnu-maintenance.scm (uri->kde-path-pattern): New procedure. (latest-kde-release): Use pattern to search for file. --- guix/gnu-maintenance.scm | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm index 730e2519ee..d76ef2c5aa 100644 --- a/guix/gnu-maintenance.scm +++ b/guix/gnu-maintenance.scm @@ -657,14 +657,36 @@ releases are on gnu.org." entries))))) entries))) +(define (uri->kde-path-pattern uri) + + (define version-regexp + (make-regexp + (string-join '("^([0-9]+\\.)+[0-9]+-?" ;; 5.12.90, 4.2.0-preview + "^[0-9]+$" ;; 20031002 + ".*-([0-9]+\\.)+[0-9]+$") ;; kdepim-4.6.1 + "|"))) + + (define (version->pattern part) + ;; If a path element might be a version´, replace it by a catch-all part + (if (regexp-exec version-regexp part) + "[^/]+" + part)) + + (let* ((path (uri-path uri)) + (directory-parts (string-split (dirname path) #\/))) + (make-regexp + (string-append + (string-join (map version->pattern directory-parts) "/") + "/")))) + (define (latest-kde-release package) "Return the latest release of PACKAGE, the name of an KDE.org package." (let* ((uri (string->uri (origin-uri (package-source package)))) - (directory (dirname (dirname (uri-path uri)))) + (path-rx (uri->kde-path-pattern uri)) (name (package-upstream-name package)) (files (download.kde.org-files)) (relevant (filter (lambda (file) - (and (string-prefix? directory file) + (and (regexp-exec path-rx file) (release-file? name (basename file)) )) files))) -- 2.21.0 From debbugs-submit-bounces@debbugs.gnu.org Sat Aug 17 17:01:21 2019 Received: (at 36919) by debbugs.gnu.org; 17 Aug 2019 21:01:21 +0000 Received: from localhost ([127.0.0.1]:56894 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hz5pB-0007sN-Cd for submit@debbugs.gnu.org; Sat, 17 Aug 2019 17:01:21 -0400 Received: from eggs.gnu.org ([209.51.188.92]:51881) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1hz5p9-0007sB-IX for 36919@debbugs.gnu.org; Sat, 17 Aug 2019 17:01:20 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48843) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hz5p4-0000jc-0t; Sat, 17 Aug 2019 17:01:14 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51936 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1hz5p3-0004TW-Eg; Sat, 17 Aug 2019 17:01:13 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Hartmut Goebel Subject: Re: [bug#36919] [PATCH 1/2] gnu-maintenance: KDE updater no longer relies on FTP access. References: <20190804102856.32609-1-h.goebel@crazy-compilers.com> <20190804102856.32609-2-h.goebel@crazy-compilers.com> Date: Sat, 17 Aug 2019 23:01:11 +0200 In-Reply-To: <20190804102856.32609-2-h.goebel@crazy-compilers.com> (Hartmut Goebel's message of "Sun, 4 Aug 2019 12:28:55 +0200") Message-ID: <87k1bbcxrc.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 36919 Cc: 36919@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi Hartmut, Hartmut Goebel skribis: > * guix/gnu-maintenance.scm (%kde-file-list-uri): New variable. > (download.kde.org-files): New procedure. > (latest-kde-release): Change to use DOWNLOAD.KDE.ORG-FILES and search > for files in this list. Nice! How about moving this code to (guix import kde) as was done for (guix import gnome) when we discussed it back then? (See .) > +(define download.kde.org-files > + (mlambda () > + "Return the list of files available at download.kde.org." > + ;; XXX: Memoize the whole procedure to work around the fact that > + ;; 'http-fetch/cached' caches the bzip2-compressed version. > + > + (define (canonicalize-path path) > + (if (string-prefix? "/srv/archives/ftp/" path) > + (set! path (string-drop path 17))) > + (if (string-suffix? ":" path) > + (set! path (string-drop-right path 1))) > + (if (not (string-suffix? "/" path)) > + (set! path (string-append path "/"))) > + path) As a rule of thumb we don=E2=80=99t use =E2=80=98set!=E2=80=99 in Guix, exc= ept in special circumstances. In this case you can write: (define (canonicalize-path path) (cond ((string-prefix? =E2=80=A6) (string-drop path 17)) ((string-suffix? =E2=80=A6) (string-drop-right path 1)) =E2=80=A6)) > + (define (ls-lR-line->filename path line) > + ;; remove mode, blocks, user, group, size, date, time and one space > + (regexp-substitute > + #f (string-match "^(\\S+\\s+){6}\\S+\\s" line) path 'post)) > + > + (let ((entries `()) > + (port (decompressed-port > + 'bzip2 > + (http-fetch/cached %kde-file-list-uri #:ttl 3600)))) What about passing =E2=80=98http-fetch/cached=E2=80=99 a custom #:write-cac= he, as is done in (guix cve)? That would allow us to store the cached file list in a pre-processed (and possibly decompressed) format, speeding up operation on cache hits. > + (do ((path (read-line port) (read-line port))) > + ((or (eof-object? path) (string=3D path ""))) > + (set! path (canonicalize-path path)) I also recommend against =E2=80=98do=E2=80=99. You can use a =E2=80=9Cname= d let=E2=80=9D loop instead, as in: (let loop ((files '())) (match (read-line port) ((? eof-object?) (reverse files)) (line (loop (cons =E2=80=A6 files))))) That=E2=80=99s about it. Thanks! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Aug 27 04:11:08 2019 Received: (at 36919) by debbugs.gnu.org; 27 Aug 2019 08:11:08 +0000 Received: from localhost ([127.0.0.1]:47639 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i2WZI-0007cp-Dc for submit@debbugs.gnu.org; Tue, 27 Aug 2019 04:11:08 -0400 Received: from mail-out.m-online.net ([212.18.0.10]:51656) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i2WZG-0007cf-LC for 36919@debbugs.gnu.org; Tue, 27 Aug 2019 04:11:07 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 46HhQd0dszz1rGS1; Tue, 27 Aug 2019 10:11:04 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 46HhQc4xrQz1qqkp; Tue, 27 Aug 2019 10:11:04 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id fsH5wqBIRsAp; Tue, 27 Aug 2019 10:11:03 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-55-68.dynamic.mnet-online.de [188.174.55.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS; Tue, 27 Aug 2019 10:11:03 +0200 (CEST) Received: from [192.168.110.2] (lenashee.goebel-consult.de [192.168.110.2]) by hermia.goebel-consult.de (Postfix) with ESMTP id 9BEEF6018D; Tue, 27 Aug 2019 10:12:39 +0200 (CEST) Subject: Re: [bug#36919] [PATCH 1/2] gnu-maintenance: KDE updater no longer relies on FTP access. To: =?UTF-8?Q?Ludovic_Court=c3=a8s?= References: <20190804102856.32609-1-h.goebel@crazy-compilers.com> <20190804102856.32609-2-h.goebel@crazy-compilers.com> <87k1bbcxrc.fsf@gnu.org> From: Hartmut Goebel Openpgp: preference=signencrypt Organization: crazy-compilers.com Message-ID: Date: Tue, 27 Aug 2019 10:11:02 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <87k1bbcxrc.fsf@gnu.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 36919 Cc: 36919@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) Am 17.08.19 um 23:01 schrieb Ludovic Courtès: > Nice! Thansk :-) > How about moving this code to (guix import kde) as was done for (guix > import gnome) when we discussed it back then? (See > .) I'll be fine with this. I just wonder whether we/I should refactor the new code to be more flexible for other ls-lR cases and keep the common parts in gnu-maintenance.scm. OTOH currently there is no other use-case -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | From debbugs-submit-bounces@debbugs.gnu.org Tue Aug 27 04:30:39 2019 Received: (at 36919) by debbugs.gnu.org; 27 Aug 2019 08:30:39 +0000 Received: from localhost ([127.0.0.1]:47658 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i2WsA-00086A-Mz for submit@debbugs.gnu.org; Tue, 27 Aug 2019 04:30:38 -0400 Received: from mail-out.m-online.net ([212.18.0.9]:46147) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i2Ws8-000860-Ix for 36919@debbugs.gnu.org; Tue, 27 Aug 2019 04:30:37 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 46Hhs62LH1z1sBSK; Tue, 27 Aug 2019 10:30:34 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 46Hhs61VYwz1qqkp; Tue, 27 Aug 2019 10:30:34 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id oz7WeXa5hlrZ; Tue, 27 Aug 2019 10:30:33 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-55-68.dynamic.mnet-online.de [188.174.55.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS; Tue, 27 Aug 2019 10:30:33 +0200 (CEST) Received: from [192.168.110.2] (lenashee.goebel-consult.de [192.168.110.2]) by hermia.goebel-consult.de (Postfix) with ESMTP id 2F468601CF; Tue, 27 Aug 2019 10:32:09 +0200 (CEST) Subject: Re: [bug#36919] [PATCH 1/2] gnu-maintenance: KDE updater no longer relies on FTP access. To: =?UTF-8?Q?Ludovic_Court=c3=a8s?= References: <20190804102856.32609-1-h.goebel@crazy-compilers.com> <20190804102856.32609-2-h.goebel@crazy-compilers.com> <87k1bbcxrc.fsf@gnu.org> From: Hartmut Goebel Openpgp: preference=signencrypt Organization: crazy-compilers.com Message-ID: <80cc20cb-589b-30df-4a3e-896a21d42eba@crazy-compilers.com> Date: Tue, 27 Aug 2019 10:30:32 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <87k1bbcxrc.fsf@gnu.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 36919 Cc: 36919@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) Hi Ludo, thanks for the coding advice. This was what I've been asking for :-) Just one point: Am 17.08.19 um 23:01 schrieb Ludovic Courtès: > As a rule of thumb we don’t use ‘set!’ in Guix, except in special > circumstances. In this case you can write: > > (define (canonicalize-path path) > (cond ((string-prefix? …) > (string-drop path 17)) > ((string-suffix? …) > (string-drop-right path 1)) > …)) AFAIK, `cond` only processes the first expression where `test ` is true. In this case, we need to process *all* cases where the test is true. This means we need to nest the evaluation, which is ugly and hard to read IMHO. Is there some more "linear" syntax? (BTW: The manual [1] is not quite precise on `cond`, so I needed to test it. Maybe I did it wrong.) [1] https://www.gnu.org/software/guile/manual/html_node/Conditionals.html -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | From debbugs-submit-bounces@debbugs.gnu.org Sun Sep 01 15:44:00 2019 Received: (at 36919) by debbugs.gnu.org; 1 Sep 2019 19:44:01 +0000 Received: from localhost ([127.0.0.1]:57254 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i4VlY-0003KW-LE for submit@debbugs.gnu.org; Sun, 01 Sep 2019 15:44:00 -0400 Received: from eggs.gnu.org ([209.51.188.92]:43980) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i4VlX-0003KK-U3 for 36919@debbugs.gnu.org; Sun, 01 Sep 2019 15:44:00 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:51439) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1i4VlS-0007B6-Bl; Sun, 01 Sep 2019 15:43:54 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=40590 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1i4VlR-0003Ds-Ow; Sun, 01 Sep 2019 15:43:54 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Hartmut Goebel Subject: Re: [bug#36919] [PATCH 1/2] gnu-maintenance: KDE updater no longer relies on FTP access. References: <20190804102856.32609-1-h.goebel@crazy-compilers.com> <20190804102856.32609-2-h.goebel@crazy-compilers.com> <87k1bbcxrc.fsf@gnu.org> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 15 Fructidor an 227 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Sun, 01 Sep 2019 21:43:51 +0200 In-Reply-To: (Hartmut Goebel's message of "Tue, 27 Aug 2019 10:11:02 +0200") Message-ID: <87mufnajk8.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 36919 Cc: 36919@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi, Hartmut Goebel skribis: >> How about moving this code to (guix import kde) as was done for (guix >> import gnome) when we discussed it back then? (See >> .) > > I'll be fine with this. > > I just wonder whether we/I should refactor the new code to be more > flexible for other ls-lR cases and keep the common parts in > gnu-maintenance.scm. OTOH currently there is no other use-case Yeah, we=E2=80=99d have to identify what common parts exist. On IRC we dis= cussed utility procedures like =E2=80=98file-sans-extension=E2=80=99, which would = be worth factorizing. Other things may not be good candidates=E2=80=94for instance, the GNU thing= is probably close to what you=E2=80=99d write for KDE, but it=E2=80=99s still = not exactly the same. Since there=E2=80=99s usually fine-tuning to be done, it may be = best to keep them separate. > Am 17.08.19 um 23:01 schrieb Ludovic Court=C3=A8s: >> As a rule of thumb we don=E2=80=99t use =E2=80=98set!=E2=80=99 in Guix, = except in special >> circumstances. In this case you can write: >> >> (define (canonicalize-path path) >> (cond ((string-prefix? =E2=80=A6) >> (string-drop path 17)) >> ((string-suffix? =E2=80=A6) >> (string-drop-right path 1)) >> =E2=80=A6)) > > AFAIK, `cond` only processes the first expression where `test ` is true. > In this case, we need to process *all* cases where the test is true. > This means we need to nest the evaluation, which is ugly and hard to > read IMHO. Is there some more "linear" syntax? Oh I see. You could roughly have one procedure for each clause and chain them. A macro might help make that more readable (Clojure has =E2=80=98->=E2=80=99). HTH! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Sep 03 08:24:54 2019 Received: (at 36919) by debbugs.gnu.org; 3 Sep 2019 12:24:54 +0000 Received: from localhost ([127.0.0.1]:59384 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i57ri-0000Zs-13 for submit@debbugs.gnu.org; Tue, 03 Sep 2019 08:24:54 -0400 Received: from mail-out.m-online.net ([212.18.0.9]:47952) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i57rg-0000Zg-B6 for 36919@debbugs.gnu.org; Tue, 03 Sep 2019 08:24:52 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 46N5kC28CCz1rLPq for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:51 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 46N5kC0kpQz1qqkB for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:51 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id juP7ZZnLqEgY for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:50 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-49-62.dynamic.mnet-online.de [188.174.49.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:50 +0200 (CEST) Received: from lenashee.goebel-consult.de (lenashee.goebel-consult.de [192.168.110.2]) by hermia.goebel-consult.de (Postfix) with ESMTP id 597AB600AA for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:26:42 +0200 (CEST) From: Hartmut Goebel To: 36919@debbugs.gnu.org Subject: [Patch v2 0/4] Make the KDE updater find packages in subdirectories Date: Tue, 3 Sep 2019 14:24:45 +0200 Message-Id: <20190903122449.409-1-h.goebel@crazy-compilers.com> X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 36919 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) Relevant changes: * Moved kde code into a separete module. THis is done *after* the first change ("no longer relies on FTP access"9, since otherwise a lot of FTP-releated identifiers would have had to be exported in gnu-maintenance.scm. * Using a custom write-cache to cache the evaluated file list, as suggested by Ludo * Removed usage of 'set!' * Using named let instead of 'do' Hartmut Goebel (4): guix: Rename and move sans-extension to tarball-sans-extension. gnu-maintenance: KDE updater no longer relies on FTP access. upstream: Move KDE updater into a separate module. import: KDE updater finds packages even in sub-directory. Makefile.am | 1 + guix/gnu-maintenance.scm | 44 +++------ guix/import/kde.scm | 190 +++++++++++++++++++++++++++++++++++++++ guix/utils.scm | 7 ++ 4 files changed, 210 insertions(+), 32 deletions(-) create mode 100644 guix/import/kde.scm -- 2.21.0 From debbugs-submit-bounces@debbugs.gnu.org Tue Sep 03 08:24:55 2019 Received: (at 36919) by debbugs.gnu.org; 3 Sep 2019 12:24:55 +0000 Received: from localhost ([127.0.0.1]:59387 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i57rj-0000a6-Ag for submit@debbugs.gnu.org; Tue, 03 Sep 2019 08:24:55 -0400 Received: from mail-out.m-online.net ([212.18.0.10]:55266) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i57rh-0000Zp-Gl for 36919@debbugs.gnu.org; Tue, 03 Sep 2019 08:24:53 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 46N5kD6JWkz1rGRd for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:52 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 46N5kD3XDNz1qqkB for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:52 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id edJkyJPtY-Ph for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:51 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-49-62.dynamic.mnet-online.de [188.174.49.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:51 +0200 (CEST) Received: from lenashee.goebel-consult.de (lenashee.goebel-consult.de [192.168.110.2]) by hermia.goebel-consult.de (Postfix) with ESMTP id 6E0A1602BE for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:26:42 +0200 (CEST) From: Hartmut Goebel To: 36919@debbugs.gnu.org Subject: [Patch v2 1/4] guix: Rename and move sans-extension to tarball-sans-extension. Date: Tue, 3 Sep 2019 14:24:46 +0200 Message-Id: <20190903122449.409-2-h.goebel@crazy-compilers.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190903122449.409-1-h.goebel@crazy-compilers.com> References: <20190903122449.409-1-h.goebel@crazy-compilers.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 36919 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) * guix/gnu-maintenance.scm (sans-extension): Move and rename to ... * guix/utils.scm (tarball-sans-extension): ... here. --- guix/gnu-maintenance.scm | 26 ++++++++++++-------------- guix/utils.scm | 7 +++++++ 2 files changed, 19 insertions(+), 14 deletions(-) diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm index d63d44f629..8fce956c60 100644 --- a/guix/gnu-maintenance.scm +++ b/guix/gnu-maintenance.scm @@ -230,12 +230,6 @@ network to check in GNU's database." (or (assoc-ref (package-properties package) 'ftp-directory) (string-append "/gnu/" name))))) -(define (sans-extension tarball) - "Return TARBALL without its .tar.* or .zip extension." - (let ((end (or (string-contains tarball ".tar") - (string-contains tarball ".zip")))) - (substring tarball 0 end))) - (define %tarball-rx ;; The .zip extensions is notably used for freefont-ttf. ;; The "-src" pattern is for "TeXmacs-1.0.7.9-src.tar.gz". @@ -261,14 +255,15 @@ true." (string-append project "-src"))))))) (not (regexp-exec %alpha-tarball-rx file)) - (let ((s (sans-extension file))) + (let ((s (tarball-sans-extension file))) (regexp-exec %package-name-rx s)))) (define (tarball->version tarball) "Return the version TARBALL corresponds to. TARBALL is a file name like \"coreutils-8.23.tar.xz\"." (let-values (((name version) - (gnu-package-name->name+version (sans-extension tarball)))) + (gnu-package-name->name+version + (tarball-sans-extension tarball)))) version)) (define* (releases project @@ -492,8 +487,9 @@ return the corresponding signature URL, or #f it signatures are unavailable." (and (string=? url (basename url)) ;relative reference? (release-file? package url) (let-values (((name version) - (package-name->name+version (sans-extension url) - #\-))) + (package-name->name+version + (tarball-sans-extension url) + #\-))) (upstream-source (package name) (version version) @@ -565,14 +561,16 @@ list available from %GNU-FILE-LIST-URI over HTTP(S)." (release-file? name (basename file)))) files))) (match (sort relevant (lambda (file1 file2) - (version>? (sans-extension (basename file1)) - (sans-extension (basename file2))))) + (version>? (tarball-sans-extension + (basename file1)) + (tarball-sans-extension + (basename file2))))) ((and tarballs (reference _ ...)) (let* ((version (tarball->version reference)) (tarballs (filter (lambda (file) - (string=? (sans-extension + (string=? (tarball-sans-extension (basename file)) - (sans-extension + (tarball-sans-extension (basename reference)))) tarballs))) (upstream-source diff --git a/guix/utils.scm b/guix/utils.scm index f480c3291f..1f99c5b3f5 100644 --- a/guix/utils.scm +++ b/guix/utils.scm @@ -91,6 +91,7 @@ arguments-from-environment-variable file-extension file-sans-extension + tarball-sans-extension compressed-file? switch-symlinks call-with-temporary-output-file @@ -578,6 +579,12 @@ minor version numbers from version-string." (substring file 0 dot) file))) +(define (tarball-sans-extension tarball) + "Return TARBALL without its .tar.* or .zip extension." + (let ((end (or (string-contains tarball ".tar") + (string-contains tarball ".zip")))) + (substring tarball 0 end))) + (define (compressed-file? file) "Return true if FILE denotes a compressed file." (->bool (member (file-extension file) -- 2.21.0 From debbugs-submit-bounces@debbugs.gnu.org Tue Sep 03 08:25:02 2019 Received: (at 36919) by debbugs.gnu.org; 3 Sep 2019 12:25:02 +0000 Received: from localhost ([127.0.0.1]:59389 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i57rk-0000aL-Qj for submit@debbugs.gnu.org; Tue, 03 Sep 2019 08:25:02 -0400 Received: from mail-out.m-online.net ([212.18.0.9]:59014) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i57ri-0000a4-N6 for 36919@debbugs.gnu.org; Tue, 03 Sep 2019 08:24:55 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 46N5kG1yF0z1rLPq for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:54 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 46N5kG1nstz1qqkB for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:54 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id G7_OaEYnWsGC for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:53 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-49-62.dynamic.mnet-online.de [188.174.49.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:53 +0200 (CEST) Received: from lenashee.goebel-consult.de (lenashee.goebel-consult.de [192.168.110.2]) by hermia.goebel-consult.de (Postfix) with ESMTP id 7F1066035D for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:26:42 +0200 (CEST) From: Hartmut Goebel To: 36919@debbugs.gnu.org Subject: [Patch v2 2/4] gnu-maintenance: KDE updater no longer relies on FTP access. Date: Tue, 3 Sep 2019 14:24:47 +0200 Message-Id: <20190903122449.409-3-h.goebel@crazy-compilers.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190903122449.409-1-h.goebel@crazy-compilers.com> References: <20190903122449.409-1-h.goebel@crazy-compilers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 36919 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Fetch the ls-lR.bz2 file list for download.kde.org, convert it into a list of file paths and cache the list. * guix/gnu-maintenance.scm (%kde-file-list-uri): New variable. (download.kde.org-files): New procedure. (latest-kde-release): Change to use DOWNLOAD.KDE.ORG-FILES and search for files in this list. --- guix/gnu-maintenance.scm | 100 +++++++++++++++++++++++++++++++++++---- 1 file changed, 92 insertions(+), 8 deletions(-) diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm index 8fce956c60..9ce06508a3 100644 --- a/guix/gnu-maintenance.scm +++ b/guix/gnu-maintenance.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Ludovic Courtès ;;; Copyright © 2012, 2013 Nikita Karetnikov +;;; Copyright © 2019 Hartmut Goebel ;;; ;;; This file is part of GNU Guix. ;;; @@ -24,6 +25,7 @@ #:use-module (sxml simple) #:use-module (ice-9 regex) #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) @@ -613,15 +615,97 @@ releases are on gnu.org." (define gnu-hosted? (url-prefix-predicate "mirror://gnu/")) +(define %kde-file-list-uri + ;; URI of the file list (ls -lR format) for download.kde.org. + (string->uri "https://download.kde.org/ls-lR.bz2")) + +(define (download.kde.org-files) + ;;"Return the list of files available at download.kde.org." + + (define (ls-lR-line->filename path line) + ;; remove mode, blocks, user, group, size, date, time and one space + (regexp-substitute + #f (string-match "^(\\S+\\s+){6}\\S+\\s" line) path 'post)) + + (define (canonicalize path) + (let* ((path (if (string-prefix? "/srv/archives/ftp/" path) + (string-drop path (string-length "/srv/archives/ftp")) + path)) + (path (if (string-suffix? ":" path) + (string-drop-right path 1) + path)) + (path (if (not (string-suffix? "/" path)) + (string-append path "/") + path))) + path)) + + (define (write-cache input cache) + "Read bzipped ls-lR from INPUT, and write it as a list of file paths to +CACHE." + + (call-with-decompressed-port 'bzip2 input + (lambda (input) + (let loop_dirs ((files '())) + (let ((path (read-line input))) + (if + (or (eof-object? path) (string= path "")) + (write (reverse files) cache)) + (let loop_entries ((path (canonicalize path)) + (files files)) + (let ((line (read-line input))) + (cond + ((eof-object? line) + (write (reverse files) cache)) + ((string-prefix? "-" line) + (loop_entries path + (cons (ls-lR-line->filename path line) files))) + ((not (string= line "")) + (loop_entries path files)) + (#t (loop_dirs files)))))))))) + + (define (cache-miss uri) + (format (current-error-port) "fetching ~a...~%" (uri->string uri))) + + (let* ((port (http-fetch/cached %kde-file-list-uri + #:ttl 3600 + #:write-cache write-cache + #:cache-miss cache-miss)) + (files (read port))) + (close-port port) + files)) + (define (latest-kde-release package) - "Return the latest release of PACKAGE, the name of an KDE.org package." - (let ((uri (string->uri (origin-uri (package-source package))))) - (false-if-ftp-error - (latest-ftp-release - (package-upstream-name package) - #:server "ftp.mirrorservice.org" - #:directory (string-append "/sites/ftp.kde.org/pub/kde/" - (dirname (dirname (uri-path uri)))))))) + "Return the latest release of PACKAGE, a KDE package, or #f if it could not +be determined." + (let* ((uri (string->uri (origin-uri (package-source package)))) + (directory (dirname (dirname (uri-path uri)))) + (name (package-upstream-name package)) + (files (download.kde.org-files)) + (relevant (filter (lambda (file) + (and (string-prefix? directory file) + (release-file? name (basename file)))) + files))) + (match (sort relevant (lambda (file1 file2) + (version>? (tarball-sans-extension + (basename file1)) + (tarball-sans-extension + (basename file2))))) + ((and tarballs (reference _ ...)) + (let* ((version (tarball->version reference)) + (tarballs (filter (lambda (file) + (string=? (tarball-sans-extension + (basename file)) + (tarball-sans-extension + (basename reference)))) + tarballs))) + (upstream-source + (package name) + (version version) + (urls (map (lambda (file) + (string-append "mirror://kde/" file)) + tarballs))))) + (() + #f)))) (define (latest-xorg-release package) "Return the latest release of PACKAGE, the name of an X.org package." -- 2.21.0 From debbugs-submit-bounces@debbugs.gnu.org Tue Sep 03 08:25:09 2019 Received: (at 36919) by debbugs.gnu.org; 3 Sep 2019 12:25:09 +0000 Received: from localhost ([127.0.0.1]:59394 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i57rr-0000bC-BC for submit@debbugs.gnu.org; Tue, 03 Sep 2019 08:25:09 -0400 Received: from mail-out.m-online.net ([212.18.0.9]:49592) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i57rk-0000aK-Py for 36919@debbugs.gnu.org; Tue, 03 Sep 2019 08:25:02 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 46N5kJ2Z52z1rLPq for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:56 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 46N5kJ2Ppvz1qqkB for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:56 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id nxVCfn43NeyX for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:54 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-49-62.dynamic.mnet-online.de [188.174.49.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:54 +0200 (CEST) Received: from lenashee.goebel-consult.de (lenashee.goebel-consult.de [192.168.110.2]) by hermia.goebel-consult.de (Postfix) with ESMTP id 917E7603AC for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:26:42 +0200 (CEST) From: Hartmut Goebel To: 36919@debbugs.gnu.org Subject: [Patch v2 3/4] upstream: Move KDE updater into a separate module. Date: Tue, 3 Sep 2019 14:24:48 +0200 Message-Id: <20190903122449.409-4-h.goebel@crazy-compilers.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190903122449.409-1-h.goebel@crazy-compilers.com> References: <20190903122449.409-1-h.goebel@crazy-compilers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 1.5 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: As it was done for (guix import gnome). * guix/import/kde.scm: New file. * Makefile.am (MODULES): Add it. * guix/gnu-maintenance.scm (%kde-updater) (%kde-file-list-uri) (download.kde.org-files) (latest-kde-release): Remove. --- Makefile.am [...] Content analysis details: (1.5 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: makefile.am] 1.5 RCVD_IN_SORBS_WEB RBL: SORBS: sender is an abusable web server [188.174.49.62 listed in dnsbl.sorbs.net] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.0 SPF_NONE SPF: sender does not publish an SPF Record X-Debbugs-Envelope-To: 36919 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) As it was done for (guix import gnome). * guix/import/kde.scm: New file. * Makefile.am (MODULES): Add it. * guix/gnu-maintenance.scm (%kde-updater) (%kde-file-list-uri) (download.kde.org-files) (latest-kde-release): Remove. --- Makefile.am | 1 + guix/gnu-maintenance.scm | 102 ------------------------- guix/import/kde.scm | 158 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 159 insertions(+), 102 deletions(-) create mode 100644 guix/import/kde.scm diff --git a/Makefile.am b/Makefile.am index fa6bf8fe80..c8366c0421 100644 --- a/Makefile.am +++ b/Makefile.am @@ -218,6 +218,7 @@ MODULES = \ guix/import/gnu.scm \ guix/import/hackage.scm \ guix/import/json.scm \ + guix/import/kde.scm \ guix/import/launchpad.scm \ guix/import/opam.scm \ guix/import/print.scm \ diff --git a/guix/gnu-maintenance.scm b/guix/gnu-maintenance.scm index 9ce06508a3..ef067704ad 100644 --- a/guix/gnu-maintenance.scm +++ b/guix/gnu-maintenance.scm @@ -1,7 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Ludovic Courtès ;;; Copyright © 2012, 2013 Nikita Karetnikov -;;; Copyright © 2019 Hartmut Goebel ;;; ;;; This file is part of GNU Guix. ;;; @@ -25,7 +24,6 @@ #:use-module (sxml simple) #:use-module (ice-9 regex) #:use-module (ice-9 match) - #:use-module (ice-9 rdelim) #:use-module (srfi srfi-1) #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) @@ -64,7 +62,6 @@ %gnu-updater %gnu-ftp-updater - %kde-updater %xorg-updater %kernel.org-updater)) @@ -615,98 +612,6 @@ releases are on gnu.org." (define gnu-hosted? (url-prefix-predicate "mirror://gnu/")) -(define %kde-file-list-uri - ;; URI of the file list (ls -lR format) for download.kde.org. - (string->uri "https://download.kde.org/ls-lR.bz2")) - -(define (download.kde.org-files) - ;;"Return the list of files available at download.kde.org." - - (define (ls-lR-line->filename path line) - ;; remove mode, blocks, user, group, size, date, time and one space - (regexp-substitute - #f (string-match "^(\\S+\\s+){6}\\S+\\s" line) path 'post)) - - (define (canonicalize path) - (let* ((path (if (string-prefix? "/srv/archives/ftp/" path) - (string-drop path (string-length "/srv/archives/ftp")) - path)) - (path (if (string-suffix? ":" path) - (string-drop-right path 1) - path)) - (path (if (not (string-suffix? "/" path)) - (string-append path "/") - path))) - path)) - - (define (write-cache input cache) - "Read bzipped ls-lR from INPUT, and write it as a list of file paths to -CACHE." - - (call-with-decompressed-port 'bzip2 input - (lambda (input) - (let loop_dirs ((files '())) - (let ((path (read-line input))) - (if - (or (eof-object? path) (string= path "")) - (write (reverse files) cache)) - (let loop_entries ((path (canonicalize path)) - (files files)) - (let ((line (read-line input))) - (cond - ((eof-object? line) - (write (reverse files) cache)) - ((string-prefix? "-" line) - (loop_entries path - (cons (ls-lR-line->filename path line) files))) - ((not (string= line "")) - (loop_entries path files)) - (#t (loop_dirs files)))))))))) - - (define (cache-miss uri) - (format (current-error-port) "fetching ~a...~%" (uri->string uri))) - - (let* ((port (http-fetch/cached %kde-file-list-uri - #:ttl 3600 - #:write-cache write-cache - #:cache-miss cache-miss)) - (files (read port))) - (close-port port) - files)) - -(define (latest-kde-release package) - "Return the latest release of PACKAGE, a KDE package, or #f if it could not -be determined." - (let* ((uri (string->uri (origin-uri (package-source package)))) - (directory (dirname (dirname (uri-path uri)))) - (name (package-upstream-name package)) - (files (download.kde.org-files)) - (relevant (filter (lambda (file) - (and (string-prefix? directory file) - (release-file? name (basename file)))) - files))) - (match (sort relevant (lambda (file1 file2) - (version>? (tarball-sans-extension - (basename file1)) - (tarball-sans-extension - (basename file2))))) - ((and tarballs (reference _ ...)) - (let* ((version (tarball->version reference)) - (tarballs (filter (lambda (file) - (string=? (tarball-sans-extension - (basename file)) - (tarball-sans-extension - (basename reference)))) - tarballs))) - (upstream-source - (package name) - (version version) - (urls (map (lambda (file) - (string-append "mirror://kde/" file)) - tarballs))))) - (() - #f)))) - (define (latest-xorg-release package) "Return the latest release of PACKAGE, the name of an X.org package." (let ((uri (string->uri (origin-uri (package-source package))))) @@ -754,13 +659,6 @@ be determined." (pure-gnu-package? package)))) (latest latest-release*))) -(define %kde-updater - (upstream-updater - (name 'kde) - (description "Updater for KDE packages") - (pred (url-prefix-predicate "mirror://kde/")) - (latest latest-kde-release))) - (define %xorg-updater (upstream-updater (name 'xorg) diff --git a/guix/import/kde.scm b/guix/import/kde.scm new file mode 100644 index 0000000000..927ecc8263 --- /dev/null +++ b/guix/import/kde.scm @@ -0,0 +1,158 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2016 David Craven +;;; Copyright © 2016, 2017 Ludovic Courtès +;;; Copyright © 2019 Hartmut Goebel +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see . + +(define-module (guix import kde) + #:use-module (guix http-client) + #:use-module (guix memoization) + #:use-module (guix gnu-maintenance) + #:use-module (guix packages) + #:use-module (guix upstream) + #:use-module (guix utils) + #:use-module (ice-9 match) + #:use-module (ice-9 rdelim) + #:use-module (ice-9 regex) + #:use-module (srfi srfi-11) + #:use-module (web uri) + + #:export (%kde-updater)) + +;;; Commentary: +;;; +;;; This package provides not an actual importer but simply an updater for +;;; KDE packages. It grabs available files from the 'ls-lR.bz2' file +;;; available on download.kde.org. +;;; +;;; Code: + +(define (tarball->version tarball) + "Return the version TARBALL corresponds to. TARBALL is a file name like +\"coreutils-8.23.tar.xz\"." + (let-values (((name version) + (gnu-package-name->name+version + (tarball-sans-extension tarball)))) + version)) + +(define %kde-file-list-uri + ;; URI of the file list (ls -lR format) for download.kde.org. + (string->uri "https://download.kde.org/ls-lR.bz2")) + +(define (download.kde.org-files) + ;;"Return the list of files available at download.kde.org." + + (define (ls-lR-line->filename path line) + ;; Remove mode, blocks, user, group, size, date, time and one space, + ;; then prepend PATH + (regexp-substitute + #f (string-match "^(\\S+\\s+){6}\\S+\\s" line) path 'post)) + + (define (canonicalize path) + (let* ((path (if (string-prefix? "/srv/archives/ftp/" path) + (string-drop path (string-length "/srv/archives/ftp")) + path)) + (path (if (string-suffix? ":" path) + (string-drop-right path 1) + path)) + (path (if (not (string-suffix? "/" path)) + (string-append path "/") + path))) + path)) + + (define (write-cache input cache) + "Read bzipped ls-lR from INPUT, and write it as a list of file paths to +CACHE." + (call-with-decompressed-port 'bzip2 input + (lambda (input) + (let loop_dirs ((files '())) + ;; process a new directory block + (let ((path (read-line input))) + (if + (or (eof-object? path) (string= path "")) + (write (reverse files) cache) + (let loop_entries ((path (canonicalize path)) + (files files)) + ;; process entries within the directory block + (let ((line (read-line input))) + (cond + ((eof-object? line) + (write (reverse files) cache)) + ((string-prefix? "-" line) + ;; this is a file entry: prepend to FILES, then re-enter + ;; the loop for remaining entries + (loop_entries path + (cons (ls-lR-line->filename path line) files) + )) + ((not (string= line "")) + ;; this is a non-file entry: ignore it, just re-enter the + ;; loop for remaining entries + (loop_entries path files)) + ;; empty line: directory block end, re-enter the outer + ;; loop for the next block + (#t (loop_dirs files))))))))))) + + (define (cache-miss uri) + (format (current-error-port) "fetching ~a...~%" (uri->string uri))) + + (let* ((port (http-fetch/cached %kde-file-list-uri + #:ttl 3600 + #:write-cache write-cache + #:cache-miss cache-miss)) + (files (read port))) + (close-port port) + files)) + +(define (latest-kde-release package) + "Return the latest release of PACKAGE, a KDE package, or #f if it could +not be determined." + (let* ((uri (string->uri (origin-uri (package-source package)))) + (directory (dirname (dirname (uri-path uri)))) + (name (package-upstream-name package)) + (files (download.kde.org-files)) + (relevant (filter (lambda (file) + (and (string-prefix? directory file) + (release-file? name (basename file)))) + files))) + (match (sort relevant (lambda (file1 file2) + (version>? (tarball-sans-extension + (basename file1)) + (tarball-sans-extension + (basename file2))))) + ((and tarballs (reference _ ...)) + (let* ((version (tarball->version reference)) + (tarballs (filter (lambda (file) + (string=? (tarball-sans-extension + (basename file)) + (tarball-sans-extension + (basename reference)))) + tarballs))) + (upstream-source + (package name) + (version version) + (urls (map (lambda (file) + (string-append "mirror://kde/" file)) + tarballs))))) + (() + #f)))) + +(define %kde-updater + (upstream-updater + (name 'kde) + (description "Updater for KDE packages") + (pred (url-prefix-predicate "mirror://kde/")) + (latest latest-kde-release))) -- 2.21.0 From debbugs-submit-bounces@debbugs.gnu.org Tue Sep 03 08:25:09 2019 Received: (at 36919) by debbugs.gnu.org; 3 Sep 2019 12:25:09 +0000 Received: from localhost ([127.0.0.1]:59398 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i57rx-0000bm-H7 for submit@debbugs.gnu.org; Tue, 03 Sep 2019 08:25:09 -0400 Received: from mail-out.m-online.net ([212.18.0.10]:43943) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i57rm-0000aN-HM for 36919@debbugs.gnu.org; Tue, 03 Sep 2019 08:25:02 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 46N5kL0fSbz1rGRd for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:58 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 46N5kL0Xgtz1qqkP for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:58 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id 9cTxd1Be99Pw for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:56 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-49-62.dynamic.mnet-online.de [188.174.49.62]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:24:56 +0200 (CEST) Received: from lenashee.goebel-consult.de (lenashee.goebel-consult.de [192.168.110.2]) by hermia.goebel-consult.de (Postfix) with ESMTP id A4AAD603FB for <36919@debbugs.gnu.org>; Tue, 3 Sep 2019 14:26:42 +0200 (CEST) From: Hartmut Goebel To: 36919@debbugs.gnu.org Subject: [Patch v2 4/4] import: KDE updater finds packages even in sub-directory. Date: Tue, 3 Sep 2019 14:24:49 +0200 Message-Id: <20190903122449.409-5-h.goebel@crazy-compilers.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190903122449.409-1-h.goebel@crazy-compilers.com> References: <20190903122449.409-1-h.goebel@crazy-compilers.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.8 (/) X-Debbugs-Envelope-To: 36919 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) Fixes and finally fixes . Formerly packages living in a path like /stable/frameworks/5.60/portingAids/kross-5.60.0.tar.xz have not been found. * guix/import/kde.scm (uri->kde-path-pattern): New procedure. (latest-kde-release): Use pattern to search for file. --- guix/import/kde.scm | 36 ++++++++++++++++++++++++++++++++++-- 1 file changed, 34 insertions(+), 2 deletions(-) diff --git a/guix/import/kde.scm b/guix/import/kde.scm index 927ecc8263..6873418d62 100644 --- a/guix/import/kde.scm +++ b/guix/import/kde.scm @@ -117,15 +117,47 @@ CACHE." (close-port port) files)) +(define (uri->kde-path-pattern uri) + "Build a regexp from the package's URI suitable for matching the package +path version-agnostic. + +Example: +Input: + mirror://kde//stable/frameworks/5.55/portingAids/kross-5.55.0.zip +Output: + //stable/frameworks/[^/]+/portingAids/ +" + + (define version-regexp + ;; regexp for matching versions as used in the ld-lR file + (make-regexp + (string-join '("^([0-9]+\\.)+[0-9]+-?" ;; 5.12.90, 4.2.0-preview + "^[0-9]+$" ;; 20031002 + ".*-([0-9]+\\.)+[0-9]+$") ;; kdepim-4.6.1 + "|"))) + + (define (version->pattern part) + ;; If a path element might be a version, replace it by a catch-all part + (if (regexp-exec version-regexp part) + "[^/]+" + part)) + + (let* ((path (uri-path uri)) + (directory-parts (string-split (dirname path) #\/))) + (make-regexp + (string-append + (string-join (map version->pattern directory-parts) "/") + "/")))) + (define (latest-kde-release package) "Return the latest release of PACKAGE, a KDE package, or #f if it could not be determined." (let* ((uri (string->uri (origin-uri (package-source package)))) - (directory (dirname (dirname (uri-path uri)))) + (path-rx (uri->kde-path-pattern uri)) (name (package-upstream-name package)) (files (download.kde.org-files)) (relevant (filter (lambda (file) - (and (string-prefix? directory file) + (and (regexp-exec path-rx file) (release-file? name (basename file)))) files))) (match (sort relevant (lambda (file1 file2) -- 2.21.0 From debbugs-submit-bounces@debbugs.gnu.org Mon Sep 09 18:44:53 2019 Received: (at 36919) by debbugs.gnu.org; 9 Sep 2019 22:44:53 +0000 Received: from localhost ([127.0.0.1]:40711 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i7SOy-0006tb-UG for submit@debbugs.gnu.org; Mon, 09 Sep 2019 18:44:53 -0400 Received: from eggs.gnu.org ([209.51.188.92]:45658) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i7SOx-0006tO-MO for 36919@debbugs.gnu.org; Mon, 09 Sep 2019 18:44:52 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:44327) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1i7SOs-0005Jk-83; Mon, 09 Sep 2019 18:44:46 -0400 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=51870 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1i7SOr-0006Oq-Oa; Mon, 09 Sep 2019 18:44:46 -0400 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Hartmut Goebel Subject: Re: [bug#36919] [Patch v2 0/4] Make the KDE updater find packages in subdirectories References: <20190804102559.32296-1-h.goebel@crazy-compilers.com> <20190903122449.409-1-h.goebel@crazy-compilers.com> Date: Tue, 10 Sep 2019 00:44:44 +0200 In-Reply-To: <20190903122449.409-1-h.goebel@crazy-compilers.com> (Hartmut Goebel's message of "Tue, 3 Sep 2019 14:24:45 +0200") Message-ID: <871rwpxf6r.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 36919 Cc: 36919@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello Hartmut, Hartmut Goebel skribis: > Relevant changes: > > * Moved kde code into a separete module. THis is done *after* the first c= hange > ("no longer relies on FTP access"9, since otherwise a lot of FTP-releat= ed > identifiers would have had to be exported in gnu-maintenance.scm. > * Using a custom write-cache to cache the evaluated file list, as suggest= ed by > Ludo > * Removed usage of 'set!' > * Using named let instead of 'do' > > > Hartmut Goebel (4): > guix: Rename and move sans-extension to tarball-sans-extension. > gnu-maintenance: KDE updater no longer relies on FTP access. > upstream: Move KDE updater into a separate module. > import: KDE updater finds packages even in sub-directory. All looks good to me, thank you! Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Tue Sep 10 13:09:57 2019 Received: (at 36919-close) by debbugs.gnu.org; 10 Sep 2019 17:09:58 +0000 Received: from localhost ([127.0.0.1]:41672 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i7jeP-0003bQ-NA for submit@debbugs.gnu.org; Tue, 10 Sep 2019 13:09:57 -0400 Received: from mail-out.m-online.net ([212.18.0.10]:42234) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1i7jeM-0003bG-Vv for 36919-close@debbugs.gnu.org; Tue, 10 Sep 2019 13:09:55 -0400 Received: from frontend01.mail.m-online.net (unknown [192.168.8.182]) by mail-out.m-online.net (Postfix) with ESMTP id 46SWjs6Tx3z1rW60; Tue, 10 Sep 2019 19:09:53 +0200 (CEST) Received: from localhost (dynscan1.mnet-online.de [192.168.6.70]) by mail.m-online.net (Postfix) with ESMTP id 46SWjs3Y1Fz1qqkj; Tue, 10 Sep 2019 19:09:53 +0200 (CEST) X-Virus-Scanned: amavisd-new at mnet-online.de Received: from mail.mnet-online.de ([192.168.8.182]) by localhost (dynscan1.mail.m-online.net [192.168.6.70]) (amavisd-new, port 10024) with ESMTP id zMtiKNNS0i-s; Tue, 10 Sep 2019 19:09:52 +0200 (CEST) Received: from hermia.goebel-consult.de (ppp-188-174-61-50.dynamic.mnet-online.de [188.174.61.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.mnet-online.de (Postfix) with ESMTPS; Tue, 10 Sep 2019 19:09:52 +0200 (CEST) Received: from [192.168.110.2] (lenashee.goebel-consult.de [192.168.110.2]) by hermia.goebel-consult.de (Postfix) with ESMTP id BDD5C6028B; Tue, 10 Sep 2019 19:12:00 +0200 (CEST) Subject: Re: [bug#36919] [Patch v2 0/4] Make the KDE updater find packages in subdirectories To: =?UTF-8?Q?Ludovic_Court=c3=a8s?= References: <20190804102559.32296-1-h.goebel@crazy-compilers.com> <20190903122449.409-1-h.goebel@crazy-compilers.com> <871rwpxf6r.fsf@gnu.org> From: Hartmut Goebel Openpgp: preference=signencrypt Organization: crazy-compilers.com Message-ID: Date: Tue, 10 Sep 2019 19:09:52 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <871rwpxf6r.fsf@gnu.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Content-Language: de-DE X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 36919-close Cc: 36919-close@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.7 (-) Pushed as 4eb69bf0d33810886ee118f38989cef696e4c868 Thanks for tutoring and for the review -- Regards Hartmut Goebel | Hartmut Goebel | h.goebel@crazy-compilers.com | | www.crazy-compilers.com | compilers which you thought are impossible | From unknown Sun Jun 15 08:51:36 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Wed, 09 Oct 2019 11:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator