From unknown Wed Jun 18 23:17:11 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#45774 <45774@debbugs.gnu.org> To: bug#45774 <45774@debbugs.gnu.org> Subject: Status: [PATCH core-updates 1/1] guix: packages: Allow patch-and-repack to work with plain files. Reply-To: bug#45774 <45774@debbugs.gnu.org> Date: Thu, 19 Jun 2025 06:17:11 +0000 retitle 45774 [PATCH core-updates 1/1] guix: packages: Allow patch-and-repa= ck to work with plain files. reassign 45774 guix-patches submitter 45774 Maxim Cournoyer severity 45774 normal tag 45774 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 10 15:05:55 2021 Received: (at submit) by debbugs.gnu.org; 10 Jan 2021 20:05:56 +0000 Received: from localhost ([127.0.0.1]:55008 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kygyJ-0005K5-9L for submit@debbugs.gnu.org; Sun, 10 Jan 2021 15:05:55 -0500 Received: from lists.gnu.org ([209.51.188.17]:39152) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kygyH-0005Jx-Iw for submit@debbugs.gnu.org; Sun, 10 Jan 2021 15:05:54 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:39302) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kygyH-0007dn-ET for guix-patches@gnu.org; Sun, 10 Jan 2021 15:05:53 -0500 Received: from mail-io1-xd30.google.com ([2607:f8b0:4864:20::d30]:43262) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kygyF-0004HY-6H for guix-patches@gnu.org; Sun, 10 Jan 2021 15:05:53 -0500 Received: by mail-io1-xd30.google.com with SMTP id o6so15590782iob.10; Sun, 10 Jan 2021 12:05:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=R6+qyBU2MlyCz6OQiOW6ONTwLEzahy4VP0r/oAUu5nQ=; b=YpMkyZUaSoWPz3xPvVVWe2vcdOM6uml0Co6l7dju+0O67SI9nXFjdUweoScGQa6obv pqYcak9DCrQcw2RkhyZzr7wK+LncgIB0Ve+xT65lX9F50WQgsn7eanpw1HUuxo1spzyB CNM7qu3CCzSOUzkfcczGuEz4p6knh4kPzWgRtyle+V/smLu/8UV1+CZsvGcJnD8a2CPZ Vvtcx2lKtfocypmE0Q1HoRuUXNuteaLmEcEIKjQLgk9BtBHOWmPUwByu/9+jipRbgbuP OPg7pRldRa9xaiY/IAVFNS9i1IQYKfAjXtjQTwMyAnRD5sWyGzDBw3Tvwx7JxSjih+tE wEAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=R6+qyBU2MlyCz6OQiOW6ONTwLEzahy4VP0r/oAUu5nQ=; b=iTsfnx3fD8tqtTpncgwUQ0NXzg/Nj2vzlvjfQT5NFNKmO71OetvEWiycRbRvT8QWi4 PpuPT29+23WO2WJxcPpAvCBiTcUF4PQ5/HFDydxyCwXAZgkvF71NtrXlWAgKm2AXj0m0 tnvTkchGPmraPQH/w64ivbt7r+tmmySd/jYrMiC4i9nEWRTdu0DUfqJQT1hAYupuHbIN Jiv4Nv717eaAVWVZHuM9AXXV+5EcSDryD/CeF8fBDU1W5hpsBZ57246m1hzyr31ZrvZZ 4WAHmPfDW5F+mRPeAcpyMdHdXwRVd6KznBSINqS38jTrc0I/BdKv4VWCXZFRGSIcncsi XAsg== X-Gm-Message-State: AOAM533/gDlnPlhF3MSzTEGLxZno1eNq5EkwHPyQ7yK1P2+94yUwBH+W vbfmq3O+PWkxG//huWGq2qjT2WA8rNOUQQ== X-Google-Smtp-Source: ABdhPJwoqIVHqO0KVyCKjzEk5ftMq6HkycPdjFpFyXl+WzfepacmRWDHWzt+yBDEpcTSZAw+GcGL/g== X-Received: by 2002:a05:6638:19c:: with SMTP id a28mr11629868jaq.76.1610309149372; Sun, 10 Jan 2021 12:05:49 -0800 (PST) Received: from localhost.localdomain (dsl-205-233-124-188.b2b2c.ca. [205.233.124.188]) by smtp.gmail.com with ESMTPSA id n77sm9516249iod.48.2021.01.10.12.05.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jan 2021 12:05:48 -0800 (PST) From: Maxim Cournoyer To: 45773@gnu.org, guix-patches@gnu.org Subject: [PATCH core-updates 1/1] guix: packages: Allow patch-and-repack to work with plain files. Date: Sun, 10 Jan 2021 15:05:35 -0500 Message-Id: <20210110200535.24377-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 Content-Type: text/plain; charset=yes Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=2607:f8b0:4864:20::d30; envelope-from=maxim.cournoyer@gmail.com; helo=mail-io1-xd30.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit Cc: Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) This change allows the use of the snippet field on a single file origin. Previously, the patch-and-repack procedure would fail on plain files, as it would end up invoking tar when attempting to extract non-tarballs. * guix/packages.scm (patch-and-repack): Only add the compressor utility to the PATH when the file is compressed. Bind more inputs in the mlet, and use them for decompressing single files. Adjust decompression and compression routines. [decompression-type]: Return #f when no known compression extension is used. [tarball?]: New nested procedure. * tests/packages.scm: Add tests. Add missing copyright year for Jan. --- guix/packages.scm | 96 +++++++++++++++++++++++++++++++--------------- tests/packages.scm | 87 +++++++++++++++++++++++++++++++++++++++-- 2 files changed, 149 insertions(+), 34 deletions(-) diff --git a/guix/packages.scm b/guix/packages.scm index 93407c143c..f6336e7345 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -5,7 +5,7 @@ ;;; Copyright © 2016 Alex Kost ;;; Copyright © 2017, 2019, 2020 Efraim Flashner ;;; Copyright © 2019 Marius Bakke -;;; Copyright © 2020 Maxim Cournoyer +;;; Copyright © 2020, 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -615,7 +615,8 @@ specifies modules in scope when evaluating SNIPPET." ((string-suffix? "bz2" source-file-name) "bzip2") ((string-suffix? "lz" source-file-name) "lzip") ((string-suffix? "zip" source-file-name) "unzip") - (else "xz"))) + ((string-suffix? "xz" source-file-name) "xz") + (else #f))) ;no compression used (define original-file-name ;; Remove the store prefix plus the slash, hash, and hyphen. @@ -653,19 +654,29 @@ specifies modules in scope when evaluating SNIPPET." (lower-object patch system)))) (mlet %store-monad ((tar -> (lookup-input "tar")) + (gzip -> (lookup-input "gzip")) + (bzip2 -> (lookup-input "bzip2")) + (lzip -> (lookup-input "lzip")) (xz -> (lookup-input "xz")) (patch -> (lookup-input "patch")) (locales -> (lookup-input "locales")) - (decomp -> (lookup-input decompression-type)) + (decomp -> (and=> decompression-type lookup-input)) (patches (sequence %store-monad (map instantiate-patch patches)))) (define build (with-imported-modules '((guix build utils)) #~(begin (use-modules (ice-9 ftw) + (ice-9 match) + (ice-9 regex) (srfi srfi-1) + (srfi srfi-26) (guix build utils)) + (define (tarball? file-name) + ;; Return true if FILE-NAME has a tar extension. + (string-match "\\.tar(\\..*)?$" file-name)) + ;; The --sort option was added to GNU tar in version 1.28, released ;; 2014-07-28. During bootstrap we must cope with older versions. (define tar-supports-sort? @@ -702,12 +713,15 @@ specifies modules in scope when evaluating SNIPPET." (package-version locales))))) (setlocale LC_ALL "en_US.utf8")) - (setenv "PATH" (string-append #+xz "/bin" ":" - #+decomp "/bin")) + (setenv "PATH" + (string-append #+xz "/bin" + (if #+decomp + (string-append ":" #+decomp "/bin") + ""))) (setenv "XZ_DEFAULTS" (string-join (%xz-parallel-args))) - ;; SOURCE may be either a directory or a tarball. + ;; SOURCE may be either a directory, a tarball or a simple file. (if (file-is-directory? #+source) (let* ((store (%store-directory)) (len (+ 1 (string-length store))) @@ -716,31 +730,51 @@ specifies modules in scope when evaluating SNIPPET." (directory (string-drop base (+ 1 dash)))) (mkdir directory) (copy-recursively #+source directory)) - #+(if (string=? decompression-type "unzip") - #~(invoke "unzip" #+source) - #~(invoke (string-append #+tar "/bin/tar") - "xvf" #+source))) - - (let ((directory (first-file "."))) - (format (current-error-port) - "source is under '~a'~%" directory) - (chdir directory) - - (for-each apply-patch '#+patches) - - #+(if snippet - #~(let ((module (make-fresh-user-module))) - (module-use-interfaces! - module - (map resolve-interface '#+modules)) - ((@ (system base compile) compile) - '#+snippet - #:to 'value - #:opts %auto-compilation-options - #:env module)) - #~#t) - - (chdir "..") + ;; File is *not* a directory. + (cond + ((tarball? #+source) + (invoke (string-append #+tar "/bin/tar") + "xvf" #+source)) + ((and=> #+decompression-type (cut string= "unzip" <>)) + ("unzip" (invoke "unzip" #+source))) + (else + ;; A simple file, either compressed or not. + (match #+decompression-type + ;; Note: Referring to the store unzip here (#+unzip) + ;; introduces a cycle. + ("unzip" (invoke "unzip" #+source)) + (else + ;; bzip2, gzip, lzip and xz share a common CLI. + (let ((name (strip-store-file-name #+source)) + (command (and=> #+decomp + (cut string-append <> "/bin/" + #+decompression-type)))) + (copy-file #+source name) + (when command + (invoke command "--decompress" name)))))))) + + + (let* ((file (first-file ".")) + (directory (if (file-is-directory? file) + file + "."))) + (format (current-error-port) "source is at '~a'~%" file) + + (with-directory-excursion directory + + (for-each apply-patch '#+patches) + + #+(if snippet + #~(let ((module (make-fresh-user-module))) + (module-use-interfaces! + module + (map resolve-interface '#+modules)) + ((@ (system base compile) compile) + '#+snippet + #:to 'value + #:opts %auto-compilation-options + #:env module)) + #~#t)) (unless tar-supports-sort? (call-with-output-file ".file_list" diff --git a/tests/packages.scm b/tests/packages.scm index a867f2fd6d..5c84dbf4b8 100644 --- a/tests/packages.scm +++ b/tests/packages.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès -;;; Copyright © Jan (janneke) Nieuwenhuizen +;;; Copyright © 2018 Jan (janneke) Nieuwenhuizen +;;; Copyright © 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -17,12 +18,12 @@ ;;; You should have received a copy of the GNU General Public License ;;; along with GNU Guix. If not, see . -(define-module (test-packages) +(define-module (tests packages) #:use-module (guix tests) #:use-module (guix store) #:use-module (guix monads) #:use-module (guix grafts) - #:use-module ((guix gexp) #:select (local-file local-file-file)) + #:use-module (guix gexp) #:use-module (guix utils) #:use-module ((guix diagnostics) ;; Rename the 'location' binding to allow proper syntax @@ -32,6 +33,7 @@ (else name)))) #:use-module ((gcrypt hash) #:prefix gcrypt:) #:use-module (guix derivations) + #:use-module (guix download) #:use-module (guix packages) #:use-module (guix grafts) #:use-module (guix search-paths) @@ -576,6 +578,11 @@ (build-derivations %store (list drv)) (call-with-input-file output get-string-all))) + +;;; +;;; Source derivation with snippets. +;;; + (unless (network-reachable?) (test-skip 1)) (test-equal "package-source-derivation, snippet" "OK" @@ -631,6 +638,80 @@ (and (build-derivations %store (list (pk 'snippet-drv drv))) (call-with-input-file out get-string-all)))) +;; Note: lzip is not part of bootstrap-coreutils&co, so is not included to +;; avoid having to rebuild the world. +(define compressors '(("gzip" . "gz") + ("xz" . "xz") + ("bzip2" . "bz2") + (#f . #f))) + +(for-each + (match-lambda + ((comp . ext) + (unless (network-reachable?) (test-skip 1)) + (test-equal (string-append "origin->derivation, single file with snippet " + "(compression: " (if comp comp "None") ")") + "2 + 2 = 4" + (let* ((name "maths") + (compressed-name (if comp + (string-append name "." ext) + name)) + (command #~(if #+comp + (string-append #+%bootstrap-coreutils&co + "/bin/" #+comp) + #f)) + (f (with-imported-modules '((guix build utils)) + (computed-file compressed-name + #~(begin + (use-modules (guix build utils) + (rnrs io simple)) + (with-output-to-file #+name + (lambda _ + (format #t "2 + 2 = 5"))) + (when #+command + (invoke #+command #+name)) + (copy-file #+compressed-name #$output))))) + (file-drv (run-with-store %store (lower-object f))) + (file (derivation->output-path file-drv)) + (file-drv-outputs (derivation-outputs file-drv)) + (_ (build-derivations %store (list file-drv))) + (file-hash (derivation-output-hash + (assoc-ref file-drv-outputs "out"))) + ;; Create an origin using the above computed file and its hash. + (source (origin + (method url-fetch) + (uri (string-append "file://" file)) + (file-name compressed-name) + (patch-inputs `(("tar" ,%bootstrap-coreutils&co) + ("xz" ,%bootstrap-coreutils&co) + ("bzip2" ,%bootstrap-coreutils&co) + ("gzip" ,%bootstrap-coreutils&co))) + (patch-guile %bootstrap-guile) + (modules '((guix build utils))) + (snippet `(substitute* ,name + (("5") "4"))) + (hash (content-hash file-hash)))) + ;; Build origin. + (drv (run-with-store %store (origin->derivation source))) + (out (derivation->output-path drv))) + ;; Decompress the resulting tar.xz and return its content. + (and (build-derivations %store (list drv)) + (let* ((bin #~(string-append #+%bootstrap-coreutils&co + "/bin")) + (f (computed-file + name + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils)) + (setenv "PATH" #+bin) + (invoke "tar" "xvf" #+out) + (copy-file #+name #$output))))) + (drv (run-with-store %store (lower-object f))) + (_ (build-derivations %store (list drv)))) + (call-with-input-file (derivation->output-path drv) + get-string-all))))))) + compressors) + (test-assert "return value" (let ((drv (package-derivation %store (dummy-package "p")))) (and (derivation? drv) -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Sun Jan 10 17:00:09 2021 Received: (at control) by debbugs.gnu.org; 10 Jan 2021 22:00:09 +0000 Received: from localhost ([127.0.0.1]:55067 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kyikr-00057E-I5 for submit@debbugs.gnu.org; Sun, 10 Jan 2021 17:00:09 -0500 Received: from mail-il1-f169.google.com ([209.85.166.169]:35923) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kyikp-000552-JT for control@debbugs.gnu.org; Sun, 10 Jan 2021 17:00:08 -0500 Received: by mail-il1-f169.google.com with SMTP id x15so618763ilk.3 for ; Sun, 10 Jan 2021 14:00:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:to:from:subject; bh=6k9B2g2tmU9SpiWIcoswIQb8dz6ZryZBBc8k7OwG9yc=; b=ZyGmNAsfRFpS2tfz5Blk0i6fs3G/H2t4Ebxa6i1ldjq8IAwMmZy9L45bOtRV4AiVC+ 0Gyvh3OryXYTwO7oXPigMQWkpvIEun2A/GjTWV0EQ/tT7xbAUpXGE9zKIciZiOEFEmLH Oy1kKLrAEr+sJHhYWDCyca/3vvyGTYB2ZS4mBJRmIq/Rwop+jkRsFW0gg+qdsWzrCAFp k812FBvHucgNY/eBl83uRw/Z5HHKcqANzeaXV9mkN9vpOVXGkUY+Q1SwFpTfDGiFdXxL Mc+2jmTQFt3gkH4NlpZckQoZ3pkjXYirbbroDXEsUobQcBRGkBnEwf38776ztE5hy7V5 9cMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:to:from:subject; bh=6k9B2g2tmU9SpiWIcoswIQb8dz6ZryZBBc8k7OwG9yc=; b=HoHB0zAVrBxoD52U3rt/vy8EQmMzTM+gxN/vtwe4InlMkgZCNKEyr8sYjZRbFGGEwr /mTZxX4ERCtfDntI6oIetq7j1lbiLvTnJLQJoTCtL8eDUo4wFxKt23Q+BvefcobQ7rzO HkVopmwPIUuDtseMnc8MyH8EtdsuwdTSoC+MD4ans4efZc+3SEJUn0xep4wpwYH1K9q7 cXTxL7W0Sdmv2HIQnWmV/NLsHlCThiL+6awKRA92grdqti0T+A/GdrlCXzg9GqdmE5Kb yQUQSlVo3qb1EVwHN+oDJGQIahvhU/uj6RYtbvJ6lmTN7RLpI7hYb6nbL1pI/2Qf9W/O ztUw== X-Gm-Message-State: AOAM532dtFKvAvZOLybK4RJD24QcICQlciqwysfkY4KbKQKZNEKRvL35 IaWWFcuqvkXb+f8IoACj3pqA71/8ZvNKOg== X-Google-Smtp-Source: ABdhPJx95HjAnNlOZwNdZEIEoxJNsq+qU68iHEjk9e5rwo6ylqvYFHYKZQU11W371gT9K8olDZ7ZHA== X-Received: by 2002:a05:6e02:50e:: with SMTP id d14mr9685891ils.248.1610316001646; Sun, 10 Jan 2021 14:00:01 -0800 (PST) Received: from hurd (dsl-205-233-124-188.b2b2c.ca. [205.233.124.188]) by smtp.gmail.com with ESMTPSA id t14sm10099354iof.23.2021.01.10.14.00.01 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jan 2021 14:00:01 -0800 (PST) Date: Sun, 10 Jan 2021 17:00:00 -0500 Message-Id: <87sg784fmn.fsf@gmail.com> To: control@debbugs.gnu.org From: Maxim Cournoyer Subject: control message for bug #45774 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) forcemerge 45774 45773 quit From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 13 12:41:01 2021 Received: (at 45774) by debbugs.gnu.org; 13 Jan 2021 17:41:01 +0000 Received: from localhost ([127.0.0.1]:36506 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kzk8f-0008FA-Fb for submit@debbugs.gnu.org; Wed, 13 Jan 2021 12:41:01 -0500 Received: from mail-qk1-f178.google.com ([209.85.222.178]:37400) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kzk8Y-0008Ep-DH for 45774@debbugs.gnu.org; Wed, 13 Jan 2021 12:40:56 -0500 Received: by mail-qk1-f178.google.com with SMTP id h4so2897862qkk.4 for <45774@debbugs.gnu.org>; Wed, 13 Jan 2021 09:40:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=yuKfDyOo7COq0f9iSZabfLktV2sm6tlW50y80agqyW8=; b=i6/CUpWLf+84uERr7yby/38V0fvLxG0fFwEKdP+ILowSRAm0qe8Bm4rsQN2Iqg/rZ+ SkQHhTCeHUfLOORzV7lYqqlxkFWDFykc1gQ4XTfLHPquY3e+KVY+VWUo4SVCIS6TXOZh D7bifLzbl+wbzMjcvWSrvd5w8VlAI0A10hg6wT/YFB8OqtNCbdZpLYfJDiCXUnUd9ii3 AQHULNDXoD7HpU1WSw0NI8Ag6AM5DytEUOQfJtxR49pzQf0tt308YbytqYQhuFdKRoAb SEzrLuvFUxM5JWLmOedIXRvBXuRzibQAWaOIaV+eomkYKjzWp83QV0nDooS0837CWEaN 5gJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=yuKfDyOo7COq0f9iSZabfLktV2sm6tlW50y80agqyW8=; b=F5+t7R0GcnJYXl0cq5V9NyNqp8o3lqJP9v8VaeMh0VwmtEacxChgJfIvK1LNOGKT93 xMWXQlkZf2P2s6UXqmD3q03vJNcnihcSgvFhHkh+RoVH061Wi55qOiqrGJx+DBXcJCgM CJVSgE+ccGtMX440OSkTPD+4HXL+cn0YWvyftCDyLcqxkDT+2i4CZXyREuAdebPhHD31 nZgDVMxMc4ADeUjcWIrIRoZnqgBrATXe0eLg0T4JJzYwk/9uODKytMJG5v5BtWZQyLmI 9+HIrAjH3R0AdpblJ6f5ayVzWbF+xZ7VQenhRKWyu51wU80uMLimNep6C2GJNm5OY4jw YmSQ== X-Gm-Message-State: AOAM530QfVDnAXnAYFCNgaPVFXqlLBaBMj8LEkKqxWWOThkOfOuAM7CD AxDmAkPYR8jHFX+aOqJ7GFyBdkbZzq2Ilw== X-Google-Smtp-Source: ABdhPJzqrP0a0vm/3SI+avE71WFu9QxCgBkpKfLUQvmcR/ScgkI9ywe8MzFLogdv3iLWWJngmJTrfw== X-Received: by 2002:a37:a747:: with SMTP id q68mr3231312qke.352.1610559643886; Wed, 13 Jan 2021 09:40:43 -0800 (PST) Received: from localhost.localdomain (dsl-157-252.b2b2c.ca. [66.158.157.252]) by smtp.gmail.com with ESMTPSA id d190sm1419001qkc.14.2021.01.13.09.40.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Jan 2021 09:40:43 -0800 (PST) From: Maxim Cournoyer To: 45774@debbugs.gnu.org Subject: [PATCH core-updates v2] build-systems/gnu: Allow unpacking/repacking more kind of files. Date: Wed, 13 Jan 2021 12:40:21 -0500 Message-Id: <20210113174021.23960-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 Content-Type: text/plain; charset=yes Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45774 Cc: Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Before this change, only plain directories, tar or zip archives were supported as the source of a package for the GNU build system; anything else would cause the unpack phase to fail. Origins relying on snippets would suffer from the same problem. This change adds the support to use files of the following extensions: .gz, .Z, .bz2, .lz, and .xz, even when they are not tarballs. Files of unknown extensions are treated as uncompressed files and supported as well. * guix/packages.scm (patch-and-repack): Only add the compressor utility to the PATH when the file is compressed. Bind more inputs in the mlet, and use them for decompressing single files. Adjust decompression and compression routines. [decompression-type]: Return #f when no known compression extension is used. [tarball?]: New nested procedure. * guix/build/utils.scm (compressor, tarball?): New procedures. Move %xz-parallel-args to the new 'compression helpers' section. * tests/packages.scm: Add tests. Add missing copyright year for Jan. * guix/build/gnu-build-system.scm (first-subdirectory): Return #f when no sub-directory was found. (unpack): Support more file types, including uncompressed plain files. --- guix/build/gnu-build-system.scm | 24 ++++++-- guix/build/utils.scm | 47 ++++++++++----- guix/packages.scm | 100 +++++++++++++++++--------------- guix/tests.scm | 40 ++++++++++++- tests/builders.scm | 40 ++++++++++++- tests/packages.scm | 69 +++++++++++++++++++++- 6 files changed, 247 insertions(+), 73 deletions(-) diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm index e556457db9..d36aecb75e 100644 --- a/guix/build/gnu-build-system.scm +++ b/guix/build/gnu-build-system.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès ;;; Copyright © 2018 Mark H Weaver ;;; Copyright © 2020 Brendan Tildesley +;;; Copyright © 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -60,13 +61,15 @@ See https://reproducible-builds.org/specs/source-date-epoch/." (setenv "SOURCE_DATE_EPOCH" "1")) (define (first-subdirectory directory) - "Return the file name of the first sub-directory of DIRECTORY." + "Return the file name of the first sub-directory of DIRECTORY or false, when +there are none." (match (scandir directory (lambda (file) (and (not (member file '("." ".."))) (file-is-directory? (string-append directory "/" file))))) - ((first . _) first))) + ((first . _) first) + (_ #f))) (define* (set-paths #:key target inputs native-inputs (search-paths '()) (native-search-paths '()) @@ -155,10 +158,19 @@ working directory." (copy-recursively source "." #:keep-mtime? #t)) (begin - (if (string-suffix? ".zip" source) - (invoke "unzip" source) - (invoke "tar" "xvf" source)) - (chdir (first-subdirectory "."))))) + (cond + ((string-suffix? ".zip" source) + (invoke "unzip" source)) + ((tarball? source) + (invoke "tar" "xvf" source)) + (else + (let ((name (strip-store-file-name source)) + (command (compressor source))) + (copy-file source name) + (when command + (invoke command "--decompress" name))))) + ;; Attempt to change into child directory. + (and=> (first-subdirectory ".") chdir)))) (define* (bootstrap #:key bootstrap-scripts #:allow-other-keys) diff --git a/guix/build/utils.scm b/guix/build/utils.scm index 76180e67e0..5e7ac74033 100644 --- a/guix/build/utils.scm +++ b/guix/build/utils.scm @@ -6,7 +6,7 @@ ;;; Copyright © 2018 Arun Isaac ;;; Copyright © 2018, 2019 Ricardo Wurmus ;;; Copyright © 2020 Efraim Flashner -;;; Copyright © 2020 Maxim Cournoyer +;;; Copyright © 2020, 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -51,6 +51,10 @@ package-name->name+version parallel-job-count + compressor + tarball? + %xz-parallel-args + directory-exists? executable-file? symbolic-link? @@ -113,9 +117,7 @@ make-desktop-entry-file - locale-category->string - - %xz-parallel-args)) + locale-category->string)) ;;; @@ -137,6 +139,32 @@ (module-replace! (current-module) '(setvbuf))) (else #f)) + +;;; +;;; Compression helpers. +;;; + +(define (compressor file-name) + "Return the name of the compressor package/binary used to compress or +decompress FILE-NAME, based on its file extension, else false." + (cond ((string-suffix? "gz" file-name) "gzip") + ((string-suffix? "Z" file-name) "gzip") + ((string-suffix? "bz2" file-name) "bzip2") + ((string-suffix? "lz" file-name) "lzip") + ((string-suffix? "zip" file-name) "unzip") + ((string-suffix? "xz" file-name) "xz") + (else #f))) ;no compression used/unknown file extension + +(define (tarball? file-name) + "True when FILE-NAME has a tar file extension." + (string-match "\\.(tar(\\..*)?|tgz|tbz)$" file-name)) + +(define (%xz-parallel-args) + "The xz arguments required to enable bit-reproducible, multi-threaded +compression." + (list "--memlimit=50%" + (format #f "--threads=~a" (max 2 (parallel-job-count))))) + ;;; ;;; Directories. @@ -1536,17 +1564,6 @@ returned." LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE LC_TIME))) - -;;; -;;; Others. -;;; - -(define (%xz-parallel-args) - "The xz arguments required to enable bit-reproducible, multi-threaded -compression." - (list "--memlimit=50%" - (format #f "--threads=~a" (max 2 (parallel-job-count))))) - ;;; Local Variables: ;;; eval: (put 'call-with-output-file/atomic 'scheme-indent-function 1) ;;; eval: (put 'call-with-ascii-input-file 'scheme-indent-function 1) diff --git a/guix/packages.scm b/guix/packages.scm index 4caaa9cb79..8f4dc7ec36 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -5,7 +5,7 @@ ;;; Copyright © 2016 Alex Kost ;;; Copyright © 2017, 2019, 2020 Efraim Flashner ;;; Copyright © 2019 Marius Bakke -;;; Copyright © 2020 Maxim Cournoyer +;;; Copyright © 2020, 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -23,6 +23,7 @@ ;;; along with GNU Guix. If not, see . (define-module (guix packages) + #:use-module ((guix build utils) #:select (compressor tarball?)) #:use-module (guix utils) #:use-module (guix records) #:use-module (guix store) @@ -609,14 +610,6 @@ specifies modules in scope when evaluating SNIPPET." ((package) package) (#f #f))))) - (define decompression-type - (cond ((string-suffix? "gz" source-file-name) "gzip") - ((string-suffix? "Z" source-file-name) "gzip") - ((string-suffix? "bz2" source-file-name) "bzip2") - ((string-suffix? "lz" source-file-name) "lzip") - ((string-suffix? "zip" source-file-name) "unzip") - (else "xz"))) - (define original-file-name ;; Remove the store prefix plus the slash, hash, and hyphen. (let* ((sans (string-drop source-file-name @@ -653,17 +646,24 @@ specifies modules in scope when evaluating SNIPPET." (lower-object patch system)))) (mlet %store-monad ((tar -> (lookup-input "tar")) + (gzip -> (lookup-input "gzip")) + (bzip2 -> (lookup-input "bzip2")) + (lzip -> (lookup-input "lzip")) (xz -> (lookup-input "xz")) (patch -> (lookup-input "patch")) (locales -> (lookup-input "locales")) - (decomp -> (lookup-input decompression-type)) + (comp -> (and=> (compressor source-file-name) + lookup-input)) (patches (sequence %store-monad (map instantiate-patch patches)))) (define build (with-imported-modules '((guix build utils)) #~(begin (use-modules (ice-9 ftw) + (ice-9 match) + (ice-9 regex) (srfi srfi-1) + (srfi srfi-26) (guix build utils)) ;; The --sort option was added to GNU tar in version 1.28, released @@ -702,45 +702,53 @@ specifies modules in scope when evaluating SNIPPET." (package-version locales))))) (setlocale LC_ALL "en_US.utf8")) - (setenv "PATH" (string-append #+xz "/bin" ":" - #+decomp "/bin")) + (setenv "PATH" + (string-append #+xz "/bin" + (if #+comp + (string-append ":" #+comp "/bin") + ""))) (setenv "XZ_DEFAULTS" (string-join (%xz-parallel-args))) - ;; SOURCE may be either a directory or a tarball. - (if (file-is-directory? #+source) - (let* ((store (%store-directory)) - (len (+ 1 (string-length store))) - (base (string-drop #+source len)) - (dash (string-index base #\-)) - (directory (string-drop base (+ 1 dash)))) - (mkdir directory) - (copy-recursively #+source directory)) - #+(if (string=? decompression-type "unzip") - #~(invoke "unzip" #+source) - #~(invoke (string-append #+tar "/bin/tar") - "xvf" #+source))) - - (let ((directory (first-file "."))) - (format (current-error-port) - "source is under '~a'~%" directory) - (chdir directory) - - (for-each apply-patch '#+patches) - - #+(if snippet - #~(let ((module (make-fresh-user-module))) - (module-use-interfaces! - module - (map resolve-interface '#+modules)) - ((@ (system base compile) compile) - '#+snippet - #:to 'value - #:opts %auto-compilation-options - #:env module)) - #~#t) - - (chdir "..") + ;; SOURCE may be either a directory, a tarball or a simple file. + (let ((name (strip-store-file-name #+source)) + (command (and=> #+comp (cut string-append <> "/bin/" + (compressor #+source))))) + (if (file-is-directory? #+source) + (copy-recursively #+source name) + (cond + ((tarball? #+source) + (invoke (string-append #+tar "/bin/tar") "xvf" #+source)) + ((and=> (compressor #+source) (cut string= "unzip" <>)) + ;; Note: Referring to the store unzip here (#+unzip) + ;; would introduce a cycle. + ("unzip" (invoke "unzip" #+source))) + (else + (copy-file #+source name) + (when command + (invoke command "--decompress" name)))))) + + (let* ((file (first-file ".")) + (directory (if (file-is-directory? file) + file + "."))) + (format (current-error-port) "source is at '~a'~%" file) + + (with-directory-excursion directory + + (for-each apply-patch '#+patches) + + #+(if snippet + #~(let ((module (make-fresh-user-module))) + (module-use-interfaces! + module + (map resolve-interface '#+modules)) + ((@ (system base compile) compile) + '#+snippet + #:to 'value + #:opts %auto-compilation-options + #:env module)) + #~#t)) (unless tar-supports-sort? (call-with-output-file ".file_list" diff --git a/guix/tests.scm b/guix/tests.scm index fc3d521163..daf4a4d15f 100644 --- a/guix/tests.scm +++ b/guix/tests.scm @@ -20,12 +20,13 @@ #:use-module ((guix config) #:select (%storedir %localstatedir)) #:use-module (guix store) #:use-module (guix derivations) + #:use-module (guix gexp) #:use-module (guix packages) #:use-module (guix base32) #:use-module (guix serialization) #:use-module (guix monads) #:use-module ((guix utils) #:select (substitute-keyword-arguments)) - #:use-module ((guix build utils) #:select (mkdir-p)) + #:use-module ((guix build utils) #:select (mkdir-p compressor)) #:use-module ((gcrypt hash) #:hide (sha256)) #:use-module (guix build-system gnu) #:use-module (gnu packages base) @@ -60,7 +61,9 @@ dummy-package dummy-origin - gnu-make-for-tests)) + gnu-make-for-tests + + test-file)) ;;; Commentary: ;;; @@ -435,6 +438,39 @@ default values, and with EXTRA-FIELDS set as specified." (native-inputs '()) ;no need for 'pkg-config' (inputs %bootstrap-inputs-for-tests)))) + +;;; +;;; Test utility procedures. + +(define (test-file store name content) + "Create a simple file in STORE with CONTENT (a string), compressed according +to its file name extension. Return both its file name and its hash." + (let* ((name-sans-ext (string-take name (string-index-right name #\.))) + (comp (compressor name)) + (command #~(if #+comp + (string-append #+%bootstrap-coreutils&co + "/bin/" #+comp) + #f)) + (f (with-imported-modules '((guix build utils)) + (computed-file name + #~(begin + (use-modules (guix build utils) + (rnrs io simple)) + (with-output-to-file #+name-sans-ext + (lambda _ + (format #t #+content))) + (when #+command + (invoke #+command #+name-sans-ext)) + (copy-file #+name #$output))))) + (file-drv (run-with-store store (lower-object f))) + (file (derivation->output-path file-drv)) + (file-drv-outputs (derivation-outputs file-drv)) + (_ (build-derivations store (list file-drv))) + (file-hash (derivation-output-hash + (assoc-ref file-drv-outputs "out")))) + (values file file-hash))) + +;;; ;; Local Variables: ;; eval: (put 'call-with-derivation-narinfo 'scheme-indent-function 1) ;; eval: (put 'call-with-derivation-substitute 'scheme-indent-function 2) diff --git a/tests/builders.scm b/tests/builders.scm index fdcf38ded3..7b9cfca362 100644 --- a/tests/builders.scm +++ b/tests/builders.scm @@ -17,7 +17,7 @@ ;;; along with GNU Guix. If not, see . -(define-module (test-builders) +(define-module (tests builders) #:use-module (guix download) #:use-module (guix build-system) #:use-module (guix build-system gnu) @@ -78,4 +78,42 @@ (test-assert "gnu-build-system" (build-system? gnu-build-system)) +;;; FIXME: The following fails because the store file names of the test-env +;;; environment are longer than the real ones, so strip-store-file-name +;;; doesn't work as intended. + +;; (use-modules (guix build gnu-build-system) +;; (guix build utils) +;; (ice-9 textual-ports) +;; (srfi srfi-11)) + +;; (define unpack (assoc-ref %standard-phases 'unpack)) +; +;; (define compressors '(("gzip" . "gz") +;; ("xz" . "xz") +;; ("bzip2" . "bz2") +;; (#f . #f))) + +;; (for-each +;; (match-lambda +;; ((comp . ext) + +;; (unless (network-reachable?) (test-skip 1)) ;for bootstrap binaries +;; (test-equal (string-append "gnu-build-system unpack phase, " +;; "single file (compression: " +;; (if comp comp "None") ")") +;; "expected text" +;; (let*-values +;; (((name) "test") +;; ((compressed-name) (if ext +;; (string-append name "." ext) +;; name)) +;; ((file hash) (test-file %store compressed-name "expected text"))) +;; (call-with-temporary-directory +;; (lambda (dir) +;; (with-directory-excursion dir +;; (unpack #:source file) +;; (call-with-input-file name get-string-all)))))))) +;; compressors) + (test-end "builders") diff --git a/tests/packages.scm b/tests/packages.scm index a867f2fd6d..423485b68e 100644 --- a/tests/packages.scm +++ b/tests/packages.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès -;;; Copyright © Jan (janneke) Nieuwenhuizen +;;; Copyright © 2018 Jan (janneke) Nieuwenhuizen +;;; Copyright © 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -17,12 +18,12 @@ ;;; You should have received a copy of the GNU General Public License ;;; along with GNU Guix. If not, see . -(define-module (test-packages) +(define-module (tests packages) #:use-module (guix tests) #:use-module (guix store) #:use-module (guix monads) #:use-module (guix grafts) - #:use-module ((guix gexp) #:select (local-file local-file-file)) + #:use-module (guix gexp) #:use-module (guix utils) #:use-module ((guix diagnostics) ;; Rename the 'location' binding to allow proper syntax @@ -32,6 +33,7 @@ (else name)))) #:use-module ((gcrypt hash) #:prefix gcrypt:) #:use-module (guix derivations) + #:use-module (guix download) #:use-module (guix packages) #:use-module (guix grafts) #:use-module (guix search-paths) @@ -50,6 +52,7 @@ #:use-module (gnu packages version-control) #:use-module (gnu packages xml) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) @@ -576,6 +579,11 @@ (build-derivations %store (list drv)) (call-with-input-file output get-string-all))) + +;;; +;;; Source derivation with snippets. +;;; + (unless (network-reachable?) (test-skip 1)) (test-equal "package-source-derivation, snippet" "OK" @@ -631,6 +639,61 @@ (and (build-derivations %store (list (pk 'snippet-drv drv))) (call-with-input-file out get-string-all)))) +;; Note: lzip is not part of bootstrap-coreutils&co, so is not included to +;; avoid having to rebuild the world. +(define compressors '(("gzip" . "gz") + ("xz" . "xz") + ("bzip2" . "bz2") + (#f . #f))) + +(for-each + (match-lambda + ((comp . ext) + (unless (network-reachable?) (test-skip 1)) + (test-equal (string-append "origin->derivation, single file with snippet " + "(compression: " (if comp comp "None") ")") + "2 + 2 = 4" + (let*-values + (((name) "maths") + ((compressed-name) (if comp + (string-append name "." ext) + name)) + ((file hash) (test-file %store compressed-name "2 + 2 = 5")) + ;; Create an origin using the above computed file and its hash. + ((source) (origin + (method url-fetch) + (uri (string-append "file://" file)) + (file-name compressed-name) + (patch-inputs `(("tar" ,%bootstrap-coreutils&co) + ("xz" ,%bootstrap-coreutils&co) + ("bzip2" ,%bootstrap-coreutils&co) + ("gzip" ,%bootstrap-coreutils&co))) + (patch-guile %bootstrap-guile) + (modules '((guix build utils))) + (snippet `(substitute* ,name + (("5") "4"))) + (hash (content-hash hash)))) + ;; Build origin. + ((drv) (run-with-store %store (origin->derivation source))) + ((out) (derivation->output-path drv))) + ;; Decompress the resulting tar.xz and return its content. + (and (build-derivations %store (list drv)) + (let* ((bin #~(string-append #+%bootstrap-coreutils&co + "/bin")) + (f (computed-file + name + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils)) + (setenv "PATH" #+bin) + (invoke "tar" "xvf" #+out) + (copy-file #+name #$output))))) + (drv (run-with-store %store (lower-object f))) + (_ (build-derivations %store (list drv)))) + (call-with-input-file (derivation->output-path drv) + get-string-all))))))) + compressors) + (test-assert "return value" (let ((drv (package-derivation %store (dummy-package "p")))) (and (derivation? drv) -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Thu Jan 14 12:48:14 2021 Received: (at 45774) by debbugs.gnu.org; 14 Jan 2021 17:48:14 +0000 Received: from localhost ([127.0.0.1]:39125 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l06jG-0007y0-EA for submit@debbugs.gnu.org; Thu, 14 Jan 2021 12:48:14 -0500 Received: from eggs.gnu.org ([209.51.188.92]:42986) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l06jE-0007xi-Ir; Thu, 14 Jan 2021 12:48:12 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:55144) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l06j9-0006a2-AT; Thu, 14 Jan 2021 12:48:07 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=49450 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1l06j8-0004Kw-Ip; Thu, 14 Jan 2021 12:48:07 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxim Cournoyer Subject: Re: bug#45773: [PATCH core-updates 0/1] Allow patch-and-repack to work with plain files. References: <20210110200535.24377-1-maxim.cournoyer@gmail.com> <20210113174021.23960-1-maxim.cournoyer@gmail.com> Date: Thu, 14 Jan 2021 18:48:04 +0100 In-Reply-To: <20210113174021.23960-1-maxim.cournoyer@gmail.com> (Maxim Cournoyer's message of "Wed, 13 Jan 2021 12:40:21 -0500") Message-ID: <877dof4dgr.fsf_-_@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45774 Cc: 45774@debbugs.gnu.org, 45773@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hi! Maxim Cournoyer skribis: > Before this change, only plain directories, tar or zip archives were supp= orted > as the source of a package for the GNU build system; anything else would = cause > the unpack phase to fail. Origins relying on snippets would suffer from = the > same problem. > > This change adds the support to use files of the following extensions: .g= z, > .Z, .bz2, .lz, and .xz, even when they are not tarballs. Files of unknown > extensions are treated as uncompressed files and supported as well. > > * guix/packages.scm (patch-and-repack): Only add the compressor utility t= o the > PATH when the file is compressed. Bind more inputs in the mlet, and use = them > for decompressing single files. Adjust decompression and compression rou= tines. > [decompression-type]: Return #f when no known compression extension is us= ed. > [tarball?]: New nested procedure. > * guix/build/utils.scm (compressor, tarball?): New procedures. Move > %xz-parallel-args to the new 'compression helpers' section. > * tests/packages.scm: Add tests. Add missing copyright year for Jan. > * guix/build/gnu-build-system.scm (first-subdirectory): Return #f when no > sub-directory was found. > (unpack): Support more file types, including uncompressed plain files. > --- > guix/build/gnu-build-system.scm | 24 ++++++-- > guix/build/utils.scm | 47 ++++++++++----- > guix/packages.scm | 100 +++++++++++++++++--------------- > guix/tests.scm | 40 ++++++++++++- > tests/builders.scm | 40 ++++++++++++- > tests/packages.scm | 69 +++++++++++++++++++++- > 6 files changed, 247 insertions(+), 73 deletions(-) How frequent is it for an origin to be a regular file other than an archive? The underlying question for me is: will this generalization and increased complexity pay off? WDYT? There are aspects of the patch that I find welcome regardless, such as the improved handling of compression helpers. Thanks, Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Thu Jan 14 13:43:54 2021 Received: (at 45774) by debbugs.gnu.org; 14 Jan 2021 18:43:54 +0000 Received: from localhost ([127.0.0.1]:39201 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l07b8-0000s3-0n for submit@debbugs.gnu.org; Thu, 14 Jan 2021 13:43:54 -0500 Received: from mail-qv1-f41.google.com ([209.85.219.41]:44801) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l07b6-0000rk-I0; Thu, 14 Jan 2021 13:43:52 -0500 Received: by mail-qv1-f41.google.com with SMTP id d11so2776289qvo.11; Thu, 14 Jan 2021 10:43:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=i679i1YM9Pc1tsnNCnVzfHyLBuHp7YFj3ZdxMElAqmg=; b=bon9fCKv7pSfe+x1pcKPeLxi9jaHSKfGmn9hOKhM4J70m1ZkC0+BD6Xapdh48VfxAF CVTr58nVpdiwj9q6RCMFnBI3PNnD5i7/cwlVws7FWwE8KuPddpRBP1j7PHIAjutpFuaB Njx2LFZsCJgY0y4jjOe2z+sk8ha0L6KJbSxAJw4UWa2+dJ+8ZfUASVvgRmeLzndqCVMU rEhDzSmS75jcEmWhqXx3x20l5KVc08GG02ZHHxonihy1387fLHx3tXGIziRmQb++h/tA es+HkYik544yc6SgenUGMpcjCvdwZ9Vr6c1FKiICahb1iMRJeZEE894cwKlVmMe5TtZg PzJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=i679i1YM9Pc1tsnNCnVzfHyLBuHp7YFj3ZdxMElAqmg=; b=cgMH1rG84iZqojXUJBPc6MPzwgJK1D61GdfnvtzJIul5m515Cst5GRjNNcnmtTlAU0 LHsQ0JSBATXa8n3KEpMDUa/8KQCHO2ojU+o+mRagZsRcj359iryLFA8ZvES5ubgDdBwK GVALeZl9I9GjgyU/2TikyU0onS0VeEsVHXQZ/5p9ohMHr16g1zMrNyX0k7ZagEY7mWi0 dt2rFHPUbkjfPdFVUaqqmkVI1bLrp2ci2zIabW16a2xY3JE7vUkhcv1AQ8OK+ZHGSARF e7VSfytDe7tcXewW7sQqfjgFlmVxYcCTAhr1WC65VKKBlUHvUEYLEPcKe8/3jTTPbKsL gNQw== X-Gm-Message-State: AOAM533cI1HNPW/5cJX/TaDzd+kRjxzkrxL3fTbR1LWSJLXmk5fV55oK mlRvwCNvokKW6dX7G/lDgo47tQL0mKpaAw== X-Google-Smtp-Source: ABdhPJxdUj83w0epSv7j9Ag45A2ITlVqsP55iDxSLuVtxwQsh8Kk4w+0FjmzMdWO87j2vCJ3k2Kn1g== X-Received: by 2002:a0c:a789:: with SMTP id v9mr8309957qva.41.1610649826924; Thu, 14 Jan 2021 10:43:46 -0800 (PST) Received: from hurd (dsl-157-252.b2b2c.ca. [66.158.157.252]) by smtp.gmail.com with ESMTPSA id z9sm3360485qts.66.2021.01.14.10.43.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Jan 2021 10:43:44 -0800 (PST) From: Maxim Cournoyer To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#45773: [PATCH core-updates 0/1] Allow patch-and-repack to work with plain files. References: <20210110200535.24377-1-maxim.cournoyer@gmail.com> <20210113174021.23960-1-maxim.cournoyer@gmail.com> <877dof4dgr.fsf_-_@gnu.org> Date: Thu, 14 Jan 2021 13:43:43 -0500 In-Reply-To: <877dof4dgr.fsf_-_@gnu.org> ("Ludovic =?utf-8?Q?Court=C3=A8s?= =?utf-8?Q?=22's?= message of "Thu, 14 Jan 2021 18:48:04 +0100") Message-ID: <87k0sfl5pc.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45774 Cc: 45774@debbugs.gnu.org, 45773@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludovic, Ludovic Court=C3=A8s writes: > Hi! > > Maxim Cournoyer skribis: > >> Before this change, only plain directories, tar or zip archives were sup= ported >> as the source of a package for the GNU build system; anything else would= cause >> the unpack phase to fail. Origins relying on snippets would suffer from= the >> same problem. >> >> This change adds the support to use files of the following extensions: .= gz, >> .Z, .bz2, .lz, and .xz, even when they are not tarballs. Files of unkno= wn >> extensions are treated as uncompressed files and supported as well. >> >> * guix/packages.scm (patch-and-repack): Only add the compressor utility = to the >> PATH when the file is compressed. Bind more inputs in the mlet, and use= them >> for decompressing single files. Adjust decompression and compression ro= utines. >> [decompression-type]: Return #f when no known compression extension is u= sed. >> [tarball?]: New nested procedure. >> * guix/build/utils.scm (compressor, tarball?): New procedures. Move >> %xz-parallel-args to the new 'compression helpers' section. >> * tests/packages.scm: Add tests. Add missing copyright year for Jan. >> * guix/build/gnu-build-system.scm (first-subdirectory): Return #f when no >> sub-directory was found. >> (unpack): Support more file types, including uncompressed plain files. >> --- >> guix/build/gnu-build-system.scm | 24 ++++++-- >> guix/build/utils.scm | 47 ++++++++++----- >> guix/packages.scm | 100 +++++++++++++++++--------------- >> guix/tests.scm | 40 ++++++++++++- >> tests/builders.scm | 40 ++++++++++++- >> tests/packages.scm | 69 +++++++++++++++++++++- >> 6 files changed, 247 insertions(+), 73 deletions(-) > > How frequent is it for an origin to be a regular file other than an > archive? The underlying question for me is: will this generalization > and increased complexity pay off? WDYT? I think consistency is the main driver here. The url-fetch method supports single file sources; it makes sense that the other components handling sources support it as well. It's hard to judge of the popularity of such a feature when it's never been available; but some use cases come to mind such as single Emacs package file. I've made use of such feature for the new texlive-updmap.cfg definition in . > There are aspects of the patch that I find welcome regardless, such as > the improved handling of compression helpers. Great! Thanks for looking at it. Maxim From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 18 11:52:00 2021 Received: (at 45774) by debbugs.gnu.org; 18 Jan 2021 16:52:00 +0000 Received: from localhost ([127.0.0.1]:48143 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Xl1-0000kv-Oi for submit@debbugs.gnu.org; Mon, 18 Jan 2021 11:51:59 -0500 Received: from mail-qt1-f175.google.com ([209.85.160.175]:34054) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Xl0-0000kJ-5n for 45774@debbugs.gnu.org; Mon, 18 Jan 2021 11:51:58 -0500 Received: by mail-qt1-f175.google.com with SMTP id c1so11702062qtc.1 for <45774@debbugs.gnu.org>; Mon, 18 Jan 2021 08:51:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=OT+qmkYntdzslMt0aEa0WPokhkPkZCvzMN7Y3wqDnes=; b=MGIIzkkXNRDniu9tx6tBn4mtgzV58V/BdDbuln+Auo/2t3wpbwSWkqyu0y9RR2GjYJ CNnp0qZn6zHTaO389bUYy1c5ulSBKKwrXm/vkcn3hOId+GM89NIKCEE2y9UFl6liTQ05 cW6hKgmGJdyViAw0zoi17bOc2f3a8mktZMM1C/bn8x9TMeJVfHb4rlsVGbF8p2MsqZHq DZB8gEG8RLfbL5WWNxsNl3zSy+k50HdokJ0B4KC3leuv4L5mm8oQTSYAo215mwDR867/ DkHRO6OeSlHLVYxJqV2bNdJWozcKxlGKo5DBRY45DNimf8h6hq9Kf3gW9IAwA0yqsTn6 cqhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=OT+qmkYntdzslMt0aEa0WPokhkPkZCvzMN7Y3wqDnes=; b=keuPsI5opZj9y/evXy4A8lxFSy70QqdaGXeG1t1nnHHsEU+qYVEILCRpno3b03ffvj GY/Qcg+P+jC2afzI7CFC8uQVbli/g5y9BqcSc4bIRA2uuwIGG85sNS1x+K7bxRYlaS68 TLvpRG5PLkERjY5E0/keUg/7OSpfONrgwCLXVQ69vULmoSH6otJTQE8DFq/aFi6ZJNmz 3l5IFhDQJP/gK2tzayO+0M2mMQEI8bxb2hR9h32hVDTjeD8TXugy5ujJIotpSOFhb5Ey TjvV7ifzJvUR4uCaG/qLq1WHGIC46tzx2ALyHsj0n5ReksMB7+o9/nztlNksPS5801Bu O7nA== X-Gm-Message-State: AOAM531PjuciNxD7bXD/Nzu72YFSQHDkuPx+yLNBB1ESHbr0WzF7sd0x L1VmuLfCIPIQTGZf18zDwvs= X-Google-Smtp-Source: ABdhPJyfKMzqWFsk/bgqojpzFO8Y4j46+Op/f5deK+0ZPoI3p49cLfQ1DPxnDQAIXRpUlNxBr/2k8Q== X-Received: by 2002:ac8:6edd:: with SMTP id f29mr454223qtv.213.1610988712453; Mon, 18 Jan 2021 08:51:52 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id m64sm10992602qkb.90.2021.01.18.08.51.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jan 2021 08:51:51 -0800 (PST) From: Maxim Cournoyer To: guix-patches@gnu.org, 45774@debbugs.gnu.org Subject: [PATCH core-updates v3 1/2] utils: Retrieve the store prefix from NIX_STORE_DIR, not STORE_DIR. Date: Mon, 18 Jan 2021 11:51:20 -0500 Message-Id: <20210118165122.25813-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45774 Cc: ludo@gnu.org, Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On the daemon side, nixStore gets set to the environment variable NIX_STORE_DIR, else the environment variable NIX_STORE else the compile time macro NIX_STORE_DIR (see the Settings::processEnvironment method in nix/libstore/globals.cc). Hence, it is more appropriate to lookup the environment variable NIX_STORE_DIR than NIX_STORE in (guix build utils). * guix/build/utils.scm (%store-directory): Call getenv with NIX_STORE_DIR instead of NIX_STORE. --- guix/build/utils.scm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/guix/build/utils.scm b/guix/build/utils.scm index 76180e67e0..2cbdb31505 100644 --- a/guix/build/utils.scm +++ b/guix/build/utils.scm @@ -144,7 +144,7 @@ (define (%store-directory) "Return the directory name of the store." - (or (getenv "NIX_STORE") + (or (getenv "NIX_STORE_DIR") "/gnu/store")) (define (store-file-name? file) -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 18 11:52:12 2021 Received: (at 45774) by debbugs.gnu.org; 18 Jan 2021 16:52:13 +0000 Received: from localhost ([127.0.0.1]:48152 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Xl7-0000lp-Im for submit@debbugs.gnu.org; Mon, 18 Jan 2021 11:52:12 -0500 Received: from mail-qk1-f181.google.com ([209.85.222.181]:37288) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l1Xl4-0000kV-62 for 45774@debbugs.gnu.org; Mon, 18 Jan 2021 11:52:04 -0500 Received: by mail-qk1-f181.google.com with SMTP id h4so19237327qkk.4 for <45774@debbugs.gnu.org>; Mon, 18 Jan 2021 08:52:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=93I9hcIBZAt02KLcqyLBdmz5tQhsR8FszOh0O4hOiyQ=; b=WDdX8RRtz/eK/xRIAyPzvjJzT30QHm98zTuYjUX6HgIQSUfY2a8QOR+uePMBg6QuC2 cpFz22CLVMYsTgII4uaGsyEySv43R7iXH/mejRU+XADCRVea3V3sBUxoMUi+SDVva7Wo hGaz292cbM1en7QkpPGz/Y1WgH5uh3mt38+R1K8DYCyuFncUpFxk4lDutck8dWp1bZ+9 OywB/BUxu0mvmxNH7LCvyesIK30LKIvx0kQQHeLM9uH2T9FdYTO3H5/PYNvPWpt201Q4 ng3iycIj7eVDzZQs80NAt0m1Hg/GF9wt11P5Lcqv2usw0fnRMr40W26N0aqikr3lBjLr yQyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=93I9hcIBZAt02KLcqyLBdmz5tQhsR8FszOh0O4hOiyQ=; b=NFQIOr+GeMoJEhXxm1TYaWjupmrRc35vGpQqeyXJEQ+HbEwyPTjaDuyKAubi7tTZNV iubo67rD4RiFvXZVQZPOqR5larShID+3wVCWi4VGJtlqhmoPyg6GBrwiAq2MPS2JLZKR 7JYoK3cnlHE6fiCAjjG3YZx/j+PEAg58OSMSx7UvdyAMSuvj9y0kCaByTtWT1DMyVdaK p6vMpz1f+JKk0YRvtB2JsN2d60h24USBRsj+d0mhgZjn+TAz9ZUPDKAoDa+wCm3VmFb8 DBBzWTHUCMTOx2IaCfTUz4KRPHlawMGfgMEIVCaiceqbXxZrRupVJYkWA3L287VcMRD0 0cEw== X-Gm-Message-State: AOAM533wZKo2rCsnyIZGROr2ECS3kS8Q/Cknx6djQqNnUAFjas0Z3H/G ttwERFOjOTDXwNU6sYmJH/Y= X-Google-Smtp-Source: ABdhPJw/ydeNhDSWz1dGaEA/U62gQoq7BcIzXeX3EwA2AJhvXQU8QfMe5FGZmBGtgsjAuw/vMxulLQ== X-Received: by 2002:a05:620a:947:: with SMTP id w7mr432151qkw.389.1610988716327; Mon, 18 Jan 2021 08:51:56 -0800 (PST) Received: from localhost.localdomain (dsl-149-228.b2b2c.ca. [66.158.149.228]) by smtp.gmail.com with ESMTPSA id m64sm10992602qkb.90.2021.01.18.08.51.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jan 2021 08:51:55 -0800 (PST) From: Maxim Cournoyer To: guix-patches@gnu.org, 45774@debbugs.gnu.org Subject: [PATCH core-updates v3 2/2] build-systems/gnu: Allow unpacking/repacking more kind of files. Date: Mon, 18 Jan 2021 11:51:21 -0500 Message-Id: <20210118165122.25813-2-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210118165122.25813-1-maxim.cournoyer@gmail.com> References: <20210118165122.25813-1-maxim.cournoyer@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=yes Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45774 Cc: ludo@gnu.org, Maxim Cournoyer X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Before this change, only plain directories, tar or zip archives were supported as the source of a package for the GNU build system; anything else would cause the unpack phase to fail. Origins relying on snippets would suffer from the same problem. This change adds the support to use files of the following extensions: .gz, .Z, .bz2, .lz, and .xz, even when they are not tarballs. Files of unknown extensions are treated as uncompressed files and supported as well. * guix/packages.scm (patch-and-repack): Only add the compressor utility to the PATH when the file is compressed. Bind more inputs in the mlet, and use them for decompressing single files. Adjust the decompression and compression routines. [decompression-type]: Remove nested variable. * guix/build/utils.scm (compressor, tarball?): New procedures. Move %xz-parallel-args to the new 'compression helpers' section. * tests/packages.scm: Add tests. Add missing copyright year for Jan. * guix/build/gnu-build-system.scm (first-subdirectory): Return #f when no sub-directory was found. (unpack): Support more file types, including uncompressed plain files. --- guix/build/gnu-build-system.scm | 24 +++++-- guix/build/utils.scm | 47 +++++++++---- guix/packages.scm | 117 ++++++++++++++++++-------------- guix/tests.scm | 43 +++++++++++- tests/builders.scm | 35 +++++++++- tests/packages.scm | 72 +++++++++++++++++++- 6 files changed, 259 insertions(+), 79 deletions(-) diff --git a/guix/build/gnu-build-system.scm b/guix/build/gnu-build-system.scm index f8e8a46854..66edd2de2d 100644 --- a/guix/build/gnu-build-system.scm +++ b/guix/build/gnu-build-system.scm @@ -2,6 +2,7 @@ ;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès ;;; Copyright © 2018 Mark H Weaver ;;; Copyright © 2020 Brendan Tildesley +;;; Copyright © 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -60,13 +61,15 @@ See https://reproducible-builds.org/specs/source-date-epoch/." (setenv "SOURCE_DATE_EPOCH" "1")) (define (first-subdirectory directory) - "Return the file name of the first sub-directory of DIRECTORY." + "Return the file name of the first sub-directory of DIRECTORY or false, when +there are none." (match (scandir directory (lambda (file) (and (not (member file '("." ".."))) (file-is-directory? (string-append directory "/" file))))) - ((first . _) first))) + ((first . _) first) + (_ #f))) (define* (set-paths #:key target inputs native-inputs (search-paths '()) (native-search-paths '()) @@ -155,10 +158,19 @@ working directory." (copy-recursively source "." #:keep-mtime? #t)) (begin - (if (string-suffix? ".zip" source) - (invoke "unzip" source) - (invoke "tar" "xvf" source)) - (chdir (first-subdirectory "."))))) + (cond + ((string-suffix? ".zip" source) + (invoke "unzip" source)) + ((tarball? source) + (invoke "tar" "xvf" source)) + (else + (let ((name (strip-store-file-name source)) + (command (compressor source))) + (copy-file source name) + (when command + (invoke command "--decompress" name))))) + ;; Attempt to change into child directory. + (and=> (first-subdirectory ".") chdir)))) (define* (bootstrap #:key bootstrap-scripts #:allow-other-keys) diff --git a/guix/build/utils.scm b/guix/build/utils.scm index 2cbdb31505..01904785bd 100644 --- a/guix/build/utils.scm +++ b/guix/build/utils.scm @@ -6,7 +6,7 @@ ;;; Copyright © 2018 Arun Isaac ;;; Copyright © 2018, 2019 Ricardo Wurmus ;;; Copyright © 2020 Efraim Flashner -;;; Copyright © 2020 Maxim Cournoyer +;;; Copyright © 2020, 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -51,6 +51,10 @@ package-name->name+version parallel-job-count + compressor + tarball? + %xz-parallel-args + directory-exists? executable-file? symbolic-link? @@ -113,9 +117,7 @@ make-desktop-entry-file - locale-category->string - - %xz-parallel-args)) + locale-category->string)) ;;; @@ -137,6 +139,32 @@ (module-replace! (current-module) '(setvbuf))) (else #f)) + +;;; +;;; Compression helpers. +;;; + +(define (compressor file-name) + "Return the name of the compressor package/binary used to compress or +decompress FILE-NAME, based on its file extension, else false." + (cond ((string-suffix? "gz" file-name) "gzip") + ((string-suffix? "Z" file-name) "gzip") + ((string-suffix? "bz2" file-name) "bzip2") + ((string-suffix? "lz" file-name) "lzip") + ((string-suffix? "zip" file-name) "unzip") + ((string-suffix? "xz" file-name) "xz") + (else #f))) ;no compression used/unknown file extension + +(define (tarball? file-name) + "True when FILE-NAME has a tar file extension." + (string-match "\\.(tar(\\..*)?|tgz|tbz)$" file-name)) + +(define (%xz-parallel-args) + "The xz arguments required to enable bit-reproducible, multi-threaded +compression." + (list "--memlimit=50%" + (format #f "--threads=~a" (max 2 (parallel-job-count))))) + ;;; ;;; Directories. @@ -1536,17 +1564,6 @@ returned." LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE LC_TIME))) - -;;; -;;; Others. -;;; - -(define (%xz-parallel-args) - "The xz arguments required to enable bit-reproducible, multi-threaded -compression." - (list "--memlimit=50%" - (format #f "--threads=~a" (max 2 (parallel-job-count))))) - ;;; Local Variables: ;;; eval: (put 'call-with-output-file/atomic 'scheme-indent-function 1) ;;; eval: (put 'call-with-ascii-input-file 'scheme-indent-function 1) diff --git a/guix/packages.scm b/guix/packages.scm index cd2cded9ee..73f605f3a9 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -5,7 +5,7 @@ ;;; Copyright © 2016 Alex Kost ;;; Copyright © 2017, 2019, 2020 Efraim Flashner ;;; Copyright © 2019 Marius Bakke -;;; Copyright © 2020 Maxim Cournoyer +;;; Copyright © 2020, 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -23,6 +23,7 @@ ;;; along with GNU Guix. If not, see . (define-module (guix packages) + #:use-module ((guix build utils) #:select (compressor tarball?)) #:use-module (guix utils) #:use-module (guix records) #:use-module (guix store) @@ -609,14 +610,6 @@ specifies modules in scope when evaluating SNIPPET." ((package) package) (#f #f))))) - (define decompression-type - (cond ((string-suffix? "gz" source-file-name) "gzip") - ((string-suffix? "Z" source-file-name) "gzip") - ((string-suffix? "bz2" source-file-name) "bzip2") - ((string-suffix? "lz" source-file-name) "lzip") - ((string-suffix? "zip" source-file-name) "unzip") - (else "xz"))) - (define original-file-name ;; Remove the store prefix plus the slash, hash, and hyphen. (let* ((sans (string-drop source-file-name @@ -651,17 +644,24 @@ specifies modules in scope when evaluating SNIPPET." (lower-object patch system)))) (mlet %store-monad ((tar -> (lookup-input "tar")) + (gzip -> (lookup-input "gzip")) + (bzip2 -> (lookup-input "bzip2")) + (lzip -> (lookup-input "lzip")) (xz -> (lookup-input "xz")) (patch -> (lookup-input "patch")) (locales -> (lookup-input "locales")) - (decomp -> (lookup-input decompression-type)) + (comp -> (and=> (compressor source-file-name) + lookup-input)) (patches (sequence %store-monad (map instantiate-patch patches)))) (define build (with-imported-modules '((guix build utils)) #~(begin (use-modules (ice-9 ftw) + (ice-9 match) + (ice-9 regex) (srfi srfi-1) + (srfi srfi-26) (guix build utils)) ;; The --sort option was added to GNU tar in version 1.28, released @@ -723,54 +723,67 @@ specifies modules in scope when evaluating SNIPPET." (package-version locales))))) (setlocale LC_ALL "en_US.utf8")) - (setenv "PATH" (string-append #+xz "/bin" ":" - #+decomp "/bin")) + (setenv "PATH" + (string-append #+xz "/bin" + (if #+comp + (string-append ":" #+comp "/bin") + ""))) (setenv "XZ_DEFAULTS" (string-join (%xz-parallel-args))) - ;; SOURCE may be either a directory or a tarball. - (if (file-is-directory? #+source) - (let* ((store (%store-directory)) - (len (+ 1 (string-length store))) - (base (string-drop #+source len)) - (dash (string-index base #\-)) - (directory (string-drop base (+ 1 dash)))) - (mkdir directory) - (copy-recursively #+source directory)) - #+(if (string=? decompression-type "unzip") - #~(invoke "unzip" #+source) - #~(invoke (string-append #+tar "/bin/tar") - "xvf" #+source))) - - (let ((directory (first-file "."))) - (format (current-error-port) - "source is under '~a'~%" directory) - (chdir directory) - - (for-each apply-patch '#+patches) - - #+(if snippet - #~(let ((module (make-fresh-user-module))) - (module-use-interfaces! - module - (map resolve-interface '#+modules)) - ((@ (system base compile) compile) - '#+snippet - #:to 'value - #:opts %auto-compilation-options - #:env module)) - #~#t) - - (chdir "..") + ;; SOURCE may be either a directory, a tarball or a simple file. + (let ((name (strip-store-file-name #+source)) + (command (and=> #+comp (cut string-append <> "/bin/" + (compressor #+source))))) + (if (file-is-directory? #+source) + (copy-recursively #+source name) + (cond + ((tarball? #+source) + (invoke (string-append #+tar "/bin/tar") "xvf" #+source)) + ((and=> (compressor #+source) (cut string= "unzip" <>)) + ;; Note: Referring to the store unzip here (#+unzip) + ;; would introduce a cycle. + ("unzip" (invoke "unzip" #+source))) + (else + (copy-file #+source name) + (when command + (invoke command "--decompress" name)))))) + + (let* ((file (first-file ".")) + (directory (if (file-is-directory? file) + file + "."))) + (format (current-error-port) "source is at '~a'~%" file) + + (with-directory-excursion directory + + (for-each apply-patch '#+patches) + + #+(if snippet + #~(let ((module (make-fresh-user-module))) + (module-use-interfaces! + module + (map resolve-interface '#+modules)) + ((@ (system base compile) compile) + '#+snippet + #:to 'value + #:opts %auto-compilation-options + #:env module)) + #~#t)) ;; If SOURCE is a directory (such as a checkout), return a ;; directory. Otherwise create a tarball. - (if (file-is-directory? #+source) - (copy-recursively directory #$output - #:log (%make-void-port "w")) - (repack directory #$output)))))) - - (let ((name (if (checkout? original-file-name) + (cond + ((file-is-directory? #+source) + (copy-recursively directory #$output + #:log (%make-void-port "w"))) + ((not #+comp) + (copy-file file #$output)) + (else + (repack directory #$output))))))) + + (let ((name (if (or (checkout? original-file-name) + (not (compressor original-file-name))) original-file-name (tarxz-name original-file-name)))) (gexp->derivation name build diff --git a/guix/tests.scm b/guix/tests.scm index fc3d521163..da75835099 100644 --- a/guix/tests.scm +++ b/guix/tests.scm @@ -20,12 +20,13 @@ #:use-module ((guix config) #:select (%storedir %localstatedir)) #:use-module (guix store) #:use-module (guix derivations) + #:use-module (guix gexp) #:use-module (guix packages) #:use-module (guix base32) #:use-module (guix serialization) #:use-module (guix monads) #:use-module ((guix utils) #:select (substitute-keyword-arguments)) - #:use-module ((guix build utils) #:select (mkdir-p)) + #:use-module ((guix build utils) #:select (mkdir-p compressor)) #:use-module ((gcrypt hash) #:hide (sha256)) #:use-module (guix build-system gnu) #:use-module (gnu packages base) @@ -60,7 +61,9 @@ dummy-package dummy-origin - gnu-make-for-tests)) + gnu-make-for-tests + + test-file)) ;;; Commentary: ;;; @@ -435,6 +438,42 @@ default values, and with EXTRA-FIELDS set as specified." (native-inputs '()) ;no need for 'pkg-config' (inputs %bootstrap-inputs-for-tests)))) + +;;; +;;; Test utility procedures. + +(define (test-file store name content) + "Create a simple file in STORE with CONTENT (a string), compressed according +to its file name extension. Return both its file name and its hash." + (let* ((ext (string-index-right name #\.)) + (name-sans-ext (if ext + (string-take name (string-index-right name #\.)) + name)) + (comp (compressor name)) + (command #~(if #+comp + (string-append #+%bootstrap-coreutils&co + "/bin/" #+comp) + #f)) + (f (with-imported-modules '((guix build utils)) + (computed-file name + #~(begin + (use-modules (guix build utils) + (rnrs io simple)) + (with-output-to-file #+name-sans-ext + (lambda _ + (format #t #+content))) + (when #+command + (invoke #+command #+name-sans-ext)) + (copy-file #+name #$output))))) + (file-drv (run-with-store store (lower-object f))) + (file (derivation->output-path file-drv)) + (file-drv-outputs (derivation-outputs file-drv)) + (_ (build-derivations store (list file-drv))) + (file-hash (derivation-output-hash + (assoc-ref file-drv-outputs "out")))) + (values file file-hash))) + +;;; ;; Local Variables: ;; eval: (put 'call-with-derivation-narinfo 'scheme-indent-function 1) ;; eval: (put 'call-with-derivation-substitute 'scheme-indent-function 2) diff --git a/tests/builders.scm b/tests/builders.scm index fdcf38ded3..624547500a 100644 --- a/tests/builders.scm +++ b/tests/builders.scm @@ -17,10 +17,12 @@ ;;; along with GNU Guix. If not, see . -(define-module (test-builders) +(define-module (tests builders) #:use-module (guix download) #:use-module (guix build-system) #:use-module (guix build-system gnu) + #:use-module (guix build gnu-build-system) + #:use-module (guix build utils) #:use-module (guix store) #:use-module (guix utils) #:use-module (guix base32) @@ -32,7 +34,9 @@ package-derivation package-native-search-paths)) #:use-module (gnu packages bootstrap) #:use-module (ice-9 match) + #:use-module (ice-9 textual-ports) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) #:use-module (srfi srfi-64)) ;; Test the higher-level builders. @@ -78,4 +82,33 @@ (test-assert "gnu-build-system" (build-system? gnu-build-system)) +(define unpack (assoc-ref %standard-phases 'unpack)) + +(define compressors '(("gzip" . "gz") + ("xz" . "xz") + ("bzip2" . "bz2") + (#f . #f))) + +(for-each + (match-lambda + ((comp . ext) + + (unless (network-reachable?) (test-skip 1)) ;for bootstrap binaries + (test-equal (string-append "gnu-build-system unpack phase, " + "single file (compression: " + (if comp comp "None") ")") + "expected text" + (let*-values + (((name) "test") + ((compressed-name) (if ext + (string-append name "." ext) + name)) + ((file hash) (test-file %store compressed-name "expected text"))) + (call-with-temporary-directory + (lambda (dir) + (with-directory-excursion dir + (unpack #:source file) + (call-with-input-file name get-string-all)))))))) + compressors) + (test-end "builders") diff --git a/tests/packages.scm b/tests/packages.scm index a867f2fd6d..b3ccd98e48 100644 --- a/tests/packages.scm +++ b/tests/packages.scm @@ -1,6 +1,7 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright © 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020 Ludovic Courtès -;;; Copyright © Jan (janneke) Nieuwenhuizen +;;; Copyright © 2018 Jan (janneke) Nieuwenhuizen +;;; Copyright © 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -17,13 +18,14 @@ ;;; You should have received a copy of the GNU General Public License ;;; along with GNU Guix. If not, see . -(define-module (test-packages) +(define-module (tests packages) #:use-module (guix tests) #:use-module (guix store) #:use-module (guix monads) #:use-module (guix grafts) - #:use-module ((guix gexp) #:select (local-file local-file-file)) + #:use-module (guix gexp) #:use-module (guix utils) + #:use-module ((guix build utils) #:select (tarball?)) #:use-module ((guix diagnostics) ;; Rename the 'location' binding to allow proper syntax ;; matching when setting the 'location' field of a package. @@ -32,6 +34,7 @@ (else name)))) #:use-module ((gcrypt hash) #:prefix gcrypt:) #:use-module (guix derivations) + #:use-module (guix download) #:use-module (guix packages) #:use-module (guix grafts) #:use-module (guix search-paths) @@ -50,6 +53,7 @@ #:use-module (gnu packages version-control) #:use-module (gnu packages xml) #:use-module (srfi srfi-1) + #:use-module (srfi srfi-11) #:use-module (srfi srfi-26) #:use-module (srfi srfi-34) #:use-module (srfi srfi-35) @@ -576,6 +580,11 @@ (build-derivations %store (list drv)) (call-with-input-file output get-string-all))) + +;;; +;;; Source derivation with snippets. +;;; + (unless (network-reachable?) (test-skip 1)) (test-equal "package-source-derivation, snippet" "OK" @@ -631,6 +640,63 @@ (and (build-derivations %store (list (pk 'snippet-drv drv))) (call-with-input-file out get-string-all)))) +;; Note: lzip is not part of bootstrap-coreutils&co, so is not included to +;; avoid having to rebuild the world. +(define compressors '(("gzip" . "gz") + ("xz" . "xz") + ("bzip2" . "bz2") + (#f . #f))) + +(for-each + (match-lambda + ((comp . ext) + (unless (network-reachable?) (test-skip 1)) + (test-equal (string-append "origin->derivation, single file with snippet " + "(compression: " (if comp comp "None") ")") + "2 + 2 = 4" + (let*-values + (((name) "maths") + ((compressed-name) (if comp + (string-append name "." ext) + name)) + ((file hash) (test-file %store compressed-name "2 + 2 = 5")) + ;; Create an origin using the above computed file and its hash. + ((source) (origin + (method url-fetch) + (uri (string-append "file://" file)) + (file-name compressed-name) + (patch-inputs `(("tar" ,%bootstrap-coreutils&co) + ("xz" ,%bootstrap-coreutils&co) + ("bzip2" ,%bootstrap-coreutils&co) + ("gzip" ,%bootstrap-coreutils&co))) + (patch-guile %bootstrap-guile) + (modules '((guix build utils))) + (snippet `(substitute* ,name + (("5") "4"))) + (hash (content-hash hash)))) + ;; Build origin. + ((drv) (run-with-store %store (origin->derivation source))) + ((out) (derivation->output-path drv))) + ;; Decompress the resulting tar.xz and return its content. + (and (build-derivations %store (list drv)) + (if (tarball? out) + (let* ((bin #~(string-append #+%bootstrap-coreutils&co + "/bin")) + (f (computed-file + name + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils)) + (setenv "PATH" #+bin) + (invoke "tar" "xvf" #+out) + (copy-file #+name #$output))))) + (drv (run-with-store %store (lower-object f))) + (_ (build-derivations %store (list drv)))) + (call-with-input-file (derivation->output-path drv) + get-string-all)) + (call-with-input-file out get-string-all))))))) + compressors) + (test-assert "return value" (let ((drv (package-derivation %store (dummy-package "p")))) (and (derivation? drv) -- 2.29.2 From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 25 09:20:20 2021 Received: (at control) by debbugs.gnu.org; 25 Jan 2021 14:20:20 +0000 Received: from localhost ([127.0.0.1]:38314 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l42j6-000347-0r for submit@debbugs.gnu.org; Mon, 25 Jan 2021 09:20:20 -0500 Received: from mail-qt1-f171.google.com ([209.85.160.171]:34160) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l42j4-00033s-00 for control@debbugs.gnu.org; Mon, 25 Jan 2021 09:20:18 -0500 Received: by mail-qt1-f171.google.com with SMTP id c1so9749145qtc.1 for ; Mon, 25 Jan 2021 06:20:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:to:from:subject; bh=gCFvmYJDzj2a+bw8ruHcvq/HWZbcD8xbFxFcvg/Jlrs=; b=YUijOT+J8gaQTUITUjj6UF912hnAXlSU5GSoUVdSPtuIgbFB4Gl977s/j933yX9OUx UM52wxwk7mCu7JndM1f1rdJ98uW1qLcdTT8aGKR//O6TdJWgcniiVlk0KJGaoPkUMn2Y Xb3fe/FLHMUSFQBccSkjGJodguFGVXQyOtds+7735V4c6IMpmS854/bEbEKFZmqBWQgP FNLL68eKSDd4YBUuJQVVD68W2Ek6lrEWEBpF0F+PRovppQIDBFo/Nadgui8mzboaDtCD 1tAwEq32e1CWyG15lCfmI5+Lt7VBlC3icMXIiITjlW+eoKALTT9YkMBEM5Bkr1AiXoGT iPOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:to:from:subject; bh=gCFvmYJDzj2a+bw8ruHcvq/HWZbcD8xbFxFcvg/Jlrs=; b=StOVxHg3moyXi8BWYq/6CYpE8cI0dLxgpz4mjf5PT4Xoo8indseDPVqSjQHy7a2PLh F89NpPFgS0Pj+1/QrhsTVpLBITKjVklzU4NcZieXEFa3qT+lBGf3/KidTpCZzfi3dIJD zE51/Dnfv5fo6aZZMxmb+ZVAwG37ET+nBWcOptF8mVxZqZBIGRor9NgyvOal0QSED8bP oOJWpvAsjHw2/i07RT5uquUGAsPcGUK8KBNSepUS0A4FZ23bfoyhgqwt7nopfeZRHHnJ wc7g2/LNvoKQNq+ip6hO8mF84lIHk1esBL+MS6ZElTYwg+SeuC3Vlmt34//q/hhrmqZa cGPQ== X-Gm-Message-State: AOAM532gglYf+gXSqbm7VB8XESGpVC75hsBpk2dB8AHZNd9b7PyqBgeD mDsPgCEpujhWGuZ8+A0VSl0q/52ssNY= X-Google-Smtp-Source: ABdhPJwF78J+GetwexlgAbOzXk20EoD2vbnEjyWKEMVIsu2DzoDONo3PaAz/c00HP7kSef0r+/Zlwg== X-Received: by 2002:ac8:4313:: with SMTP id z19mr619360qtm.225.1611584412328; Mon, 25 Jan 2021 06:20:12 -0800 (PST) Received: from hurd (dsl-10-132-34.b2b2c.ca. [72.10.132.34]) by smtp.gmail.com with ESMTPSA id n139sm7487670qke.89.2021.01.25.06.20.10 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 06:20:10 -0800 (PST) Date: Mon, 25 Jan 2021 09:20:09 -0500 Message-Id: <87ft2p3xpi.fsf@gmail.com> To: control@debbugs.gnu.org From: Maxim Cournoyer Subject: control message for bug #45959 X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) forcemerge 45959 45773 quit From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 26 22:57:32 2021 Received: (at 45774-done) by debbugs.gnu.org; 27 Jan 2021 03:57:32 +0000 Received: from localhost ([127.0.0.1]:43256 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l4bxU-0006dS-4w for submit@debbugs.gnu.org; Tue, 26 Jan 2021 22:57:32 -0500 Received: from mail-qk1-f178.google.com ([209.85.222.178]:46641) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l4bxS-0006d8-3u; Tue, 26 Jan 2021 22:57:30 -0500 Received: by mail-qk1-f178.google.com with SMTP id a7so569266qkb.13; Tue, 26 Jan 2021 19:57:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=9zRkkNeuLpkXYdFUAaY8o6Tp2v1Ai3QiaQn+qxzV3Dg=; b=ZoSdJqGWA9UIao9OywvK9R4S6ULo/z0spH5P++3Gufmd21Wy19dHJ2wNN7Ypc3k2Nd OD2M79bA1pI0s139X5WQGjm7GrpoVu+gj8iegNKSe8AEKSynC5SnuA1Ek+uLVWtHW/xJ R9E820yAUJaU1J5yuCP2MkxR3mq09v6Zw7FicsepgJUIUgeBIbLVYtr/KiNey4K8wnDp uUMtv00sWGOxnbk2qTd2U4aYFoFApXOK2pPVCUS61bjHbV7kUMedn8Udk7l+L3/fUp+S zZ0HUQ23zu+JKWXLBBnYSbulqp3FUTR1eNeCoVdr7ooZehKX+7hng+VpILS8WrKzB/ih tTPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=9zRkkNeuLpkXYdFUAaY8o6Tp2v1Ai3QiaQn+qxzV3Dg=; b=tH4NeYm2V/dOMQmAyc3qq7nfkOJVdNJa6Uox9MfB7BoMpZHmH3jAXpwms7B3UZfdOs h1mQ0bedihRZ1aGNue6TuquaktHclRGx6hwWH16snVb1x5UgwazD8gN2LPSxydEhfHCs 3qftWoWk2lQy9gwYPB/Bpji1G0h3G+/N0yJQWPnka7e9amme5qBneLts644ggZ4YA7cI WGHhs8g6ScKBvodYQhNo6fWR+p+sqKK2NpcIqMTSPxB5lVKRdm8ndU6GTG4ESH4pbRz1 qL7pJpzMnw3b5InrIjYq2FRnmFdUOIrEpoUKVUBZAnTKuwRgbVaOsX+gTRzD6wUq0Uo6 Qa/Q== X-Gm-Message-State: AOAM532SUPQCS4UtZsop5ls0VACqRcjjv7oAHjMXetGfLSoZHwn7s2Tw F/YqAU7YUB2sgssmPE0NNQuSe5Zw7ws= X-Google-Smtp-Source: ABdhPJz9bM2PDS1wwOaov/2NfN9ez3t4v2bjkSeEwtH3VDOJZLtFqilUsIpEHZjr5ezLD54USzxexg== X-Received: by 2002:a05:620a:2105:: with SMTP id l5mr9025465qkl.126.1611719844267; Tue, 26 Jan 2021 19:57:24 -0800 (PST) Received: from hurd (dsl-10-132-34.b2b2c.ca. [72.10.132.34]) by smtp.gmail.com with ESMTPSA id n94sm549163qtd.28.2021.01.26.19.57.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Jan 2021 19:57:23 -0800 (PST) From: Maxim Cournoyer To: Ludovic =?utf-8?Q?Court=C3=A8s?= Subject: Re: bug#45773: [PATCH core-updates 0/1] Allow patch-and-repack to work with plain files. References: <20210110200535.24377-1-maxim.cournoyer@gmail.com> <20210113174021.23960-1-maxim.cournoyer@gmail.com> <877dof4dgr.fsf_-_@gnu.org> <87k0sfl5pc.fsf@gmail.com> Date: Tue, 26 Jan 2021 22:57:22 -0500 In-Reply-To: <87k0sfl5pc.fsf@gmail.com> (Maxim Cournoyer's message of "Thu, 14 Jan 2021 13:43:43 -0500") Message-ID: <87wnvzyqu5.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45774-done Cc: 45774-done@debbugs.gnu.org, 45773-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi Ludo, Maxim Cournoyer writes: > Hi Ludovic, > > Ludovic Court=C3=A8s writes: > >> Hi! >> >> Maxim Cournoyer skribis: >> >>> Before this change, only plain directories, tar or zip archives were su= pported >>> as the source of a package for the GNU build system; anything else woul= d cause >>> the unpack phase to fail. Origins relying on snippets would suffer fro= m the >>> same problem. >>> >>> This change adds the support to use files of the following extensions: = .gz, >>> .Z, .bz2, .lz, and .xz, even when they are not tarballs. Files of unkn= own >>> extensions are treated as uncompressed files and supported as well. >>> >>> * guix/packages.scm (patch-and-repack): Only add the compressor utility= to the >>> PATH when the file is compressed. Bind more inputs in the mlet, and us= e them >>> for decompressing single files. Adjust decompression and compression r= outines. >>> [decompression-type]: Return #f when no known compression extension is = used. >>> [tarball?]: New nested procedure. >>> * guix/build/utils.scm (compressor, tarball?): New procedures. Move >>> %xz-parallel-args to the new 'compression helpers' section. >>> * tests/packages.scm: Add tests. Add missing copyright year for Jan. >>> * guix/build/gnu-build-system.scm (first-subdirectory): Return #f when = no >>> sub-directory was found. >>> (unpack): Support more file types, including uncompressed plain files. >>> --- >>> guix/build/gnu-build-system.scm | 24 ++++++-- >>> guix/build/utils.scm | 47 ++++++++++----- >>> guix/packages.scm | 100 +++++++++++++++++--------------- >>> guix/tests.scm | 40 ++++++++++++- >>> tests/builders.scm | 40 ++++++++++++- >>> tests/packages.scm | 69 +++++++++++++++++++++- >>> 6 files changed, 247 insertions(+), 73 deletions(-) >> >> How frequent is it for an origin to be a regular file other than an >> archive? The underlying question for me is: will this generalization >> and increased complexity pay off? WDYT? > > I think consistency is the main driver here. The url-fetch method > supports single file sources; it makes sense that the other components > handling sources support it as well. It's hard to judge of the > popularity of such a feature when it's never been available; but some > use cases come to mind such as single Emacs package file. I've made use > of such feature for the new texlive-updmap.cfg definition in > . I've been building a sizable part of core-updates on top of this change now for nearly two weeks, and in this time it's already come handy twice. I've made sure the tests ran fine and pushed to core-updates as commit cfcead2e515c0dae02127e5a76496463898be6b6. Let me know if anything breaks :-). Maxim From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 26 22:59:01 2021 Received: (at 45774-done) by debbugs.gnu.org; 27 Jan 2021 03:59:01 +0000 Received: from localhost ([127.0.0.1]:43274 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l4byu-0006gs-Tp for submit@debbugs.gnu.org; Tue, 26 Jan 2021 22:59:01 -0500 Received: from mail-qv1-f51.google.com ([209.85.219.51]:41445) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l4byu-0006gc-3X; Tue, 26 Jan 2021 22:59:00 -0500 Received: by mail-qv1-f51.google.com with SMTP id h21so483879qvb.8; Tue, 26 Jan 2021 19:58:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=Zt9DiiPCWqltC7WcZm2o694F5WAmWxbhJfqf1wqFUnk=; b=Sg7I4n/+q4aBhc6LuB/Mw8Vfw0VCvHvbzc+IgKaqZoG/Z0sWQLf6JVgZPAlkvUo1rY Q/W6wVYqKa5NoOFLOy6Qb8X75rc8wovCFcJAOs8fjJqBLJ0/vCT6voqTuo/XHWAtZEf9 VaKme7amAjKEw6xyMrkeMY6VmUek50E9W0GcF3bYzeH2dx8UMaXfuet28nU8ZIZN5E7G M2xyNzpuUvN+EkIZzQvVUw9mGpLguQMuBfIghXfQLulEpt0LEb6YIPmyb5BFn2nZuwEE m8BE41BzLoOCBY3vbGH3KLop3yk49fjAAV15zIrAai1Zo4VRPOnk0zAZwpjMrusw4nfd +OFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=Zt9DiiPCWqltC7WcZm2o694F5WAmWxbhJfqf1wqFUnk=; b=XgB9QdZCpDwNC+IyrAdNtugEyhYlsWGhTEiUscA1VGAPSp5muJTLHmf2w6B6lCRvsm evAjFKNrwY+0imMMKKip24vrcQngZfxXK3h8LmsAyfHL2XST4p4qjFjoLmBt1NOOWglw X21IVDGpsA2iNKYIeGY5yDJIC/BQGOxuEo7v/GWcMPuBBo00jXlJz7GETIKCEJ+DVc2g Ym4x2KwEejCAWs0aaqFn3SO9oepzZLvmyeV+9wZM6kEnFURLUM1gCFkX2MOAdod535S2 7kr7H9/Fg4NK4vDStapGJPsHk36cvYnYYs1B0cGPcLezlLtT9DuY7JQWVAK6UMPMBQ9k gW3Q== X-Gm-Message-State: AOAM531vMtHWJy+DcOEgEoYOMC58yqV4GgfhBl6pyq199g1VzrHGbnuK 8j8Q6QeaXOQ/C4bcgJ41Z3k= X-Google-Smtp-Source: ABdhPJzu/n/90ZMYIIIC+HPqkL6/Yfs11EzuqyFA2HD6jwJYaddoxN/MnzFJq+qqv/SrP5bE+uGlzg== X-Received: by 2002:a0c:b38b:: with SMTP id t11mr8827143qve.58.1611719934559; Tue, 26 Jan 2021 19:58:54 -0800 (PST) Received: from hurd (dsl-10-132-34.b2b2c.ca. [72.10.132.34]) by smtp.gmail.com with ESMTPSA id e185sm405684qkb.127.2021.01.26.19.58.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Jan 2021 19:58:54 -0800 (PST) From: Maxim Cournoyer To: 45958-done@debbugs.gnu.org Subject: Re: bug#45958: [PATCH core-updates v3 1/2] utils: Retrieve the store prefix from NIX_STORE_DIR, not STORE_DIR. References: <20210118165122.25813-1-maxim.cournoyer@gmail.com> Date: Tue, 26 Jan 2021 22:58:53 -0500 In-Reply-To: <20210118165122.25813-1-maxim.cournoyer@gmail.com> (Maxim Cournoyer's message of "Mon, 18 Jan 2021 11:51:20 -0500") Message-ID: <87sg6nyqrm.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 45774-done Cc: 45774-done@debbugs.gnu.org, ludo@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello, Maxim Cournoyer writes: > On the daemon side, nixStore gets set to the environment variable > NIX_STORE_DIR, else the environment variable NIX_STORE else the compile time > macro NIX_STORE_DIR (see the Settings::processEnvironment method in > nix/libstore/globals.cc). Hence, it is more appropriate to lookup the > environment variable NIX_STORE_DIR than NIX_STORE in (guix build utils). > > * guix/build/utils.scm (%store-directory): Call getenv with NIX_STORE_DIR > instead of NIX_STORE. > --- > guix/build/utils.scm | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/guix/build/utils.scm b/guix/build/utils.scm > index 76180e67e0..2cbdb31505 100644 > --- a/guix/build/utils.scm > +++ b/guix/build/utils.scm > @@ -144,7 +144,7 @@ > > (define (%store-directory) > "Return the directory name of the store." > - (or (getenv "NIX_STORE") > + (or (getenv "NIX_STORE_DIR") > "/gnu/store")) > > (define (store-file-name? file) I found that both environment variables should be honored and pushed with a revised explanation in 47a6a938c3c4d0bbe7b6a3c64ff75d7bfb2f24fb. Closing, Maxim From debbugs-submit-bounces@debbugs.gnu.org Wed Jan 27 18:11:03 2021 Received: (at 45774-done) by debbugs.gnu.org; 27 Jan 2021 23:11:03 +0000 Received: from localhost ([127.0.0.1]:46427 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l4txn-0008QN-F5 for submit@debbugs.gnu.org; Wed, 27 Jan 2021 18:11:03 -0500 Received: from eggs.gnu.org ([209.51.188.92]:44092) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1l4txl-0008Pf-85; Wed, 27 Jan 2021 18:11:01 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]:57235) by eggs.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1l4txg-0004lJ-3H; Wed, 27 Jan 2021 18:10:56 -0500 Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=55796 helo=ribbon) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from ) id 1l4txf-0005Jx-Mo; Wed, 27 Jan 2021 18:10:55 -0500 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: Maxim Cournoyer Subject: Re: bug#45958: [PATCH core-updates v3 1/2] utils: Retrieve the store prefix from NIX_STORE_DIR, not STORE_DIR. References: <20210118165122.25813-1-maxim.cournoyer@gmail.com> <87sg6nyqrm.fsf@gmail.com> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 9 =?utf-8?Q?Pluvi=C3=B4se?= an 229 de la =?utf-8?Q?R?= =?utf-8?Q?=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-pc-linux-gnu Date: Thu, 28 Jan 2021 00:10:53 +0100 In-Reply-To: <87sg6nyqrm.fsf@gmail.com> (Maxim Cournoyer's message of "Tue, 26 Jan 2021 22:58:53 -0500") Message-ID: <874kj2t1qa.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 45774-done Cc: 45774-done@debbugs.gnu.org, 45958-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Maxim Cournoyer skribis: >> (define (%store-directory) >> "Return the directory name of the store." >> - (or (getenv "NIX_STORE") >> + (or (getenv "NIX_STORE_DIR") >> "/gnu/store")) >>=20=20 >> (define (store-file-name? file) > > I found that both environment variables should be honored and pushed > with a revised explanation in 47a6a938c3c4d0bbe7b6a3c64ff75d7bfb2f24fb. Sounds good to me, thanks! Ludo=E2=80=99. From unknown Wed Jun 18 23:17:11 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Thu, 25 Feb 2021 12:24:11 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator