From unknown Sat Jun 21 10:20:37 2025 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.509 (Entity 5.509) Content-Type: text/plain; charset=utf-8 From: bug#68405 <68405@debbugs.gnu.org> To: bug#68405 <68405@debbugs.gnu.org> Subject: Status: [PATCH v2] guix: download: Add support for git repositories. Reply-To: bug#68405 <68405@debbugs.gnu.org> Date: Sat, 21 Jun 2025 17:20:37 +0000 retitle 68405 [PATCH v2] guix: download: Add support for git repositories. reassign 68405 guix-patches submitter 68405 Romain GARBAGE severity 68405 normal tag 68405 patch thanks From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 12 10:15:41 2024 Received: (at submit) by debbugs.gnu.org; 12 Jan 2024 15:15:41 +0000 Received: from localhost ([127.0.0.1]:37228 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOJG0-0000OE-1i for submit@debbugs.gnu.org; Fri, 12 Jan 2024 10:15:41 -0500 Received: from lists.gnu.org ([2001:470:142::17]:51918) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOJFu-0008UB-5m for submit@debbugs.gnu.org; Fri, 12 Jan 2024 10:15:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rOJFp-0006zu-M4 for guix-patches@gnu.org; Fri, 12 Jan 2024 10:15:29 -0500 Received: from mail2-relais-roc.national.inria.fr ([192.134.164.83]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rOJFl-0001SK-Hd for guix-patches@gnu.org; Fri, 12 Jan 2024 10:15:29 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=6sc6ecIQQ2ow4Y3j6Yyp6bddX6EdIF/IbahhuOo8ACE=; b=oIWHTqz7IJrT2mwFZQJqR7AcGoyMZzaItJJBTdBLXyncIIMyzxYRyouB gG3OW7IEDgPzxzXf7S78CKs0bG5R+RIf5tRUvZ1mjl2tAsZp2wKfbNMlE LhjFKlFttoDgxYhupIltqGXpSof3JXhUWb1rSTOuV5OHZnrQEklsKDCVO M=; Authentication-Results: mail2-relais-roc.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; spf=None smtp.helo=postmaster@guix-A102.bordeaux.inria.fr Received-SPF: SoftFail (mail2-relais-roc.national.inria.fr: domain of romain.garbage@inria.fr is inclined to not designate 193.50.110.153 as permitted sender) identity=mailfrom; client-ip=193.50.110.153; receiver=mail2-relais-roc.national.inria.fr; envelope-from="romain.garbage@inria.fr"; x-sender="romain.garbage@inria.fr"; x-conformance=spf_only; x-record-type="v=spf1"; x-record-text="v=spf1 include:mailout.safebrands.com a:basic-mail.safebrands.com a:basic-mail01.safebrands.com a:basic-mail02.safebrands.com ip4:128.93.142.0/24 ip4:192.134.164.0/24 ip4:128.93.162.160 ip4:89.107.174.7 mx ~all" Received-SPF: None (mail2-relais-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@guix-A102.bordeaux.inria.fr) identity=helo; client-ip=193.50.110.153; receiver=mail2-relais-roc.national.inria.fr; envelope-from="romain.garbage@inria.fr"; x-sender="postmaster@guix-A102.bordeaux.inria.fr"; x-conformance=spf_only X-IronPort-AV: E=Sophos;i="6.04,189,1695679200"; d="scan'208";a="146367628" Received: from unknown (HELO guix-A102.bordeaux.inria.fr) ([193.50.110.153]) by mail2-relais-roc.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jan 2024 16:15:21 +0100 From: Romain GARBAGE To: guix-patches@gnu.org Subject: [PATCH v2] guix: download: Add support for git repositories. Date: Fri, 12 Jan 2024 16:12:19 +0100 Message-ID: <20240112151411.22470-2-romain.garbage@inria.fr> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=192.134.164.83; envelope-from=romain.garbage@inria.fr; helo=mail2-relais-roc.national.inria.fr X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: submit Cc: Romain GARBAGE X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Added `--recursive' option. Removed `pk' call. * guix/scripts/download.scm (git-download-to-store*): Add new variable. (copy-recursively-without-dot-git): New variable. (git-download-to-file): Add new variable. (show-help): Add 'git', 'commit', 'branch' and 'recursive'options help message. (%default-options): Add default value for 'git-reference' and 'recursive' options. (%options): Add 'git', 'commit', 'branch' and 'recursive' command line options. (guix-download) [hash]: Compute hash with 'file-hash*' instead of 'port-hash' from (gcrypt hash) module. This allows us to compute hashes for directories. * doc/guix.texi (Invoking guix-download): Add @item entries for `git', `commit', `branch' and `recursive' options. Add a paragraph in the introduction. * tests/guix-download.sh: New tests. --- doc/guix.texi | 23 ++++++ guix/scripts/download.scm | 146 ++++++++++++++++++++++++++++++++++---- tests/guix-download.sh | 42 +++++++++++ 3 files changed, 199 insertions(+), 12 deletions(-) diff --git a/doc/guix.texi b/doc/guix.texi index 3002cdfa13..d3b40e878b 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -13983,6 +13983,9 @@ the certificates of X.509 authorities from the directory pointed to by the @env{SSL_CERT_DIR} environment variable (@pxref{X.509 Certificates}), unless @option{--no-check-certificate} is used. +Alternatively, @command{guix download} can also retrieve a Git +repository, possibly a specific commit, tag, or branch. + The following options are available: @table @code @@ -14007,6 +14010,26 @@ URL, which makes you vulnerable to ``man-in-the-middle'' attacks. @itemx -o @var{file} Save the downloaded file to @var{file} instead of adding it to the store. + +@item --git +@itemx -g +Checkout the Git repository at the latest commit on the default branch. + +@item --commit=@var{commit-or-tag} +Checkout the Git repository at @var{commit-or-tag}. + +@var{commit-or-tag} can be either a tag or a commit defined in the Git +repository. + +@item --branch=@var{branch} +Checkout the Git repository at @var{branch}. + +The repository will be checked out at the latest commit of @var{branch}, +which must be a valid branch of the Git repository. + +@item --recursive +@itemx -r +Recursively clone the Git repository. @end table @node Invoking guix hash diff --git a/guix/scripts/download.scm b/guix/scripts/download.scm index 19052d5652..50c9a43791 100644 --- a/guix/scripts/download.scm +++ b/guix/scripts/download.scm @@ -22,17 +22,23 @@ (define-module (guix scripts download) #:use-module (guix scripts) #:use-module (guix store) #:use-module (gcrypt hash) + #:use-module (guix hash) #:use-module (guix base16) #:use-module (guix base32) #:autoload (guix base64) (base64-encode) #:use-module ((guix download) #:hide (url-fetch)) + #:use-module ((guix git) + #:select (latest-repository-commit + update-cached-checkout)) #:use-module ((guix build download) #:select (url-fetch)) + #:use-module (guix build utils) #:use-module ((guix progress) #:select (current-terminal-columns)) #:use-module ((guix build syscalls) #:select (terminal-columns)) #:use-module (web uri) + #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (srfi srfi-1) #:use-module (srfi srfi-26) @@ -54,6 +60,57 @@ (define (download-to-file url file) (url-fetch url file #:mirrors %mirrors))) file)) +;; This is a simplified version of 'copy-recursively'. +;; It allows us to filter out the ".git" subfolder. +;; TODO: Remove when 'copy-recursively' supports '#:select?'. +(define (copy-recursively-without-dot-git source destination) + (define strip-source + (let ((len (string-length source))) + (lambda (file) + (substring file len)))) + + (file-system-fold (lambda (file stat result) ; enter? + (not (string-suffix? "/.git" file))) + (lambda (file stat result) ; leaf + (let ((dest (string-append destination + (strip-source file)))) + (case (stat:type stat) + ((symlink) + (let ((target (readlink file))) + (symlink target dest))) + (else + (copy-file file dest))))) + (lambda (dir stat result) ; down + (let ((target (string-append destination + (strip-source dir)))) + (mkdir-p target))) + (const #t) ; up + (const #t) ; skip + (lambda (file stat errno result) + (format (current-error-port) "i/o error: ~a: ~a~%" + file (strerror errno)) + #f) + #t + source)) + +(define (git-download-to-file url file reference recursive?) + "Download the git repo at URL to file, checked out at REFERENCE. +REFERENCE must be a pair argument as understood by 'latest-repository-commit'. +Return FILE." + ;; TODO: Support recursive repos. + ;; 'libgit2' doesn't support the URL format generated by 'uri->string' so + ;; we have to do a little fixup. Dropping completely the 'file:' protocol + ;; part gives better performance. + (let ((url* (cond ((string-prefix? "file://" url) + (string-drop url (string-length "file://"))) + ((string-prefix? "file:" url) + (string-drop url (string-length "file:"))) + (else url)))) + (copy-recursively-without-dot-git + (update-cached-checkout url* #:ref reference #:recursive? recursive?) + file)) + file) + (define (ensure-valid-store-file-name name) "Replace any character not allowed in a store name by an underscore." @@ -67,17 +124,36 @@ (define valid name)) -(define* (download-to-store* url #:key (verify-certificate? #t)) +(define* (download-to-store* url #:key (verify-certificate? #t) #:allow-other-keys) (with-store store (download-to-store store url (ensure-valid-store-file-name (basename url)) #:verify-certificate? verify-certificate?))) +(define* (git-download-to-store* url reference recursive? #:key (verify-certificate? #t)) + "Download the git repository at URL to the store, checked out at REFERENCE. +URL must specify a protocol (i.e https:// or file://), REFERENCE must be a +pair argument as understood by 'latest-repository-commit'." + ;; Ensure the URL string is properly formatted when using the 'file' protocol: + ;; URL is generated using 'uri->string', which returns "file:/path/to/file" instead of + ;; "file:///path/to/file", which in turn makes 'git-download-to-store' fail. + (let* ((file? (string-prefix? "file:" url)) + (url* (if (and file? + (not (string-prefix? "file:///" url))) + (string-append "file://" (string-replace url "" 0 (string-length "file:"))) + url))) + (with-store store + ;; TODO: Support recursive repos. + ;; TODO: Verify certificate support and deactivation. + (latest-repository-commit store url* #:recursive? recursive? #:ref reference)))) + (define %default-options ;; Alist of default option values. `((format . ,bytevector->nix-base32-string) (hash-algorithm . ,(hash-algorithm sha256)) (verify-certificate? . #t) + (git-reference . #f) + (recursive? . #f) (download-proc . ,download-to-store*))) (define (show-help) @@ -97,6 +173,19 @@ (define (show-help) do not validate the certificate of HTTPS servers ")) (format #t (G_ " -o, --output=FILE download to FILE")) + (format #t (G_ " + -g, --git download the default branch's latest commit of the + git repository at URL")) + (format #t (G_ " + --commit=COMMIT_OR_TAG + download the given commit or tag of the git + repository at URL")) + (format #t (G_ " + --branch=BRANCH download the given branch of the git repository + at URL")) + (format #t (G_ " + -r, --recursive download a git repository recursively")) + (newline) (display (G_ " -h, --help display this help and exit")) @@ -105,6 +194,13 @@ (define (show-help) (newline) (show-bug-report-information)) +(define (add-git-download-option result) + (alist-cons 'download-proc + ;; XXX: #:verify-certificate? currently ignored. + (lambda* (url #:key verify-certificate? ref recursive?) + (git-download-to-store* url ref recursive?)) + (alist-delete 'download result))) + (define %options ;; Specifications of the command-line options. (list (option '(#\f "format") #t #f @@ -136,11 +232,36 @@ (define fmt-proc (alist-cons 'verify-certificate? #f result))) (option '(#\o "output") #t #f (lambda (opt name arg result) - (alist-cons 'download-proc - (lambda* (url #:key verify-certificate?) - (download-to-file url arg)) - (alist-delete 'download result)))) - + (let* ((git + (assoc-ref result 'git-reference))) + (if git + (alist-cons 'download-proc + (lambda* (url #:key verify-certificate? ref recursive?) + (git-download-to-file url arg (assoc-ref result 'git-reference) recursive?)) + (alist-delete 'download result)) + (alist-cons 'download-proc + (lambda* (url #:key verify-certificate? #:allow-other-keys) + (download-to-file url arg)) + (alist-delete 'download result)))))) + (option '(#\g "git") #f #f + (lambda (opt name arg result) + ;; Ignore this option if 'commit' or 'branch' has + ;; already been provided + (if (assoc-ref result 'git-reference) + result + (alist-cons 'git-reference '() + (add-git-download-option result))))) + (option '("commit") #t #f + (lambda (opt name arg result) + (alist-cons 'git-reference `(tag-or-commit . ,arg) + (add-git-download-option result)))) + (option '("branch") #t #f + (lambda (opt name arg result) + (alist-cons 'git-reference `(branch . ,arg) + (alist-delete 'git-reference result)))) + (option '(#\r "recursive") #f #f + (lambda (opt name arg result) + (alist-cons 'recursive? #t result))) (option '(#\h "help") #f #f (lambda args (leave-on-EPIPE (show-help)) @@ -183,12 +304,13 @@ (define (parse-options) (terminal-columns))) (fetch (uri->string uri) #:verify-certificate? - (assq-ref opts 'verify-certificate?)))) - (hash (call-with-input-file - (or path - (leave (G_ "~a: download failed~%") - arg)) - (cute port-hash (assoc-ref opts 'hash-algorithm) <>))) + (assq-ref opts 'verify-certificate?) + #:ref (assq-ref opts 'git-reference) + #:recursive? (assq-ref opts 'recursive?)))) + (hash (let* ((path* (or path + (leave (G_ "~a: download failed~%") + arg)))) + (file-hash* path* #:algorithm (assoc-ref opts 'hash-algorithm)))) (fmt (assq-ref opts 'format))) (format #t "~a~%~a~%" path (fmt hash)) #t))) diff --git a/tests/guix-download.sh b/tests/guix-download.sh index f4cb335eef..3bf63c4b12 100644 --- a/tests/guix-download.sh +++ b/tests/guix-download.sh @@ -45,4 +45,46 @@ cmp "$output" "$abs_top_srcdir/README" # This one should fail. guix download "file:///does-not-exist" "file://$abs_top_srcdir/README" && false +# Test git support with local repository +test_directory="$(mktemp -d)" +trap 'chmod -Rf +w "$test_directory"; rm -rf "$test_directory" ; rm -f "$output"' EXIT + +# Create a dummy git repo in the temporary directory +( + cd $test_directory + git init + touch test + git config user.name "User" + git config user.email "user@domain" + git add test + git commit -m "Commit" + git tag -a -m "v1" v1 +) + +# Extract commit number +commit=$((cd $test_directory && git log) | head -n 1 | cut -f2 -d' ') + +# We expect that guix hash is working properly or at least that the output of +# 'guix download' is consistent with 'guix hash' +expected_hash=$(guix hash -rx $test_directory) + +# Test the different options +for option in "" "--commit=$commit" "--commit=v1" "--branch=master" +do + command_output="$(guix download --git $option "file://$test_directory")" + computed_hash="$(echo $command_output | cut -f2 -d' ')" + store_path="$(echo $command_output | cut -f1 -d' ')" + [ "$expected_hash" = "$computed_hash" ] + diff -r -x ".git" $test_directory $store_path +done + +# Should fail +guix download --git --branch=non_existent "file://$test_directory" && false + +# Same but download to file instead of store +tmpdir="t-archive-dir-$$" +trap 'chmod -Rf +w "$test_directory"; rm -rf "$test_directory" ; rm -f "$output" ; rm -rf "$tmpdir"' EXIT +guix download --git "file://$test_directory" -o $tmpdir +diff -r -x ".git" $test_directory $tmpdir + exit 0 -- 2.41.0 From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 12 10:55:57 2024 Received: (at control) by debbugs.gnu.org; 12 Jan 2024 15:55:58 +0000 Received: from localhost ([127.0.0.1]:37287 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOJsz-00087H-On for submit@debbugs.gnu.org; Fri, 12 Jan 2024 10:55:57 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:60884) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOJsx-00086x-W8 for control@debbugs.gnu.org; Fri, 12 Jan 2024 10:55:56 -0500 Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rOJsu-0002HO-LB for control@debbugs.gnu.org; Fri, 12 Jan 2024 10:55:52 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=MIME-version:Subject:From:To:Date:in-reply-to: references; bh=JxF7jVJA+eJ62XSN41SrTzutb3yRy9hhh+5gNfTIr30=; b=aIleIjmB0nFMIe ELcZKL1GTTizpZXUru6O3eD1zErwvGFXjD4jpcVTB1uXxDTk4UDg/J0Jl46FSLc4hF1+qJGtCX7iN rrLpGAcqETV2IHo9ZE8fxbWys3dhUSVlQi9lVGZXvVSHQaBjVk/wsuHAiZ+I2TinJXaHRC4HgWBD6 0AIynhTanhhmhtdLkidvXKfpLYxzLxZ8kBIdoTC6wncb75AdOBpV/R3Uyw9P38vT1+QcFjXojtnVs EDlfs3eR4LqBail+wyeZjgl9La/pBKaRs35KVrppuVXegKvYebb5jREO5zgmui68V3lyVSNgV+ALR lYk3GbeNJQGTdKqWpRtQ==; Date: Fri, 12 Jan 2024 16:55:47 +0100 Message-Id: <87zfxaim0s.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #68404 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) merge 68404 68405 quit From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 12 10:58:06 2024 Received: (at 68405) by debbugs.gnu.org; 12 Jan 2024 15:58:06 +0000 Received: from localhost ([127.0.0.1]:37303 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOJv4-0008Bl-CP for submit@debbugs.gnu.org; Fri, 12 Jan 2024 10:58:06 -0500 Received: from mail2-relais-roc.national.inria.fr ([192.134.164.83]:64110) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rOJv0-0008B5-Ir for 68405@debbugs.gnu.org; Fri, 12 Jan 2024 10:58:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:in-reply-to:references:date: message-id:mime-version:content-transfer-encoding; bh=S0OYBiDazU6MuOtfYiWP80siFfsYW3FFKXtmu3YvBBE=; b=CAbPe8I72geRlgmDFHfkvKIHFOP1suklDgLDYMh6tICxxw//RWn+HrhW mPp8MyCsVa+N0IbaOatMvdLASfbzwjF2SN56r9mCJWshBNNb2LXk+oV07 1R1bSwhBtm/nGDaWHgoeHgUCRSnkC1N7L5Tw46qifyMyngbK3jGJAiqYg w=; Authentication-Results: mail2-relais-roc.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=ludovic.courtes@inria.fr; dmarc=fail (p=none dis=none) d=inria.fr X-IronPort-AV: E=Sophos;i="6.04,190,1695679200"; d="scan'208";a="146377462" Received: from unknown (HELO ribbon) ([193.50.110.52]) by mail2-relais-roc.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jan 2024 16:57:58 +0100 From: =?utf-8?Q?Ludovic_Court=C3=A8s?= To: 68405@debbugs.gnu.org Subject: Re: bug#68405: [PATCH v2] guix: download: Add support for git repositories. In-Reply-To: <20240112151411.22470-2-romain.garbage@inria.fr> (Romain GARBAGE's message of "Fri, 12 Jan 2024 16:12:19 +0100") References: <20240112151411.22470-2-romain.garbage@inria.fr> Date: Fri, 12 Jan 2024 16:57:57 +0100 Message-ID: <87sf32ilx6.fsf@gnu.org> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 68405 Cc: Romain GARBAGE X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello! Romain GARBAGE skribis: > * guix/scripts/download.scm (git-download-to-store*): Add new variable. > (copy-recursively-without-dot-git): New variable. > (git-download-to-file): Add new variable. > (show-help): Add 'git', 'commit', 'branch' and 'recursive'options > help message. > (%default-options): Add default value for 'git-reference' and > 'recursive' options. > (%options): Add 'git', 'commit', 'branch' and 'recursive' command > line options. > (guix-download) [hash]: Compute hash with 'file-hash*' instead of > 'port-hash' from (gcrypt hash) module. This allows us to compute > hashes for directories. > * doc/guix.texi (Invoking guix-download): Add @item entries for > `git', `commit', `branch' and `recursive' options. Add a paragraph in > the introduction. > * tests/guix-download.sh: New tests. Full disclosure: Romain and I work together at Inria. I=E2=80=99ve reviewe= d the changes and they LGTM, but we=E2=80=99ll leave time for others to chime in. Ludo=E2=80=99. From debbugs-submit-bounces@debbugs.gnu.org Thu Jan 18 23:16:57 2024 Received: (at 68405) by debbugs.gnu.org; 19 Jan 2024 04:16:57 +0000 Received: from localhost ([127.0.0.1]:57154 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rQgJL-0000vr-Nh for submit@debbugs.gnu.org; Thu, 18 Jan 2024 23:16:57 -0500 Received: from mail-qk1-x72f.google.com ([2607:f8b0:4864:20::72f]:55679) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rQgJI-0000pO-RX for 68405@debbugs.gnu.org; Thu, 18 Jan 2024 23:16:53 -0500 Received: by mail-qk1-x72f.google.com with SMTP id af79cd13be357-7833b6bb41bso31898885a.3 for <68405@debbugs.gnu.org>; Thu, 18 Jan 2024 20:16:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705637805; x=1706242605; darn=debbugs.gnu.org; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=ZzkFu4aDGH2Yp3208PqbfKpfSdSdXdu+FvJexNYdQKo=; b=YBeDaq6jm8UDKVK8Vb8QVCwFFWKwi5E5VV8D/gXBztdbrLCFkMdfVBk0i3K4CcFIOm g8pxFpRKzN3myAtM5ZGvi0lO9eja0XlM5go8H/VfJv8PcrufvZIiwVJsUBbmi8H2/wBq 8st2rxg+DTVSuiG8yZmGaN/h/05yNTx3EUnSfiH2Ur/7IF8uEEf3u4g/LmEaZ09aHa7g YqmHzPYgBplhDPaP5/TUY0rN2f/plQEv6HnlC/ziO4YNRZAtnlInySvRRhIzH2jBH3e2 Ev6/3X4ZHqpHr3BJCdZFGPajVtCx3R5nO0Tq7ZiRb/jZD3NbmltgcJ1toW0a6liartiG HQHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705637805; x=1706242605; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ZzkFu4aDGH2Yp3208PqbfKpfSdSdXdu+FvJexNYdQKo=; b=ef8xcMYe9c8oBZ0lxCMaIWf9NAe97A0tnvRbj+hX7LCgCd/VC6PXjI2+SyuanrifRV Iq/CRDbquKNpZ2G8GSSvnPEn4npOFxBXRSpHICWDbyG/rpZxuLc6hkStyYZJSz4xeP4U YQWd3CrcDsDw3YYta2iV4twkJ0v4V8HHj8P8M0X4PjF1C9RC0lDWv5Pj9M84ixyP8YGf MIm1G0zSXCHaaINqtJknQYZbVuU6c1CDGveDzlLso5ZdOXNPSCUYa4k6yuNQZGvc8pr1 NeCmM/JrbYDss4a3cFgz5KbZxRho4PmCYsM7uAfRHvvGv0kWqzZKv7t1NsYP/w3hP9B8 E7aQ== X-Gm-Message-State: AOJu0YyEj6O/CwBy2bN+PYqF5Yx1wssZ7I6t7NwU/0tFjIGnma6Wfdmk 6uhlSyzd+5KBUxoYv6A1C5+fYvkNYTlXcBj363C/d1kyyAIeMI+6 X-Google-Smtp-Source: AGHT+IEP9aFnEYvqy9M+VPqoJigRowByPgu+CrKSlpdGapMLjbdS0EwqB/juxkYNUZopFUOPAafXfQ== X-Received: by 2002:a05:620a:1a17:b0:783:7fd1:dc90 with SMTP id bk23-20020a05620a1a1700b007837fd1dc90mr742314qkb.59.1705637804981; Thu, 18 Jan 2024 20:16:44 -0800 (PST) Received: from hurd (dsl-10-132-160.b2b2c.ca. [72.10.132.160]) by smtp.gmail.com with ESMTPSA id ha11-20020a05622a2b0b00b003f6ac526568sm7352889qtb.39.2024.01.18.20.16.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Jan 2024 20:16:44 -0800 (PST) From: Maxim Cournoyer To: Romain GARBAGE Subject: Re: bug#68405: [PATCH v2] guix: download: Add support for git repositories. In-Reply-To: <20240112151411.22470-2-romain.garbage@inria.fr> (Romain GARBAGE's message of "Fri, 12 Jan 2024 16:12:19 +0100") References: <20240112151411.22470-2-romain.garbage@inria.fr> Date: Thu, 18 Jan 2024 23:16:42 -0500 Message-ID: <87jzo6j6tx.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 68405 Cc: 68405@debbugs.gnu.org, =?utf-8?Q?Ludovic_Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello, Romain GARBAGE writes: > Added `--recursive' option. I still see a TODO about supporting recursive repos in the code. Is that still the case? > Removed `pk' call. > > * guix/scripts/download.scm (git-download-to-store*): Add new variable. > (copy-recursively-without-dot-git): New variable. > (git-download-to-file): Add new variable. > (show-help): Add 'git', 'commit', 'branch' and 'recursive'options > help message. > (%default-options): Add default value for 'git-reference' and > 'recursive' options. > (%options): Add 'git', 'commit', 'branch' and 'recursive' command > line options. > (guix-download) [hash]: Compute hash with 'file-hash*' instead of > 'port-hash' from (gcrypt hash) module. This allows us to compute > hashes for directories. > * doc/guix.texi (Invoking guix-download): Add @item entries for > `git', `commit', `branch' and `recursive' options. Add a paragraph in > the introduction. > * tests/guix-download.sh: New tests. This sounds good and is something that I'm many many of us have wanted for some time. Thank you for working on it! Nitpick about the commit message: the convention seems to be to not use a hanging indent when writing GNU ChangeLog messages. > --- > doc/guix.texi | 23 ++++++ > guix/scripts/download.scm | 146 ++++++++++++++++++++++++++++++++++---- > tests/guix-download.sh | 42 +++++++++++ > 3 files changed, 199 insertions(+), 12 deletions(-) > > diff --git a/doc/guix.texi b/doc/guix.texi > index 3002cdfa13..d3b40e878b 100644 > --- a/doc/guix.texi > +++ b/doc/guix.texi > @@ -13983,6 +13983,9 @@ the certificates of X.509 authorities from the directory pointed to by > the @env{SSL_CERT_DIR} environment variable (@pxref{X.509 > Certificates}), unless @option{--no-check-certificate} is used. > > +Alternatively, @command{guix download} can also retrieve a Git > +repository, possibly a specific commit, tag, or branch. > + > The following options are available: > > @table @code > @@ -14007,6 +14010,26 @@ URL, which makes you vulnerable to ``man-in-the-middle'' attacks. > @itemx -o @var{file} > Save the downloaded file to @var{file} instead of adding it to the > store. > + > +@item --git > +@itemx -g > +Checkout the Git repository at the latest commit on the default branch. > + > +@item --commit=@var{commit-or-tag} > +Checkout the Git repository at @var{commit-or-tag}. > + > +@var{commit-or-tag} can be either a tag or a commit defined in the Git > +repository. > + > +@item --branch=@var{branch} > +Checkout the Git repository at @var{branch}. > + > +The repository will be checked out at the latest commit of @var{branch}, > +which must be a valid branch of the Git repository. > + > +@item --recursive > +@itemx -r > +Recursively clone the Git repository. > @end table > > @node Invoking guix hash > diff --git a/guix/scripts/download.scm b/guix/scripts/download.scm > index 19052d5652..50c9a43791 100644 > --- a/guix/scripts/download.scm > +++ b/guix/scripts/download.scm > @@ -22,17 +22,23 @@ (define-module (guix scripts download) > #:use-module (guix scripts) > #:use-module (guix store) > #:use-module (gcrypt hash) > + #:use-module (guix hash) > #:use-module (guix base16) > #:use-module (guix base32) > #:autoload (guix base64) (base64-encode) > #:use-module ((guix download) #:hide (url-fetch)) > + #:use-module ((guix git) > + #:select (latest-repository-commit > + update-cached-checkout)) > #:use-module ((guix build download) > #:select (url-fetch)) > + #:use-module (guix build utils) > #:use-module ((guix progress) > #:select (current-terminal-columns)) > #:use-module ((guix build syscalls) > #:select (terminal-columns)) > #:use-module (web uri) > + #:use-module (ice-9 ftw) > #:use-module (ice-9 match) > #:use-module (srfi srfi-1) > #:use-module (srfi srfi-26) > @@ -54,6 +60,57 @@ (define (download-to-file url file) > (url-fetch url file #:mirrors %mirrors))) > file)) > > +;; This is a simplified version of 'copy-recursively'. > +;; It allows us to filter out the ".git" subfolder. > +;; TODO: Remove when 'copy-recursively' supports '#:select?'. Is a #:select? planned for copy-recursively? (in the works?) > +(define (copy-recursively-without-dot-git source destination) > + (define strip-source > + (let ((len (string-length source))) > + (lambda (file) > + (substring file len)))) > + > + (file-system-fold (lambda (file stat result) ; enter? > + (not (string-suffix? "/.git" file))) > + (lambda (file stat result) ; leaf > + (let ((dest (string-append destination > + (strip-source file)))) > + (case (stat:type stat) > + ((symlink) > + (let ((target (readlink file))) > + (symlink target dest))) > + (else > + (copy-file file dest))))) > + (lambda (dir stat result) ; down > + (let ((target (string-append destination > + (strip-source dir)))) > + (mkdir-p target))) > + (const #t) ; up > + (const #t) ; skip > + (lambda (file stat errno result) > + (format (current-error-port) "i/o error: ~a: ~a~%" > + file (strerror errno)) > + #f) > + #t > + source)) > + > +(define (git-download-to-file url file reference recursive?) > + "Download the git repo at URL to file, checked out at REFERENCE. > +REFERENCE must be a pair argument as understood by 'latest-repository-commit'. > +Return FILE." > + ;; TODO: Support recursive repos. > + ;; 'libgit2' doesn't support the URL format generated by 'uri->string' so > + ;; we have to do a little fixup. Dropping completely the 'file:' protocol > + ;; part gives better performance. > > + (let ((url* (cond ((string-prefix? "file://" url) > + (string-drop url (string-length "file://"))) > + ((string-prefix? "file:" url) > + (string-drop url (string-length "file:"))) > + (else url)))) > + (copy-recursively-without-dot-git > + (update-cached-checkout url* #:ref reference #:recursive? recursive?) > + file)) > + file) > + > (define (ensure-valid-store-file-name name) > "Replace any character not allowed in a store name by an underscore." > > @@ -67,17 +124,36 @@ (define valid > name)) > > > -(define* (download-to-store* url #:key (verify-certificate? #t)) > +(define* (download-to-store* url #:key (verify-certificate? #t) #:allow-other-keys) > (with-store store > (download-to-store store url > (ensure-valid-store-file-name (basename url)) > #:verify-certificate? verify-certificate?))) > > +(define* (git-download-to-store* url reference recursive? #:key (verify-certificate? #t)) > + "Download the git repository at URL to the store, checked out at REFERENCE. > +URL must specify a protocol (i.e https:// or file://), REFERENCE must be a > +pair argument as understood by 'latest-repository-commit'." > + ;; Ensure the URL string is properly formatted when using the 'file' protocol: > + ;; URL is generated using 'uri->string', which returns "file:/path/to/file" instead of > + ;; "file:///path/to/file", which in turn makes 'git-download-to-store' fail. > + (let* ((file? (string-prefix? "file:" url)) > + (url* (if (and file? > + (not (string-prefix? "file:///" url))) > + (string-append "file://" (string-replace url "" 0 (string-length "file:"))) > + url))) > + (with-store store > + ;; TODO: Support recursive repos. > + ;; TODO: Verify certificate support and deactivation. > + (latest-repository-commit store url* #:recursive? recursive? #:ref reference)))) > + Some lines look like > 80 chars here. Please break long lines accordingly. > (define %default-options > ;; Alist of default option values. > `((format . ,bytevector->nix-base32-string) > (hash-algorithm . ,(hash-algorithm sha256)) > (verify-certificate? . #t) > + (git-reference . #f) > + (recursive? . #f) > (download-proc . ,download-to-store*))) > > (define (show-help) > @@ -97,6 +173,19 @@ (define (show-help) > do not validate the certificate of HTTPS servers ")) > (format #t (G_ " > -o, --output=FILE download to FILE")) > + (format #t (G_ " > + -g, --git download the default branch's latest commit of the > + git repository at URL")) > + (format #t (G_ " > + --commit=COMMIT_OR_TAG > + download the given commit or tag of the git > + repository at URL")) > + (format #t (G_ " > + --branch=BRANCH download the given branch of the git repository > + at URL")) > + (format #t (G_ " > + -r, --recursive download a git repository recursively")) > + > (newline) > (display (G_ " > -h, --help display this help and exit")) > @@ -105,6 +194,13 @@ (define (show-help) > (newline) > (show-bug-report-information)) > > +(define (add-git-download-option result) > + (alist-cons 'download-proc > + ;; XXX: #:verify-certificate? currently ignored. > + (lambda* (url #:key verify-certificate? ref recursive?) > + (git-download-to-store* url ref recursive?)) > + (alist-delete 'download result))) > + > (define %options > ;; Specifications of the command-line options. > (list (option '(#\f "format") #t #f > @@ -136,11 +232,36 @@ (define fmt-proc > (alist-cons 'verify-certificate? #f result))) > (option '(#\o "output") #t #f > (lambda (opt name arg result) > - (alist-cons 'download-proc > - (lambda* (url #:key verify-certificate?) > - (download-to-file url arg)) > - (alist-delete 'download result)))) > - > + (let* ((git > + (assoc-ref result 'git-reference))) > + (if git > + (alist-cons 'download-proc > + (lambda* (url #:key verify-certificate? ref recursive?) > + (git-download-to-file url arg (assoc-ref result 'git-reference) recursive?)) > + (alist-delete 'download result)) > + (alist-cons 'download-proc > + (lambda* (url #:key verify-certificate? #:allow-other-keys) > + (download-to-file url arg)) > + (alist-delete 'download result)))))) > + (option '(#\g "git") #f #f > + (lambda (opt name arg result) > + ;; Ignore this option if 'commit' or 'branch' has > + ;; already been provided > + (if (assoc-ref result 'git-reference) > + result > + (alist-cons 'git-reference '() > + (add-git-download-option result))))) > + (option '("commit") #t #f > + (lambda (opt name arg result) > + (alist-cons 'git-reference `(tag-or-commit . ,arg) > + (add-git-download-option result)))) > + (option '("branch") #t #f > + (lambda (opt name arg result) > + (alist-cons 'git-reference `(branch . ,arg) > + (alist-delete 'git-reference result)))) > + (option '(#\r "recursive") #f #f > + (lambda (opt name arg result) > + (alist-cons 'recursive? #t result))) > (option '(#\h "help") #f #f > (lambda args > (leave-on-EPIPE (show-help)) > @@ -183,12 +304,13 @@ (define (parse-options) > (terminal-columns))) > (fetch (uri->string uri) > #:verify-certificate? > - (assq-ref opts 'verify-certificate?)))) > - (hash (call-with-input-file > - (or path > - (leave (G_ "~a: download failed~%") > - arg)) > - (cute port-hash (assoc-ref opts 'hash-algorithm) <>))) > + (assq-ref opts 'verify-certificate?) > + #:ref (assq-ref opts 'git-reference) > + #:recursive? (assq-ref opts 'recursive?)))) > + (hash (let* ((path* (or path > + (leave (G_ "~a: download failed~%") > + arg)))) > + (file-hash* path* #:algorithm (assoc-ref opts 'hash-algorithm)))) Here also there are some too long lines in the above hunks; please break long lines so they fit within the 80 characters limit. > (fmt (assq-ref opts 'format))) > (format #t "~a~%~a~%" path (fmt hash)) > #t))) > diff --git a/tests/guix-download.sh b/tests/guix-download.sh > index f4cb335eef..3bf63c4b12 100644 > --- a/tests/guix-download.sh > +++ b/tests/guix-download.sh > @@ -45,4 +45,46 @@ cmp "$output" "$abs_top_srcdir/README" > # This one should fail. > guix download "file:///does-not-exist" "file://$abs_top_srcdir/README" && false > > +# Test git support with local repository Nitpick: please punctuate standalone comments (here, a missing period). > +test_directory="$(mktemp -d)" > +trap 'chmod -Rf +w "$test_directory"; rm -rf "$test_directory" ; rm -f "$output"' EXIT the 'chmod' doesn't seem to be useful; since we force removing with -f ? And where did the $output variable come from? > + > +# Create a dummy git repo in the temporary directory > +( > + cd $test_directory > + git init > + touch test > + git config user.name "User" > + git config user.email "user@domain" > + git add test > + git commit -m "Commit" > + git tag -a -m "v1" v1 > +) > + > +# Extract commit number > +commit=$((cd $test_directory && git log) | head -n 1 | cut -f2 -d' ') > + > +# We expect that guix hash is working properly or at least that the output of > +# 'guix download' is consistent with 'guix hash' > +expected_hash=$(guix hash -rx $test_directory) > + > +# Test the different options > +for option in "" "--commit=$commit" "--commit=v1" "--branch=master" > +do > + command_output="$(guix download --git $option "file://$test_directory")" > + computed_hash="$(echo $command_output | cut -f2 -d' ')" > + store_path="$(echo $command_output | cut -f1 -d' ')" > + [ "$expected_hash" = "$computed_hash" ] > + diff -r -x ".git" $test_directory $store_path > +done > + > +# Should fail > +guix download --git --branch=non_existent "file://$test_directory" && false > + > +# Same but download to file instead of store > +tmpdir="t-archive-dir-$$" > +trap 'chmod -Rf +w "$test_directory"; rm -rf "$test_directory" ; rm -f "$output" ; rm -rf "$tmpdir"' EXIT It'd look nicer if there was a single global trap call at the top of these tests. Don't forget to punctuate your comments :-). Otherwise, it looks good to me, although I haven't tried it. -- Thanks, Maxim From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 19 03:53:35 2024 Received: (at 68405) by debbugs.gnu.org; 19 Jan 2024 08:53:35 +0000 Received: from localhost ([127.0.0.1]:57465 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rQkd5-0006xG-4N for submit@debbugs.gnu.org; Fri, 19 Jan 2024 03:53:35 -0500 Received: from mail2-relais-roc.national.inria.fr ([192.134.164.83]:20383) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rQkd1-0006x1-R6 for 68405@debbugs.gnu.org; Fri, 19 Jan 2024 03:53:33 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=date:from:to:cc:message-id:in-reply-to:references: subject:mime-version:content-transfer-encoding; bh=ndD69BxGdMZZgIOho0ujkhjqD1uTuQ4lUCbNLxbSdNg=; b=RmJOb5gfGz8+0JwWrfyTXi9TbZgpyz9lqgEDsvFV5WxYI2gi0pe9A6eb YOO7Us9c/tPUvJgfeT4lmelEacWneMQ93VWE8A4Wu27YTNfKcfH1X0ukK zK2mNeDR+YmghPQfbhJicq7rSKKP9eKtEe5U+EC5izcE3zFKp/2m+FwPd Y=; Authentication-Results: mail2-relais-roc.national.inria.fr; dkim=none (message not signed) header.i=none; spf=Pass smtp.mailfrom=romain.garbage@inria.fr; spf=None smtp.helo=postmaster@zcs-store5.inria.fr Received-SPF: Pass (mail2-relais-roc.national.inria.fr: domain of romain.garbage@inria.fr designates 128.93.142.32 as permitted sender) identity=mailfrom; client-ip=128.93.142.32; receiver=mail2-relais-roc.national.inria.fr; envelope-from="romain.garbage@inria.fr"; x-sender="romain.garbage@inria.fr"; x-conformance=spf_only; x-record-type="v=spf1"; x-record-text="v=spf1 include:mailout.safebrands.com a:basic-mail.safebrands.com a:basic-mail01.safebrands.com a:basic-mail02.safebrands.com ip4:128.93.142.0/24 ip4:192.134.164.0/24 ip4:128.93.162.160 ip4:89.107.174.7 mx ~all" Received-SPF: None (mail2-relais-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@zcs-store5.inria.fr) identity=helo; client-ip=128.93.142.32; receiver=mail2-relais-roc.national.inria.fr; envelope-from="romain.garbage@inria.fr"; x-sender="postmaster@zcs-store5.inria.fr"; x-conformance=spf_only X-IronPort-AV: E=Sophos;i="6.05,204,1701126000"; d="scan'208";a="147528175" X-MGA-submission: =?us-ascii?q?MDGteFqhghlwCbLmUtkHW6QUieLXm5j2j4hXx5?= =?us-ascii?q?Gbb94vOmh9hezeIAH+/xGgMT67HA/ATboEQrXXIxts7yZEI2GK8y5e8T?= =?us-ascii?q?m8BbMFjOqucqrBBZQTw81i745X3UoeWGEzmvmf2VqVX7b0qAaAFlePLA?= =?us-ascii?q?U0ESfnGYt2TKLgVauAgm39tA=3D=3D?= Received: from zcs-store5.inria.fr ([128.93.142.32]) by mail2-relais-roc.national.inria.fr with ESMTP; 19 Jan 2024 09:53:23 +0100 Date: Fri, 19 Jan 2024 09:53:23 +0100 (CET) From: Romain Garbage To: Maxim Cournoyer Message-ID: <1558292173.14259694.1705654403237.JavaMail.zimbra@inria.fr> In-Reply-To: <87jzo6j6tx.fsf@gmail.com> References: <20240112151411.22470-2-romain.garbage@inria.fr> <87jzo6j6tx.fsf@gmail.com> Subject: Re: bug#68405: [PATCH v2] guix: download: Add support for git repositories. MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [193.50.110.153] X-Mailer: Zimbra 8.8.15_GA_4581 (ZimbraWebClient - FF115 (Linux)/8.8.15_GA_4581) Thread-Topic: bug#68405: [PATCH v2] guix: download: Add support for git repositories. Thread-Index: edmRLhNKdSNTvmR1+gEErvpsUs1oLg== X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 68405 Cc: 68405@debbugs.gnu.org, Ludovic =?iso-8859-1?Q?Court=E8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) ----- Mail original ----- > De: "Maxim Cournoyer" > =C0: "Romain Garbage" > Cc: 68405@debbugs.gnu.org, "Ludovic Court=E8s" > Envoy=E9: Vendredi 19 Janvier 2024 04:16:42 > Objet: Re: bug#68405: [PATCH v2] guix: download: Add support for git repo= sitories. > Hello, Hi Maxim, Thank you very much for your review.=20 I actually pushed a v3 of this patch last Tuesday, somehow the issues have = not been merged together. The new patch is available here: https://issues.guix.gnu.org/68499 I will address your comments below. > Romain GARBAGE writes: >=20 >> Added `--recursive' option. >=20 > I still see a TODO about supporting recursive repos in the code. Is > that still the case? It was removed in v3. =20 >> Removed `pk' call. >> >> * guix/scripts/download.scm (git-download-to-store*): Add new variable. >> (copy-recursively-without-dot-git): New variable. >> (git-download-to-file): Add new variable. >> (show-help): Add 'git', 'commit', 'branch' and 'recursive'options >> help message. >> (%default-options): Add default value for 'git-reference' and >> 'recursive' options. >> (%options): Add 'git', 'commit', 'branch' and 'recursive' command >> line options. >> (guix-download) [hash]: Compute hash with 'file-hash*' instead of >> 'port-hash' from (gcrypt hash) module. This allows us to compute >> hashes for directories. >> * doc/guix.texi (Invoking guix-download): Add @item entries for >> `git', `commit', `branch' and `recursive' options. Add a paragraph in >> the introduction. >> * tests/guix-download.sh: New tests. >=20 > This sounds good and is something that I'm many many of us have wanted > for some time. Thank you for working on it! >=20 > Nitpick about the commit message: the convention seems to be to not use > a hanging indent when writing GNU ChangeLog messages. I'll remove it then :) =20 [...] >> +;; This is a simplified version of 'copy-recursively'. >> +;; It allows us to filter out the ".git" subfolder. >> +;; TODO: Remove when 'copy-recursively' supports '#:select?'. >=20 > Is a #:select? planned for copy-recursively? (in the works?) For the record, it is the issue #68406 (thanks for reviewing it too!) [...] =20 > Some lines look like > 80 chars here. Please break long lines > accordingly. Will fix. [...] > Here also there are some too long lines in the above hunks; please break > long lines so they fit within the 80 characters limit. Ditto. =20 >> (fmt (assq-ref opts 'format))) >> (format #t "~a~%~a~%" path (fmt hash)) >> #t))) >> diff --git a/tests/guix-download.sh b/tests/guix-download.sh >> index f4cb335eef..3bf63c4b12 100644 >> --- a/tests/guix-download.sh >> +++ b/tests/guix-download.sh >> @@ -45,4 +45,46 @@ cmp "$output" "$abs_top_srcdir/README" >> # This one should fail. >> guix download "file:///does-not-exist" "file://$abs_top_srcdir/README" = && false >> >> +# Test git support with local repository >=20 > Nitpick: please punctuate standalone comments (here, a missing period). Will do. >> +test_directory=3D"$(mktemp -d)" >> +trap 'chmod -Rf +w "$test_directory"; rm -rf "$test_directory" ; rm -f >> "$output"' EXIT >=20 > the 'chmod' doesn't seem to be useful; since we force removing with -f ? I copied it from another test :) I will remove it. > And where did the $output variable come from? It comes from L39 and is used in an already existing test. [...] >> +# Same but download to file instead of store >> +tmpdir=3D"t-archive-dir-$$" >> +trap 'chmod -Rf +w "$test_directory"; rm -rf "$test_directory" ; rm -f >> "$output" ; rm -rf "$tmpdir"' EXIT >=20 > It'd look nicer if there was a single global trap call at the top of > these tests. Don't forget to punctuate your comments :-). Ok, so I'll move all the temporary file/directory creation/definition to th= e top together with the trap call definition. I'll submit a v4 with these changes. Thanks again for reviewing. --=20 Romain From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 19 04:54:09 2024 Received: (at control) by debbugs.gnu.org; 19 Jan 2024 09:54:10 +0000 Received: from localhost ([127.0.0.1]:57546 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rQlZh-0002RC-MK for submit@debbugs.gnu.org; Fri, 19 Jan 2024 04:54:09 -0500 Received: from mail2-relais-roc.national.inria.fr ([192.134.164.83]:58562) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rQlZd-0002Qe-HG for control@debbugs.gnu.org; Fri, 19 Jan 2024 04:54:08 -0500 Authentication-Results: mail2-relais-roc.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=ludo@gnu.org; dmarc=fail (p=none dis=none) d=gnu.org X-IronPort-AV: E=Sophos;i="6.05,204,1701126000"; d="scan'208";a="147544855" Received: from unknown (HELO ribbon) ([193.50.110.52]) by mail2-relais-roc.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2024 10:53:57 +0100 Date: Fri, 19 Jan 2024 10:53:56 +0100 Message-Id: <87h6j9bqdn.fsf@gnu.org> To: control@debbugs.gnu.org From: =?utf-8?Q?Ludovic_Court=C3=A8s?= Subject: control message for bug #68405 MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) merge 68405 68499 quit From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 19 05:24:50 2024 Received: (at 68405) by debbugs.gnu.org; 19 Jan 2024 10:24:50 +0000 Received: from localhost ([127.0.0.1]:57594 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rQm3N-0000AR-D5 for submit@debbugs.gnu.org; Fri, 19 Jan 2024 05:24:50 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:38230) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rQm3J-0000A1-Ng for 68405@debbugs.gnu.org; Fri, 19 Jan 2024 05:24:48 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=P08SjKfGyb58HbVtAouxVKmqR4LifKGuV97BDZFYlXk=; b=uVRnPOemAdA+adQibKhyvs8PqiRHB4cK6/uCgIO/VHGriznU0MPAEYbc cJGMgPijjiSpVubq7A01Wv0HYeU5RUc066LB9LUp2ppK4jJAaV+8Z3zLL 6nOUel3CeDog57arG93Jrk0f3Nwfx2ajQc5tVhz4dH3+rD+OJzyEz7cIb g=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; spf=None smtp.helo=postmaster@guix-A102.bordeaux.inria.fr Received-SPF: SoftFail (mail3-relais-sop.national.inria.fr: domain of romain.garbage@inria.fr is inclined to not designate 193.50.111.147 as permitted sender) identity=mailfrom; client-ip=193.50.111.147; receiver=mail3-relais-sop.national.inria.fr; envelope-from="romain.garbage@inria.fr"; x-sender="romain.garbage@inria.fr"; x-conformance=spf_only; x-record-type="v=spf1"; x-record-text="v=spf1 include:mailout.safebrands.com a:basic-mail.safebrands.com a:basic-mail01.safebrands.com a:basic-mail02.safebrands.com ip4:128.93.142.0/24 ip4:192.134.164.0/24 ip4:128.93.162.160 ip4:89.107.174.7 mx ~all" Received-SPF: None (mail3-relais-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@guix-A102.bordeaux.inria.fr) identity=helo; client-ip=193.50.111.147; receiver=mail3-relais-sop.national.inria.fr; envelope-from="romain.garbage@inria.fr"; x-sender="postmaster@guix-A102.bordeaux.inria.fr"; x-conformance=spf_only X-IronPort-AV: E=Sophos;i="6.05,204,1701126000"; d="scan'208";a="77262144" Received: from reserved-service-dhcp-pool-bso.bordeaux.inria.fr (HELO guix-A102.bordeaux.inria.fr) ([193.50.111.147]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jan 2024 11:24:37 +0100 From: Romain GARBAGE To: 68405@debbugs.gnu.org Subject: [PATCH v4] guix: download: Add support for git repositories. Date: Fri, 19 Jan 2024 11:19:23 +0100 Message-ID: <20240119102417.17155-1-romain.garbage@inria.fr> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 68405 Cc: Romain GARBAGE , maxim.cournoyer@gmail.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * guix/scripts/download.scm (git-download-to-store*): Add new variable. (copy-recursively-without-dot-git): New variable. (git-download-to-file): Add new variable. (show-help): Add 'git', 'commit', 'branch' and 'recursive'options help message. (%default-options): Add default value for 'git-reference' and 'recursive' options. (%options): Add 'git', 'commit', 'branch' and 'recursive' command line options. (guix-download) [hash]: Compute hash with 'file-hash*' instead of 'port-hash' from (gcrypt hash) module. This allows us to compute hashes for directories. * doc/guix.texi (Invoking guix-download): Add @item entries for `git', `commit', `branch' and `recursive' options. Add a paragraph in the introduction. * tests/guix-download.sh: New tests. Move variables and trap definition to the top of the file. Change-Id: Ic2c428dca4cfcb0d4714ed361a4c46609339140a --- doc/guix.texi | 23 ++++++ guix/scripts/download.scm | 154 +++++++++++++++++++++++++++++++++++--- tests/guix-download.sh | 45 ++++++++++- 3 files changed, 208 insertions(+), 14 deletions(-) Changes from v3->v4 * Capitalized `git' in help message * Replaced underscore with dash in help message * Fixed url shadowing * Wrapped long lines * Added missing punctuation * Moved variables and trap call definition to the top of the test file * Renamed some variables in the test file to be more descriptive * Removed unnecessary call to `chmod' in the test file diff --git a/doc/guix.texi b/doc/guix.texi index a66005ee9d..6e5f801a1e 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -14020,6 +14020,9 @@ the certificates of X.509 authorities from the directory pointed to by the @env{SSL_CERT_DIR} environment variable (@pxref{X.509 Certificates}), unless @option{--no-check-certificate} is used. +Alternatively, @command{guix download} can also retrieve a Git +repository, possibly a specific commit, tag, or branch. + The following options are available: @table @code @@ -14044,6 +14047,26 @@ URL, which makes you vulnerable to ``man-in-the-middle'' attacks. @itemx -o @var{file} Save the downloaded file to @var{file} instead of adding it to the store. + +@item --git +@itemx -g +Checkout the Git repository at the latest commit on the default branch. + +@item --commit=@var{commit-or-tag} +Checkout the Git repository at @var{commit-or-tag}. + +@var{commit-or-tag} can be either a tag or a commit defined in the Git +repository. + +@item --branch=@var{branch} +Checkout the Git repository at @var{branch}. + +The repository will be checked out at the latest commit of @var{branch}, +which must be a valid branch of the Git repository. + +@item --recursive +@itemx -r +Recursively clone the Git repository. @end table @node Invoking guix hash diff --git a/guix/scripts/download.scm b/guix/scripts/download.scm index 19052d5652..ce2ed68248 100644 --- a/guix/scripts/download.scm +++ b/guix/scripts/download.scm @@ -22,17 +22,24 @@ (define-module (guix scripts download) #:use-module (guix scripts) #:use-module (guix store) #:use-module (gcrypt hash) + #:use-module (guix hash) #:use-module (guix base16) #:use-module (guix base32) #:autoload (guix base64) (base64-encode) #:use-module ((guix download) #:hide (url-fetch)) + #:use-module ((guix git) + #:select (latest-repository-commit + update-cached-checkout + with-git-error-handling)) #:use-module ((guix build download) #:select (url-fetch)) + #:use-module (guix build utils) #:use-module ((guix progress) #:select (current-terminal-columns)) #:use-module ((guix build syscalls) #:select (terminal-columns)) #:use-module (web uri) + #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (srfi srfi-1) #:use-module (srfi srfi-26) @@ -54,6 +61,57 @@ (define (download-to-file url file) (url-fetch url file #:mirrors %mirrors))) file)) +;; This is a simplified version of 'copy-recursively'. +;; It allows us to filter out the ".git" subfolder. +;; TODO: Remove when 'copy-recursively' supports '#:select?'. +(define (copy-recursively-without-dot-git source destination) + (define strip-source + (let ((len (string-length source))) + (lambda (file) + (substring file len)))) + + (file-system-fold (lambda (file stat result) ; enter? + (not (string-suffix? "/.git" file))) + (lambda (file stat result) ; leaf + (let ((dest (string-append destination + (strip-source file)))) + (case (stat:type stat) + ((symlink) + (let ((target (readlink file))) + (symlink target dest))) + (else + (copy-file file dest))))) + (lambda (dir stat result) ; down + (let ((target (string-append destination + (strip-source dir)))) + (mkdir-p target))) + (const #t) ; up + (const #t) ; skip + (lambda (file stat errno result) + (format (current-error-port) "i/o error: ~a: ~a~%" + file (strerror errno)) + #f) + #t + source)) + +(define (git-download-to-file url file reference recursive?) + "Download the git repo at URL to file, checked out at REFERENCE. +REFERENCE must be a pair argument as understood by 'latest-repository-commit'. +Return FILE." + ;; 'libgit2' doesn't support the URL format generated by 'uri->string' so + ;; we have to do a little fixup. Dropping completely the 'file:' protocol + ;; part gives better performance. + (let ((url (cond ((string-prefix? "file://" url) + (string-drop url (string-length "file://"))) + ((string-prefix? "file:" url) + (string-drop url (string-length "file:"))) + (else url)))) + (copy-recursively-without-dot-git + (with-git-error-handling + (update-cached-checkout url #:ref reference #:recursive? recursive?)) + file)) + file) + (define (ensure-valid-store-file-name name) "Replace any character not allowed in a store name by an underscore." @@ -67,17 +125,42 @@ (define valid name)) -(define* (download-to-store* url #:key (verify-certificate? #t)) +(define* (download-to-store* url + #:key (verify-certificate? #t) + #:allow-other-keys) (with-store store (download-to-store store url (ensure-valid-store-file-name (basename url)) #:verify-certificate? verify-certificate?))) +(define* (git-download-to-store* url + reference + recursive? + #:key (verify-certificate? #t)) + "Download the git repository at URL to the store, checked out at REFERENCE. +URL must specify a protocol (i.e https:// or file://), REFERENCE must be a +pair argument as understood by 'latest-repository-commit'." + ;; Ensure the URL string is properly formatted when using the 'file' + ;; protocol: URL is generated using 'uri->string', which returns + ;; "file:/path/to/file" instead of "file:///path/to/file", which in turn + ;; makes 'git-download-to-store' fail. + (let* ((file? (string-prefix? "file:" url)) + (url (if (and file? + (not (string-prefix? "file:///" url))) + (string-append "file://" (string-drop url (string-length "file:"))) + url))) + (with-store store + ;; TODO: Verify certificate support and deactivation. + (with-git-error-handling + (latest-repository-commit store url #:recursive? recursive? #:ref reference))))) + (define %default-options ;; Alist of default option values. `((format . ,bytevector->nix-base32-string) (hash-algorithm . ,(hash-algorithm sha256)) (verify-certificate? . #t) + (git-reference . #f) + (recursive? . #f) (download-proc . ,download-to-store*))) (define (show-help) @@ -97,6 +180,19 @@ (define (show-help) do not validate the certificate of HTTPS servers ")) (format #t (G_ " -o, --output=FILE download to FILE")) + (format #t (G_ " + -g, --git download the default branch's latest commit of the + Git repository at URL")) + (format #t (G_ " + --commit=COMMIT-OR-TAG + download the given commit or tag of the Git + repository at URL")) + (format #t (G_ " + --branch=BRANCH download the given branch of the Git repository + at URL")) + (format #t (G_ " + -r, --recursive download a Git repository recursively")) + (newline) (display (G_ " -h, --help display this help and exit")) @@ -105,6 +201,13 @@ (define (show-help) (newline) (show-bug-report-information)) +(define (add-git-download-option result) + (alist-cons 'download-proc + ;; XXX: #:verify-certificate? currently ignored. + (lambda* (url #:key verify-certificate? ref recursive?) + (git-download-to-store* url ref recursive?)) + (alist-delete 'download result))) + (define %options ;; Specifications of the command-line options. (list (option '(#\f "format") #t #f @@ -136,11 +239,36 @@ (define fmt-proc (alist-cons 'verify-certificate? #f result))) (option '(#\o "output") #t #f (lambda (opt name arg result) - (alist-cons 'download-proc - (lambda* (url #:key verify-certificate?) - (download-to-file url arg)) - (alist-delete 'download result)))) - + (let* ((git + (assoc-ref result 'git-reference))) + (if git + (alist-cons 'download-proc + (lambda* (url #:key verify-certificate? ref recursive?) + (git-download-to-file url arg (assoc-ref result 'git-reference) recursive?)) + (alist-delete 'download result)) + (alist-cons 'download-proc + (lambda* (url #:key verify-certificate? #:allow-other-keys) + (download-to-file url arg)) + (alist-delete 'download result)))))) + (option '(#\g "git") #f #f + (lambda (opt name arg result) + ;; Ignore this option if 'commit' or 'branch' has + ;; already been provided + (if (assoc-ref result 'git-reference) + result + (alist-cons 'git-reference '() + (add-git-download-option result))))) + (option '("commit") #t #f + (lambda (opt name arg result) + (alist-cons 'git-reference `(tag-or-commit . ,arg) + (add-git-download-option result)))) + (option '("branch") #t #f + (lambda (opt name arg result) + (alist-cons 'git-reference `(branch . ,arg) + (add-git-download-option result)))) + (option '(#\r "recursive") #f #f + (lambda (opt name arg result) + (alist-cons 'recursive? #t result))) (option '(#\h "help") #f #f (lambda args (leave-on-EPIPE (show-help)) @@ -183,12 +311,14 @@ (define (parse-options) (terminal-columns))) (fetch (uri->string uri) #:verify-certificate? - (assq-ref opts 'verify-certificate?)))) - (hash (call-with-input-file - (or path - (leave (G_ "~a: download failed~%") - arg)) - (cute port-hash (assoc-ref opts 'hash-algorithm) <>))) + (assq-ref opts 'verify-certificate?) + #:ref (assq-ref opts 'git-reference) + #:recursive? (assq-ref opts 'recursive?)))) + (hash (let* ((path* (or path + (leave (G_ "~a: download failed~%") + arg)))) + (file-hash* path* + #:algorithm (assoc-ref opts 'hash-algorithm)))) (fmt (assq-ref opts 'format))) (format #t "~a~%~a~%" path (fmt hash)) #t))) diff --git a/tests/guix-download.sh b/tests/guix-download.sh index f4cb335eef..d4cd2ea6b9 100644 --- a/tests/guix-download.sh +++ b/tests/guix-download.sh @@ -16,6 +16,12 @@ # You should have received a copy of the GNU General Public License # along with GNU Guix. If not, see . +# Define some files/folders needed for the tests. +output="t-download-$$" +test_git_repo="$(mktemp -d)" +output_dir="t-archive-dir-$$" +trap 'rm -rf "$test_git_repo" ; rm -f "$output" ; rm -rf "$output_dir"' EXIT + # # Test the `guix download' command-line utility. # @@ -36,8 +42,6 @@ guix download "file://$abs_top_srcdir/README" guix download "$abs_top_srcdir/README" # This one too, even if it cannot talk to the daemon. -output="t-download-$$" -trap 'rm -f "$output"' EXIT GUIX_DAEMON_SOCKET="/nowhere" guix download -o "$output" \ "file://$abs_top_srcdir/README" cmp "$output" "$abs_top_srcdir/README" @@ -45,4 +49,41 @@ cmp "$output" "$abs_top_srcdir/README" # This one should fail. guix download "file:///does-not-exist" "file://$abs_top_srcdir/README" && false +# Test git support with local repository. +# First, create a dummy git repo in the temporary directory. +( + cd $test_git_repo + git init + touch test + git config user.name "User" + git config user.email "user@domain" + git add test + git commit -m "Commit" + git tag -a -m "v1" v1 +) + +# Extract commit number. +commit=$((cd $test_git_repo && git log) | head -n 1 | cut -f2 -d' ') + +# We expect that guix hash is working properly or at least that the output of +# 'guix download' is consistent with 'guix hash'. +expected_hash=$(guix hash -rx $test_git_repo) + +# Test the different options +for option in "" "--commit=$commit" "--commit=v1" "--branch=master" +do + command_output="$(guix download --git $option "file://$test_git_repo")" + computed_hash="$(echo $command_output | cut -f2 -d' ')" + store_path="$(echo $command_output | cut -f1 -d' ')" + [ "$expected_hash" = "$computed_hash" ] + diff -r -x ".git" $test_git_repo $store_path +done + +# Should fail. +guix download --git --branch=non_existent "file://$test_git_repo" && false + +# Same but download to file instead of store. +guix download --git "file://$test_git_repo" -o $output_dir +diff -r -x ".git" $test_git_repo $output_dir + exit 0 -- 2.41.0 From debbugs-submit-bounces@debbugs.gnu.org Fri Jan 19 22:23:14 2024 Received: (at 68405) by debbugs.gnu.org; 20 Jan 2024 03:23:14 +0000 Received: from localhost ([127.0.0.1]:60602 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rR1ww-0000Gy-1b for submit@debbugs.gnu.org; Fri, 19 Jan 2024 22:23:14 -0500 Received: from mail-qk1-x72f.google.com ([2607:f8b0:4864:20::72f]:53682) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rR1wt-0000Gj-I8 for 68405@debbugs.gnu.org; Fri, 19 Jan 2024 22:23:12 -0500 Received: by mail-qk1-x72f.google.com with SMTP id af79cd13be357-783293278adso104149985a.3 for <68405@debbugs.gnu.org>; Fri, 19 Jan 2024 19:23:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1705720983; x=1706325783; darn=debbugs.gnu.org; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=YV5bQkU6I643bcNzV5lhfOvZ7wMtT3B8YpiUOiDaTuQ=; b=Uh8lITUxnhQ1o7v7+Zpufa6UnSj1Yi4AUg2RpdZB6heGq2jn1fF6CDVpR1lTX7CPua EgwBMhLMdxK1k0WMYqm/YOLWsOYIIPYOjW2BwtU5cE7/Md0OcP2R3uB51WUTkHYb7BCt vr0YT/EndRhEN5iWqZroHfCxlQ4nq1WVwlgRuBEVUQQfuUa3wqxHaQHLNbYOZI6WgtVa tjNYLcrWIxcUOXn3Rw7aCUPJUanUZ6v5VqrlzVUPXRLwqLPDPmzSWHYhrNoMbtGU2WB+ 4bw371NoMPujtvLP6Vjx0Wl6qz6uRIUsN83Zhp2jC9BKv0Rs9OUEPaqqBkYZsNILYXfV 0/VQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705720983; x=1706325783; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=YV5bQkU6I643bcNzV5lhfOvZ7wMtT3B8YpiUOiDaTuQ=; b=G9DUggzYUGeQt/YTPo+ZGhhDBBb7EYbgtkXWw2K33YFklhg1LToTKdQNprfUPpfhyX xTNCL0zq6n1rqySU55TzkJqdShNpox1dArVXPjcq9dFNHworO4VdvSluYsVtoak8K0iH mskZwgc4nkHqh6vTu5MoQshxZnzvdh35cCa9UTWwYGl/pstTh+1lKtziWuN7j6/++cZG A5iExH5Tmc5c/6Ml1fI6AKdJGNftsyEhwa8H+ksp1Q67MYu34HFPpsRmQ6mPisuRge2Z fzE4VeLRSWYLCgwQkMrsYE91uThmjq/ScjZ4pFkYTcDp92ApDIRt7EXkkJ1OrmcUyaas h70A== X-Gm-Message-State: AOJu0Yy6dreF1SZ7nt8sQ04n8hFPetDUV/CzVP+ZRhmDn0dYicxYZIUb Iy9fRfNxnGOAgCgwt7/Y4Dgp9CKM4HprWEI6EV5TeBiPKTWu7ibw X-Google-Smtp-Source: AGHT+IHSJdmc+tRpnlUfMnyEqL0tLs6fN14nlCbI2w5ayxzc7S6QEyQP1p8fXyC9pAe6Sm2WnL8WTg== X-Received: by 2002:ae9:e918:0:b0:783:54b4:7763 with SMTP id x24-20020ae9e918000000b0078354b47763mr983026qkf.87.1705720983220; Fri, 19 Jan 2024 19:23:03 -0800 (PST) Received: from hurd (dsl-10-132-160.b2b2c.ca. [72.10.132.160]) by smtp.gmail.com with ESMTPSA id vy4-20020a05620a490400b00781df19c062sm251944qkn.59.2024.01.19.19.23.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Jan 2024 19:23:02 -0800 (PST) From: Maxim Cournoyer To: Romain GARBAGE Subject: Re: [PATCH v4] guix: download: Add support for git repositories. In-Reply-To: <20240119102417.17155-1-romain.garbage@inria.fr> (Romain GARBAGE's message of "Fri, 19 Jan 2024 11:19:23 +0100") References: <20240119102417.17155-1-romain.garbage@inria.fr> Date: Fri, 19 Jan 2024 22:23:01 -0500 Message-ID: <87v87ohene.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 68405 Cc: 68405@debbugs.gnu.org, =?utf-8?Q?Ludovic_Court=C3=A8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hello! Romain GARBAGE writes: > * guix/scripts/download.scm (git-download-to-store*): Add new variable. > (copy-recursively-without-dot-git): New variable. > (git-download-to-file): Add new variable. > (show-help): Add 'git', 'commit', 'branch' and 'recursive'options > help message. > (%default-options): Add default value for 'git-reference' and > 'recursive' options. > (%options): Add 'git', 'commit', 'branch' and 'recursive' command > line options. > (guix-download) [hash]: Compute hash with 'file-hash*' instead of > 'port-hash' from (gcrypt hash) module. This allows us to compute > hashes for directories. > * doc/guix.texi (Invoking guix-download): Add @item entries for > `git', `commit', `branch' and `recursive' options. Add a paragraph in > the introduction. > * tests/guix-download.sh: New tests. Move variables and trap definition > to the top of the file. > > Change-Id: Ic2c428dca4cfcb0d4714ed361a4c46609339140a [...] > +(define* (git-download-to-store* url > + reference > + recursive? > + #:key (verify-certificate? #t)) > + "Download the git repository at URL to the store, checked out at REFERENCE. > +URL must specify a protocol (i.e https:// or file://), REFERENCE must be a > +pair argument as understood by 'latest-repository-commit'." > + ;; Ensure the URL string is properly formatted when using the 'file' > + ;; protocol: URL is generated using 'uri->string', which returns > + ;; "file:/path/to/file" instead of "file:///path/to/file", which in turn > + ;; makes 'git-download-to-store' fail. > + (let* ((file? (string-prefix? "file:" url)) > + (url (if (and file? > + (not (string-prefix? "file:///" url))) > + (string-append "file://" (string-drop url (string-length "file:"))) > + url))) > + (with-store store > + ;; TODO: Verify certificate support and deactivation. > + (with-git-error-handling > + (latest-repository-commit store url #:recursive? recursive? #:ref reference))))) The above contains too long lines still :-). > + > (define %default-options > ;; Alist of default option values. > `((format . ,bytevector->nix-base32-string) > (hash-algorithm . ,(hash-algorithm sha256)) > (verify-certificate? . #t) > + (git-reference . #f) > + (recursive? . #f) > (download-proc . ,download-to-store*))) > > (define (show-help) > @@ -97,6 +180,19 @@ (define (show-help) > do not validate the certificate of HTTPS servers ")) > (format #t (G_ " > -o, --output=FILE download to FILE")) > + (format #t (G_ " > + -g, --git download the default branch's latest commit of the > + Git repository at URL")) > + (format #t (G_ " > + --commit=COMMIT-OR-TAG > + download the given commit or tag of the Git > + repository at URL")) > + (format #t (G_ " > + --branch=BRANCH download the given branch of the Git repository > + at URL")) > + (format #t (G_ " > + -r, --recursive download a Git repository recursively")) > + > (newline) > (display (G_ " > -h, --help display this help and exit")) > @@ -105,6 +201,13 @@ (define (show-help) > (newline) > (show-bug-report-information)) > > +(define (add-git-download-option result) > + (alist-cons 'download-proc > + ;; XXX: #:verify-certificate? currently ignored. > + (lambda* (url #:key verify-certificate? ref recursive?) > + (git-download-to-store* url ref recursive?)) > + (alist-delete 'download result))) > + > (define %options > ;; Specifications of the command-line options. > (list (option '(#\f "format") #t #f > @@ -136,11 +239,36 @@ (define fmt-proc > (alist-cons 'verify-certificate? #f result))) > (option '(#\o "output") #t #f > (lambda (opt name arg result) > - (alist-cons 'download-proc > - (lambda* (url #:key verify-certificate?) > - (download-to-file url arg)) > - (alist-delete 'download result)))) > - > + (let* ((git > + (assoc-ref result 'git-reference))) > + (if git > + (alist-cons 'download-proc > + (lambda* (url #:key verify-certificate? ref recursive?) > + (git-download-to-file url arg (assoc-ref result 'git-reference) recursive?)) > + (alist-delete 'download result)) > + (alist-cons 'download-proc > + (lambda* (url #:key verify-certificate? #:allow-other-keys) > + (download-to-file url arg)) > + (alist-delete 'download result)))))) Here as well. Otherwise, I've tested it with: --8<---------------cut here---------------start------------->8--- ./pre-inst-env guix download -gr https://git.jami.net/savoirfairelinux/jami-client-qt -o /tmp/jami --8<---------------cut here---------------end--------------->8--- and it worked as advertised; very nice! Some idea for the future: the --recurse option could take an optional argument that'd be a comma-separated list of submodules to fetch, e.g. --8<---------------cut here---------------start------------->8--- ./pre-inst-env guix download --git \ --recurse=daemon,3rdparty/SortFilterProxyModel \ https://git.jami.net/savoirfairelinux/jami-client-qt -o /tmp/jami --8<---------------cut here---------------end--------------->8--- and it'd recurse *only* the listed submodules. This would be useful as some projects contain submodules for windows or other platforms we do not care about and they may be very large (heavy) to download. The same idea could be implemented for our git-reference, where recursive? could accept a git submodule names list. But back to the current scope: Reviewed-by: Maxim Cournoyer -- Thanks, Maxim From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 22 05:33:44 2024 Received: (at 68405) by debbugs.gnu.org; 22 Jan 2024 10:33:44 +0000 Received: from localhost ([127.0.0.1]:39946 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRrcd-0000rV-Qe for submit@debbugs.gnu.org; Mon, 22 Jan 2024 05:33:44 -0500 Received: from mail3-relais-sop.national.inria.fr ([192.134.164.104]:9209) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRrca-0000rE-EE for 68405@debbugs.gnu.org; Mon, 22 Jan 2024 05:33:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=XyxxLkiH64B4zTKzPHdn5jwN6kNVnwpiXSHGWMbkzy8=; b=lB6h+sDQOfrsSMTtaRwlJtAB3+ujldtM++0RlTaRMHYT7wdfR2kKhMOH ePmewwXgjLIYV6PK1bjlRRoQc1AyLultR9aDdD01OO7ZVnKItes5h66sR g85sliHpG1hTavsTeQ1wFManOCxk+qalIDKZr34/T2aTlnOwnfw8KfRpl o=; Authentication-Results: mail3-relais-sop.national.inria.fr; dkim=none (message not signed) header.i=none; spf=SoftFail smtp.mailfrom=romain.garbage@inria.fr; spf=None smtp.helo=postmaster@guix-A102.bordeaux.inria.fr Received-SPF: SoftFail (mail3-relais-sop.national.inria.fr: domain of romain.garbage@inria.fr is inclined to not designate 193.50.110.153 as permitted sender) identity=mailfrom; client-ip=193.50.110.153; receiver=mail3-relais-sop.national.inria.fr; envelope-from="romain.garbage@inria.fr"; x-sender="romain.garbage@inria.fr"; x-conformance=spf_only; x-record-type="v=spf1"; x-record-text="v=spf1 include:mailout.safebrands.com a:basic-mail.safebrands.com a:basic-mail01.safebrands.com a:basic-mail02.safebrands.com ip4:128.93.142.0/24 ip4:192.134.164.0/24 ip4:128.93.162.160 ip4:89.107.174.7 mx ~all" Received-SPF: None (mail3-relais-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@guix-A102.bordeaux.inria.fr) identity=helo; client-ip=193.50.110.153; receiver=mail3-relais-sop.national.inria.fr; envelope-from="romain.garbage@inria.fr"; x-sender="postmaster@guix-A102.bordeaux.inria.fr"; x-conformance=spf_only X-IronPort-AV: E=Sophos;i="6.05,211,1701126000"; d="scan'208";a="77449155" Received: from unknown (HELO guix-A102.bordeaux.inria.fr) ([193.50.110.153]) by mail3-relais-sop.national.inria.fr with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Jan 2024 11:33:28 +0100 From: Romain GARBAGE To: 68405@debbugs.gnu.org Subject: [PATCH v5] guix: download: Add support for git repositories. Date: Mon, 22 Jan 2024 11:32:55 +0100 Message-ID: <20240122103319.8125-1-romain.garbage@inria.fr> X-Mailer: git-send-email 2.41.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 68405 Cc: Romain GARBAGE , maxim.cournoyer@gmail.com X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) * guix/scripts/download.scm (git-download-to-store*): Add new variable. (copy-recursively-without-dot-git): New variable. (git-download-to-file): Add new variable. (show-help): Add 'git', 'commit', 'branch' and 'recursive'options help message. (%default-options): Add default value for 'git-reference' and 'recursive' options. (%options): Add 'git', 'commit', 'branch' and 'recursive' command line options. (guix-download) [hash]: Compute hash with 'file-hash*' instead of 'port-hash' from (gcrypt hash) module. This allows us to compute hashes for directories. * doc/guix.texi (Invoking guix-download): Add @item entries for `git', `commit', `branch' and `recursive' options. Add a paragraph in the introduction. * tests/guix-download.sh: New tests. Move variables and trap definition to the top of the file. Change-Id: Ic2c428dca4cfcb0d4714ed361a4c46609339140a --- doc/guix.texi | 23 ++++++ guix/scripts/download.scm | 167 +++++++++++++++++++++++++++++++++++--- tests/guix-download.sh | 45 +++++++++- 3 files changed, 222 insertions(+), 13 deletions(-) Changes v4->v5 * Wrapped missed long lines diff --git a/doc/guix.texi b/doc/guix.texi index a66005ee9d..6e5f801a1e 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -14020,6 +14020,9 @@ the certificates of X.509 authorities from the directory pointed to by the @env{SSL_CERT_DIR} environment variable (@pxref{X.509 Certificates}), unless @option{--no-check-certificate} is used. +Alternatively, @command{guix download} can also retrieve a Git +repository, possibly a specific commit, tag, or branch. + The following options are available: @table @code @@ -14044,6 +14047,26 @@ URL, which makes you vulnerable to ``man-in-the-middle'' attacks. @itemx -o @var{file} Save the downloaded file to @var{file} instead of adding it to the store. + +@item --git +@itemx -g +Checkout the Git repository at the latest commit on the default branch. + +@item --commit=@var{commit-or-tag} +Checkout the Git repository at @var{commit-or-tag}. + +@var{commit-or-tag} can be either a tag or a commit defined in the Git +repository. + +@item --branch=@var{branch} +Checkout the Git repository at @var{branch}. + +The repository will be checked out at the latest commit of @var{branch}, +which must be a valid branch of the Git repository. + +@item --recursive +@itemx -r +Recursively clone the Git repository. @end table @node Invoking guix hash diff --git a/guix/scripts/download.scm b/guix/scripts/download.scm index 19052d5652..de68e6f328 100644 --- a/guix/scripts/download.scm +++ b/guix/scripts/download.scm @@ -22,17 +22,24 @@ (define-module (guix scripts download) #:use-module (guix scripts) #:use-module (guix store) #:use-module (gcrypt hash) + #:use-module (guix hash) #:use-module (guix base16) #:use-module (guix base32) #:autoload (guix base64) (base64-encode) #:use-module ((guix download) #:hide (url-fetch)) + #:use-module ((guix git) + #:select (latest-repository-commit + update-cached-checkout + with-git-error-handling)) #:use-module ((guix build download) #:select (url-fetch)) + #:use-module (guix build utils) #:use-module ((guix progress) #:select (current-terminal-columns)) #:use-module ((guix build syscalls) #:select (terminal-columns)) #:use-module (web uri) + #:use-module (ice-9 ftw) #:use-module (ice-9 match) #:use-module (srfi srfi-1) #:use-module (srfi srfi-26) @@ -54,6 +61,57 @@ (define (download-to-file url file) (url-fetch url file #:mirrors %mirrors))) file)) +;; This is a simplified version of 'copy-recursively'. +;; It allows us to filter out the ".git" subfolder. +;; TODO: Remove when 'copy-recursively' supports '#:select?'. +(define (copy-recursively-without-dot-git source destination) + (define strip-source + (let ((len (string-length source))) + (lambda (file) + (substring file len)))) + + (file-system-fold (lambda (file stat result) ; enter? + (not (string-suffix? "/.git" file))) + (lambda (file stat result) ; leaf + (let ((dest (string-append destination + (strip-source file)))) + (case (stat:type stat) + ((symlink) + (let ((target (readlink file))) + (symlink target dest))) + (else + (copy-file file dest))))) + (lambda (dir stat result) ; down + (let ((target (string-append destination + (strip-source dir)))) + (mkdir-p target))) + (const #t) ; up + (const #t) ; skip + (lambda (file stat errno result) + (format (current-error-port) "i/o error: ~a: ~a~%" + file (strerror errno)) + #f) + #t + source)) + +(define (git-download-to-file url file reference recursive?) + "Download the git repo at URL to file, checked out at REFERENCE. +REFERENCE must be a pair argument as understood by 'latest-repository-commit'. +Return FILE." + ;; 'libgit2' doesn't support the URL format generated by 'uri->string' so + ;; we have to do a little fixup. Dropping completely the 'file:' protocol + ;; part gives better performance. + (let ((url (cond ((string-prefix? "file://" url) + (string-drop url (string-length "file://"))) + ((string-prefix? "file:" url) + (string-drop url (string-length "file:"))) + (else url)))) + (copy-recursively-without-dot-git + (with-git-error-handling + (update-cached-checkout url #:ref reference #:recursive? recursive?)) + file)) + file) + (define (ensure-valid-store-file-name name) "Replace any character not allowed in a store name by an underscore." @@ -67,17 +125,46 @@ (define valid name)) -(define* (download-to-store* url #:key (verify-certificate? #t)) +(define* (download-to-store* url + #:key (verify-certificate? #t) + #:allow-other-keys) (with-store store (download-to-store store url (ensure-valid-store-file-name (basename url)) #:verify-certificate? verify-certificate?))) +(define* (git-download-to-store* url + reference + recursive? + #:key (verify-certificate? #t)) + "Download the git repository at URL to the store, checked out at REFERENCE. +URL must specify a protocol (i.e https:// or file://), REFERENCE must be a +pair argument as understood by 'latest-repository-commit'." + ;; Ensure the URL string is properly formatted when using the 'file' + ;; protocol: URL is generated using 'uri->string', which returns + ;; "file:/path/to/file" instead of "file:///path/to/file", which in turn + ;; makes 'git-download-to-store' fail. + (let* ((file? (string-prefix? "file:" url)) + (url (if (and file? + (not (string-prefix? "file:///" url))) + (string-append "file://" + (string-drop url (string-length "file:"))) + url))) + (with-store store + ;; TODO: Verify certificate support and deactivation. + (with-git-error-handling + (latest-repository-commit store + url + #:recursive? recursive? + #:ref reference))))) + (define %default-options ;; Alist of default option values. `((format . ,bytevector->nix-base32-string) (hash-algorithm . ,(hash-algorithm sha256)) (verify-certificate? . #t) + (git-reference . #f) + (recursive? . #f) (download-proc . ,download-to-store*))) (define (show-help) @@ -97,6 +184,19 @@ (define (show-help) do not validate the certificate of HTTPS servers ")) (format #t (G_ " -o, --output=FILE download to FILE")) + (format #t (G_ " + -g, --git download the default branch's latest commit of the + Git repository at URL")) + (format #t (G_ " + --commit=COMMIT-OR-TAG + download the given commit or tag of the Git + repository at URL")) + (format #t (G_ " + --branch=BRANCH download the given branch of the Git repository + at URL")) + (format #t (G_ " + -r, --recursive download a Git repository recursively")) + (newline) (display (G_ " -h, --help display this help and exit")) @@ -105,6 +205,13 @@ (define (show-help) (newline) (show-bug-report-information)) +(define (add-git-download-option result) + (alist-cons 'download-proc + ;; XXX: #:verify-certificate? currently ignored. + (lambda* (url #:key verify-certificate? ref recursive?) + (git-download-to-store* url ref recursive?)) + (alist-delete 'download result))) + (define %options ;; Specifications of the command-line options. (list (option '(#\f "format") #t #f @@ -136,10 +243,46 @@ (define fmt-proc (alist-cons 'verify-certificate? #f result))) (option '(#\o "output") #t #f (lambda (opt name arg result) - (alist-cons 'download-proc - (lambda* (url #:key verify-certificate?) - (download-to-file url arg)) - (alist-delete 'download result)))) + (let* ((git + (assoc-ref result 'git-reference))) + (if git + (alist-cons 'download-proc + (lambda* (url + #:key + verify-certificate? + ref + recursive?) + (git-download-to-file + url + arg + (assoc-ref result 'git-reference) + recursive?)) + (alist-delete 'download result)) + (alist-cons 'download-proc + (lambda* (url + #:key verify-certificate? + #:allow-other-keys) + (download-to-file url arg)) + (alist-delete 'download result)))))) + (option '(#\g "git") #f #f + (lambda (opt name arg result) + ;; Ignore this option if 'commit' or 'branch' has + ;; already been provided + (if (assoc-ref result 'git-reference) + result + (alist-cons 'git-reference '() + (add-git-download-option result))))) + (option '("commit") #t #f + (lambda (opt name arg result) + (alist-cons 'git-reference `(tag-or-commit . ,arg) + (add-git-download-option result)))) + (option '("branch") #t #f + (lambda (opt name arg result) + (alist-cons 'git-reference `(branch . ,arg) + (add-git-download-option result)))) + (option '(#\r "recursive") #f #f + (lambda (opt name arg result) + (alist-cons 'recursive? #t result))) (option '(#\h "help") #f #f (lambda args @@ -183,12 +326,14 @@ (define (parse-options) (terminal-columns))) (fetch (uri->string uri) #:verify-certificate? - (assq-ref opts 'verify-certificate?)))) - (hash (call-with-input-file - (or path - (leave (G_ "~a: download failed~%") - arg)) - (cute port-hash (assoc-ref opts 'hash-algorithm) <>))) + (assq-ref opts 'verify-certificate?) + #:ref (assq-ref opts 'git-reference) + #:recursive? (assq-ref opts 'recursive?)))) + (hash (let* ((path* (or path + (leave (G_ "~a: download failed~%") + arg)))) + (file-hash* path* + #:algorithm (assoc-ref opts 'hash-algorithm)))) (fmt (assq-ref opts 'format))) (format #t "~a~%~a~%" path (fmt hash)) #t))) diff --git a/tests/guix-download.sh b/tests/guix-download.sh index f4cb335eef..d4cd2ea6b9 100644 --- a/tests/guix-download.sh +++ b/tests/guix-download.sh @@ -16,6 +16,12 @@ # You should have received a copy of the GNU General Public License # along with GNU Guix. If not, see . +# Define some files/folders needed for the tests. +output="t-download-$$" +test_git_repo="$(mktemp -d)" +output_dir="t-archive-dir-$$" +trap 'rm -rf "$test_git_repo" ; rm -f "$output" ; rm -rf "$output_dir"' EXIT + # # Test the `guix download' command-line utility. # @@ -36,8 +42,6 @@ guix download "file://$abs_top_srcdir/README" guix download "$abs_top_srcdir/README" # This one too, even if it cannot talk to the daemon. -output="t-download-$$" -trap 'rm -f "$output"' EXIT GUIX_DAEMON_SOCKET="/nowhere" guix download -o "$output" \ "file://$abs_top_srcdir/README" cmp "$output" "$abs_top_srcdir/README" @@ -45,4 +49,41 @@ cmp "$output" "$abs_top_srcdir/README" # This one should fail. guix download "file:///does-not-exist" "file://$abs_top_srcdir/README" && false +# Test git support with local repository. +# First, create a dummy git repo in the temporary directory. +( + cd $test_git_repo + git init + touch test + git config user.name "User" + git config user.email "user@domain" + git add test + git commit -m "Commit" + git tag -a -m "v1" v1 +) + +# Extract commit number. +commit=$((cd $test_git_repo && git log) | head -n 1 | cut -f2 -d' ') + +# We expect that guix hash is working properly or at least that the output of +# 'guix download' is consistent with 'guix hash'. +expected_hash=$(guix hash -rx $test_git_repo) + +# Test the different options +for option in "" "--commit=$commit" "--commit=v1" "--branch=master" +do + command_output="$(guix download --git $option "file://$test_git_repo")" + computed_hash="$(echo $command_output | cut -f2 -d' ')" + store_path="$(echo $command_output | cut -f1 -d' ')" + [ "$expected_hash" = "$computed_hash" ] + diff -r -x ".git" $test_git_repo $store_path +done + +# Should fail. +guix download --git --branch=non_existent "file://$test_git_repo" && false + +# Same but download to file instead of store. +guix download --git "file://$test_git_repo" -o $output_dir +diff -r -x ".git" $test_git_repo $output_dir + exit 0 -- 2.41.0 From debbugs-submit-bounces@debbugs.gnu.org Mon Jan 22 05:40:08 2024 Received: (at 68405) by debbugs.gnu.org; 22 Jan 2024 10:40:08 +0000 Received: from localhost ([127.0.0.1]:39959 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRrio-00011p-RP for submit@debbugs.gnu.org; Mon, 22 Jan 2024 05:40:08 -0500 Received: from mail2-relais-roc.national.inria.fr ([192.134.164.83]:2690) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rRrij-000119-Rw for 68405@debbugs.gnu.org; Mon, 22 Jan 2024 05:40:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=inria.fr; s=dc; h=date:from:to:cc:message-id:in-reply-to:references: subject:mime-version:content-transfer-encoding; bh=JEejeQ+1XmRYs0yZ40E5/6P9v3ApsKua9EJb1RBqP+4=; b=s3rCT5QsYPfXTZcVhiVCl4QY/tJjSp7b0de+ilDEmiVp4Lob+xoN43ET Uz8HEMgEztaIjTk5okowuYJXvJhuYvmAzH/2SNkF826mPEruPf6jHtQ00 zFhAJHwAqrlhM2YQMGS8wDbXWR3nNfmoRGAhvQIjvbesnUnGSoGIJqnXe Q=; Authentication-Results: mail2-relais-roc.national.inria.fr; dkim=none (message not signed) header.i=none; spf=Pass smtp.mailfrom=romain.garbage@inria.fr; spf=None smtp.helo=postmaster@zcs-store5.inria.fr Received-SPF: Pass (mail2-relais-roc.national.inria.fr: domain of romain.garbage@inria.fr designates 128.93.142.32 as permitted sender) identity=mailfrom; client-ip=128.93.142.32; receiver=mail2-relais-roc.national.inria.fr; envelope-from="romain.garbage@inria.fr"; x-sender="romain.garbage@inria.fr"; x-conformance=spf_only; x-record-type="v=spf1"; x-record-text="v=spf1 include:mailout.safebrands.com a:basic-mail.safebrands.com a:basic-mail01.safebrands.com a:basic-mail02.safebrands.com ip4:128.93.142.0/24 ip4:192.134.164.0/24 ip4:128.93.162.160 ip4:89.107.174.7 mx ~all" Received-SPF: None (mail2-relais-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@zcs-store5.inria.fr) identity=helo; client-ip=128.93.142.32; receiver=mail2-relais-roc.national.inria.fr; envelope-from="romain.garbage@inria.fr"; x-sender="postmaster@zcs-store5.inria.fr"; x-conformance=spf_only X-IronPort-AV: E=Sophos;i="6.05,211,1701126000"; d="scan'208";a="147916808" X-MGA-submission: =?us-ascii?q?MDEw/ZnSN4P2DkQJ1BITx4hhD8GJHBn+nE+Dx7?= =?us-ascii?q?YZyDWCFC9hIC1Peqf6eRFKDi4LnFvaZT9XP/Dk4rPEFkvOo3h+RahPII?= =?us-ascii?q?wUyJucXPo4xHZGlXD9Z2njG0kdjJdmJKRZuYG1UByZd8ImXAiyzjiRMP?= =?us-ascii?q?JYla/WCxUByQ3ot/Ae1U1YfQ=3D=3D?= Received: from zcs-store5.inria.fr ([128.93.142.32]) by mail2-relais-roc.national.inria.fr with ESMTP; 22 Jan 2024 11:39:51 +0100 Date: Mon, 22 Jan 2024 11:39:51 +0100 (CET) From: Romain Garbage To: Maxim Cournoyer Message-ID: <555736077.15526795.1705919991769.JavaMail.zimbra@inria.fr> In-Reply-To: <87v87ohene.fsf@gmail.com> References: <20240119102417.17155-1-romain.garbage@inria.fr> <87v87ohene.fsf@gmail.com> Subject: Re: [PATCH v4] guix: download: Add support for git repositories. MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [193.50.110.153] X-Mailer: Zimbra 8.8.15_GA_4581 (ZimbraWebClient - FF115 (Linux)/8.8.15_GA_4581) Thread-Topic: guix: download: Add support for git repositories. Thread-Index: Y3e7xAkhoyuNBFtWlJeR9d26F7Rdew== X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 68405 Cc: 68405 <68405@debbugs.gnu.org>, Ludovic =?iso-8859-1?Q?Court=E8s?= X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) Hello, ----- Mail original ----- > De: "Maxim Cournoyer" > =C0: "Romain Garbage" > Cc: "68405" <68405@debbugs.gnu.org>, "Ludovic Court=E8s" > Envoy=E9: Samedi 20 Janvier 2024 03:23:01 > Objet: Re: [PATCH v4] guix: download: Add support for git repositories. [...] >> + (with-store store >> + ;; TODO: Verify certificate support and deactivation. >> + (with-git-error-handling >> + (latest-repository-commit store url #:recursive? recursive? #:re= f >> reference))))) >=20 > The above contains too long lines still :-). Sorry, I missed them, I sent a v5 hopefully wrapping all the lines longer t= han 80 characters :) [...] =20 > Otherwise, I've tested it with: >=20 > --8<---------------cut here---------------start------------->8--- > ./pre-inst-env guix download -gr > https://git.jami.net/savoirfairelinux/jami-client-qt -o /tmp/jami > --8<---------------cut here---------------end--------------->8--- >=20 > and it worked as advertised; very nice! >=20 > Some idea for the future: the --recurse option could take an optional > argument that'd be a comma-separated list of submodules to fetch, e.g. >=20 > --8<---------------cut here---------------start------------->8--- > ./pre-inst-env guix download --git \ > --recurse=3Ddaemon,3rdparty/SortFilterProxyModel \ > https://git.jami.net/savoirfairelinux/jami-client-qt -o /tmp/jami > --8<---------------cut here---------------end--------------->8--- >=20 > and it'd recurse *only* the listed submodules. This would be useful as > some projects contain submodules for windows or other platforms we do > not care about and they may be very large (heavy) to download. >=20 > The same idea could be implemented for our git-reference, where > recursive? could accept a git submodule names list. That's actually a nice idea. I will see if I can find some time to add it t= o a later patch. > But back to the current scope: >=20 > Reviewed-by: Maxim Cournoyer Thank you for your review ! --=20 Romain From debbugs-submit-bounces@debbugs.gnu.org Tue Jan 23 09:06:48 2024 Received: (at 68405-done) by debbugs.gnu.org; 23 Jan 2024 14:06:48 +0000 Received: from localhost ([127.0.0.1]:42462 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rSHQN-0007Wq-JQ for submit@debbugs.gnu.org; Tue, 23 Jan 2024 09:06:48 -0500 Received: from mail-qk1-x72e.google.com ([2607:f8b0:4864:20::72e]:45464) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rSHQK-0007WW-O8; Tue, 23 Jan 2024 09:06:45 -0500 Received: by mail-qk1-x72e.google.com with SMTP id af79cd13be357-7831386ee01so357436285a.3; Tue, 23 Jan 2024 06:06:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706018794; x=1706623594; darn=debbugs.gnu.org; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:from:to:cc:subject:date:message-id:reply-to; bh=C6SnuTaoZC5mhYLQu0aQ3/Yk3VjOhRJDdeoY73cHoSw=; b=JAe1zEPkW7gdSar0JOoJMsRGleGjgTU2HBNR7VM9fBufQpap5h4Q1ng56xmjjavWsa xqRC30bOtMY/C6D2d8FgbNtxdxpQugERARWHaxcbhtH41lrd7W/X7qloeyF6cKWQ7XT6 fw9mPXMgISAs+NKApKF4kZ995z88PkbsDPFtWgtaawj4Fx58g3vUx5rff4t4vSGZusfz 4GwJSTLNu5CWBSl+OB+jX+QzjO3lpsaXjRZ62K0EulIJ6vyDwb8UGnU/KwAXgrHv54xS bNvI0uO6n9qeXrCubYpuw0Sm4+cNxIokg1NWqh5ywxyi6Z2afUc8kGwinjVWwcv1BPra C0/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706018794; x=1706623594; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=C6SnuTaoZC5mhYLQu0aQ3/Yk3VjOhRJDdeoY73cHoSw=; b=CHTkt6QngN2FLcdTIESaDJxsyDAigU1qsMq0wlbJjQSyK9tYi5K21sTFraZSgN0UV2 acfQbWbtvYdcEvbDwxOrq9hfbEFODH5PFSvrIzSNxIq4qN753DNK99ksLGgYILR29V8U 1LhJy55XZZE8ZUr2pOgE3ioXGsXsiIAzWKyLsFjF0u5i5jN6arr5PYYhGO+oNDrJPDQY JznCjLiJlHr6UfkcVsnAzlSwVYkbn55ak1r6xRnHRfB9Dz5U60SYEPYDIbTkftQzNVhq gytGT5YhmetbfXpgTfTa8zOEZlWgq16nalUgkXKGiimFgFbUf2cWF6QOqpZnjV7H7o97 tMEQ== X-Gm-Message-State: AOJu0YxDUvhsHbVc58HQUIvVqqtUb/sObge2VIg0nk3NC22ufBqvJyRZ fW9Pi822BSjCqKCP0j/OCElYvbZ+JSN85lT1aef8qA5dVExxTylD X-Google-Smtp-Source: AGHT+IHdRw7iBO9sD1ZRxWQxqWcB9alKhqVTavat0VeGKglFHzP35eWBT4nRBPdIZkbpN5QVuOXOgQ== X-Received: by 2002:a05:620a:280d:b0:783:8701:5183 with SMTP id f13-20020a05620a280d00b0078387015183mr8424146qkp.95.1706018794233; Tue, 23 Jan 2024 06:06:34 -0800 (PST) Received: from hurd (dsl-158-18.b2b2c.ca. [66.158.158.18]) by smtp.gmail.com with ESMTPSA id dc55-20020a05620a523700b00781de2da438sm3155575qkb.107.2024.01.23.06.06.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 06:06:33 -0800 (PST) From: Maxim Cournoyer To: Romain GARBAGE Subject: Re: bug#68499: [PATCH v3] guix: download: Add support for git repositories. In-Reply-To: <20240122103319.8125-1-romain.garbage@inria.fr> (Romain GARBAGE's message of "Mon, 22 Jan 2024 11:32:55 +0100") References: <20240112151411.22470-2-romain.garbage@inria.fr> <20240122103319.8125-1-romain.garbage@inria.fr> Date: Tue, 23 Jan 2024 09:06:32 -0500 Message-ID: <87zfww16vr.fsf_-_@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 68405-done Cc: 68405-done@debbugs.gnu.org, =?utf-8?Q?Ludovic_Court=C3=A8s?= , 68499-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) Hi, Romain GARBAGE writes: > * guix/scripts/download.scm (git-download-to-store*): Add new variable. > (copy-recursively-without-dot-git): New variable. > (git-download-to-file): Add new variable. > (show-help): Add 'git', 'commit', 'branch' and 'recursive'options > help message. > (%default-options): Add default value for 'git-reference' and > 'recursive' options. > (%options): Add 'git', 'commit', 'branch' and 'recursive' command > line options. > (guix-download) [hash]: Compute hash with 'file-hash*' instead of > 'port-hash' from (gcrypt hash) module. This allows us to compute > hashes for directories. > * doc/guix.texi (Invoking guix-download): Add @item entries for > `git', `commit', `branch' and `recursive' options. Add a paragraph in > the introduction. > * tests/guix-download.sh: New tests. Move variables and trap definition > to the top of the file. > > Change-Id: Ic2c428dca4cfcb0d4714ed361a4c46609339140a Applied to master with commit 916fb5347a. Thank you for this useful contribution. -- Maxim From unknown Sat Jun 21 10:20:37 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Wed, 21 Feb 2024 12:24:05 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator